Probably the most important point is that a SVFR clearance is initiated by the pilot. CAP413:
1.5 Special VFR Flights
1.5.1 Special VFR clearances are only issued for flights within Control Zones and are
normally at the request of the pilot.
therefore; if the conditions were not within your limitations you would not request it!