Given ATC is only providing traffic separation services but not obstacle/terrain separation (and SVFR is available to pilots and aircraft unable to undertake any radio navigation), it seems a reasonable requirement to be in visual contact with the ground.
Interestingly, this is not actually required in the FARs (but is required for visual and contact approaches). It is a requirement in the ANO, however, it is in the slightly obscure Section 1 (Interpretation) part of the ANO.