The farce that is E airspace...
This, from Wikipedia, sums it up nicely:
For example, consider Class E airspace. An aircraft operating under VFR may not be in communication with ATC, so it is imperative that its pilot be able to see and avoid other aircraft (and vice versa). That includes IFR flights emerging from a cloud, so the VFR flight must keep a designated distance from the edges of clouds above, below, and laterally, and must maintain at least a designated visibility, to give the two aircraft time to observe and avoid each other. The low-level speed limit of 250 knots does not apply above 10,000 feet (3,000 m), so the visibility requirements are higher.
So VFR needs to stay in VMC so they can spot and avoid me as I come barrelling out of a cloud on descent at 250KIAS. Please. Talk about wings being thicker on the ends...