The difference in philosophy between UK and US airspace structure is about as close to a religious divide as you get in aviation (yes, I'm including the Airbus vs Boeing rubbish!).
The fundamental premise underlying class E airspace is that if you're in VMC you can see the other (silent VFR) guy coming, and if you're in IMC you can't, so you'd better get help from ATC i.e. that see and avoid is to some extent effective. If you don't believe in see and avoid, you might as well make the airspace class G rather than E.
The FAA seems to believe in the premise, to the extent that most of the lower airspace in the CONUS is class E. It also has been known to hand out suspensions to pilots who fly without a clearance in class G, on the careless and reckless catchall.
The UK CAA by contrast seems to not believe in the premise. Most of our airspace is either A/B (separate everything regardless of conditions) or F/G (separate nothing, unless it chooses to participate).
The premise of see-and-avoid makes good sense, and, like many other premises that make good sense, is completely unsupported by scientific evidence. All the evidence points to the conclusion that unassisted see-and-avoid has very low efficacy enroute, and that traffic density is the dominant factor in the probability of collision. Met conditions are irrelevant, except in as much as the traffic density is usually lower in IMC!
Of course I paint that as a black-and-white picture. Risk management is a myriad of shades of grey, but I think the extremes help to follow the principles.
To answer your specific question of why class E makes things harder, the UK does not have the ATC infrastructure to support universal class E as in the US. Any class E would have to be specific zones, which would make navigation in already congested airspace even more taxing for both VFR and IFR, and would make enroute IFR outside the airways structure even more difficult than it already is.