You have two problems without radar:
One being unknown unknowns.
The other being what is Class E Airspace? As I said before it is neither controlled nor is it uncontrolled.
At least with Radar you eliminate the unknowns and some direction can be given to IFR aircraft on the best course of action.
I would suggest that in the US the vast majority of Class E is under radar coverage and whilst the parts without it would be pretty quiet.
I am personally not so bothered by having non Radar Class E at higher levels but to have Class E to low level without some sort of radar coverage is downright dangerous.