FFF, in Class D airspace, we know who is there be you IFR, VFR or SVFR, you're right, we don't separate IFR and VFR, but we do pass traffic information.
Clas E airspace is a whole different ballgame, it is controlled airspace but you do not need a clearance to enetr it VFR, therefore any radar contact we have in Class E airspace, which we do not know about, we have to try and avoid as best we can with IFR traffic, the stuff we have to avoid is generally a ppl out for a jolly, but because he's VFR, he could just pop up ahead of IFR traffic doing 300kts and I've no time to try and get my IFR traffic out of the way, I have to hope that if he's at a imiliar level, he can SEE & AVOID that traffic, at 300kts, it's unlikely he'd have time to react, hence the 250kts speed limit, which LEGALLY I cannot lift.