In practice, this is exactly what happens Bluskis.
The only issue that arises, as others have written is when ATC need to mix genuine IFR traffic, with the VFR boys. Under these circumstances, the VFR boys will normally get the boot for a while to allow the IFR arrival to take place.