Brize also will not allow VFR traffic to self separate from it's IFR traffic.
That is because it has to provide protection for its IFR traffic and it can't do that if VFR traffic isn't doing what it is told. You don't know what the clearance of the IFR traffic is and where it is going but ATC does - that is why they will control you to ensure safe separation.