FFF
in this particular instance I can only imagine that the aircraft in question was on the verge of missing his slot, that being the case it would get priority over any other IFR or VFR aircraft (except emergencies of course).
Otherwise, I can't see why you would have been delayed.
At my place, we try and treat everyone equally, but in the event of a queue forming, as a general rule of thumb, IFR gets priority over VFR, additionally, like it or not, commercial realities at our airport and many others apply.
For a fuller explanation, (as Aluminium Persauder says), bring biscuits and all will become clear....