FFF,
The other thing might be that the controller was doing a few other things and didnt have time to crunch up the possibilites that well. As niknak says, if he was very tight for the slot, that really is the priority for most of us and the controller probably didnt think about the reality of getting you up first.
Maybe we all just think of the idea of SVFR not restricting IFR traffic. Ive had a quick look and cant see any reference to VFR traffic, but some controllers are just used to bundling them all into that rule?!