Sam, if the only reason you needed a VFR flight plan was because it was an international flight, then it makes even less sense than at first glance. If you had cancelled your vfr flight plan, you could have departed vfr without delay. Or even asked for a vfr departure without using your flight plan then asked to activate it once airbourne. In neither situation would you be doing anything wrong. So no real/obvious reason for brussels to delay your departure. And indeed nothing to stop you filing an airbourne flight plan after departure. I think that you are right about the fact that no rules exist in general re early departure and I think that the guy in brussels has some explaining to do ! Also wondering about other situations requiring VFR flight plans, eg spain in general, night flight in some circumstances in europe etc, but in general VFR flight plans have no time dependent/slot/coordination issues so it makes sense that there are rules about late departure because there has to be a cut off where the fpl gets dropped if you aren't going to use it, but no issues round early usage. ( and trivial to cancel and refile since it requires no validation of route etc).
It woudl definitely be worth re-posting this in the ATC issues section. There is a host of knowledgeable people there that might not see this in private flying.