You could be restricted since you were flying out of EU? I do not know how EU-Swiss regulations are, but between Norway and EU (Sweden) there are certain requirements.
When we arrive in Norway we are not allowed to leave the aircraft before the arrival time in the flight plan unless we call Norwegian customs and get approval. This is because customs want to have the possibility to do random inspections and they do their planning according to the times filed in the flightplan. Usually I solve this by trying to arrive on time, or a tiny bit late, good training in flightplanning.
For departures from Norway the customs needs to be notified about the departure time at least 60 minutes before departure if we are traveling outside Norway (when departing from a airport with a manned customs office). I guess this is because they want to be able to do spot check of departing planes as well (i.e. large amounts of cash requires reporting to authorities before export).
It could be similar restrictions from the local customs office in Brussels that was the reason you were not allowed to leave early?