If there is no departure sequence, meaning the aircraft concerned is the only departure you have, then our duty as ATC is to get you airborne bang on your CTOT. (Arrivals permitting) Its that simple.
All ATC units with departures subject flow control from NMOC (previously Eurocontrol CFMU) must follow these rules.
So yes, sometimes it feels pretty pointless to wait at the holding point for 5 minutes when theres noone else around you but the system is there to protect both ATC and operating agencies.