HD,
Had the Spanish Controller explained any of the hypothetical reasons you mention above that would have been acceptable.
However, he made it perfectly clear that we were not departing because our FPL time was on the hour and as it was time 53 when we arrived at the holding point, we had to hold.
I am trying to ascertain if he was complying with the rules or not.