Splitduty.... There are many reasons why your "problem" could have occurred. Local traffic, for example. Just because the runway is empty does not mean anyone can just take off. Most times one can, but there will, inevitably, be times when other traffic will dictate what happens. Where I worked there were numerous aerial activities which sometimes delayed our traffic - military surveys, police and ambulance helicopters, Royal flights, even loonies doing sky-diving! Crews never questioned us because they knew would get them away as fast as poss.
I know that things are different in other countries, but where I worked there was always a perfectly good reason for how the traffic was handled and the only people who knew the reason were the controllers.