<<not just at the whim of the controller>>
Well, yes, it is the controller who makes the ultimate decision about the departure order. However, he takes into account many factors, some of which you already mentioned. He has to determine separation, based on the route after take-off (SID - left or right turn-out), type of aircraft (speed consideration) and whether wake turbulence separation is required. Slot times play a part too, which is why aircraft are sometimes taken out of the queue to depart ahead of those in front.
It is a complex job but, much like the work of a pilot, it becomes second nature to properly trained controllers.