Mathers, I think you'll find the Capt
does have the final say.
I can offload the bags if I feel I have good reason ie. no shows,
or I can wait, if I feel there is good reason to, ie. info that pax are stuck at security or waiting for special assistance.
An on time departure slips down the list of priority if there are other issues
over pax or technical , despite the dispatchers desire for an on time departure.