I'm with ACMS. Apply common sense at all times. However, you have to be able to do this with the assistance of cabin crew. They will have to re-check that the aircraft is secure for take-off. On a small aircraft, a 'comfort stop' might only take a few minutes but on a large aircraft it might take some time, more than you really want to spend.
And Agg_karan's, I afraid your problem passengers won't be deterred by a little seat belt sign. If they are already on board, your goose is cooked. And regarding covering yourself, I'd look for a different employer or a stronger union. We are always responsible, whether we do it by the book or apply common sense.