I've had a 180HP 4 seater for 5 years and flown 500 hours in it and never had anyone in the back seats, I decided to remove the back seat to make more room for cargo, bicycle etc. I think back to some of the crazy stuff I did when first getting my PPL and shudder, like loading myself and 3 other big guys into a 150hp warrior on a 90F day and taking off, luckily it was a long runway.
Don't do anything your are doubtful about or uncomfortable with best to err on the side of caution, take them up one at a time and shut down the engine during stops.