Yes, that's exactly how I reason this one, too.
If I'm taking passengers I always do a meticulous W&B schedule. Because I do that I cannot claim that I am ignorant of the situation.
If it's marginal, calculate reduced fuel, and therefore reduced flight endurances, extra landings, alternates, etc.. If in this situation, I take a set of scales to my rendezvous point with my passengers, insist that they let me weight them, and separately, their baggage, and then recalculate.
I realise its time-consuming, and I am by no means immune from impatience. However, I am very cautious about this, and have to sacrifice time for fore-knowledge and the consequent ability to make an informed decision. (I have before now, made passengers leave items of baggage behind in the car, etc..) Perhaps I'm too cautious, and lacking in a spirit of adventure - but I'd rather keep within the stated operational limits, within the law, and in a safe mode when flying friends and relations about.