Perhaps flying clubs should follow the airline tech log policy where you put a delay code in the tech log or notes after the flight.
Thereby, anyone picking up the aircraft would see why it was late. In your example, it could be the first flight was delayed due to weather, or instructor needing a pee, and then you would put your delay down as a reaction, since you didn't delay any further than the delay you inherited.
It would certainly stop you feeling bad when handing over the keys late to the next guy, as there is a paper trail showing it's not your fault.