Did they need to take over? Are you not as safe as you think? How much experience do you have and do they have?
I used to fly with one bloke who was quite inexperienced, and when we'd come into land I'd tighten my seat belt, stretch my legs and flex my fingers.....Then guaranteed in the last 50 feet (when he started to flare...yea 50'!) I'd have to grab the stick and save the landing and the aeroplane. Either that or I'd end up shouting AIRSPEED AIRSPEED at him as the needle started to sink to the bottom of the dial on approach.
It is called self preservation ! He wanted to fly with me for the experience, I wanted to go home after the flight.
Perhaps your mates are talking about you right now, saying how lucky they are to be alive?