Yes, I do the same. If I am giving a ride to anybody who I don't know well or has no experience I always take the duals out. If my passenger is either a pilot of any sort or a frequent occupant then I don't mind having them installed if they happen to be there already.
However, if they are out as they normally are, I will leave them out and only specially fit them if I am flying with another helicopter pilot.
I once refitted them before I knew that the pedals were a left and right version, and if you get them reversed the back of the RHS one hits the LHS one. The full and free check showed that my passenger's camera bag was on the pedals, and after he had moved it I did not repeat the check. Big Mistake! When I lifted into the hover I had no left pedal travel and had to very quickly dump the lever. I always always check that left and right are in the correct positions now.