BTW, in the UK it's clearly stated here:
The Air Navigation Order 2009
"50.—(1) Subject to the exceptions set out in articles 51 to 60, a person must not act as a member of the flight crew of an aircraft registered in the United Kingdom without holding an appropriate licence granted or rendered valid under this Order.
(2) An appropriate licence for the purposes of this Part means a licence which entitles the holder to perform the functions being undertaken in relation to the aircraft concerned on the particular flight."
There are exceptions listed next, but you won't find any exception for "passengers who want to have a little bit of harmless fun". Exceptions mostly deal with training.
The rules refer to "acting as member of the flight crew of an aircraft". If manipulating the primary controls does not qualify one as a "member of the flight crew" then the next one is on me.
If operating the ailerons and stabiliser is sufficient to make you part of the flight crew, then we be better start getting auto pilots to do skill test.
I think you are seeing what you want to you rather than what is written.