That sounds perfectly reasonable, to manipulate the controls is to "pilot" an aeroplane however; the UK ANO only defines "Pilot in Command"
‘Pilot in command’ means a person who for the time being is in charge of the piloting of an aircraft without being under the direction of any other pilot in the aircraft;
i.e their responsibility is for the "operation" of the aeroplane not just the piloting as in the EASA definition:
‘Pilot-in-command’ (PIC) means the pilot designated as being in command and charged with the safe conduct of the flight.
.
We are so often told by Authority that there can only be one pilot in a single pilot aeroplane, which is the authorised person in charge, the PIC.
If a pasenger has a go, they are piloting in a practical sense, but they are not acting as PIC and therefore have no requirement for a licence. Nothing in the law prohibits this and unless there was an overiding safety case it is unlikely that any court would rule otherwise.