Pulling ICAO, EASA and the ANO together, the strict interpretation is that a pilot is a person manipulating the controls of an aircraft. Such a person requires an appropriate licence unless undergoing flight training in accordance with EASA Aircrew regulation as amended from time to time. It follows that a passenger who manipulates the controls of an aircraft is acting at that time as pilot and needs to hold the appropriate valid licence unless undergoing flight training.
Having had the discussion about 'letting passengers have a go' with the policy department in the CAA I would agree with the above quote from Legalapproach.
ifitaint...