Even if the POH specifies the PIC must sit in the LHS, there seems no reason why the passenger can't be the sole manipulator.
By virtue of being a passenger one is no longer a member of the crew Art 255
'Passenger' means a person other than a member of the crew;
In order to become the sole manipulator for the purposes of the exercise, you must be either PIC or Dual and unless the other pilot is an instructor, they automatically become a passenger whilst you manipulate the controls for the purpose of recency, and if you are outside 90 days then you are illegal! Plain simple and unambiguous.