See LASORS A19.
In addition, before acting as pilot-in-command of a foreign
registered aircraft, the licensing requirements of the state
of registration must be met in accordance with ANO
Article 26(3). For ‘N’ registered aircraft, U.S. Federal Air
Regulations (FARs) 61.3(a)(1) states that ‘when the aircraft
is operated within a foreign country a current pilot licence
issued by that country in which the aircraft is operated
may be used’.
However, the grey area is that a Medical Declaration does not meet the normal standards required by the state of registration (USA) for private flight. So I recommend that you don't attempt to fly a N-reg aeroplane without holding a JAA Class 2 medical until this matter is clarified by the Authority.
UPDATE Advice from a chap at the CAA is that, unless the pilot concerned could provide evidence in the form of a letter, signed by an authorised officer of the FAA, that the FAA would find a UK issued sub-ICAO licence acceptable for the purpose of flying an N registered SSEA in the UK, the answer would be no, you may not do so.
Expect to see some guidance concerning this in a forthcoming Advice Circular from the CAA.