CAA also confirm that while there is no problem in French PPL's coming to UK in French registered aircraft, there is a problem, in fact its not legaly possible for a French PPL to hire and fly a UK registered aircraft, say for example on holiday in the UK. His French class 2 medical is not JAA and therefore not accepted as valid in the UK.
I don't know who told you that but its utter nonsence. So long as you hold a license and medical that comply with ICAO Annex 1 and its issued by an ICAO State e.g. France, then you can fly a UK registered aircraft in accordance with Article 26. They could of course say your UK issued licence is not valid with the French medical.