This was an issue some years ago and a friend of mine got as far as challenging them to arrest him or let him go having spent an unplanned overnight in France (they let him go).
I'm neither accountant nor lawyer, but if you have the arrival certificate from the UK I think the French will accept that any VAT issue is for the UK authorities not them.
As others have said, VAT exempt is VAT paid for GAR purposes and the Aircraft being on C of A is thus in free circulation within the EU