BB - yes, you do need to have a proof that VAT has been paid. The French flying customs are very hot on that.
G-reg is OK ( but it's as well to have the chit from HM Customs to confirm the VAT-paid status). For N-reg it's vital.
I have heard horror stories about French officialdom giving pilots of N-reg aircraft a hard time, but I've never had that experience.
You save a little money (not a lot) - mostly the CAA fees for everything. In theory you are "required" to do less maintenance, but in reality you still have to keep the aircraft airworthy. So apart from the silliness of Notice 75s on props, there isn't all that much in it.
The big plus is that an FAA IR is achievable to an ordinary human being...