You will be rampchecked sooner or later .. This is mainly for vatreasons. Once you are in the clear on that they will log it in their database and you will not be bothered anymore.
On this issue I had a funny encounter as well. I am based in rotterdam which is a customs airport and on a nice day i filed and flew to Den Helder. I was straight away ramp checked by 2 guys who were send specifically from the amsterdam office (1.5 hours away). Why they never checked me in Rotterdam but did decide to do it on this occasion I have no idea.
The French are mostly interested in the VAT aspect as you say. We have had customs inspect us in Cannes. I've also been ramp checked in Rotterdam too, by the AMS office staff. They were very friendly but extremely thorough.
The OP most certainly needs an FAA IR.