'What's new?', I hear you say. But is this not unlawful?
Not that I'm aware of.
Surely, speaking in French in the flight deck is one thing, but if you communicate with an ATCO you must speak English, otherwise you exclude other pilots on that frequency who do not speak/understand French from understanding your conversation, which could be a safety issue?
The country's language (if it's an ICAO language) is the official language used by that country's ATC. However, if someone calls ATC in English, ATC must reply in English. The "safety issue" bit was discussed here in PPRuNe not too long ago.