You don't need to contact Special Branch to fly to France. You only contact them if you're going to Ireland, the Channel Islands, and I think the Isle of Man (not sure about that one).
If you want to keep it really simple for your first trip, and you're only going for a weekend, go to Le Touquet. They are used to British visitors; they get loads and loads of them. They speak English well, on the radio and on the ground, and will help you out, even find you a place to stay if necessary. And it's a nice town.
If you want to go further, stick to large airports, and you won't have to bother with aviation French. Deauville is easy, and you can go to Honfleur; it's a rather touristy fishing village, but lovely. Or fly along the Normandy beaches and then go to Dinard, also a nice place.
Save the small airfields and French cribsheets etc for your next trip. I'm a great believer in taking things slowly; after all, this is supposed to be fun.
Don't be afraid to ask anyone and everyone for help. I can't remember how many people at how many airfields have helped me fill out a flight plan form correctly. I do it once a year, and I never remember how to by the following year.
Flying to France is not that difficult, but it's not that easy either. Don't try to do too much, and you'll have a great time.