If you don't have enough time to go 'down town' I would suggest to fly to Jersey as the Flying Club is rather nice to hang out in and does good food.
Guernsey as a destination is nicer if you can go off airport and the taxi is only 10 mins or so to the heart of St Peter Port. Plenty of different outlets to suit all tastes and budgets.
Used to go DCT to Cherbourg but after the hassle with the ZITs decided that there is little difference between that and DCT ORTAC as others have suggested.
They usually keep you low (ie below 1000ft) if you are VFR but do ask as when possible they allow you higher, which to a coward like me does feel a lot more comfy over all that water!
FD