In my experience having operated both Lycoming 540 and Continetal 520 engines, here are some of my observations. The continental is smoother and in my opinion easier to start. The continental fuel injection system is slightly different to Lycoming as is the starting technique. You can use the boost pump in a continental with the mixture at idle cut off to circulate cold fuel right up to the fuel control unit without getting any fuel in the cylinders. This makes heat soaked engines a doddle to start by getting rid of vapour lock problems. Continental seems to be able to run smoothly with lower fuel flows too. Could have something to do with a better induction system? Some engineers I have spoken to also say the other major difference is that in a lycoming, the camshaft is in the top of the crankcase and continental has it in the bottom. This can lead to the cam in under utilised lycomings wearing prematurely as the oil runs off into the sump after shut down and takes a relatively long time to get good oil flow way up there after start.
Reliability wise, nothing in it in my opinion. Both suffer from poor factory parts quality every now and then (check out quality control related ADs probably more so lycoming) and both suffer from operation by muppets!
My personal preference is for continental for a couple of reasons: Smoother, it's a better looking engine! and they sound better
