if you look at it from a thermodynamic point of view, all 4 stroke petrol engines have approximately the same efficiency.
Yes, and for that reason I think the bottom line is that from a technical perspective, airframe and operational considerations are more fertile ground to gain in overall aircraft efficiency than engine changes. Airframe weight reduction, increased wing aspect ratio, higher altitude cruise etc all tend to have a good payback.
Changes to engine design that increase efficiency tend to have associated drawbacks. Water cooled heads increase efficiency at climb power by allowing leaner mixtures, but the advantage tends to go away in cruise and you're left with a more complex design. Within the modern era of small piston aircraft engines, Continental did water cooled cylinders in the early 80s, but apparently didn't see fit to pursue the idea commercially. Rotax started with water cooled heads on higher rpm four strokes in 1987, and at full power the efficiency benefit of water cooling more than offsets the efficiency disadvantage of higher rpm. Everything looks comparatively good until you get to cruise, at which point you could lean out a simple, air cooled, lower speed engine and it would start looking OK again, without the extra stuff. You can actually achieve the same thing in climb on an air cooled engine using water injection but nobody has bothered for decades, instead using extra fuel as the evaporative coolant for full power climb.
Geared engines tend to be lighter for the same power, which helps reduce overall weight... a good thing. But as with the Rotax, cooling issues then become more critical, which may necessitate water cooling, and you may also need dual carbs or EFI - both changes adding complexity and some measure of weight. Propeller choice and gearbox resonances can also become an issue, which is one reason Rotaxes mostly use wooden props - they are both lightweight and like a big sponge for vibration, but they are also less aerodynamically efficient and less durable than metal props. The net result for the Rotax is a lightweight engine that isn't as 'industrially tough' as it might otherwise be. Good for sport planes.
Diesels have high efficiency for the reasons described by previous posts but are intrinsically heavy. To get them anywhere near competitive in weight for an aircraft they become complex (turbo'd, geared, water cooled, and non-rebuildable), maintenance intensive, and extremely expensive. Few private buyers benefit financially over the life cycle of the engine, even in areas where fuel tax is extreme. I think personally that their best application is a flight school that flies the aircraft a lot of hours/month and has a full time mechanic. The biggest current user of diesel aircraft engines, the US Army, shares those characteristics. Also benefiting would be those buyers who simply cannot buy aviation gasoline locally due to lack of infrastructure in their area - which is also true for US Army operations, and is the reason they went to diesel UAVs.
To me, the most promising change for privately owned and operated aircraft engines right now is simply to incorporate variable ignition timing, as per Lightspeed ignition which advances spark in the partial power, high altitude configuration and is thereby effective in the same regime where the airframe is most efficient, and where the plane operates for the longest period during a cross-country flight. The offsetting disadvantage, which is usually the necessity for an aircraft electrical system, is no longer a significant driver for modern aircraft. But again, the efficiency gains that can be realized with a lightweight, aerodynamically efficient airframe seem to exceed what can on balance be achieved with engine changes for the average user.