Relyon, some interesting thoughts there...
Most engines now aim to keep combustion below 1800-2000 Kelvin (depending on peak pressure), since this is the temperature where NOx forms. Common rail reduces peak temperature (by allowing injection during combustion), and and Exhaust Gas Recirculation reduces free oxygen. The downside is (like the turbine) limiting combustion temperatures also limits thermal efficiency, but the latest Volvo trucks engines are up to about 45% total efficiency (almost as good as ship prime movers of 50%). The future will bring particulate oxydation catalysts and NOx Selective Catalytic Reduction.
VolksWagen have done a lot of interesting work on high turbo boost petrols, where the patial load choking losses can be reduced by simply altering boost. In fact per kg petrol, diesel and kerosine are very similar in specific energy, but you are right about there being some volume advantage to the heavy fuels.
The limiting factor for turbo-diesel power/mass is the shear stress on the piston skirt, which undergoes mixed wet/dry lubrication. Crosshead designs help, but to my mind more innovative crank mechanisms are ultimately required (perhaps epicycloidal for aero usage). The ultimate will be laser initiated combustion, which should alleviate the traditional diesel knock (which is actually the sound of the piston being banged over centre at TDC).
The feature of increased turbo boost (with the required intercooling) is the reason that i am suprised turbine manufactures aren't revisiting regeneration. Heat exchanger technology has progressed, so that lighter designs are now possible. A simple interim solution would be to use reheat for short duration high power requirements - this would make the core engine more efficient in cruise.