For propellers, googling "propeller advance ratio" provides an introduction to their complexities.
For the engine efficiency, I think density altitude alone is not enough, you need to consider both density and pressure. This old pprune thread is an interesting introduction:
Piston engine power at altitude [Archive] - PPRuNe Forums
One more thought: all the Flight Sims have modeled it somehow, you could try looking there.
As others have said, it isn't simple, especially in real life!