There is a lot of design work goes into this, but simply put the aircraft has an "optimum altitude" for each combination of weight and temperature. It will be tabulated in the AFM. Off optimum you lose, it's that simple. If you're low you can claw back a bit by flying slower.
The gas turbine engine has a design point of compressor speeds which is most efficient and it (they) hang off an airframe which has a most efficient AoA. At optimum altitude these two points are in harmony. As weight reduces, less thrust is required as well as less lift meaning that both the wing and the engine tend to move away from design points for sensible cruise speeds. Climbing tends to restore this balance but generally the engine has the most influence so we climb more to ensure the engine is operating at designed RPM and hence efficiency. Wings tend to be more tolerant of off design conditions than engines.
In general terms as altitude increases, air density and (importantly) temperature both decrease. Lower density air means less drag but of course less lift. Lower temperatures mean better engine efficiency. As has been previously posted, there is a reduction in TAS but fuel flow falls faster while flight times increase. See the trades now needing to be balanced?
A quick and dirty reply from me but there are some on these fora who have devoted a good part of their working lives to such number juggling and I'm sure that one at least will correct any errors I am posting here.