Another way to think about it is this:
When the atmosphere is cold, it is (like most things in life
) thinner, so for a given altimeter
reading you end up flying lower down.
This incidentally is one reason why GPS altitude can differ significantly from the altimeter, at reasonable heights e.g. 10,000ft, even if you have exactly the right QNH of a station directly below. The GPS is probably within 100ft but the altimeter might be 500ft out. It doesn't matter in reality because everybody uses the baro figure so inter-aircraft separation is maintained.