Think in terms of a vertical column air between you and the altimeter setting source. Cold air is more dense. Thus, the column compresses with cold temperatures. The longer the column (height above altimeter setting source) and the colder the air, the greater the column is compressed.
Old aviators adage, "Cold or low, look out below."