You could also visualise the situation by imagining a column of air extending upwards from the surface of the earth. There is a certain mass of air in the column and your aircraft is in the comumn at some fixed height above the earth.
The static pressure around your aircraft is determined by the mass of air that is in the comun above it. Your altimeter indication is determined by that static pressure.
If the air suddenly gets colder in will contract. Gravity is pulling it down, so more air will sink down into the part of the column that is below you. This means that there will be less air above you, so your static pressure will decrease. This will cause your altimeter to indicate a higher altitude.
If the air gets hotter it expands. It cannot expand downwards because the earth is in the way. It cannot expand sideways because other air is in the way. So it expands upwards. You now have a greater mass of air above you. So your static pressure increases and your indicated altitude decreases.