You are thinking in the right direction. Since colder air is also denser, pressure layers within the atmosphere tend to be closer to each other, which basically means that instead of 30ft/hPa (near MSL), you could get 20ft/hPa (very emphasized, I'm not in the mood to do calculations, but I think it's too much for a real case). This basically means that 850 hPa pressure layer is located lower in cold air than in ISA conditions. And since your altimeter is just a pressure sensor, which has a calibrated scale in feet (or meters some times), it senses the correct pressure, but since it's calibrated for ISA temperature, it displays the wrong (higher) indicated altitude. The same thing goes for warmer-than-ISA air, which has a tendency to increase the distance between pressure layers and you get let's say 40ft/hPa instead of 30ft/hPa, which means that 850 hPa pressure layer is actually higher in warmer air than in ISA conditions. If you're looking for an easy way to remember this, just think of piece of metal or anything similar: if you heat it, it's length increases and if you put it in a freezer, it shrinks
The practical issue is, we never correct altitudes for temperatures greater than ISA, but just for lower than ISA, since by flying higher than our altimeter says, we are on the safe side (obstacle clearance increases), while we have to correct for temperatures lower than ISA, since if it's very cold, our obstacle clearance could converge to zero and we would have a CFIT with altimeter indicating "safe" altitude.