It is only really significant when it gets down to -16C (in Canada, anyway).
For example, at -20°C at 500 feet, your altimeter should be reading 570 ft to ensure a height above ground of 500 ft. Just remember that, when temperatures are less than ISA, you will be lower than the altimeter reading, so you must add a correction.