In the northern hemisphere, port drift means you’re flying towards higher pressure, so the altimeter will under-read and your true altitude will be higher than indicated. Starboard drift means you’re flying towards low pressure so the altimeter will over-read which could be dangerous.
In the Southern Hemisphere it’s the other way around.
The reason is to do with the direction of air circulation around a pressure centre.
In the northern hemisphere, the circulation is anti-clockwise around a low and clockwise around a high.
Vice-versa in the Southern Hemisphere.
If you draw a few diagrams you’ll soon get the idea.