With the confusing bit being "does my altimeter go up or down?"
If you change from QNH (sea level readings) to QFE (ground level readings) you don't go up, but the ground comes up to meet you, (most of the time), so your "clearance distance" reduces. By about 30ft per millibar.
I wonder if I could have made that any more confusing than it looks.