If you look at the altimeter as a measuring instrument, you set its reference point by adjusting the subscale to make it read the altitude at a particular station. That is then the QNH or altimeter setting for the station, and any properly calibrated altimeter will also read the station elevation with that QNH set.
As you move away from the station elevation, the temperature of the air between station elevation and your altitude starts to affect the accuracy of the altimeter. So what matters is the vertical distance between the station at which the QNH was determined and the altitude in question.
For almost all practical purposes, the QNH will be determined at the airport elevation at which you are about to land, so it's height aal that's important. I suppose there might be a case where you're using a remote altimeter setting for an airport at a different elevation, but the lateral barometric differences are likely to be as great as the vertical temperature errors.