Correct, but an IFR GPS contains (as a cert requirement) correction for this. It can be done with a polynomial but
AFAIK most units do it with a lookup table which is interpolated.
At high altitudes, a GPS is potentially far more accurate than a baro altimeter because baro altimetry assumes things like the temperature gradient. One can often see this, by dialling the QNH of an airport one is overflying at say FL150; the GPS and the altimeter can differ by 500ft quite easily, while a GPS should be within say 20ft of the known airport elevation when on the ground (and better with EGNOS).