By way of experiment I have sat on the ground at a point of known elevation and compared a number of GPS receivers that I have, including a couple specifically intended for aviation use. The altitudes indicated have varied significantly, and we're not just talking about that known constant 200ft error relating to the geoid model in certain chipsets. GPS altitudes varied with time, and varied between units at the same time. The altitude accuracy was not something I would stake my life or licence on.
No doubt in future GPS receivers will be better able to accurately measure true altitude - but at the moment altimetry is not, and is not intended to be, based on true altitude. We have a well established, reliable and consistent system of altimetry based on pressure altitude. The 'receivers' are robust, simple, capable of very good accuracy, instant reponse, require no power for their core function, etc etc. There are a number of very good reasons pressure altimeters are here to stay for the forseeable future.
Back to the OP, I think it would be very unlikely that you would ever not have access to a good enough QNH/RPS such that using GPS altitude was preferable. Blind faith in GPS can stitch you up.
Would I ever refer to GPS alt? Well perhaps if I had a static blockage, no alternate source, no standby alt and I was in IMC, then I would probably let down over the sea until VFR using GPS alt, but I can guarantee that I'd descend the last 1000ft or so at a snail's pace. It would take a lot of bad luck to get into that situation though.