If you leave the Q setting on the altimeter the same, it measures the pressure height difference between that pressure level and where you actually are (i.e. what the altimeter SENSES).
If the Q setting does not refer to the actual ground Q, then you are not measuring (i.e. having displayed by the altimeter) the correct difference in height.
So as the actual Q value increases at the ground, if you leave it the same on the altimeter, the instrument is measuring an ever smaller and smaller difference, thereby displaying an ever decreasing altitude on the instrument.
As this appears to represent a descent on the dial, you will try to maintain your previous altitude by gradually climbing over a period of time. This is why over a cross-country flight with the wrong Q set, flying from an area of Low pressure to High pressure, you end up gradually climbing, i.e. higher than you think you are.
High to low, read high, be low
Low to high, read low, be high