Jeepers!
Just like the DG which is "tied" to the magnetic compass, the altimeter needs to be set off data (either elevation or QNH if from the atis) to start with to give you the correct reading. Moving the dial moves all the pointers and the subscale in the same direction, but has no effect on the capsule. The capsule will only change with actual changes in pressure by either changing altitude or the natural pressure variations .