I don't believe that's the case.
First of all, if the difference between your altimeters is more than 200ft in-flight, you should declare yourself as being unable to fly in RVSM airspace.
Second, the ATC should not have problem with your transponder indicating let's say FL372. They could ask you to switch to the other transponder/alt source for their convenience, but you would still be RVSM capable regardless.
To the OP: I wouldn't change AP for short hand-over of controls (e.g. when you do the briefing), but if you do switch it for a prolonged period of time, switching the transponder/alt source to the AP side is appropriate.