As a Load Controller, I am quite perturbed that Cabin Crew sometimes allow this.
The Final Loadsheet on an allocated seating flight; uses the allocated seating distribution along with where baggage and cargo is loaded to calculate settings used for positioning the Stab Trim for Take Off.
The cabin crew and indeed these passengers that decide to move around the cabin without permission are changing the trim of the aircraft to that being set for take Off. On, many larger aircraft this effect may be neglidgeable.
However, they don't know if the aircraft may be close to trim limits already before anyone moves! Also on smaller aircraft a move by two or three passengers may well take the aircraft out of trim!