Once got a jump seat ride on a 737 (those were the days) and watched the trim wheel moving as the cabin crew pushed a drinks cart from one end of the plane to the other. So 80 kg moving about the cabin does make a difference.
Have also been re-assigned seats at the gate (shock, horror: from business class to economy) for take off on a lightly loaded 767. There were very few of us, so even large planes have the same issue.
Question - if the weight sheet is done based on people sitting in assigned seats, how do airlines with free boarding cope? (I guess the slightly cynical answer would be 'they don't fly unless they are full, so the problem doesn't arise'.)