I'd have been very tempted to have stayed in the seat, told the late pax that I'd been told to sit there by that cabin crew member (which would only have been stretching the truth slightly) and directed the late pax to the rear galley to see him. Chances are that it would have resulted in the late pax also being told to take some other empty seat ...