Like you, I don't have a problem looking at the N1/EPRs and seeing the auto-thrust changes - they are in my scan anyway.
A surprising number of pilots I have known do not understand the Airbus Ground-speed mini function. If the thrust levers moved and could be held against the clutches, probably those same pilots would get themselves into low energy situations by preventing the thrust levers from responding to Ground-speed mini demands.