The ATHR is disconnected by moving the levers so that the thrust lever angle is at current commanded thrust and then pressing both disconnect buttons on the outside of the thrust levers. If you didn't move them to current commanded position they'd spool up to climb thrust when disconnected.
As for the purpose of the ATHR pb on the FCU (AP control panel) - its purpose is to either manually arm or disarm the ATHR, reset the ATHR in case of the known glitch on some A319s with CFM engines and also potentially after a windshear - once recovered the speed can go berserk - pushing the ATHR off and on gets the thrust off whilst leaving control with the system - it's a nicer way than closing the thrust levers.