IMHO, I think you are confusing the change from speed (IAS) to mach in the climb (or descent) due to Mach crossover (which is automatic) with the A/THR mode. Two different things. [If you are in selected speed in the descent, passing mach crossover, the value in the FCU will automatically change from Mach to IAS, the FMA A/THR mode will not change]
From the FCTM (Normal Operations, Descent):
If the pilot wishes to shallow the descent path, V/S can be used. A/THR reverts to SPEED mode.
So the A/THR will signal FADEC to obtain and maintain the target speed to give you the selected V/S, hence SPEED on the FMA.
