Airbus say leave the ATHR system alone it will not increase above FLAP Limit speed.
I love the A320 but in this situation trust it as far as I can throw it.
Have never seen the situation you described - did it happen on different aircraft or the same one? My experience is that A/T is lazy to react sometimes on the A320 but never have I seen it add inappropriate power*.
Autothrust will allow speed decay below VAPP but not into VLS (in theory).
* Yes, yes, A319 speedbrake-with-flap exception...