Unfortunately, this is one of those 'environmental capture' dangers that is inherent to the Airbus. You are used to the TL's being just switches to command the Fadecs at certain phases of flight. So, Take off, Cruise and Landing become switch positions, what the engine is actually doing is being handled by the computers. Generally, I was taught to have hands on the TL's up to V1, below 1500' agl and at any time that the autothrust is off. The last one should remind you that the athr is off because it feels 'odd' or non standard to have your hands on them and requesting PM to do Hdg,MCDU and Alt changes for you. I'm aware of a similar occurrence to this one on the 330, where a break down in situational awareness with athr off resulted in a 'scare'.