I was not able to find a place in the Fcom where it clearly answer to your question, but I think this could help:
Fcom 1.22.30.pg 5
Both FDs off:
-The FD bars disappear from both PFDs.
-If no AP was engaged, lateral and vertical modes disengage. The A/THR, if active, automatically reverts to (or remains in) SPEED/MACH mode.
-If one AP was engaged when FDs are switched OFF, this AP remains engaged in the active modes but the FDs are no longer displayed
and
Fcom 4.5.30 pg 5
If a takeoff is initiated without FDs, the system responds as follow :
-There are no FD bars.
-There is no A/THR arming.
-There is no guidance available.
-The target speed on the PFD is that selected on the FCU or is defaulted to 100 kt.
-Setting the thrust levers to the CL detent does not activate A/THR.
Note : Do not engage the autothrust prior to selecting a target speed on the FCU.
So and this is just my interpretation, it will only revert to speed if active, not armed, and since FD are off during the t/0 the A/thr is not armed when the TL are moved to MCT or Toga detent