I'm right seat on the 737 so its perhaps a bit different to the A320 seeing as we don't have a detent to go to, but for me its when the CPT as PM says take-off thrust set is when he is going to take control of the lever, that's in our FCOM: "After takeoff thrust is set the Captain's hand must be on the thrust levers until V1".