He starts giving guidance to the PF ("Watch your speed") at 2:10:27, so if he had his head down in the ECAM it wasn't for any more than 22 seconds, certainly less than a minute.
He starts making comments on the PF's flight path ("Stabilise, Go back down (descend)") between 2:10:28 and 2:10:31 - so he's starting to sound unhappy about 25 seconds after the PF takes control. He makes a direct reference to lateral controls at 2:11:00, as you say - but he's clearly aware that something's not right 30 seconds before that.
Remember that the rules state that the non-relief pilot is not to take control without the Captain's say-so under most conditions. This wasn't specific to AF's Airbus crews, it was fleet-wide.