But the PNF who pressed TOGA is expecting it and can help an overloaded PF by taking control or applying pressure or trim.
It's more a psychological issue of having FOs trusting Captains and believing that their vocal inputs will get them back in the groove. I've been there - I had two events where in hindsight the Captain was more incapacitated or confused than I had perceived and taking control would have been the better course of action rather than continued vocal intervention. I think it's something that really could be helped by training in the sim, where the Captain is told quietly by the TRI to start making mistakes and misjudgements but the FO isn't made aware of the scenario, a bit like when the typical RTO with incapacitated pilot is done, but much subtler. The idea is to show that mild incapacitation might allow the PF to give all the SOP responses but to make poor decisions and fail to react to other indications and that the FO has to make a decision to take control even though the other guy appears concious. Incapacitation training seems to be far to obvious at the moment, with the incapacitations always being total.