Another possible reason for why we do what we do (PF handling all controls, PNF hands-off until fast taxi speed on an abort) is that our aircraft has FADEC so that the only power settings are either TOGA or else Climb (full or reduced-thrust takeoff). There is no power setting aside from just shoving the throttles into a detent, when the FADEC does the rest.
The interesting thing in all this, all these different approaches to the basic problem of a safe stop-go decision and actions, is that there doesn't seem to be any real consensus.
I had a similar question in a sim session about actions post-abort with a fire indication, whether to simply fire all the bottles, shut the a/c down and evacuate or else to spend about two minutes working through a rather detailed list of things to do while the pax were expected to just sit there politely waiting to be told what to do.
Then just to leaven the mixture the instructor raised the spectre of insurance company lawyers coming after us if we had converted our aircraft into slag unnecessarily, albeit in the cause of preserving human lives!