"So we are looking at the PM making approximately 40 actions in the last 7 minutes of the flight, averaged out (which it never is) is something every 10 seconds"
Excellent reason to not add more actions on the PM that the PF could easily do.
At least there is a reason in the rest of your message : possible flight path deviation due to looking at something that's not an horizon.
I will let that sink in, but at first sight it does not entirely convince me. Especially on an airbus, you can look away for a third of a second, the aircraft is supposed to be stable.
It's just like when you check "autobrake", "ecam memo" on final approach, you have to look away from the horizon for half a second or so, it usually does not destabilize the trajectory.