Depends on your ops manual as many others have said. Where I fly, the PF will, while handflying, call out for modes [select ....] and values [set...] and the PM will do it.
However, that is not to preclude the PF from selecting/setting things himself if the PM is busy, but he then has to inform PM at an appropriate time.
When the autopilot is on, PF selects modes and sets values and informs the PM correspondingly.