There is nothing wrong if the captain wants to enjoy a little or practise/refresh a little hand flying. A little CRM to communicate with the FO before knocking the A/P could have help.Nonetheless,not a cloud in the sky doesn't necessary means that there would be no other threats until touch down.How about traffic,complicated ATC vectors and of course,a weak FO that couldn't cope with the workload? These factors sure play a bigger role than having a little bit of manual flying fun.
It is also the FO's right to speak up to the captain if he is finding it hard to cope.Send him back for some training if you have to after the flight,but for that very moment,noone need a distressed right hand man.