First, if you slap my hands again then I will go to the police and do you for assault.
Secondly, "you have control". When he insists that you fly then you insist that since he has shown quite clearly that you are not qualified to fly the aircraft then he should do all of the legs. Having an OPS manual reference would help your cause in this regard, does it say anything about the % of legs an FO is to fly to keep the skills up? When he is so wrecked after all of the flying and all of your mates are doing the same thing to him then he might employ a bit of CRM and refrain from the slapping.