Certainly annoying and potentially hazardous behaviour. When I was in the right seat I flew with a character who apparently did not realise that there is limited nosewheel steering available on the FO's side through the rudder pedals.
I did not realise what he was doing until he did it on the second leg I flew of a multi day trip. After 'handing over control' to me when lining up I was having a hell of a time keeping it straight while accelerating and had almost full pedal deflection (I was thinking is there a strong crosswind I didn't notice)
then suddenly the force I was opposing was released and I swerved, overcorrecting for the lack of this but straightened it out again and took off normally.
Stealing a milliseconds glance over to his side I noticed him taking his hand off the Tiller. Realising the reason for my directional difficulties on this and my previous leg I waited until we were established in cruise flight then discussed and resolved the matter with him. I think he was so clueless he thought he was 'helping'
I strongly recommend discussing your most serious issues with this person. The right approach will nearly always resolve matters, if not is there a mechanism where he can be 'counselled' by his Peers without involving management.
The worst thing to do is sit there and 'stew' it's not good for you but worse it's not safe.