In my 9 years of flying at several airlines which had the two person rule, I’ve never had an issue. If we’re chatting with the CC member, then the cockpit speaker is turned on to aid in monitoring RT. The CC know not to be insulted if I suddenly stick up my hand to stop them speaking if ATC call us.
We also ensure that CC don’t come in until we are established in the cruise for precisely the reason another poster mentioned regarding level busts and distraction.
I’ve never found it to be an inconvenience or hassle. Ultimately I fly the way the company wants me to and if they want two people in the flight deck then I’m not about to argue the toss.