Correct. I’ve taken the chance to escape the flight deck multiple times to avoid some nutter Captain, usually ranting about politics, religion, his personal life or whatever.
So you think the Captain is a "nutter" and you leave the flight deck? Your job is to provide support to the PIC should things go pear-shaped. So if you are swanning off into cabin just because you don't like the other person the problem lies with you not them. The Captain is not there to be your BFF.