Without question, the first officer should have refused to give up his seat. If the discussion between he and his captain became heated, he could have pulled the CVR CB once on station and requested a download of the conversation, and let the facts and voice transcription speak for themselves.
It's worked in the past too. Better than he said, she said.
That said the F/O deserves some disciplinary action too.