Definitely agree flying with other pilots has lots of benefits.
My main difficulty is that since becoming an instructor, if I ever fly with anyone just for fun without my instructor's hat on the relationship becomes much more difficult. The other guy is probably more stressed about making mistakes because he knows he's got an instructor sitting next to him, and I'm trying not to be instructory and make suggestions or point out mistakes.
Most disturbing example was flying with someone last year, 4-up in a PA28. I tried to make it clear to the guy that he was P1, then he went and took off without any vital actions! I decided to shut up but just make sure everything was checked myself, then dealt with it on the ground at the other end.