First, let me say that one of the most pleasurable parts of private flying is sharing that pleasure with other people...and there's nothing wrong with letting another person have a go at flying the airplane.
So far as CRM...I have a policy of performing all checklists challenge and response, all briefings out loud verbally, regardless of whether I'm flying single pilot solo, or in a crew environment with one or more other crewmembers present. Same practices, same procedures. Even if I'm alone I perform the full brief out loud.
I firmly believe this practice and this consistency creates a safer environment. I practice this the same in a Pawnee as I do in a Cessna 172 as I do in 747.
When other pilots or crewmembers are involved, there should always be a clear understanding of what is to be accomplished, by whom, when, and where.
Don't forget that human resources are only one aspect of CRM. Cockpit resource management applies also to the arranging and preparation of charts and flight documents, radio setup, etc. CRM is just as applicable to single pilot operations, VFR or IFR, as it is to a crew cockpit environment.