I hate having anyone else in the cockpit who has a licence
I'm the other way around. I don't mind taking pax at all but the one nagging thought at the back of my mind is that if I keel over unexpectedly (I'm at that age where it happens...) then it's not just me that is going to sing with the choir invisible. The pax I take are regular pax and I always let them handle the aircraft enough so that they might feel confident to at least have a bash at getting it down.
I never tell them that's why I'm letting them have a pole around though...
On the other hand I really enjoy having another pilot on board. I have a regular flying buddy, my leg out, his leg back etc and we have it nailed down exactly who is in charge when flying and what the non flying pilot is expected to do. We brief it as well so there is no mistake. We're both ex mil and certainly in his case he has over 11,000 hrs cross cockpit experience so it works well for us.