Passengers are hard work, probably harder than you realise right now. They don't understand how much you have to concentrate, and they chat to you at the wrong times - like when you're trying to listen to an important clearance on the radio. They get excited and scared and airsick. They expect you to be able to go up in any weather conditions, and they don't like plans being changed at the last moment. Especially if they're family or friends, they don't mean to give you an extra workload, but the fact remains that they do, almost always.
That's why people are telling you to just get a little flying on your own first, and I would agree. It's not that anyone will think badly of you, and it doesn't matter what anyone else thinks anyway. It's simply for you to consolidate what you know and get a little experience, not of manipulating the controls, but of captaincy and decision-making...because you haven't done that yet!
But as with everything else in flying from now on, it's your choice. Good luck.