As a lesser experienced pilot, get the recent experience you need to be competent, and then some. Then, plan to remove as many time limitations as possible. I'm honestly not familiar with the local factors of your planned trip, but if you're watching a clock for a planned day trip, and before you start out there is a time pinch point to within an hour, you have a problem - good chance it'll not work out.
The other thing to consider and compare, is that it sounds as though you would like to be the organizer, the pilot, and a guest within the one trip. That means that your are spreading your recently restored skills in piloting within three roles, so piloting is not getting all the attention. Thinking of this differently, if your best buddies asked you to fly them to place X, so they could watch the game, while you waited with the plane to take them home when the game was over (so now you have only the one role), would you assure them that you could accomplish that one task delay free? For myself I'd enter even that with caution - without adding the other two roles on top.
If you have a contingency, that's fine (actually it's pretty well mandatory here) . But I'd plan the contingency (staying overnight) as your "Plan A", and maybe fly home same day as your "Plan B", if everything is going very well - which is unusual in day trip planning in GA. If everyone on your trip is unwilling to stay over, they certainly should not be departing with you in the first place - just trust me on that one....
I like to encourage people to use GA planes for their person adventures, but I always do it with out airline's "inside voice" motto in mind: "Time to spare, go by air".