Bexil,
I, and many others, know how you feel. I rent a/c and have done many years of taking passengers to interesting places. As Whirly says, decisions on whether to fly in marginal conditions are never easy, but I found I put two extra, unneeded, pressures on me.
1) Assumed my friends I take flying would be disappointed if I said the weather isn't up for it. True, but quickly found no one actually had a problem, as I always warned them in advance weather can be crap, or worse - can be forecast to be crap and then watch it turn lovely sitting on the ground
2) Assumed owners would be pissed if I was too cautious about Wx decisions. Well their skills are different to mine, they don't know the actual weather en route and they want the a/c back in one piece so it’s me that has to make the call.
I have found I can apply general rules on my personal minima to help in decision-making. These are based on forecast/ actual cloud base, vis etc; on my skills (& a/c kit) and distance I want to fly. I've discussed this with others and seen that most people have similar minima’s, tailored to their own skills. So now when I see someone take off in the clag, I don’t feel (so much) “oh maybe I should go”, more “I hope their OK”.
You did a good thing to turn back, well done!
PS. IO540, suggested getting an IMC is the answer to all your points. Well it doesn’t stop you having to make good decisions, but it will help if you feel more comfortable in “our” weather if you find actual =/= forecast.