On the wall at Thruxton Airfield:
"Learning HOW to fly takes around 45 hours. Learning WHEN to fly takes a lifetime."
It's usually the low hours, inexperienced, and plain BAD pilots who will criticise. Anyone else knows just how difficult that kind of decision making can be, and will applaud you for making the safe decision, even when you didn't want to.
The pilot I know who agonises most over go/no-go decisions has over 2000 hours. This person once said to me that if I couldn't decide, imagine how I would feel telling the story to someone after an accident; would it sound like a sensible decision. I find it quite a useful test.