The only thing worse, from a patient's viewpoint, than not taking off to transport them to the hospital is to take off only to return to base before reaching the scene due to deteriorating weather. Once you say you're coming the ground responders are counting on that commitment; they may cancel ground transport and they may do things (or defer things) to the patient in anticipation of your arrival. If you call them twenty minutes into your flight to tell them that you're not coming after all they're going to be not amused.
Never take off unless you know for a fact that you can complete the flight.