In over 37 years of flying, I cannot remember any crew member ever refusing to fly to a destination that "might" be affected by hurricanes, typhoons or cyclones etc.etc.. If the weather is forecast to be above your minima on arrival (or you have 2 alternates that satisfy this requirement), then surely you are obliged to operate the flight as per your contract?
Perhaps you should spare a thought for your colleagues and their passengers who will be waiting for your arrival, so that they may keep the expected schedule.
Hurricanes, typhoons, cyclones, tornadoes, blizzards and fog, not to mention earthquakes, are part of the natural world's phenomena - if the thought of encountering any of them frightens you, then don't leave home! Your training and experience should help you deal with them on duty.