"The most dangerous part of flying is the drive to the airfield" is an old wives' tale that, as topcat says, misrepresents statistics based upon airline travel.
I feel safer in in an aircraft than a car, because (with the sole and rare exception of midair collisions) I know that I am essentially invulnerable to the negligence of others and completely responsible for my own safety: which cannot be said of travelling in a car. However, statistically speaking the chances of having a flying accident are considerably great than a car crash.
See further
page 7 of the 1/98 issue of
Free Flight (reprinting an article that originally appeared in a 1993 issue of
Aerokurier), which is specifically aimed at soaring but generally applicable to all branches of recreational aviation.