Driving and flying are quite different, the only common factors being planes and cars have motors and you have to be grown up to get the licence for either.
Car driving requires little if any knowledge of the mechanics and dynamics of the machine, let alone the aerodynamics.
Weather conditions are of very little concern for the car driver in the UK for 90% of the days in a year.
Navigational errors in a car can be corrected by stopping and asking the way.
Arriving and parking in a crosswind has to my knowledge never been a problem.
Weight and balance is pretty irrelevent, even if indulging in cramming the maximum number of students into a mini.
Running out of fuel merely involves a long walk, perhaps in the rain.
Conflicting traffic is restricted to directly ahead or behind, or from the approaching side road, always reasonably well defined.
Difficult to exceed the structural limits of a car.
The only radio you have to operate is radio 2 or the CD player.
Driving is conducted on a well defined two dimensional plane.
It could go on and on and on.
Piloting requires you try to be disciplined from the moment you think about a flight, and make many judgements about all the above and more, relating them to your qualifications, legal sign ups, experience and recency and personal well being.
In running through all these thought and decision processes it should not be considered to be a lack of confidence, more a process after which you will be confident you have made the correct decision on whether to go or not.