Seems like accurate statistic are difficult to come by. But I think the overall view is that Ballooning is quite safe and good fun.
Someone said they didnt see many balloon, probably because most of them fly before you get up! Although there are also a lot flying in the evening, last couple of hours before sunset. THey dont fly in the middle of the day due to too much thermic acticvity and stronger winds, as well as hotter temperatures.
Gengis's inital comment about the safety included a statement- at least for public transport. I would imagine you would find that the sport balloonists have a better safety record than the public transport flights, at least in the UK. Ballooning is very weather dependant and many times you will find the sport balloonists deciding that the conditions are not right, so not flying, whereas the commercial operators are still flying. Some of this may be due to the commercial pilots having more experience, thus being able to land in worse conditions, but a lot of it due to the commercial pressure.
Not to mention the problems commercial balloonists have caused with landown relations, landing in crop and flying over animals (horses and pigs dont like balloons) that the sport pilots have been careful to avoid.
Im not a balloonist but have several friends who are and had some excellent flights with them. Definitly recommended.