Despite the arguments in favour of, or against composites... the simple fact that so many Airbuses and other aircraft, containing composites, have been flying for many years... without any definitive figures being produced that composite use has led to an increase in accidents... is probably the best indicator that composites are perfectly satisfactory in practice.
The fact that they are "different" to metals, in their properties, is probably the key to the belief that they are less satisfactory.
It's more difficult to develop a mental picture of what happens in a composite under stress, whereas metal deformation is relatively easily understood (and seen).
The factor that is still a concern, is the flammability of composites. Yes, a metal aircraft burns, for sure... but a composite burns more quickly and more thoroughly.