The ones I hate the most are Gliders ! They tend to be white , slow moving and circle close to cloud bases.
They are also very unlikely to be transponding and do not stay at a constant altitudes
That is not against glider pilots but a natural concern over gliders and collision risk
That's not really what you should be worrying about. My understanding is that the collision risk is almost exclusively between aircraft of the same class - gliders tend to hit gliders, powered aircraft hit powered aircraft. I guess it's because the operating profiles are different, and collisions tend to happen where there's a high density (in the circuit, gliders in thermals and so on).
A quick search of the AAIB reports database, looking for the keyword "collision", shows about 9 power-power mid airs since 2000, and 3 glider-glider ones. There's only one power-glider, and that was the Tutor near Oxford a couple of years back.
Paul