A racing glider carries ballast to go faster. If pleasure gliders carry ballast (I don't know if they do), it might be for stability (increased Va).
Yes, there is a greater drag force acting on the faster glider as it descends, but it is acting against a heavier mass, so it has the same effect.
I can prove this with trigonometry if provoked.