I have however seen Vg diagrams that 'taper' towards the right, defining lower load limits at higher speed.
I'm just speculating here, but if the Vg diagram "tapers" to the right, couldn't that be caused by the potential for flutter under high AoAs?
From what little I understand from wing design, is that the center of pressure/lift of a wing (in normal flight) deliberately coincides with the placement of the spar, to reduce/eliminate torque effects. Hence the reason that the spar is not in the middle of the wing, but placed at approximately 1/3rd from the leading edge.
At high AoA (caused, in this case, by high G loading at high speed) the center of pressure/lift moves forward and no longer coincides with the placement of the spar. This leads to a torque effect across the wing, and a slight warping of the wing across its length, which in turn increases the AoA of the outer portion of the wing. Until that portion of the wing stalls, and moves back to its original shape, after which the cycle starts again. AKA flutter.
Obviously this effect would be most pronounced when you have long slender wings.
So I can well imagine that gliders have (or rather - should theoretically have) lower G limits at high speeds than at lower speeds. (Of course other aspects of the construction may lead to a lower G limit overall than what the wings would be capable of, so this effect may not be relevant for the final Vg diagram.)