A toy glider launched in still air probably won't fly in perfectly straight trim due to a number of reasons. Due to the inbuilt stability of a free-flight model this will result in yaw and roll. However, any deviation from straight flight in still air will quite possibly be imperceptably small to begin with.
However, if a headwind is introduced, any slight turn away from the wind will induce a drift component which will greatly exaggerate the deviation from the original straight ahead, especially at the low airspeed of a hand-launched toy glider. The yaw and roll will continue and the drift will increase to a maximum as the aircraft flies to the full crosswind heading. As the wind becomes a crosswind, any gust will cause the into-wind wing to be affected more because of fuselage blanking, causing an increased angle of bank away from the gust. As the model turns out of wind, the effects will reduce again and the glider will likely as not fly off downwind. (Some will forget that the glider has inertia, so it is affected differently by a change in relative wind)
This happens to birds as well, they often swoop away downwind but always make a point of turning back into wind to land. My model gliders never had this ability, hence a trail of broken balsa wood when they landed downwind with a high groundspeed.