Wilyflier, You could be doing your downwind turn through some hilltop rotor, or getting your inboard wing into lower, slower moving air. The wing only 'sees' the air surrounding it. As you can see the ground directly beneath you from a hang glider, it is very difficult to avoid slowing the glider down as the ground rushes below at a rate of knots.
Sliverstrata is correct in saying that climbing into a tailwind can decrease your airspeed. It is the same effect as decending into a headwind,only the wind gradient is has the opposite direction as your height changes.
If you watch, from the ground, a free flight model aeroplane circling, as it drifts downwind, it will appear to slow down as it turns into wind and speed up as it turns down wind. If you were to observe the same model from above, for example, from a hot air balloon, it would appear to fly a perfect circle.