There's an interesting article somewhere about the dangers of the downwind turn (pilots have died making such a manouvre). Wind gradient can be a very real danger, but perhaps the biggest is the visual disorientation of the pilot leading to inappropriate control inputs. The other killer in such a situation at low level (so just after take off, for instance) can be the much reduced climb angle relative to the ground when flying downwind in a strong wind.
Also, flying 'down in the weeds' is visually very different to doing the same at a few hundred feet. Add a strong wind, and things can become very interesting.