Interesting how this always brings up differing opinions from all levels... To me, my simple fizzics say that it shouldn't matter, because my simple brain can cope with the 'parcel of air' theory.
But... there must be something out there that makes us all, including me, wonder. Here's a hypothesis:
Most of the 'downwind turn' bingles that I've heard of have been when someone's turned downwind early after taking off in a strong crosswind. If, when you are turning away from the wind, you are climbing through the height where the wind is increasing as it is no longer being slowed by the ground , then voila, you have a temporary decrease in airspeed. Admittedly, this is a show-off manoeuvre, but it does happen, and it does give the possibility of an illusion that can cause us to doubt the 'don't lose speed/height in a downwind turn' mantra.
If it's not this, then there must be some kind of illusion out there that's keeping us wondering. I used to fly hang gliders, and when showing off, groundspeed could change from 45 to 5 mph (and vice versa) in a trice: you don't go zooming up and down doing that.