Loss of height during a downwind turn happens only significantly when low, slow, near the stall and unpowered. Low means near enough to the ground for a small height loss to matter (or be noticed). Slow means less than 4 times the current wind-speed. Near the stall means just that, and unpowered means either no engine or a situation where thrust is not significant (ie flight idle or too late ….).
212Man gives the gist of it – it appears to be all down to inertia. Try turning a scratching K8 back downwind at circuit height (tally-ho) and then do the same with a Janus (veery carefully) and that would appear to prove it.