Why is the downwind leg in a circuit called the downwind leg, if when it is flown at anything from 500ft to 1000ft+? I think your wish to limit the use of downwind as a description is too narrow (speaking personally, of course).
You need to know where you are, relative to wind direction for a whole host of reasons and heights, don't you?