I guess you have no idea on how the protection zone of a hold is designed...
They have to take into account the different types of entries (direct, teardrop, etc), the wind drift, the accuracy of the hold fix (not the same if it's VOR, RDL-DME or RNAV based, and if it's in distance or time for the outbound leg length). Quite a lot of different factors.
And yes, the difference between FL50 and FL80 in terms of TAS may not be huge, but it still can make you go in another TMA. Where I work, it's a pain in the a** to locate holds because of that.