First you need to understand that the aerodynamics of an airplane depend on the indicated airspeed (IAS).
The slowest you can go is a particular IAS. But the ground speed for that IAS gets faster at higher altitudes.
----------------
Then you need to understand that a particular design will become overstressed and break up at a particular Mach number.
So the highest speed your airplane can fly is that particular Mach number. The ground speed for this Mach number decreases with altitude.
----------------
So at lower speeds you are worried about whether the airplane will fly at all, and this depends on the IAS.
And at higher speeds you are worried about whether the airplane will break up so you have to watch the Mach number.
----------------
At higher altitudes the margin between minimum IAS and maximum Mach becomes less so you have less choice of speed.
----------------
Hope I haven't offended too many pilots by making it simple. I'm just a dull old Engineer.
TAS is assumed the same as ground speed for this simple explanation