In my humble opinion,
the last answer by Rhys is the one that holds the truth and perfectly answers the question. It is all a matter of looking at it in terms of increased TAS with altitude and higher air density (thus higher drag) with lower altitude.
Not to forget the aerodynamical difficulties encountered with airspeeds exceeding Mach 1 : high speed stall condition for the airlayer on the extrados of the wing, air intake of the engines which needs to reverse the speed of the incoming air to subsonic speeds.
Good question.