What you say does not hold for supersonic flight.
In supersonic flight, lift-dependent wave drag increases rapidly with Mach speed (by M^2), while some other drag components decrease with altitude. So the faster you go, the higher you need to fly to minimize drag.