Ok, let' s take an example.
Imagine a Cessna 172 with a wing the size of an A380 wing, but the same mass and design as a 172 wing. This results in a very low wing loading.
Your stall speed is going to go down to 5 kts, which opens up your flight envelope, or so you think.
The problem is that with the same engine/power, the 172 will reach a top speed of 15kts because the dozens of times bigger wing is going to create more parasite drag. The increased drag will be such that the small engine will barely be able to keep up with it.
If you maintain the same design and the same lift to drag ratio as you suggest, the glide ratio stays the same, only the gliding speed will be different.
Your coffin corner or buffet boundaries won' t get any bigger because for the same wing design but X times bigger, your Mcrit will be lower.
When manufacturers design an aircraft, they first determine a small range of cruise speeds they want to work in, after conducting studies. Once they determine this range, they adapt the wing loading to meet the requirements of this range. Hence, the right wing loading for the right cruising speed and altitude.
There is a multitude of factors including engine performance that goes into determining the right cruise speed and altitude. Turbofan engines are most efficient at high subsonic speeds.
I do agree though that there is a benefit to low wing loading and that is the faster climb to cruising altitude.