Certainly a minor consideration, but I suspect that the main driver, generally, is to keep the certification stall speed down .. which involves a bit of a trade-off in loading flexibility.
The operational limitation involved in losing that portion of the envelope is, for most aircraft, preferable to the performance penalties in retaining it.
Another consideration is tail loading.