It can impact things like VMC (which gets worse at lighter weights) or gust loading cases (where the accelerations seen by the structure are higher at lighter weights, since the same airload is applied to a lighter aircraft)
Strictly, one could say that these cases are actually at "minimum flight weight" not minimum zero fuel weight, but there's a pretty close relationship between the two, and if one were being conservative one could assume that min ZFW was the minumum flight weight.