Well, not many pilots are equipped to know what the margin is between the limiting speeds and actual structural failure, and none are equipped to know how much damage a fire is causing where they can't see it.
Add to that, that the way I normally light the fire in my living room is blowing on the spark....
Add further to that that I will almost certainly want to land at the end of it, so want to be fairly close to approach speed at the end of the dirty dive.
.... and personally I want a combination of high RoD and low speed, so staying below VFE works fine for me.
G