FFF,
Like you I have often pondered why. I arrived at a pretty similar conclusion; the combination of extra drag and low speed mean you need to crab more, and more crab means more rudder authority is required for roundout. But even that seems a little weak as we're talking about one single degree for what would be a 20deg crab at 76kts with a 25kt component.
I suppose I can understand why you would want to take off some flap in gusty conditions, but not xwind.