The dense foam padding is not to prevent chips and scratches on the handle,
I never flew with the foam attached to be honest and always believed it to be for shipping purposes. To me it just looks like a bit of FOD waiting to happen.
Mind you nothing a couple of tie wraps wouldn't fix I suppose.