basenotic - Just for your interest, there are certain acceptable deficiencies for items like some fairings, winglets, panels etc. There is a specific equipment deficiency list which will say if flight is authorised without a certain item and what the penalty in performance and fuel might be. It is possible you left New York without this panel, if such a deficiency is allowable(?), bearing in mind it was in the dark you may have missed it.
As John Farley has said, it is not a common occurrence and the performance penalty would be negligible.