If you refer to the printed number "657BB" it is not a serialised component.
If you refer to this number, it even is just an access panel, not the flaperon. The chance that access panels have been swapped between tail numbers is significantly higher, than the chance that flaperons would have been swapped.
But I am quite sure that EADS-CASA identifies much more parts by individual numbers, not necessarily specific serial numbers, but at least batch numbers or modification status numbers, which could narrow down the possibilities where the flaperon belongs to significantly. The production QS will most probably use some sort of barcode-stickers or something like that to track the components that were produced, inspected (e.g. ultrasonic for delaminations) and then finally assembled to form a flaperon. It sounds like at least they do expect some "numbered component within the flaperon" to identify it further.
In light aircraft production it is standard to embed small serial number stickers within the translucent GFRP laminate, visible from the outside but not removeable. In CFRP this is a different story, but even those have GFRP layers in the contact area with metal fittings which would allow for permanent part identification stickers.