What you also need to factor in, is that the pressure distribution along the wings length becomes different. After all, the area where the flaps are (typically inboard) now delivers significantly more lift than the outboard areas. This leads to different bending moments/locations within the spar, compared to the normal configuration, and might lead to localized increased stresses. (Although in general, if you bring the lift in, the bending moment at the wing root will decrease.)
This effect is very significant in sailplanes where you use airbrakes, typically mounted fairly inboard, to destroy the lift at that location and provide a huge amount of drag. Suddenly all the lift needs to be generated by the outboard portions of the wing, increasing the wing root bending moment etc.
So that's another reason there are flap limiting speeds. But it's also the reason there are flap limiting G factors. In the R2160 I fly, the limits are normally -3 to +6G, but with flaps extended they're only 0 to +2G. (Flying inverted with flaps deployed would move the lift to the outboard portions of the wing, increasing the wing root bending effect.)
Oh, and to add to your list: The flaps themselves may be damaged due to torsion effects. On most aircraft I've seen, the flaps are actuated with some sort of actuating mechanism that's attached to the inboard edge of the flaps, not to the center. This will generate a torsion force throughout the flaps themselves. And at some airspeed, the flaps will not be able to withstand that torsion.