Steve76 is right on, the airloads on the doors are the thing that helps set the max gear down speed. I must say there is a rumour that the doors are made of stout stuff, however. A crew is said to have left the gear down during a flight to 1.2 Vne, with no ill effects. Of course, I wouldn't know WHO the pilot was.
This story, if true, means that you should respect the gear limit speed of 130 knots, but not fear that limit.