There are two conspiring physical relationships that reduce the maximum forward speed for helicopters:
1) Retreating blade stall. The blade that is sweeping downwind during its journey around the mast (the retreating blade) sees its airspeed reduced by the aircraft's foward flight speed. In a hover, the blade tips are revolving at about .7 Mach, 450 knots or so. When the helicopter is traveling forward at 150 knots, the retreating blade tip has slowed to 300 knots or so, a very slow speed for such a thin, low area wing. Of course, half way toward the rotor hub, the blade's section is flying at almost backwards, and so is quite stalled. This retreating blade stall raises drag and power tremndously, and serves as a practical limit. Since control of the helicopter relies on adding lift to that retreating blade when a roll command is given, one of the signs of retreating blade stall is a tendency of the aircraft to roll to the retreating blade side.
2) Advancing Blade Mach - The advanicing tip sees an increase in its speed so that the 450 knots in a hover plus 150 knots of aircraft speed make it experience about 600 knots, almost Mach 1. Thus, transonic effects drive the advancing blades experience, and raise havoc with that blade's aerodynamic loads. Think of the problem the way the blade sees it, about 5 times a second it goes from Mach1 to stall and back again!
The way the Lynx record was set was a very intelligent application of several factors:
1) The blades were quite wide for the job, less practical for the hover (blade width robs hover performance) but just what is needed whan only a small segment ot the blade is lifting out there at 200+ knots.
2) The blade tips were carefully shaped to help transonic drag, so the advancing tip paid less of a price than a regular blade.
3) The excess engine power that was available was used in jet thrust by shaping the engine exhausts to achieve extra thrust, which relieved the rotor of that chore and helped boost the speed. As rotor thrust is "wasted" pulling the nose down and pulling the helicopter through the air, less is available to hold the aircraft up.
How do we get speed today? Three different ways:
1) Compounds like the X3 use power, props, wings and wide blades to get speed. It is no wonder that the X3 (built with the fuselage of an EC-155 at about 12,000 lbs) uses the drive train of an EC 175 and the engines of an NH-90 to do the job.
2) Tilting rotors and wings can do the cruise task, where the rotors no longer lift, and the wing does it all. The V-22 and 609 are examples of totally practical tilt rotors. Note that power still plays a part, they need about 50% more power than a pure helicopter of the same range and payload, but easily fly 100 knots faster.
3) Rigid coaxial rotors - where the retreating side's woes are ignored, and the pair of advancing blades, one on each side, do all the real lifting work. The X2 is an example, where the rotor is slowed down in high speed flight so tip mach isn't a problem, and the retreating blade has its angle of attack reduced to reduce drag and let it loaf along. The prop pushes the aircraft forward, the rotor lifts in near-autorotation, and the whole thing has reasonable efficiency and high lift/drag, albeit with the complexity of a coaxial rotor and prop. Few coaxials can go this fast, the rotors must be very rigid to permit post stall flight.