Dave, as usual, your analysis is sandbagged in favor of your pet configuration, because you are adding apples and oranges. Nice, but poor engineering.
For an 11,000 lb class helo, the 500 HP is not the tail rotor penalty when you determine performance. At a steady hover, the penalty is 50 horsepower. The greater HP is not a tax on the engines, since they need to produce the extra only when there is 50 knots of wind through the rotor, so that the main rotor power is perhaps 600 horse power less, thus the sideward performance is perhaps 100 horsepower less than a steady hover.
Also, your 8 lb/HP is a very low disk loading (large rotor) and you have not accounted for the weight of the new rotor nor its shafting/gearboxes. Of course, by eliminating the TR you have saved that system's weight, so there is a budget to spend toward the new main rotor. For the S76, the TR and gearboxes/shafting probably weighs about 250 lbs)
Kamov wrote some papers on this to show the coax's advantages, but the extra drag of the higher head made the fuel burn far greater, so at any decent range, his coax carried less payload.