"The only location of sliding friction in the powertrain is between the worm & gear, in the Speed Controller"
But replacing worm gear with a Motor/Gen helps to recapture this power. Dig up some papers on the Toyota Motor/Gen shunt transmission system used on the "Prius" hybrid.
"It is looking like the power loss may be a curve; from 0% at transition, up to 13%, and then back down to 0% at cruise"
Sounds about right. Trouble with shunt systems is that by the time you've minimised either the losses or recirculated power you end up with an inline transmission (or 2 in this case). Don't forget a pilot in an emergency may not follow the calculated flight envelope, but the heli has to deliver without trans failure.
What were your thoughts regarding proposed method of linking prop thrust with prop RPM? I agree with variable rotor rpm being far preferable to compounding.
Mart