Would a "a proper, verified source" be classified as an engine test cell. A cell capable of even verifying an AVGAS replacement good enough for submission for certification?
It's a start. Unfortunately, the history of science is littered with examples where something that worked perfectly in the laboratory and in small scale trials had unforeseen problems in the field.
What you really need is an experiment where you run many engines to TBO, some with LOP procedures, some with the old ROP procedures. The conditions need to be as close as possible to identical for each sample i.e. no significant difference in pilot population etc. You then compare the number of engine problems in each sample.
It's a difficult experiment to run. It needs massive resources. Probably the only practical way to do it would be to put the same engine in different aircraft types, and write ROP procedures in the POH for one, and LOP procedures in another.
Wait a moment - did someone say that some aircraft had LOP procedures in the POH while the identical engine in other aircraft specified ROP procedures? Maybe someone
has been collecting data and evaluating LOP vs. ROP results in the field....
What would the results of this trial look like? The conclusion would be something like "LOP procedures resulted in a lower rate/higher rate/no significant change in engine problems reported"
In fact there is a document on the Advanced Pilot web site that quotes Lycoming: "the technique of operating lean of peak and power recovery was discontinued due to the resulting increase in service issues."
To which APS reply "Wrong". Well OK, I guess that settles it...
Just maybe Lycoming are not as dumb as APS like to make out. Maybe they have also tested LOP successfully in engine test cells, but found that it caused problems in the field. Actually, that is exactly what Lycoming say, as quoted on the APS web site.
It wouldn't surprise me if after Lycoming employees go to the APS courses, they go back to Lycoming and someone pulls a folder maked "Lycoming Confidential" out of the filing cabinet. They point out that Lycoming did the same engine tests with the same results 40 years ago, then produce the statistics from LOP field trials which show an increase in problems.
APS claim that extra education and instrumentation are required to run LOP. OK, no problem, but then you need to run another experiment where you compare the rate of problems LOP and ROP with the same level of education and instrumentation. Otherwise, any change could be due to the education, and you are not measuring the effect of LOP vs. ROP.
Science is difficult like that. It is more difficult than just running engines in a test cell. You need expert analysis of the experiment and statistics to see whether they support the conclusions. It isn't clear to me whether APS employ anyone with a science degree, or a statistician.
In summary, I believe that engines can be operated LOP as shown by APS. I believe that they can be very reliable when operated by a fastidious owner.
I
don't know whether they will be as reliable when operated by someone who was taught by his instructor who was taught by the flying school owner, who did the APS course but had a little too much lunch and dozed in the afternoon. Or the pilot who knows all about it because they read about it on the internet. Or someone who just follows the POH. Or whether LOP operations might be less tolerant of mistakes when under pressure and you have to make unexpected power adjustments etc. I don't know whether the fastidious owner would get better reliability ROP with the same instrumentation.
The Lycoming quotes on the APS website make me suspect that maybe Lycoming have gathered the data, and they
do know at least some of this, and as a result do not in general recommend LOP.