Flight deck evaluations etc are generally subjective, thus depend on ‘expertise’. This requires wide ranging experience, which enables comparisons. Evaluators also require a good working knowledge of certification regulations which define acceptable solutions. Unfortunately, due to the subjective nature of the task the regulations also tend to be subjective, e.g. CS (FAR) 25.1303, 1309, 1322, AMC 25-11.
A quantitative system for workload evaluation was devised in France in the early days of Airbus (A300-A310, 3 crew – 2 crew flight deck). This defined each panel and switch in terms of the angular displacement and distance from the crew. It also considered panel orientation, frequency and occasion of use, and together with a rating of ‘look, touch, move (switch/turn)’ provided a measure of workload, but not always in context.
Aircraft handing rating scales (see Google ‘Cooper-Harper’) for aircraft control and operation provide support for a subjective evaluation.
The ‘Bedford’ workload rating scale based on Cooper-Harper (Google ‘Bedford Workload Scale’) has been used successfully as a relative measure of workload, but this as with most rating scales, requires very careful definition of the tasks undertaken in the evaluation.
Pilot unions etc, can contribute to the design process, but IMHO not always with great success. An example was the first edition of the SAE EFIS guide which due to the individuals involved was based on the MD 11, and thus had little relevance to other manufacturers’ concepts.
My experience with ‘avionics’ customers (predominantly up-market Biz Jet operators) were that the pilots had a wish list for everything, but without actually knowing how or when they might be used. This was particularly noticeable with ‘older’ chief pilots who required the latest / best, but failed to understand the systems and in some cases could not work them.
Biz-Jets / GA flight deck system designs appear to have a different design concept from the major commercial aircraft.
The ‘lightweight’ GA systems are have been more innovative and the design led by the avionics vendor – displays available before the aircraft. More recently, many of these systems have been used successfully in regional aircraft, the choice and installation often being driven by cost.
Commercial manufacturers tend to invest more in total concept evaluation during the design process and specify what systems the avionics vendor should provide – display design done with the aircraft; thus, we see greater differences in Airbus vs Boeing. It is during these early evaluations that operators can contribute up-to-date / real life situations which more accurately define the task (see above), but they rarely contribute to the actual design concept.
There are many problems such as what appears to work in one aircraft may not be so successful in another aircraft or even in the same type where situations change (context); these require considerable design thought, simulation, and operational evaluation. Most manufactures aim for and achieve excellence, but this depends on ‘fallible’ human judgment – assumption about use and abuse of equipment and the operational situations.
Also, the industry is in a constant state of change, where new or ‘unforeseen’ situations, different standards of operation, training, knowledge, and application, alter the perception of previously well designed systems.
Human perception and operational situations change faster than the design and evaluation process; humans forget, are complacent, and design assumptions are often invalidated or overlooked due to evolving behaviours.