Peter, the statistical explanation does not clarify how a pilot is limited in the overall certification, even though in your view the math dominates.
Considering two recent accidents (A330 AF447 and 737 TK1951), the system problems originated with the sensors where known limitations of software, operating as designed, created operational problems. There was nothing to find in bench testing at whatever level was tested.
The resultant operational problems relate to the human-system interface, the situation, and human behaviour;
AFAIK behaviour cannot be modelled adequately by math / bench tests. Thus it is in the human-situation area that a pilot might aid certification.
With respect to the process of certification, the current statistical approach is limited as you describe, yet the industry seeks resilience both in systems and operation to improve safety. Does that imply that resilience cannot be achieved with statistics?
With an enormous caveat of hindsight, in the two accidents, each of the sensor faults had been previously identified and considered by the regulators; the resultant decisions lacked elements of resilience.
For the A330, I assumed that the assessed risk of loss of all airspeed was statistically remote, but this wasn’t proven for the pertaining conditions, just a judgement, but equally there wasn't a total loss of sensed speed. The inadequacy was in the design specification for sensor selection, yet this was statistically acceptable in certification. The operational question is whether this acceptability (with hindsight) was satisfactory for all scenarios – yes it’s OK on a clear day with an experienced crew, but perhaps not at night near Cbs. It is this sort of judgement which a pilot should be able to help with.
The 737 accident IMHO is clear cut – a problem of grandfather rights. Rad alt anomalies were known; new installations either use triple mix or modern dual self-monitoring sensors. This newer 737 just used the old standard, allowed by certification. However, consider which operating standard the certification assumed – what the crew will do, possibly that of the latest ‘state of the art’ system (note the similarities with the MD-80 take-off config warning). Thus there was a gap between what should happen in operation (assumption) and what actually did happen (reality); it is the nature and significance of this gap which cannot be identified by statistics, but pilot input could provide guidance, experience, intuition.
A final point on resiliency is that the concept requires organisations to ‘learn’. In both accidents, the regulators did not learn from preceding incidents. This is a weakness of both the certification process (continued airworthiness) and humans in the process; a weakness perhaps aided by the statistical approach and associated statistical thinking. Thus I would argue for the process to change, there should be a balancing contribution from non-statistical operational judgement.
If not, the industry will have to accept rare accidents such as AF447 – limitations of design and human judgement in certification, and as with TK1951 – limitations of the operating human and the certification process.
I don’t judge which end of the system, design or human, requires change, but point out that there is something in the middle where greater pilot involvement than currently recognised might help make that judgement, preferably before the event.