Merged: The Ambidji Report – CASA should get their money back!

[2b2]

Ian, you might like to fix up this note on Page 7 of your report -

% Canada's Van Nuys has 600,000 movements, so Canada total > this.

[Ian Bryce]

Thanks 2b2 for pointing out my error. The Review has been corrected.

[Dick Smith]

Owen Stanley in his Post number 243 claimed that the diagram in the Ambidji Report which has been discussed was only a “labelling error”. He believes that the correct calculations were performed by Ambidji.

I have recently had advice from a friend who has expertise in mathematics. He does not agree with Owen Stanley. This is what he says:

The graph on the next page, Fig 11.5 page 141, compares actual collision frequencies and those calculated by Ambidji from the US empirical data.

If you compare the data in this graph you get the following comparison between the values calculated by Ambidji and the correct calculation using the US FAA empirical data.

The figure on page 141 (fig 11.5) shows the following comparison:

....................................................Rate From
....................................................the Graph
........................................US.......Fig. 11.5 p.141
Location........Movements.....Calc'n....(Ambidji calc)
Archerfield.......146,000.......0.018.......0.02
Bankstown.......418,000.......0.146.......0.08
Jandakot.........414,000.......0.143.......0.09
Moorabbin.......372,000........0.115.......0.06
Parafield.........141,000........0.017.......0.025

The result for Archerfield is curious. The rest are consistent with the mistaken calculation.

The calculation should be accidents pa = 0.834 x M x M Where M = number of millions of movements
ie for 418,000 movements M = 0.418 0.834 x 0.418 x 0.418 = 0.146

It was my first assumption that the graph was simply mislabeled but Fig 11.5 is where Ambidji compare the GAAP outcomes with the US data for Class D towers shows the result was incorrectly calculated..

The final calculations by Ambidji did use the historic data as follows.

Ambidji took the most recent 9 years which included a cluster of accidents - 7 in all at 6 aerodromes. And based on this scant data calculated a separate accident frequency at 6 airports !!

They did this as follows.

Data this scarce must have a large amount of scatter. They accepted as 'correct' the high results and they filled in the low results with additional hypothetical accidents calculated from the US empirical data.

This is so scientifically invalid that one is left speechless.

[Ian Bryce]

Further Springside recommendations

Hi Clinton,
The reason I put only one recommendation in my Review was that the problem of speech clarity by ATC is my own proposal, whereas the matter of inbound reporting points has been discussed at length by pilots. In particular, Ambidji performed a very comprehensive survey of pilots, in which the problem of conflicts at IRPs, far from the scrutiny of controllers, has been well noted.

Since you ask, I will go into more detail now. My Springside model of mid air collision (MAC) rates (described in my review) has a term depending on traffic movements alone, representing circuit area conflicts. It has a second term depending also on the restrictions at the control zone boundaries, such as the way Holsworthy and Sydney ctrl zones affect Bankstown. The theoretical basis is that such constrictions concentrate traffic which increases risk. The observational basis is that the two terms when combined (and tuned) give very good agreement with the historical MACs.

Thus there is a sound basis that removing restrictions at zone boundaries would decrease the conflicts, other things being equal. Of course all the traffic must then converge in the curcuit area, but that seems to be a lesser problem and is visible to the controller.

Thus, some suggestions could be made:

- For BK, reinstate the third Reporting Point at Westmead so that traffic from the Light Aircraft Lane does not have to divert to Prospect and combine with others.

- Make IRPs not applicable to jets on VFR, so that they can fly direct to the circuit from any direction.

- Replace IRPs with several designated lanes for piston AC on VFR, so they can join at any point.

- Lower the restricted zone at Holsworthy so traffic from S and SE can track direct.

In theory these measures would reduce the hotspots away from control. Others more experienced than I would have to work out the details.
___________

[Ian Bryce]

Replying to Owen no 253:

Yes I did in fact send my Review to CASA Office of Airspace Regulation.

PS: I replied to Clinton no 252 at length, but for some reason my reply has not appeared in this list.

[OZBUSDRIVER]

If I was McCormick, I would be wandering down to my environmental monitoring outfit in AirServices and ordering them to do a study of BK and JK immeadiately. Record all the tracks within 30nm of the GAAPs and make my decisions based upon recorded data as to whether I would add IRPs or change procedures around INDIVIDUAL IRP according to actual measurments.(Ambidji mentions it....McCormick should have demanded it BEFORE making sweeping changes)

Then I would make a correlation between busiest periods and tower tapes and then make a decision on any need for training or extra resources to alleviate any congestion problems...it could mean each IRP might need an individual sector frequency leading into and out of the circuit area....with a TWR controller dedicated to just watch that bit of airspace and nothing else. Complete with a TSAD set up to best view the data and look up and straight at that bit of airspace....if that is where the risk is then that is where the resources should be placed.

Ambidji missed the trees for the forest of data.

[Ian Bryce]

Thanks for your comments Clinton.

You ask, "does your model take into account the fact that one of the benefits of IRPs is that outbound and transiting aircraft have known, identifiable points to avoid?"
The answer is no, my model it is not that elaborate. It could be further developed.

The proposal for inbound VFR lanes (marked on charts) might assist in this problem.

Re the closing of Westmead IRP - an idea - could it have been for controller convenience - they have an easier time visually scanning in 2 directions than 3?

You also say "I don’t see how restoration of Westmead would have avoided the recent MAC near TWRN". A short answer is that the numbers are too low for statistical validity in individual cases, and that it why we need a model so we can merge all the actual cases.
A second answer is the "push around effect" - closing Westmead forced traffic from the north around to PSP, and the increased density there led more traffic from the west to choose TWRN.

Regarding visibility of TWRN (and Westmead in the past) - why not place a strobe light there? (Could equally well help to locate an inbound VFR lane).

[LeadSled]

I believe I said earlier “I believe for Moorabbin, Jandakot and Bankstown, historic mid-air collision rates were used, not calculated rates based on the FAA formula.” Yes?

Owen,

I am reminded of the old saw:" You think you know what I meant, but what you didn't know was that I didn't say what I mean".

Without causing you to to get so worked up that you are in danger of popping your foofle valve, could I suggest you look carefully at what Ambidjii have done (probably more correctly --- what is so fundamentally wrong with the processes CASA has inherited from Airservices) with the data. Please have a closer look at how and when/where Ambidjii have selected data, then midcalculated. Perhaps you could explain why they have excluded the long period of time (often with higher movement rates than the last nine years) when there were ZERO MAC.

A few of us with direct experience of the matter (including a number of CFIs/ATOs/CPs at YSBK) have very supportable opinions on why we have had the recent MACs, and it IS NOT GAAP procedures at the heart of the problem.

As I have said, and many agree, the results fail the common sense test.

I know Dick's source of analysis here, a very well know and highly respected, (and published) practitioner in the risk management field, with wide industry experience, including (especially) aviation. I would go so far as to say he could lay claim to being "the father" of modern industrial risk management in Australia (a different field to reliability analysis --- where some of our self proclaimed academic aviation "risk" experts hail from), who has had significant input into the ARM model, and AS/NZ 4360:1999 and 2004. He has done some masterful analysis on the use and abuse of the ARM model, all published. He is professionally, in all respects, on a par with Professor O'Neil at ANU.

Tootle pip!!

[Ian Bryce]

A word on statistical techniques - how to predict from sparse data.
Leadsled said in post 259:

"Perhaps you could explain why [Ambidji] have excluded the long period of time (often with higher movement rates than the last nine years) when there were ZERO MACs".
A good question - it is a common problem that the actual data is too thin to analyse normally. We have 40 years data but when split into decades there are too few MACs to get a smooth curve.
One is tempted to take an average of the lot - but this gives equal weight to ancient data when practices and technologies were different.
Or one can take the latest year, decade, 20 yrs etc - but the weighting jumps from zero to full at some arbitrary date.

A sounder approach is to apply most significance to the latest data (eg decade), less to the previous decade, and even less to the previous again, etc. There is no sharp cutoff. One only needs to decide on a "relaxation period" where the technology has changed a certain amount, and where the relative weighting falls to 1/e = 0.368.

It is called "exponentially reducing weighting" and is all in Wiki.

I used this method (page 9), taking a relaxation period of 30 years (the results are not very sensitive to this). Thus to predict future MACS, we take a weighted average of all past data. The actuals for 2000-09 are weighted with 1.0, for 1990-99 with 0,717, for 1980-89 with 0.513, and for 1969-70 with 0.368. The result for BK is a rate of 0.16 pa. (The Springside MAC model, unifying all 6 aerodromes, later gives 0.14 for BK.

This method optimally combines the empty decade with the high decade, and all the other data. One rule for all sites. No discussion or selection of the "best feeling" data is necessary.

[Capn Bloggs]

this gives equal weight to ancient data when practices and technologies were different.

For a start, since we didn't have whizzbang gadgets in the cockpit such as GPS in the 70s to distract us, the only thing we could do to be entertained was to LOOK OUT, a technique which is known to help prevent MACs in low speed circuit environments.

[OZBUSDRIVER]

Ahhhhhh Bloggsy.......how does your theory explain the MACs that occured from 67 to 78?????

More likely a standard of training or a change in rules OR a reduction in services around 02.....now what happened about that time?

[Big_Binocs]

Re the closing of Westmead IRP - an idea - could it have been for controller convenience - they have an easier time visually scanning in 2 directions than 3?

If I recall correctly, Westmead was removed to reduce the amount of VCAs into the Sydney CTR, I'm not sure but I think there was also some redrawing of the SY CTR to assist this as well as some changes to improve the flow of traffic through the LOE. There were probably even other reasons as well. Nothing to do with making it "easier" for controllers, we seem to do fine watching five IRPs at Camden. However, watching 360 IRPs is going to be a pain! Although at YSBK it would really only be about 125.

There IS a strobe at 2RN, on a smaller tower near the base. And if I remember right there still are strobes in the LOE and there were even some that led you to WST when it was there, not sure though my memory ain't what it used to be.

My understanding of the actual 2RN IRP is that you would fly roughly over the arrow on the VTC, on the south side of the Mast. This is assuming the pilot of the aeroplane is sitting on the left side of the plane thereby being able to comfortably and easily look out his window and down at the mast as he approaches and make the report abeam it. That would appear to be the logical correlation between the VTC markings and the inbound procedure, and in line with the reccommended inbound tracks. I'm not a pilot so I couldn't say how an aviator might interpret it. But other than someone driving a chopper, the Optica or the P68 Observer, if you flew right over the mast how would you see it?

Anecdotally, I can say that watching aircraft inbound all day on the TSAD, that PSP must be about 6 miles wide. I would hazard a professional guesstimate and say that no more than 10% of aircraft are actually at the correct point when reporting "AT Prospect" or 2RN for that matter. Very few pilots actually give correct positions with their inbound call. The majority of PSP and 2RN traffic are usually 2-3NM from the correct IRP in a variety of directions. Although I will say that reporting for 2RN is marginally better than PSP.

I sometimes think that pilots have forgotten the art of accurate reporting or they don't realise that those calls aren't just for the benefit of the Tower controller but for situational awareness of other aircraft as well! Truly, we would be giddy with delight if everyone reported their "actual" position e.g. "2 miles west of Prospect" or "1 mile south of 2RN" or be AT the IRP when they say they are.

I'm not a statistician or a mathematical genius (I have enough trouble just counting to 6 ), but I think that going from two to three IRPs and saying that it will significantly reduce the chance of a MAC is a bit simplistic. 40 IRPs is not going to prevent two aircraft in bound from Camden and using 2RN potentially having a MAC.

Also, I would think that even if there were no fixed IRPs that schools would tend to teach their students a "standard route" to approach the field for ease of training, standardisation etc and that these would probably be as obvious and valid to other schools as well. So you would tend to get "clustering" of inbound aircraft routes anyway. Not to mention "inbound 6 miles west of YSBK" could be anywhere and is not very informative for situational awareness.

As for having no fixed IRPs, with my admittedly basic understanding gained from the ATSB and NTSB documents I have trawled from the web, MACs have an increased risk when there are shallow converging angles and same direction involved e.g. less than 30? degrees between inbound tracks. That old "constant reference in the window thingy". This is potentially more likely with no fixed IRP, how would you model that risk? Not to forget the increased risk of head-on MAC. I think people forget the segregated altitudes only apply in the CTR. Particularly off 29, most departing aircraft get to the boundary and immediately climb when in G.

Increasing the amount of IRPs or getting rid of them altogether after only one MAC seems to be more a knee-jerk reaction. In Australia, more GA MACs at aerodromes occur in the circuit and usually around final, or so I understand from the ATSB Discussion Paper B2004/0114 ,May 2004 Review of Midair Collisions Involving General Aviation Aircraft in Australia between 1961 and 2003.


The majority of YSBK traffic will naturally use 2RN and PSP because of the geography, structure of airspace, the location of the training area and the two IFR arrival routes ( Watle4 & Ric4 STARS) track through them. Anecdotally I would have to say that I think that only 20% of our traffic, at most, would actually use WST if it came back and the majority of those would be helicopters.

Anyway, just my 2c worth.

Of course these are solely my own opinion and do not necessarily reflect those of my employer or colleagues.

[LeadSled]

Ian,
With respect, I would make the following comments:

A good question - it is a common problem that the actual data is too thin to analyse normally. We have 40 years data but when split into decades there are too few MACs to get a smooth curve.

In fact, the sample is so small that we can examine the circumstances of each MAC --- and we have, and there is no common trend, each one was unique. In my view, attempting to draw any conclusions from smoothing the data does nor produce a valid outcome.

What I have already alluded to is "other factors" that have become evident in the last "9" years (MACs are only party of the story) that have resulted in some disturbing trends ---- beyond, of course, the fact that GA is dying.

There is no sharp cutoff. One only needs to decide on a "relaxation period" where the technology has changed a certain amount,

Except for one problem, there have been NO significant technology changes, indeed most of the GA fleet is 30 or more years old.

A new C-172 is just an old C-172 (including the grandfathered certification standards) with Garmin glass added, lovely stuff, but it doesn't change how you fly around a VFR circuit --- or shouldn't, and in any event there are only a relative handful of such aircraft in the civil fleet. There is no shortage of data on fleet average age.

A rigid fact of life for many flying schools is that the only affordable replacement a for a 30-40 year old C-152 is a rebuilt 30-40 year old C-152--- or a PA-23 likewise.

There are aircraft in daily use today, that I first flew "out of the box" 40+ years ago. Despite comments in another post, I don't believe pilots are "head down" mucking around with GPS in the circuit, and GPS is about the only major piece of "technology" that you could call new.

The training syllabus has been mucked around with in that time, for little benefit, but the core syllabus has not really changed much since the mid-1930 --- but how the volume and style of regulation has changed.

There has been significant technology change in the above period, as it effects air transport, both aircraft and CNS/ATM, but not General Aviation.

A mass of historical data is available, Bankstown, Archerfield and Moorrabin are quite places these days, compared to the mid-60s to the mid-70s.

Tootle pip!!

[Frank Arouet]

[Capn Bloggs]

Caused in part by incessant (and unnecessary) mucking around with what used to be a simple, straightforward, easily understood airspace system that "had" to be changed to match the rest of the world, driving people away from flying in droves. What a crock.

[OZBUSDRIVER]

In fact, the sample is so small that we can examine the circumstances of each MAC --- and we have, and there is no common trend, each one was unique. In my view, attempting to draw any conclusions from smoothing the data does nor produce a valid outcome.

-Leadsled

Statistics-The mathematics of the collection, organization, and interpretation of numerical data, especially the analysis of population characteristics by inference from sampling.

Why do we use statistics? This goes back to my highschool ed......forest for the trees stuff....this sample is so bloody small we can study each individual event and make a judgment on each individual event. We can see the DIRECT relationship between each event or aspects of each event common to all others without the need to apply a model. Bugga me, its was there right in front of me all the time. Our 17MACs over the last fourty years as opposed to a sample available in the US.

We can (and do) apply the cause(s) of each individual event as a set of rules...."Our rules are written with the blood of our martyrs." As it has always been in Australian aviation....every major prang brings about a new rule to stop us from re-enacting that terrible event. What stat will predict that!

This then presents a problem for continued changes to airspace rules...if a statistical model cannot be applied because the probability is incalculable on such a small "Sample" of events....then....how is it possible to place a value on a statistically insignificant probability of an accident occuring and the old rules or even any new rules?....or in plain english...."how much do you want to bet that an accident will NOT happen under a rule change to ICAO D/C or even FAA D?"

Leadsled, I take it that you support the retention of GAAP based on your belief that the MAC events that DID occur over the years since introduction have little, arguably so, to do with procedures.

WHERE is the US Safety case for their D and E airspace arrangements?
Considering the FAA modified and codified what was the Done Thing by US aviators. You have to ask...as does Ambidji...is the US operating in a high risk environment?

[Frank Arouet]

Caused in part by incessant (and unnecessary) mucking around with what used to be a simple, straightforward, easily understood airspace system

Capn Bloggs;

Maybe: But you ignore a succession of government’s with a culture of NOT supporting general aviation in this Country because there are no votes in it for them, an apathetic industry, over regulation and under representation, prescriptive knee jerk reactions to problems that don’t exist, an indolent-incompetent and aggressive Regulator, increasing maintenance costs, mandatory paperwork requirements, training issues, medical bastardry, regulatory corruption, cronyism, decreasing aircraft ownership, ownership costs, threats of forced technology with no appreciative benefit, ASIC, AVID, with associated perpetual costs, petty changes such as security locking devices, the security controlled airports farce, wasted taxpayers money with aggressive and adventurist legal pursuits and increasing costs to maintain and increase the bureaucracy may have something to do with it as well.

Finally something peculiar to Australian aviation, an over opinionated population of so called professional demigod’s who can’t get over the idea that they don’t own the entire sky and enthusiastically vilify and defame others may have an opinion that clashes with their own agendas. The same experts who can’t quiet grasp the concept of divide and conquer. A concept that those who are the main cause of the problems are exploiting.

[LeadSled]

Clinton,

I would make the following comments:

For example, if your observations about the GAAP MAC sample size and the consequences for the conclusions that may or may not be validly drawn from the data are correct, there is no valid basis upon which to compare Australian and US MAC rates.

In the context of the Ambidjii report, they are effectively saying that 350 or so US Class D zones are operating at an intolerable level of risk. Quite apart from any FAA/AOPA/FSF view on that matter, the aviation insurance underwriting sector does not see it that way, either.

In my opinion, only broad generalizations can be drawn by comparisons, such as (unsurprisingly) that collision risk is greater in the circuit area than en-route (despite a in my opinion hilariously inept study by Airservices, that neglected the fact that most of the US mid-west is around 5-6,000 ft AMSL).
However, MACs are rare enough in US that they too can be looked at individually.

The major conclusion I draw: that MAC is not a major cause of incidents or accident, given the "big picture", but looking at the Australia reaction to matters airspace, that are entirely uncontentious in just about any other country, one could easily get the impression that MACs were the biggest air safety problem in Australia.

In about March 2002 (or thereabouts, give or take about a year) US FSF did an interesting study on Australia's air safety record, every Australian pilot, LAME and ATC should read it.

Yet some people seem to have no compunction in extrapolating Australian accident/incident rates and concluding that Australia has a higher accident/incident rate than the US.

One of the problems we have here, is the belief that Australia has the world's best air safety record, a severe case of "rose coloured glasses". We are fond of quoting "jet fatalities", and can't even get that right, we have had one, wiped out most of the Mareeba Council, but we confine ourselves to "RPT" jet, and pat ourselves on the back. Selectively narrow down the definitions enough, and the record will always be perfect

Given the size of the whole Australian fleet, provided you don't overdo it, a study of the Australia accident and incident rates is instructive. As far as I am concerned, ICAO categories should be used, but we fiddle with that, too, to produce "more flattering" figures.

Some time ago, a study was commissioned, using well known insurance co. investigators, and independent (of underwriters and the aviation sector) statisticians. The then Minister had the whole paper checked by the US NTSB, who concurred with the results, including the caveats as to the limitations of interpretations.

The results, overall and in individual categories, did rather lift the rose coloured glasses for that Minister.

Prorata, GA came out the best of the categories vis a vie USA, the RPT accident and incident rates were not good. Australia did come out better than "JAA" countries. CIS states and Africa (except SA) were not included. I have no reason to believe, based on the record since, that we have significantly improved. Most of the "apparent" improvement in recent years is just that, apparent rather than real, a change in the balance of who is doing what within categories.

I'll leave experts like Ian to grapple with the issue, but I understand, from a number of experts in the area, that the number of hours/miles flown by Australian aircraft is still so small as to still be statistically insignificant.

There is an interesting paper that is a contribution to the current Aviation Green Paper, saying exactly that. It has been suggest that the loss of a single B737NG would catapult Australia from "the best" to close to the worst.

There are no shortage of individual airlines that have more high capacity transport aircraft that all operators in Australia combined, who have excellent accident/incident records.

In my opinion, just about the biggest threat to air safety is complacency, there is no shortage of complacency in the areas at which I am looking.

Tootle pip!!

Late PS: The differences in accident with the US/Australia study mentioned above were so great that no amount of Australian/CASA "rationalization" about errors/incompleteness of US data collection could explain away the differences.

There was NO category where Australia equaled or bettered the US.

What really surprised me was how bad the western Europe/UK figures were, leading to the obvious connection between draconian regulation and air safety outcomes ----- is there a proportional relationship with the volume of regulation and the accident/incident rate. Again, Africa excluded.

The greater the regulatory burden, the greater the accident rate ----is almost a valid conclusion from the public databases.

If the US consistently produces the world's best air safety outcomes, in all categories (and they do --- have a really good look at 20 year running averages) what are we, or the UK CAA, etc., achieving, except running aviation into the ground.

[OZBUSDRIVER]

[/REBOOT]

Leadsled could you PLEASE just stick to one side of the argument!

[Big_Binocs]

Clinton

One question for ATC'ers re Westmead: Would reinstatement of the Westmead IRP create any complications for EMS helos getting in and out of the HLS at the hospital?

That little wedge of airspace between YSBK and WST would be a very congested piece of airspace if WST were reinstated.

As it stands now there are numerous heli ops in that area with choppers operating in all directions at various altitudes, there is the heliport at RSH, the entry/exit to R405, floatplanes to the harbour and aircraft departing north from YSBK.

The Medical choppers already contend with that and manage to negotiate transit through PSP and the LOE into WST, whether they currently encounter complications with just that mix I do not know. I would suggest that if you were to add more fixed wing traffic transitting that area into and out of YSBK that there aren't enough eyes in the world to keep an effective lookout for traffic that could be anywhere and tracking any which way at various altitudes and probably totally unannounced, no alerted see-and-avoid. That's a lot of criss-crossing traffic, compressed into an operating area less than 2000ft and what about the entry/exit altitudes for YSBK?

Personally I don't see WST being reopened, or the need for it either. The VCA prevention for the SY CTR would probably rate as more important. But I also didn't think I'd see GAAP turn into D in my time either so what would I know?

As others have mentioned I think better education and compliance would be a more effective option. And not only for student pilots and instructors, there are plenty of "professionals" and private pilots who seem to have the same troubles.

Cheers