How does one know whether the SMS is working?
Thread Starter
Join Date: Jun 2008
Location: In the Auzzie outback
Posts: 164
Likes: 0
Received 0 Likes
on
0 Posts
How does one know whether the SMS is working?
How would a person know whether an operators SMS is:
A: Monitoring and measuring company safety performance; and
B: Making periodic improvements in company safety performance?
A CEO point of view would be interesting to know.
Just wanting to know what people in the industry think about their SMS and if they care enough about it to follow how it's working in their operation.
A: Monitoring and measuring company safety performance; and
B: Making periodic improvements in company safety performance?
A CEO point of view would be interesting to know.
Just wanting to know what people in the industry think about their SMS and if they care enough about it to follow how it's working in their operation.
Join Date: Jun 2008
Location: Behind a dusty desk, and in some really hot, dusty, wet and cold places subject to who is paying the bill. But mostly Gods own land.
Posts: 261
Likes: 0
Received 0 Likes
on
0 Posts
SMS can be seen as one of those things that costs but you cant see why because in reality in can be very difficult to see why your paying for it, beyond the legal requirement for it.
Even the saying "if you think safety is expensive try paying for an accident" does not explain away the cost issue of implementing an SMS.
As you say you establish a monitoring system, most companies all-ready have them in the form of, for example, Pilots do company check-flights or simulator time, engineers have their work checked by technical records or at certificate of maintenance review periods; these are required by an SMS system to be complimented by a reporting system of some sort. So what you have with a check-ride, or a review of engineers paper-work, is a proactive system of the company going out look for errors within a set system, or systems. The required introduction of a reporting system means errors can be picked up after they occure, alowing an operator to deal with an error reactively.
So as you ask, your CEO wants to know how pilots are doing... ok last year the training officer said that all his pilots suck at landings when he took them out for a check ride, one key indicator, and we had X number of reports via our reporting system saying much the same thing... the little chap who lives under the flight path is in fear of his life... a second key indicator.
So the proactive & reactive systems both suggest that pilots suck at landings; the CEO can now say I will target my money on improvements on this issue and for example insist every pilot does simulator training on landings.
This is a targeted approach to safety, having identifyed an error a course of action has been established to correct it. So how do we know it works?
The very same system that found this error should "hopefully" see the training pilots proactive reports that pilots suck at landings and the reports from the guy in fear of his life under the flight path reduce, or dissapear.
If it works simulator training becomes a standard part of pilot training, if the reports are still there you tray a different approach, more training flights, change the flight profiles, whatever but at least you can measure a corrective actions effectiveness.
Ok you asked how do you know if a sms is working, well the above is ok if you have reports. Some companies have a rubbish culture of handling reporting, no anonymous report systems, or a sack the reporter attitude, so just because there are no reports doesn't leave you high-and-dry.
For example, I have no reports from technical records or from the guy who carries out certificate of maintenance reviews that the engineers are rubbish at paperwork. Do you therefore ignore it? No you proactively go and look at the systems against the systems in place.
For example you go talk to the technical records clerk and "sample check" a work order she is processing and find that you can't read a single word on the work order. The clerk turns out to be an ex-kinder garden teacher and can read crayon scrawl with oily finger prints on it via a form of brail technique. Great so we can read "engineer" but it high-lights the issue that when that clerk leaves we may have a problem.
You move on and do a sample check of a certificate of maintenance review with the senior engineer who is an old school engineer who thinks all this safety stuff is just "a bother" "it just gets in the way!" Ok when we do our sample check we find that everything is great but there is no written procedure for carrying out a certificate of maintenance review.
These two "issues" need addressing, engineers need a targeted and "encouraged" to improve their written skills, we can provide them with pens or whatever approach is deemed best. There needs to be a process written down for the CMR because what happens when the senior engineer isn't there?
so from no reactive reports and no suggestion of an error you have proactively assessed a system, identified two "errors" and can now target a solution. This is fed back into the audit system and assessed again at the next audit where hopefully at the next assessment the errors will not be there.
Safety management is a continuous process of assessment & reassessment, both proactively & re-actively to allow a targeted allocation of resorces to correct an error.
Sorry it's long, I hope it helps
Miles
Even the saying "if you think safety is expensive try paying for an accident" does not explain away the cost issue of implementing an SMS.
As you say you establish a monitoring system, most companies all-ready have them in the form of, for example, Pilots do company check-flights or simulator time, engineers have their work checked by technical records or at certificate of maintenance review periods; these are required by an SMS system to be complimented by a reporting system of some sort. So what you have with a check-ride, or a review of engineers paper-work, is a proactive system of the company going out look for errors within a set system, or systems. The required introduction of a reporting system means errors can be picked up after they occure, alowing an operator to deal with an error reactively.
So as you ask, your CEO wants to know how pilots are doing... ok last year the training officer said that all his pilots suck at landings when he took them out for a check ride, one key indicator, and we had X number of reports via our reporting system saying much the same thing... the little chap who lives under the flight path is in fear of his life... a second key indicator.
So the proactive & reactive systems both suggest that pilots suck at landings; the CEO can now say I will target my money on improvements on this issue and for example insist every pilot does simulator training on landings.
This is a targeted approach to safety, having identifyed an error a course of action has been established to correct it. So how do we know it works?
The very same system that found this error should "hopefully" see the training pilots proactive reports that pilots suck at landings and the reports from the guy in fear of his life under the flight path reduce, or dissapear.
If it works simulator training becomes a standard part of pilot training, if the reports are still there you tray a different approach, more training flights, change the flight profiles, whatever but at least you can measure a corrective actions effectiveness.
Ok you asked how do you know if a sms is working, well the above is ok if you have reports. Some companies have a rubbish culture of handling reporting, no anonymous report systems, or a sack the reporter attitude, so just because there are no reports doesn't leave you high-and-dry.
For example, I have no reports from technical records or from the guy who carries out certificate of maintenance reviews that the engineers are rubbish at paperwork. Do you therefore ignore it? No you proactively go and look at the systems against the systems in place.
For example you go talk to the technical records clerk and "sample check" a work order she is processing and find that you can't read a single word on the work order. The clerk turns out to be an ex-kinder garden teacher and can read crayon scrawl with oily finger prints on it via a form of brail technique. Great so we can read "engineer" but it high-lights the issue that when that clerk leaves we may have a problem.
You move on and do a sample check of a certificate of maintenance review with the senior engineer who is an old school engineer who thinks all this safety stuff is just "a bother" "it just gets in the way!" Ok when we do our sample check we find that everything is great but there is no written procedure for carrying out a certificate of maintenance review.
These two "issues" need addressing, engineers need a targeted and "encouraged" to improve their written skills, we can provide them with pens or whatever approach is deemed best. There needs to be a process written down for the CMR because what happens when the senior engineer isn't there?
so from no reactive reports and no suggestion of an error you have proactively assessed a system, identified two "errors" and can now target a solution. This is fed back into the audit system and assessed again at the next audit where hopefully at the next assessment the errors will not be there.
Safety management is a continuous process of assessment & reassessment, both proactively & re-actively to allow a targeted allocation of resorces to correct an error.
Sorry it's long, I hope it helps
Miles
Thread Starter
Join Date: Jun 2008
Location: In the Auzzie outback
Posts: 164
Likes: 0
Received 0 Likes
on
0 Posts
It does help thank you Miles,
Some very good examples there.
Summed it up to assessment, repair or fix then reassessment.
Corrective actions effectiveness are measurable in this mannor too and at the end of the day the SMS will only work as good as the people who run it.
Some very good examples there.
Summed it up to assessment, repair or fix then reassessment.
Corrective actions effectiveness are measurable in this mannor too and at the end of the day the SMS will only work as good as the people who run it.
Join Date: Mar 2005
Location: Kingwood and Onalaska TX
Age: 79
Posts: 24
Likes: 0
Received 0 Likes
on
0 Posts
How would one know if their SMS is working?
Record and test. Sounds simple but requires a way to measure.
First, let’s look at SMS as a three legged milk stool. SMS is the seat and the legs represent the following;
Leg one...define the standards. Standards are the regulations, best practices, industry, insurance and company. What are we required to do?
Leg two...develop procedures to comply with the standards. How will we define what we are required to do and how are we going to do it? Empower individuals with proper authority and responsibilities to manage the process.
Leg three...develop a recording process (hazards reports, continuous improvement reports, internal audits etc.) that define your ways to address the procedures that address the standards. Finally test the process...are we doing what we say, do we close the loop (issues defined are addressed and closed).
All three legs are required to support the system.
First, let’s look at SMS as a three legged milk stool. SMS is the seat and the legs represent the following;
Leg one...define the standards. Standards are the regulations, best practices, industry, insurance and company. What are we required to do?
Leg two...develop procedures to comply with the standards. How will we define what we are required to do and how are we going to do it? Empower individuals with proper authority and responsibilities to manage the process.
Leg three...develop a recording process (hazards reports, continuous improvement reports, internal audits etc.) that define your ways to address the procedures that address the standards. Finally test the process...are we doing what we say, do we close the loop (issues defined are addressed and closed).
All three legs are required to support the system.
Join Date: Jun 2008
Location: Behind a dusty desk, and in some really hot, dusty, wet and cold places subject to who is paying the bill. But mostly Gods own land.
Posts: 261
Likes: 0
Received 0 Likes
on
0 Posts
Seawings,
I like the picture you paint, I havn't heard that one before, you approach of empowering is one a really believe in but havn't heard someone else advocate it before... have you looked at change management principles? wiki adcar or change management, if you havn't allready looked at it it may help... just a thought.
Another though about the stool analogy, when using it do you think it adequately explains the feedback, reassessment issue of an SMS?
I only ask as I've found that when trying to explain SMS, auditing, safety etc to those who are not really very enthusiastic about the subject that being able to draw the logical flow of the process help... how have you got on?
Cheers
Miles
I like the picture you paint, I havn't heard that one before, you approach of empowering is one a really believe in but havn't heard someone else advocate it before... have you looked at change management principles? wiki adcar or change management, if you havn't allready looked at it it may help... just a thought.
Another though about the stool analogy, when using it do you think it adequately explains the feedback, reassessment issue of an SMS?
I only ask as I've found that when trying to explain SMS, auditing, safety etc to those who are not really very enthusiastic about the subject that being able to draw the logical flow of the process help... how have you got on?
Cheers
Miles
Join Date: Mar 2005
Location: Kingwood and Onalaska TX
Age: 79
Posts: 24
Likes: 0
Received 0 Likes
on
0 Posts
Quote: have you looked at change management principles?
Management of Change (MOC) is an intergrel part of the “test and report” process. As one part it documents your change and the review process (test) determines the completeness of your process.
Quote : do you think it adequately explains the feedback, reassessment issue of an SMS?
The analogy of the stool is to illustrate the circle of SMS, (define, develop and test). The loop is closed when you find the procedures produced to manage the standards are effective. These are never carved into stone so continual monitoring of the processes are necessary to keep the program alive.
Quote: to those who are not really very enthusiastic about the
I have used the simple stool analogy to try and dispel the complexity of the issue. You can make it too COMPLICATED and the system will fail, however, using the phased approach used by IS-BAO (a three part certification) you can keep it in manageable portions.
Phase I is really the understanding of the basic elements and starting the process to put them into place, (FOM revision to incorporate the basics, safety program incorporating the needed elements...responsibilities, empowerments and testing procedures, ) to manage the program. DEFINING THE LOOP
Phase II looks for implementation of the elements and the application of the elements into a cultural change within the department (manuals reflect changes, processes reflect changes, records being developed to show progress and compliance). DEVELOPING THE LOOP
Finally, Phase III looks for a cultural change to have occurred, where the elements have become ingrained into the department and the processes are becoming automatic. CLOSING THE LOOP
I think that too many try to embrace the whole concept at once and not realizing that a step by step approach, taking years to fully become a culture, is required....”Rome was not built in a day”.
Management of Change (MOC) is an intergrel part of the “test and report” process. As one part it documents your change and the review process (test) determines the completeness of your process.
Quote : do you think it adequately explains the feedback, reassessment issue of an SMS?
The analogy of the stool is to illustrate the circle of SMS, (define, develop and test). The loop is closed when you find the procedures produced to manage the standards are effective. These are never carved into stone so continual monitoring of the processes are necessary to keep the program alive.
Quote: to those who are not really very enthusiastic about the
I have used the simple stool analogy to try and dispel the complexity of the issue. You can make it too COMPLICATED and the system will fail, however, using the phased approach used by IS-BAO (a three part certification) you can keep it in manageable portions.
Phase I is really the understanding of the basic elements and starting the process to put them into place, (FOM revision to incorporate the basics, safety program incorporating the needed elements...responsibilities, empowerments and testing procedures, ) to manage the program. DEFINING THE LOOP
Phase II looks for implementation of the elements and the application of the elements into a cultural change within the department (manuals reflect changes, processes reflect changes, records being developed to show progress and compliance). DEVELOPING THE LOOP
Finally, Phase III looks for a cultural change to have occurred, where the elements have become ingrained into the department and the processes are becoming automatic. CLOSING THE LOOP
I think that too many try to embrace the whole concept at once and not realizing that a step by step approach, taking years to fully become a culture, is required....”Rome was not built in a day”.
Join Date: Jun 2008
Location: Behind a dusty desk, and in some really hot, dusty, wet and cold places subject to who is paying the bill. But mostly Gods own land.
Posts: 261
Likes: 0
Received 0 Likes
on
0 Posts
Nice approach and something to think about...thanks!
Miles
Miles
Join Date: May 2001
Location: Canada
Posts: 21
Likes: 0
Received 0 Likes
on
0 Posts
We're not too thrilled about SMS in Canada...
...or the fact that Transport Canada is not meeting their ICAO obligations. SMS is being implemented in all sectors of Canadian aviation, despite that only the major airlines have been enabled by regulations to operate under SMS. Transport Canada has essentially ceased all regulatory oversight of the entire industry, including the airlines. Hope what happened in the US doesn't happen here too.
http://safeskies.ca/events/2009-04-2...e_Problems.pdf
FOR IMMEDIATE RELEASE
MAY 11, 2009
CANADA
Air Safety Problems Exposed Online
During Question Period on April 23, the Honorable Rob Merrifield, Secretary of State for Transport, rejected the claims of the NDP Transport Critic, Dennis Bevington, that Canada is not meeting international standards for aviation safety. Perhaps Mr. Merrifield has been duped by bureaucrats in the same manner as the Canadian travelling public.
In addition to evidence provided during extensive Committee hearings held in 2007 over a controversial Bill to amend the Aeronautics Act, further proof was recently aired in a landmark Round Table discussion on Air Safety held at Parliament Hill and hosted by Mr. Bevington.
The organizers have now created a new website, www.safeskies.ca, which allows media and members of the public to experience the Round Table: to see and hear the revelations made by pilots, industry insiders, whistleblowers and accident victims.
“Canadians need to know that despite lessons learned from aviation accidents and related problems in the rail industry, Transport Canada has abandoned them and left their lives in the hands of those responsible for profit margins – a dangerous strategy which is likely to have tragic consequences.”, claims the SafeSkies website.
“International Civil Aviation Organization (ICAO) says you absolutely have to retain your traditional oversight and the fact of the matter is, Transport Canada is simply not doing that and has no intention of doing it” states Greg Holbrook, National Chair of the Canadian Federal Pilots Association (CFPA), during his presentation.
Apparently the Canadian Public is travelling under a false sense of security. The major airlines have already been empowered to regulate themselves, and are operating without scrutiny from Transport Canada. A similar situation in the US resulted in national disaster, thousands of cancelled flights and huge public outcry. Business aircraft have been governing themselves since 2003 – though enabling regulations were not actually passed until 2005. Transport Canada’s own audit in 2007 revealed this has created a system plagued with troubling holes. Air Taxi and Commuter aircraft in Canada have not yet been enabled by regulations to govern themselves, yet Transport Canada is no longer overseeing this major sector of aviation either.
Other Round Table presentations highlighted the systemic problems within Transport Canada, and the known failures which have caused multiple deaths and destroyed lives.
An alliance of individuals and industry representatives is now being formed with its prime focus to seek resolution to these issues, hold Transport Canada accountable to the public, and restore Canadian’s faith in air safety. For information about future developments on these topics, check back with Safe Skies and subscribe to the newsletter.
Prepared by
Kirsten Stevens and Kirsten Brazier
Advocates for Air Safety
Canada
http://safeskies.ca/events/2009-04-2...e_Problems.pdf
FOR IMMEDIATE RELEASE
MAY 11, 2009
CANADA
Air Safety Problems Exposed Online
During Question Period on April 23, the Honorable Rob Merrifield, Secretary of State for Transport, rejected the claims of the NDP Transport Critic, Dennis Bevington, that Canada is not meeting international standards for aviation safety. Perhaps Mr. Merrifield has been duped by bureaucrats in the same manner as the Canadian travelling public.
In addition to evidence provided during extensive Committee hearings held in 2007 over a controversial Bill to amend the Aeronautics Act, further proof was recently aired in a landmark Round Table discussion on Air Safety held at Parliament Hill and hosted by Mr. Bevington.
The organizers have now created a new website, www.safeskies.ca, which allows media and members of the public to experience the Round Table: to see and hear the revelations made by pilots, industry insiders, whistleblowers and accident victims.
“Canadians need to know that despite lessons learned from aviation accidents and related problems in the rail industry, Transport Canada has abandoned them and left their lives in the hands of those responsible for profit margins – a dangerous strategy which is likely to have tragic consequences.”, claims the SafeSkies website.
“International Civil Aviation Organization (ICAO) says you absolutely have to retain your traditional oversight and the fact of the matter is, Transport Canada is simply not doing that and has no intention of doing it” states Greg Holbrook, National Chair of the Canadian Federal Pilots Association (CFPA), during his presentation.
Apparently the Canadian Public is travelling under a false sense of security. The major airlines have already been empowered to regulate themselves, and are operating without scrutiny from Transport Canada. A similar situation in the US resulted in national disaster, thousands of cancelled flights and huge public outcry. Business aircraft have been governing themselves since 2003 – though enabling regulations were not actually passed until 2005. Transport Canada’s own audit in 2007 revealed this has created a system plagued with troubling holes. Air Taxi and Commuter aircraft in Canada have not yet been enabled by regulations to govern themselves, yet Transport Canada is no longer overseeing this major sector of aviation either.
Other Round Table presentations highlighted the systemic problems within Transport Canada, and the known failures which have caused multiple deaths and destroyed lives.
An alliance of individuals and industry representatives is now being formed with its prime focus to seek resolution to these issues, hold Transport Canada accountable to the public, and restore Canadian’s faith in air safety. For information about future developments on these topics, check back with Safe Skies and subscribe to the newsletter.
Prepared by
Kirsten Stevens and Kirsten Brazier
Advocates for Air Safety
Canada
Last edited by rbaron; 18th May 2009 at 18:41. Reason: link to US Backgrounder added...
Join Date: May 2009
Location: East Kilbride, Scotland, UK
Posts: 12
Likes: 0
Received 0 Likes
on
0 Posts
Hi,
I've been a bit of a lurker for a while but have only just registered so this is my first post.
Firstly, let me state my interest - I work for a software company that publishes an integrated Safety and Quality Management solution for the aviation sector, so what follows is a reference to our product, but I think its relevant for the post.
To maintain non-commercial focus I won't mention our product by name although if you are really interested, please PM me.
Our frame-work is based on a Plan-Do-Check-Act cycle, and its the check/act part that can be difficult, although we think we've cracked it.
Incidents and occurence reporting and investigations are captured within customisable forms and can include an estimated $ risk value.
A robust analysis tool lets the accountable executive monitor the aggregate of incidents and isolate and prioritise areas of improvements.
Data from near misses can be captured, which means that preventative actions can be implemented before accidents occur and the $ saved can be calculated.
Does this help?
I've been a bit of a lurker for a while but have only just registered so this is my first post.
Firstly, let me state my interest - I work for a software company that publishes an integrated Safety and Quality Management solution for the aviation sector, so what follows is a reference to our product, but I think its relevant for the post.
To maintain non-commercial focus I won't mention our product by name although if you are really interested, please PM me.
Our frame-work is based on a Plan-Do-Check-Act cycle, and its the check/act part that can be difficult, although we think we've cracked it.
Incidents and occurence reporting and investigations are captured within customisable forms and can include an estimated $ risk value.
A robust analysis tool lets the accountable executive monitor the aggregate of incidents and isolate and prioritise areas of improvements.
Data from near misses can be captured, which means that preventative actions can be implemented before accidents occur and the $ saved can be calculated.
Does this help?
Join Date: Jul 2002
Location: UK Sometimes
Posts: 1,062
Likes: 0
Received 0 Likes
on
0 Posts
SMS Regulations and compliance thereof are pretty much wasted if organisations don't have a top-down, bottom-up, proactive safety culture.
Such a safety culture is one which is demonstrable by having a transparent just culture, an open-and-honest reporting culture and a proactive learning culture.
Without those, the organisation just has a process and a set of regulations.
Such a safety culture is one which is demonstrable by having a transparent just culture, an open-and-honest reporting culture and a proactive learning culture.
Without those, the organisation just has a process and a set of regulations.
To paraphrase James Reason (‘Safety Culture and Safety Management,’ RAeS Lecture 21 May):-
- A ‘good’ (working) SMS should reflect that human factors dominate risk – the SMS is a well informed system.
- SMS is a top down management process, but requires a complementary bottom up error management process – check that error management is in place and being used.
- Check what distinguishes your organisation from other organisations; both those who have high incident rates and those who do not.
- Everyone in the organisation is worried when there are no serious incidents or accidents – good organisational safety culture.
- The organisational culture driving the SMS (as above) should be ‘CEO proof’.
- The CEO must ‘not forget to be afraid’ – always being aware of the risk and consequences of serious incidents and accidents, i.e. the CEO is continuously reminded by the output of the SMS – not just by the presence/absence of ‘ring binders’ on the shelf.
Join Date: May 2007
Location: Vancouver Island
Age: 57
Posts: 101
Likes: 0
Received 0 Likes
on
0 Posts
SMS and your regulatory agency
I am curious how other country's regulatory agencies are going about implementing SMS.
In Canada, it has been written into the regulations in some sectors, but is on the verge of regulation for all sectors - although the associated Act has not yet been approved in Parliament. At the same time as advising all organizations that they must implement an SMS, and during the implementation, they are changing the very way they audit and inspect. The National Audit Program has been cancelled for instance, and the Frequency of Inspection Policy replaced by an SMS Surveillance Policy. Many people here are concerned that Transport Canada already had a difficult job of enforcement in such a large, geographically diverse country - and that operators who operate safely already have a form of SMS, while those who don't (and don't care to) can make any piece of paper look good to satisfy mere paper inspections.
Have other countries passed associated Acts? Have your regulatory agencies maintained the same audit and inspection policies, or changed them? What has your experience been with your agency during and after SMS implementation? Do you feel that SMS will make a difference to the operator who prefers to cut corners?
In Canada, it has been written into the regulations in some sectors, but is on the verge of regulation for all sectors - although the associated Act has not yet been approved in Parliament. At the same time as advising all organizations that they must implement an SMS, and during the implementation, they are changing the very way they audit and inspect. The National Audit Program has been cancelled for instance, and the Frequency of Inspection Policy replaced by an SMS Surveillance Policy. Many people here are concerned that Transport Canada already had a difficult job of enforcement in such a large, geographically diverse country - and that operators who operate safely already have a form of SMS, while those who don't (and don't care to) can make any piece of paper look good to satisfy mere paper inspections.
Have other countries passed associated Acts? Have your regulatory agencies maintained the same audit and inspection policies, or changed them? What has your experience been with your agency during and after SMS implementation? Do you feel that SMS will make a difference to the operator who prefers to cut corners?
