True Root Cause and Corrective Action (RCCA) Training

True RCCA Training Class Objective:

To provide a project management approach, tools, and tactics for Teams to conduct successful True root cause investigations and define and implement effective and accountable corrective action plans that prevent re-occurrence.

Empower more people and functions to lead and/ or more effectively contribute to root cause and corrective action (RCCA) projects

07/13/10

True RCCA Training Class Features:

Utilizes three primary tools:
Project Rolling Action Item List (RAIL) Issue Closure Plan (ICP) Root Cause Verification Action Plan (VAP)

Not all tools referenced are always going to be required.

There is flexibility depending on the complexity and scope of the problem. The Corrective Action Board (CAB), Integrated Corrective Action Team (ICAT), and the project champion will offer guidance. The ICP should be utilized for most issues/ projects and serve as a company record.
Less complex problems can outline and execute the full closure plan including verification actions within the ICP itself.
Ex. PCAS RAIL

Simple issues can be worked and documented within a CAT project tracking spreadsheet

The Method works even for the most complex problems.

The 8D sequence of the steps is critical and should not be cut short or skipped. The time each step will take is again a variable depending on the complexity and scope of the problem.

07/13/10

True RCCA Training Class Features (contd):

Contains notes (highlighted Blue) for key guidance and warnings relative to conducting a successful process and accomplishing the best team performance Clarifies terminology involved in the process Company record of accountability
Willingly, by name documentation Names recorded at each separate progress point since the players may be different

07/13/10

True RCCA Training Class Features (contd):

This training is based on the HBC modified 8-D problem solving steps but with additional instruction offered to increase the focus needed to assure that a team captures true root cause and confidently prevents re-occurrence. Therefore, Step 4 has been expanded into 3 sub steps.
HBC Modified 8-D Process
1. 2. 3. 4.
(steps 1 and 2 switched, Step 8 includes closure justification):

Construct the problem statement Form a team Implement interim containment Determine the root cause (RC)
1. 2. 3. Brainstorm Sort and Analyze RC Verification Action Plan (VAP)

5. 6. 7. 8.

Develop corrective action (CA) Validation of corrective action (Implement a permanent fix) Prevent recurrence, preventive action (PA) Project closure justification, recognize the team

This training assumes that you have some familiarity with the 8-D problem solving process and the basic problem solving tools.

07/13/10

True RCCA Training Class Reference the 8-D steps:

1. Construct the Problem Statement
Take the initiative Go to the floor
See and touch the part, physically examine and interview the situation first hand Talk to everyone who should know something
1. Get on the phone with suppliers if applicable a.) Make sure this is a non defensive discussion and doesnt involve liability

Take written notes

Consider the failure mode/ modes and aircraft operating conditions (consistent?) Utilize a minimum of three relevant people (you + 2 others) to confirm it is:
Correct and focused (single problem) Short/ concise as possible Simple/ clear as possible States the problem/ effect/ fault: does not infer or declare the solution Typically includes who, what, when, where, impact = 1 or 2 sentences
1. 2. Typically employ Quality Eng, Design Eng, Mfg Eng, Tool coordinators, Operators / Mechanics, Ops Team Lead, SCM (MPN, Buyer, Matl Handler) Work hard at this, expect iterations a.) Note: a problem well stated is a problem half solved b.) Note: If you cannot say it simply, you do not understand the problem Albert Einstein c.) Einstein also said: If he had one hour to save the world he would spend 55 minutes defining the problem and only 5 minutes finding the solution.
07/13/10

True RCCA Training Class

1. Construct the Problem Statement (Contd)
Start the project RAIL right away (eventually the Project Leaders responsibility)
First step in a written trackable plan Use for logging anything that needs follow up, no need to limit content Typical columns needed: - Item number (entry order) - Assigned to (responsible) - Priority number - ECD Pert Diagram - Date initiated - Date complete 1 2 - Description - Comments

3 5

Start the Issue Closure Plan (ICP) Project Leaders responsibility

First step: Record the problem statement in the ICP 6 A full written structured and trackable plan that works with the RAIL Show higher level root cause (RC), corrective action (CA), preventive action (PA) investigation and closure tasks descriptions with key indentured interdependent sub tasks Clearly reveals the critical path to closure (RAIL does not) Think Pert diagram for multi-dimensional interdependence of tasks and their sequence Note: keep asking whats the next step? Make sure all ECDs have a basis/ plan, therefore a PCD = planned completion date (or promise) PCDs are: written, scope of task analyzed, prioritized, resources committed Make sure all interdependent handoffs are clear and agreed to (dont drop the baton) Its presentation ready to brief mgmt (retains continuity of actions, 3 panel charts do not) Its a company record of project the RAIL and the VAP are placed into as attachments
07/13/10

Form 78-35140:

Issue Closure Plan (ICP)

Communications and Interim containment are 8D steps for emerging issues (normally not applicable for CAB selected projects)

The key tasks and sub tasks required to reach closure should all be separately numbered according to their interdependent sequential flow (clearly reveals the critical path)

ICP form available under Form tab at http://www.hawkerbeechcraft.com/supply_chain/contractual_flowdown/

05/5/11

True RCCA Training Class

Form 78-35140: Issue Closure Plan (ICP)
Assignee Company ECD PCD Status

As key as the Problem Statement

The key tasks and sub tasks required to reach closure should all be separately numbered according to their interdependent sequential flow (clearly reveals the critical path)

The ICP is the central hub for attached documents and the full project record

08/17/10 07/13/10

Form 78-35140:
Issue Title: Supplier Name & Supplier # Part # (1) Problem Statement:

Issue Closure Plan (ICP)

Date Opened: Status Date: Corrective Action # Technical Significance: Potential Reduction in Component Reliability Significance to Production/Delivery: No Impact Significance to Field: No Impact Project Lead: Buyer:

Product Line: (1.1) Communications 1. 2. 3. 4. (2) Form Team 1. Notify Product Line QA Lead Notify Chief Engineer Notify PC Administrator (Larry Moores Group) Red Flag Notification Released Contact Supplier QA Manager: 2. 3. (3) Interim Containment: 1. 2. 3. Name: Phone Number Email:

Assignee

Company

PCD ECD

Status

C Montgomery

HBC

John Doe, Quality Engineer, 6-XXXX Jim Smith, Design Engineer, 6-YYYY
Determine suspect lot: Stock: WIP:

ICP form available in both Excel and Word versions

08/17/10

True RCCA Training Class Reference the 8-D steps:

2. Identify and Organize the Primary Team That Will Be Needed
3-5 people is best, cross functional, needed skill sets, diverse Pick a project leader, facilitator (RH man) & champion for large complex projects
The Leaders role is to assure the team follows the process, gets everyone engaged, utilizes the tools, drives and maintains the plan, updates and communicates the project information, calls the meetings The facilitator helps the leader and the team utilize the tools and conduct more effective team meetings The champions role is to offer guidance, helps get priority for closure tasks, removes constraints, be accountable with the team for results [also the CABs role]

Approach as a Special Project thats above your day job not typically a full time assignment, Note: continuous improvement is part of everyone's job
Everyone can either lead or support these Many simple projects can be done alone but use 2 others for 3 person confirmation points

3. Interim Containment
This is not part of the True RC training class focus assumes that if the problem is a new emerging issue that containment/ remedial action are accomplished in parallel and recorded in the ICP
07/13/10

True RCCA Training Class Reference the 8-D steps: 4. Find the Root Cause
4.1 Brainstorm the potential root causes and the escape point
Assign someone to capture the thoughts (facilitator)
Recommend using a white board, at end of meeting capture with digital camera Note: the best way to have a good idea (or the correct idea) is to have a lot of ideas Everyone still takes written notes: For their personal understanding and active engagement To get started on their action items prior to RAIL update and distribution To capture anything the group dynamics may have missed Assure everything gets captured on the RAIL at the end of the groups meeting System element level block diagrams Process flow diagrams 3D illustration (from A/C Maintenance Manual (AMM), Illustrated Parts Catalog (IPC)) Digital photographs Check Lessons Learned databases for problem/ solution similarity Available data and trends
07/13/10

Focus the discussion by using:

True RCCA Training Class

4.1 Brainstorm (Contd)
Death by 1000 questions (5+ whys / maybes = Socratic debate)
Ask why did or how could that happen Followed by maybe its because. Ask whats changed over time (especially recently) Ask why doesnt it happen in another similar situation/ application Ask what contributed, when did it occur, who was involved Look for underlying causal chains or sequence of events leading to the problem (they may or may not be interdependent)

Differentiate between symptoms, effects of, and actual root causes

Symptoms are indicators or observations, they characterize whats changed
When and where it was first observed, how it was first identified, quantifies the size and trend of the problem (ex. the problem became more frequent when it got hot outside) Effect = Fault = Problem: the spar cap fractured (failure mode- vs bent, buckled, crippled) Direct cause = Defect: because it developed a crack (failure mechanism- tensil crack growth) Contributing causal chain: part was damaged during transportation (crack initiation), because handling procedures werent followed, because new employees assigned to the task and not trained True Root cause = supervisors not assuring new employees are trained on requirements Other contributing cause: undetected during subsequent inspections

Effects are something brought about by a cause, a description of a fault

Identify what you know (supported by data) versus opinion but capture/ record/ and explore everything.
07/13/10

True RCCA Training Class

4.1 Brainstorm (Contd)
Everyone participates, Repeat until no one can think of another why to ask
Encourage free thinking, dont criticize Dont jump to conclusions too quickly this causes the process to stop too soon Think intensely and at multiple levels (ref: the causal chain) Peel the onion professionally and unemotionally challenge the basis Permit no pride of authorship within the team Note: most people can only support 1-2 layers of questions with their basis for root cause or a corrective action Dont try to implement solutions before the analysis is complete Note: Our desire to act overpowers our need to understand Supplier Issues should start with a pre visit phone call Includes the full team (include SMEs) and the suppliers counter parts This is the best first step to capturing potential root causes and making any required on site visits more efficient Success test eventually cant tell who on the team came up with the original idea or the full string of the idea Note: Individuals cant be perfect but teams can
07/13/10

True RCCA Training Class

4. Find the Root Cause (Contd) 4.2 Sort and analyze the potential root causes
Start by utilizing the simple project tools during the brainstorming
Further focuses the team in a structured way to organize and clarify the interactions and interdependencies at different levels of the cause

Simple project tools

Affinity diagrams groups supporting ideas into categories and a priority structure Pareto reveals what causes or effects dominate and may lead to the priority of items to investigate Cause and effect (fishbone) diagrams collects potential root cause elements and sub elements Typically: people, materials, machines/ equipment, methods, measurements, training, environment (noise, lighting, temp) Fault Tree- a vertically oriented fishbone Problem, Process flow diagram Fault, or Sequence of events diagram
Un-desired Effect

Complex project tools

Histograms identifies patterns, nominal values, and data limits Relation diagrams identifies relationships between drivers and indicators Scatter diagram compare relationship between two variables Web diagram compare multiple variables to look for combination that could be the problem Trend charts FMEA DOE
07/13/10

True RCCA Training Class

4.2 Sort and analyze the potential root causes- (contd)
Simple project tools
Design Fab / Assy

Fault, Problem

Testing

Quality

Sys Ops

Fault Trees (essentially a vertically oriented fishbone diagram) Example fault tree elements / realms / branches:

Design (from detail components, sub systems, system level schematic)

Relevant elements for investigation Key characteristics Dimensional/Tolerance analysis (includes: temperature, vibration, dynamic loads/ deflections.)

Fabrication, Assembly, Test process and controls

Actual dimensions of relevant parts and tolerance trends (selective assembly practiced?) Property inspection results of relevant parts-hardness, surface finish, conductivity, etc. Process documentation and control Contamination control

Quality (variability reduction and Type Design compliance)

Non conformance condition history and trends of relevant parts Delegation of inspection authority Sampling rates

Maintenance System operation (see example)

07/13/10

True RCCA Training Class Fault Tree Example

1.0 Physical problems Generator main contacts remain closed

(System Operation Branch Example)

2.0 Commanded close

1.1 Physically stuck

1.2 Contacts welded closed

1.1.1 Damaged return spring

1.2.1 Over current

1.2.1.1 Short circuit 1.1.2 FOE 1.2.1.1.1. Aircraft harness 1.1.3 Damaged parts 1.2.1.2 Transient load during switching 1.2.1.3 Close/ release timing 1.2.1.3.1 Commands to relays 1.2.1.3.2 Relay operation

provides clear sub level identification numbering for VAP Can be constructed in Excel

1.2.1.4 Two power sources linked 1.2.2 Magnetized 1.2.3 Over voltage 1.2.4 External heat

07/13/10

True RCCA Training Class

4.2 Sort and analyze the potential root causes- (contd) Complex project tools
Failure Mode and Effects analysis (FMEA)
Excel spreadsheet produced during development that determines up front preventative action plans or used during an investigation to determine priority of potential root cause items. Based on a calculated Risk Priority Numbers (RPN):
Significance of the effect (severity), likeliness of occurrence, and probability of detection. RPN scoring systems can be unique for an individual project based on complexity of the problem (they can be simple)

Design/Product FMEA: built from a top level system schematic (what box talks to what other box/ components in the system?) and/ or a detail part or assy level drawings
What design characteristics/functions could lead to a particular failure mode?

Process FMEA: built from manufacturing process flow diagram

What process characteristics/functions could lead to a particular failure mode?
07/13/10

True RCCA Training class

Example FMEAs entries for leaking landing gear investigation
See Step 4.3: the VAP elements should be utilized ilo this section found in a typical FMEA

Design FMEA
Identifier Design Characteristic Item/Design Function/ Requirements Potential Failure Mode Potential Effect(s) of Failure S e v Potential Cause(s) / Mechanism(s) of Failure O c c Current Design/Process Controls D e t R P N Comments / Notes Recommended Actions Responsibility & Target Completion Date Actions Taken

1.1.2

Dimensional

Lower Bore Diam

Oversize

Lower Bearing Static Seal leakage due to insufficient seal squeeze

Poor machining practices

Air gauge inspection

75

5 QNs for Oversize

2.2.3

Surface Plating

Sealing Surface

Soft surface scores & flakes easily

Leakage

Plating too thick

Dimensional Inspection

320

High RPNs require more Mfg and QA controls

D e t R P N Responsibility & Target Completion Date

Process FMEA
Identifier Process Characteristic Item/Process Function/ Requirements Potential Failure Mode Potential Effect(s) of Failure S e v Potential Cause(s) / Mechanism(s) of Failure Non-homogenius material composition Poor inventory mgmt (FIFO) O c c Current Design/Process Controls Comments / Notes Recommended Actions Actions Taken

3.2.1

Seal physical charecteristics

Elongation

Local thinning, microcracks

Leak path thru the seal

Inspection at the supplier

15

4.2.2

Seal storage

Shelf life

Shelf life exceeded

Seal dried out, brittle- may crack

Cure date label on seal bag

15

08/24/10

True RCCA Training Class

Example RPN scoring values
Ranking 1 2 3 4 5 6 7 8 9 10 Ranking 1 2 3 4 5 6 7 8 9 10 No effect Convenience item affected; next process customer not affected Rework needed at the process station Minor dissatisfaction to next process Rework needs to be done off-station Rework that can lead to scrap Potential scrap to be decided by customer Scrap with part affected without damage to machine and tooling Scrap with machine, tooling and part affected Hazardous. Affects safety of operator Detection/Prevention Will prevent failure mode from happening with notification of prevention Will prevent failure mode from happening without notification of prevention Will prevent failure mode from happening most times Will prevent failure mode from happening sometimes Will not prevent failure mode from happening but will detect it with notification Will always detect failure mode but without notification Will detect failure mode most times Will detect failure mode sometimes (On and Off) Will not detect failure mode No controls in place Severity

07/13/10

True RCCA Training Class

Example RPN scoring values
Ranking
1 2 3 4 5 6 7 8 9 10

Possible Failure Rates

< = 1 in 1,500,000 1 in 150,000

Probability of Failure
Remote: Failure is unlikely.

Occurrence
Never happened before Happens rarely and may cause FM / effect rarely

Low: Relatively Few Failures 1 in 15,000 1 in 2,000 1 in 400 1 in 80 1 in 20 High: Repeated Failures 1 in 8 1 in 3 > = 1 in 2 Very High: Failure is almost inevitable Moderate: Occasional Failures Happens sometimes and may cause FM / effect rarely Happens every time and may cause FM / effect rarely Happens rarely and may cause FM / effect sometimes Happens sometimes and may cause FM / effect sometimes Happens every time and may cause FM / effect sometimes Happens rarely and will cause FM / effect every time Happens sometimes and will cause FM / effect every time Happens every time and will cause FM / effect every time

07/13/10

True RCCA Training Class

4.2 Sort and analyze the potential root causes- (contd)
Other potential actions from lessons learned on complex problems:
Where do we have data that could help? How does teardown evidence and lab analysis information map to the Fault Tree elements? Create a chronological map of failure events to all manufactured units (serial numbers) and delivery dates for a correlation to when something may have changed. Capture change history of all Fault Tree realms and branches Design configuration, process, specs, inspection/test methods, personnel, tooling, location, suppliers, etc. Capture field service history Pareto list of all event S/N parts to the FT branches Look for combinations of FT branches/ variables as possible RC Review the LAI/ FAI and re-qualification process and reports for change of source or manufacturing location Construct a sub tier supplier map Recent interview results, recommendations Audit reports and findings Turnover rate, business level changes Training and certifications compliance controls
07/13/10

True RCCA Training Class

4. Find the Root Cause - (Contd) - 4.3 Set Up the Root Cause Verification Action Plan (VAP)
The VAP is a comprehensive Indentured spreadsheet used to manage the verification actions required to dismiss or confirm every potential RC at each level and record the results and conclusions. Built from and matches the fishbone or fault tree branches
Columns: Indentured numbering system matching locator numbers set up for each branch of the fault tree Title of the Fault Tree element and sub elements RPNs to prioritize RC theory statement for each causal level Every potential RC must have a written theory statement of how it could have caused the problem Use the guidelines for good problem statements Verification action/ task description and applicability to confirm or dismiss Action assigned to ECD Status: A-active, C-complete Results: basis for closure and conclusion Potential RC conclusion: Yes (color cell and FT element red), No (green), Maybe (yellow)

07/13/10

True RCCA Training Class Fault Tree Example

1.0 Physical problems Generator main contacts remain closed

2.0 Commanded close

1.1 Physically stuck

1.2 Contacts welded closed

1.1.1 Damaged return spring

1.2.1 Over current

1.2.1.1 Short circuit 1.1.2 FOE 1.2.1.1.1. Aircraft harness 1.1.3 Damaged parts 1.2.1.2 Transient load during switching 1.2.1.3 Close/ release timing 1.2.1.3.1 Commands to relays 1.2.1.3.2 Relay operation

provides clear sub level identification numbering for VAP

1.2.1.4 Two power sources linked 1.2.2 Magnetized 1.2.3 Over voltage 1.2.4 External heat

07/13/10

Root Cause Verification Action Plan

PCD (Promised Completion Date)

Root Cause Verification Action Plan: "Generator main contacts remain closed"
Fault Tree Locator Number Fault Tree Element Fault Tree Subelement Fault Tree Subelement Fault Tree Subelement Fault Tree Subelement RPN

VAP Root Cause Theory Statement Verification Action/ Task Description

Status:

Assigned to:

ECD A = Active
= Complete

Results

Potential RC Conclusion

1.0 Physical problems 1.1 1.1.1 1.1.2 1.1.3 Physically stuck Damaged return spring FOE Damaged parts

Leads to
No

1.2 1.2.1 1.2.1.1 1.2.1.1.1 1.2.1.2 1.2.1.3 1.2.1.3.1 1.2.1.3.2 1.2.1.4 1.2.2 1.2.3 1.2.4

Contacts welded closed Overcurrent Short circuit Aircraft harness Transient load during switching Close/ release timing Commands to relays Relay operation Two power sources linked Magnitesed Over voltage External heat Yes Maybe

2.0 Commanded close

(A new sheet typically started for each major fault tree element)

08/24/10

Root Cause Verification Action Plan

Root Cause Verification Action Plan: "Generator main contacts remain closed"
Fault Tree Locator Number Fault Tree Element Fault Tree Subelement Fault Tree Subelement Fault Tree Subelement Fault Tree Subelement RPN

Spar Cap fractured 1.0 Physical problems

1.1 1.1.1 1.1.2 1.1.3 VAP

Developedstuck crack Physically

return Damaged during spring transportation FOE Damaged parts

Handling procedures not followed

Any level could be the root cause

New employees not trained PCD

Root Cause Theory Statement

Verification Action/ Task Description

Assigned to:

ECD

Status: A = Active C = Complete

Results

Potential RC Conclusion

No

08/24/10

True RCCA Training Class

4.3 Set up the Verification Action Plan (VAP) (contd)
Separate section/ sheet of the spreadsheet for each fault tree major element may be needed Report % complete based on number of FT elements closed Determines what verification action required to dismiss or confirm every potential RC at each level Verification = proving beforehand that the planned action will do what is intended
Ex.) analysis, tests/ trials (objective evidence), comparisons with similar activities utilizing a subject matter expert (SME) Without a completed verification action the potential RC cant be eliminated based on available data

Validation = after CA implementation, checking that the action is achieving its goal and without adding a new problem (proof that the change has worked)
Note: nothing is particularly hard if you divide it into small jobs

Ultimately verify by:

demonstrating you can turn the problem on and off The selected RC also fully explains why the problem doesnt occur in a similar applications and scenarios

A minimum of three relevant people to commit they concur the True RC is correct, comprehensive in terms of multiple levels of contributing causes, and the escape point is clear
Reference them in the ICP for future info (willingly held accountable) Should also include the champion
07/13/10

True RCCA Training Class

Reference the 8-D steps:
5. Corrective Action (CA) plan definition and implementation
Repeat the brainstorm/ death by 1000 questions
Ask and challenge why would or how will this prevent re-occurrence Peel the onion 5+ layers deep From the escape point determine where the problem should have been detected and where the future controls should be placed within the process Consider implementation point and effects on the operation Do pro/ cons if there are CA alternatives Belts and suspenders or multi-phased implementation may be worth the investment Keep the customer in mind Do the right thing even if its hard or takes a long time Add the action plan tasks to the ICP

A minimum of three relevant people to commit they concur the CA plan is correct and comprehensive in terms of appropriate action assigned to all levels of contributing cause
Reference them in the ICP for future info (willingly held accountable) Should also include the champion

6. Define what will constitute validation for the CA

Ex.) an overall effectiveness metric for periodic review, follow up audits, periodic detail inspections or tests A recalculation of the original RPN to show significant improvement ( = 0, if re-occurrence eliminated) Add the action plan tasks, including tracking the implementation to the ICP A minimum of three relevant people to commit they concur the validation plan is correct and reference them in the ICP for future info (willingly held accountable) Should also include the champion
07/13/10

True RCCA Training Class

Reference the 8-D steps:
7. Preventative Action (PA) plan definition and implementation
1. Systemic analysis where else or what else could be impacted by this problem and RC? Think both other products and processes- assure they take action 2. Is there an underlying cultural or operational problem that needs addressed? How could this have been avoided? New communication or training required Change to policies, practices, procedures 3. What potential unintended effects could occur from implementing the CA? This requires repeat of the brainstorm process and a check of the lessons learned database for applicability What will be the verification action to assure theyre avoided 4. Mistake proofing: ex. physical/ mechanical controls on tooling (see Juran training material) Add the action plan tasks to the ICP A minimum of three relevant people to commit they concur the PA plan is correct and comprehensive Reference them in the ICP for future info (willingly held accountable) Should also include the champion
07/13/10

True RCCA Training Class

Reference the 8-D steps:

8. Project Closed: The full team agrees that the predetermined closure criteria were satisfied
For each step above (as recorded in the ICP) Typically status ICP as closed once CA and PA closure plan accepted and implemented Update the ICP once the validation step is complete (Leaders responsibility) Assured lessons learned are recorded and shared across the organization for continuous improvement Appropriate recognitions and celebrations followed through.

07/13/10