Risk Informed Evaluation of Patient Safety Training (Text Version)

Slide presentation from the AHRQ 2009 conference.

On September 15, 2009, Anthony Slonim made this presentation at the 2009 Annual Conference. Select to access the PowerPoint® presentation (1,309 KB) (Plugin Software Help).

Slide 1

Risk Informed Evaluation
of Patient Safety Training

Anthony D. Slonim, MD, DrPH

Vice President Medical Affairs
Carilion Medical Center

Senior Staff, Departments of Medicine and Pediatrics
Carilion Clinic

Professor, Medicine and Pediatrics
Virginia Tech-Carilion School of Medicine

Slide 2

Objectives

  • Decision-making science How do we normally make decisions?
  • Picking up on Level 4.Can we train to improve our decision making results?

Slide 3

Decision Making Science

 

Slide 4

Bayes Theorem

P (A/B) =P(B/A) * P(A)
 P(B)

Slide 5

Pattern Recognition

Square patterned image divided into four. Each segment is an alternating color of pink and green.

  • How many squares do you see?
  • Jumping to conclusions too quickly.
  • Is there flashing in the squares?
  • Your mind will play tricks on you!

 

Slide 6

Decision-Making

Medical Decision-Making Process

  1. Perception/Data gathering (training-H and P, Labs, Rads)
    • "Amber light" is showing
  2. Interpretation (training-pattern recognition and probability)
    • "Amber light" means prepare to stop, maybe
  3. Decision making (based on probability + experience)
    • Stop or go
  4. Action taking (reflex/"gut level response"/programming)
    • Hit the brake or accelerator

Marx D and Slonim AD: Assessing patient safety risk before the injury occurs: An Introduction to Socio-Technical Probabilistic Risk Assessment. Quality and Safety in Healthcare 2003; 12 Suppl 2: 33-38.

Slide 7

Medical Decision-Making

Perception/Data Gathering

Image depicting a treatment process with Triage Nurse in the center of a square, with ED Nurse outside the Triage Nurse. Finally Physician is on the outside of the ED Nurse.

Actions:

  • Make a Dx and Treat
  • Get Help: Cardiology Consultation
  • Do more testing-which test? (Pre-test probabilities)

Slide 8

Medical Decision-Making

Same image as above but with word written on top of it:

What are the results? Did we get it right?

 

Slide 9

Expert Decision Making:
Practice, Practice, Practice

  • Expert - pattern matching against large mental library, quick, accurate if confirm correct answer
  • Novice - library is empty - slow, error prone process
  • Certain Diagnoses are Favored- Frequent, Recent, Serious
  • Heuristics - fixating on the wrong pattern

 

Slide 10

Pattern Recognition

Picture of a triangle and 3 red circles at each point.

Slide 11

Picking up on Level 4:
Can we train for results?

 

Slide 12

Kirkpatrick's Levels

  • Level I Reactions
    • How well trainees liked training
  • Level II Learning
    • The extent to which trainees understand and retain principles, facts, and techniques
  • Level III Behavior
    • The extent to which behavior changes as a result of training
  • Level IV Results
    • Impact of training on organizational criteria

 

Slide 13

Elements of Quality Programs

Flow chart of Quality Functions

  • Program Identifcation and Prioritization
  • Data Analytics
  • Process Improvements
  • Change Management
  • Improved Outcomes
  • Quality Functions
  • Research
  • Education / Training

 

Slide 14

Why is there a safety problem?

  • Considerable variation in practice
    • Based on opinion or consensus
    • Evidence-based guidelines-unsupported
    • Failure to create fail-safe processes
    • Our providers may not know their work
      • Policies and procedures
    • We're learning to work together
    • We're not sure of the results we're looking for

 

Slide 15

Process Analysis

  • Processes:
    • A series of sequential steps governing interactions
      • Between patients and providers
      • Between providers and providers
  • Examples of process analysis techniques:
    • Root cause analysis-retrospective
    • HAACP (hazard analysis and crit control points)
    • FMEA (failure mode effects)
    • PI methodology

 

Slide 16

Low-frequency, High Impact Events

  • Low frequency, high-impact events
    • Variable processes and practices
      • Wrong site surgery
      • The abduction of children from hospitals
      • Deaths or major harm
  • Process analysis helps to identify risk and prioritize interventions
  • Decision support helps to guide decision making

 

Slide 17

Probabilistic Risk Assessment

  • A hybrid between process analysis and decision support
  • Identifies risk points and directs to interventions
  • Is hierarchical and probabilistic
    • Allows disentanglement of patient level risks, provider level risks, and system level risks
    • Assigns probabilities for prioritization of risk reduction strategies
    • Includes sociotechnical components into the models

 

Slide 18

Conceptual Framework

Probabilistic Risk Assessment identifing Qualitative methods

  • The Institution
  • The Providers
  • The Patient

Slide 19

The Prospective Risk Model

Diagram showing the top three risks.
 

 

Slide 20

Training Evaluation

  • Definition
    • The systematic collection of descriptive and judgmental information necessary to make decisions related to instructional activities
  • Ensures training
    • Meets its stated objectives
    • Changes trainee attitudes
    • Increases trainee knowledge
    • Develops trainee skills
    • Transfers results to the job

 

Slide 21

Training Evaluation

  • Important variables to consider:
    • Organizational Factors
    • Individual Factors
    • Trainee Knowledge, Skills, and Attitudes
    • Training Transfer
    • Organizational Outcomes

 

Slide 22

Merging Kirkpatrick and ST PRA

  • Socio-Technical Probabilistic Risk Assessment
    • Good for examining low base rate events (Six Sigma)
    • Models contributing causes
      • Procedural tasks
      • Team tasks
  • Identifies the impact of an intervention
    • Evidence base
    • Empirically based
  • Adjust and test the model
    • Monte Carlo
    • Changes in the likelihood of outcomes

 

Slide 23

Traditional Approaches

  • Quick Wins
  • "Fire-fighting"
  • Burn-out / Fatigue
  • Difficult-to-Sustain, Short-Term Results

 

Slide 24

Quality Fusion Approach

Graph demonstrating fusion results of impact over time.

 

Slide 25

Example

 

Slide 26

What is Escalation?

  • Failure to rescue associated with
    • Interpretation problems
    • Throughput problems
    • Put another way.
  • When you do not realize the patient is in trouble OR you know the patient is in trouble, but you don't respond as needed.

 

Slide 27

Common Course

Ideally, we track the illness. As the patient gets worse (line goes up), we respond. As the patient improves we adjust.
 

 

Slide 28

Going Off Course

The defect rate in our model
is caused by failures to properly track
the course of the illness.
 

 

Slide 29

Never On Course

Sometimes, we're off course right from the beginning and it's difficult to get back on course.

Image of a red and blue line with an arrow in between them to signify being off course.

Slide 30

The Prospective Risk Model

The Top Three Risks
 

 

Slide 31

Conclusions

  • A focus on results helps providers and patients
  • Training on risk points can improve performance
    • Leads to better results
    • Requires alterations in decision making
    • Enhances empiric data for better understanding training
Current as of December 2009
Internet Citation: Risk Informed Evaluation of Patient Safety Training (Text Version). December 2009. Agency for Healthcare Research and Quality, Rockville, MD. http://www.ahrq.gov/news/events/conference/2009/slonim/index.html