Development and Use of Ambulatory Adverse Event Trigger Tools (Text Version)

Slide presentation from the AHRQ 2010 conference.

On September 28, 2010, Vera Rosenthal made this presentation at the 2010 Annual Conference. Select to access the PowerPoint® presentation (190 KB). Free PowerPoint® Viewer (Plugin Software Help).


Slide 1

Development and Use of Ambulatory Adverse Event Trigger Tools

Development and Use of Ambulatory Adverse Event Trigger Tools

Amy K. Rosen, PhD

AHRQ Conference
Sept. 29, 2010

Boston University School of Public Health, Boston, MA,
VA Center for Organization, Leadership and Management Research
(COLMR), Boston MA
akrosen@bu.edu

Slide 2

Acknowledgements

Acknowledgements

Sponsored by AHRQ Contract No. HHSA290200600012, Task Order Officer: Amy Helwig, MD.

  • PI Amy Rosen, PhD
  • Co-PI Jonathan Nebeker, MD, MS
  • Co-Investigators:
    • Stephan Gaehde, MD
    • Haytham Kaafarani, MD, MPH
    • Brenna Long, MA
    • Hillary Mull, MPP
    • Brian Nordberg, BS
    • Steve Pickard, MS
    • Peter Rivard, PhD
    • Lucy Savitz, PhD, MBA
    • Chris Shanahan, MD, MPH
    • Stephanie Shimada, PhD

Slide 3

Project Goal and Settings

Project Goal and Settings

  • Goal: Develop adverse event (AE) triggers for the outpatient setting:
    • Outpatient surgery AEs.
    • Outpatient diagnostic testing and loss to follow-up AEs.
    • Outpatient adverse drug events (ADEs).
  • Three sites for patient data:
    • Boston Medical Center (BMC).
    • Intermountain Healthcare.
    • Western region of the Veterans Health Administration (VA).

Slide 4

Background

Background

  • Trigger tool: A screen applied to healthcare data that triggers review of a patient record for a potential adverse event.
  • Trigger: An algorithm that uses electronic medical record data to identify patterns consistent with a possible adverse event.
    • Example: combination of a lab value threshold and an active prescription.
  • AE specific trigger: A trigger intended to identify a specific adverse event.

Slide 5

Methods

Methods

Develop Triggers:

  • Clinicians review literature to develop potential outpatient AE triggers.
  • Review these triggers with focus groups.
  • Hold Delphi process to refine trigger logic, develop new triggers.
  • Establish list of outpatient AE triggers to test.

Merge Data:

  • Obtain de-identified data extracts from each site.
  • Merge variables from each site into trigger project database.
  • Iterative process of checking data quality and obtaining new data extracts.

Program triggers and perform chart classification:

  • Program trigger algorithms to run on data in project database and flag cases.
  • Iterative process of running trigger and checking data quality.
  • Build mock electronic medical record (EMR) to interface with project database Perform chart classification.

Analyze trigger performance

  • Calculate positive predictive value (PPV) and 95% confidence intervals for each trigger

Slide 6

Methods: Outpatient Surgery Triggers

Methods: Outpatient Surgery Triggers

  • Sample:
    • N=17,492 ambulatory surgeries from three institutions in CY05.
    • Sampling for chart review: 17 trigger-flagged cases from each site to the extent the data were available.
  • Definition of AE:
    • NSQIP AEs: Surgeries with an event that meets the National Surgical Quality Improvement Program criteria for an AE (e.g., postoperative urinary tract infection).
    • Other AE: Surgery with an AE determined by nurse clinical judgment (e.g., postoperative iatrogenic injury).
  • Analysis:
    • PPV = number of patients with AE/number of positive triggers

Slide 7

Methods: Loss to Follow-up Trigger

Methods: Loss to Follow-up Trigger

  • Definition of AE:
    • Failure to follow-up on abnormal fecal occult blood test (FOBT) results, which could potentially lead to a failure to detect colorectal cancer or pre-cancerous growths in a timely manner.
      • In the event a patient has a positive FOBT, the standard of care is a colonoscopy within 2 months.
      • We designed the trigger to exclude bleeding that might potentially be due to a recently diagnosed ulcer or GI surgery.
  • Analysis:
    • PPV was calculated for cases without a colonoscopy and cases missing an appropriate colonoscopy.

Slide 8

Methods: Focus Groups

Methods: Focus Groups

  • We conducted 90-minute focus group and multiple 30-45 minute interviews at each of the sites between October and December 2009.
  • Participants were frontline staff, middle managers and executives including:
    • Physicians
    • Nurses
    • Informatics and information systems experts
    • Pharmacists
    • Quality and patient safety experts

Slide 9

Results: Outpatient Surgery Triggers

Results: Outpatient Surgery Triggers

Trigger NameTrigger RuleCases Flagged
N (%)
Flagged Cases Sampled for ReviewPPV for NSQIP AEs
(95% CI)
PPV for Any AE
(95% CI)
Emergency Department (ED)Outpatient surgery AND subsequent ED visit ≤ 21 days519
(3%)
10012%
(6-20%)
49%
(39-59%)
AdmissionOutpatient surgery AND admission to hospital ≤ 30 days3,846
(22%)
5112%
(4-24%)
41%
(26-56%)
ProcedureOutpatient surgery AND procedure (interventional radiological OR urological OR cardiac OR gastroenterological) OR re-operation ≤ 30 days1,132
(7%)
516%
(1-16%)
24%
(13-37%)
Length of Stay
(LOS)
Outpatient surgery AND subsequent admission with length of stay > 24 hours1,002
(7%)
516%
(1-16%)
25%
(14-40%)

Slide 10

Results: Outpatient Surgery Triggers

Results: Outpatient Surgery Triggers

Trigger NameTrigger RuleCases Flagged
N (%)
Flagged Cases Sampled for ReviewPPV for NSQIP AEs (95% CI)PPV for Any AE
(95% CI)
Pulmonary Embolism (PE)/ Deep Vein Thrombosis (DVT)Outpatient surgery AND ICD-9-CM code for DVT OR PE ≤ 30 days189
(1%)
15058%
(50-66%)
65%
(56-72%)
PE/DVT: 52%
(43-60%)
Surgical Site Infection (SSI)Outpatient surgery AND antibiotics started or continued > 24 hours OR wound debridement CPT code within 5-30 days1,408
(8%)
51SSI: 4%
(0.5-13%)
18%
(8-28%)
HematocritOutpatient surgery AND hematocrit check ≤ 14 days and hematocrit drop > 6 units from preoperative baseline ≤ 1 year before the index event451
(3%)
NANA100%

Slide 11

Results: Loss to Follow-up Trigger

Results: Loss to Follow-up Trigger

  • Sample:
    • N=995 cases with positive FOBT tests in CY05 from one institution
Trigger Rule# trigger-positive 2 months post FOBTCriteriaNumber meeting criteria at 6 monthsPPV
(95% CI)
Positive FOBT
AND (serum Fe<60 mcg/dL OR vplasma/serum ferritin<40 ng/mL OR
MCV<80 fL OR HCT<36%)
AND age greater than 21 AND no colonoscopy at two months after FOBT
AND no (diagnosis code for upper GI ulcer OR
CPT code for GI surgery) ≤1 month prior
85Number with no colonoscopy5666%
(5-76%)
Number with no colonoscopy AND with no clinical reason why not2934%
(24-45%)
Number with no colonoscopy AND with no clinical reason AND no documented refusal2226%
(17-37%)

Slide 12

Results: Focus Group/Interviews

Results: Focus Group/Interviews

Triggers ranked from highest to lowest likelihood of adoption.

Image: A bar graph detailing triggers ranked from highest to lowest likelihood of adoption, based on participants’ initial impressions of the trigger tools.

Slide 13

Results: Focus Group/Interviews

Results: Focus Group/Interviews

Triggers ranked from highest to lowest perceived ease of implementation.

Image: A bar graph detailing triggers ranked from highest to lowest perceived ease of implementation, based on participants' initial impressions of the trigger tools.

Slide 14

Summary

Summary

  • The set of surgical triggers (excluding 'Hematocrit') required review of 454 cases and detected at least one AE in 204 ambulatory surgeries (PPV for any AE=45%).
    • Many surgeries had more than one AE detected.
  • Triggers best suited for adoption:
    • Admission: flag rate=22%; PPV for detection of any AE=41%; popular in focus groups.
    • PE/DVT: flag rate=1%; PPV for detection of any AE=65%; very popular in focus groups.
    • FOBT: flag rate=8%; PPV for detection of any cases without colonoscopy after 6 months=66%; very popular in focus groups.

Slide 15

Implications

Implications

  • Triggers may have potential to screen for outpatient AEs using a focused sample of cases.
  • Triggers could complement existing screening programs used to detect AEs.
  • Potential for real-time AE detection, particularly with electronic medical records.

Slide 16

Dissemination to Date

Dissemination to Date

  • Rosen AK & Nebeker JR, Use of Trigger Tools to Identify Risks and Hazards to Patient Safety, Sept. 2010. Speakers, AHRQ 2010 Annual Conference, Bethesda, MD.
  • Long BL, Pickard S, Mull HJ, Hoffman JM, Rosen AK & Nebeker JN, Reliability of AHRQ Harm Scale Used with Explicit Criteria in Retrospective ADE Classification, Sept. 2010. Poster, AHRQ 2010 Annual Conference, Bethesda, MD.
  • Shimada S, Mull HJ, Kaafarani H, Rosen AK, Nebeker JR, Nordberg B, Pickard S & Singh H. Development and Assessment of a Trigger Tool to Detect Patients with Positive Fecal Occult Blood Test (FOBT) Results Who Are Lost to Follow-Up. Accepted poster, AMIA 2010 National Symposium, Nov 2010, Washington, DC.
  • Kaafarani H, Rosen AK, Nebeker JR, Shimada SL, Mull HJ, Rivard PE, Savitz L, Helwig A, Shin MH & Itani KMF. Development of Trigger Tools for Surveillance of Adverse Events in Ambulatory Surgery, 2010. Quality and Safety in Health Care. [E-pub ahead of print] http://www.ncbi.nlm.nih.gov/pubmed/20513790
  • Mull H & Nebeker JR, Informatics Tools for the Development of Action-Oriented Adverse Drug Event Triggers, 2008. AMIA Annual Symposium Proceedings. Nov 6:505-9. http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2655939/
  • Mull HJ, Nordberg B, Nebeker JR, Using Electronic Medical Records Data for Health Services Research, Case Study: Development and Use of Ambulatory Adverse Event Trigger Tools. Speaker, AcademyHealth Annual Research Meeting, June 2010, Boston, MA.

Slide 17

Dissemination to Date (cont'd)

Dissemination to Date (cont'd)

  • Shimada S, Rivard P, Nebeker JR, Savitz L, Shanahan C, Gaehde S & Rosen AK. Priorities & Preferences of Potential Ambulatory Trigger Tool Users. Speaker, AcademyHealth Annual Research Meeting, June 2009, Chicago, IL.
  • Kaafarani H, Rosen AK, Nebeker JR, Shimada SL, Rivard PE, Mull HJ, Long B, Shin MH, Savitz L & Itani KMF. Developing Trigger Tools for Surveillance of Adverse Events in Same-Day Surgery: A Literature-Based, End-User Inspired and Expert-Evaluated Methodology, Sept 2009. Poster, AHRQ 2009 Annual Meeting, Bethesda, MD.
  • Mull HJ, Shimada S, Nebeker JR & Rosen AK, A Review of the Trigger Literature: Adverse Events Targeted and Gaps in Detection, June 2008. Proceedings of the Trigger and TIDS Expert Meeting, Agency for Healthcare Research and Quality, Bethesda, MD. http://www.ahrq.gov/news/events/other/triggers/triggers1.html.
  • Nebeker JR, Stoddard GJ & Rosen AK, Considering Sensitivity and Positive Predictive Value in Comparing the Performance of Triggers Systems for Iatrogenic Adverse Events, Proceedings of the Trigger and TIDS Expert Meeting, Agency for Healthcare Research and Quality, June 2008. Bethesda, MD. http://www.ahrq.gov/news/events/other/triggers/triggers2.html.
  • Shimada S, Rivard PE, Mull HJ, Nebeker JR & Rosen AK, Triggers and Targeted Injury Detection Systems: Aiming for the Right Target With the Appropriate Tool, June 2008. Proceedings of the Trigger and TIDS Expert Meeting, Agency for Healthcare Research and Quality, Bethesda, MD. http://www.ahrq.gov/news/events/other/triggers/triggers3.html.
Current as of December 2010
Internet Citation: Development and Use of Ambulatory Adverse Event Trigger Tools (Text Version). December 2010. Agency for Healthcare Research and Quality, Rockville, MD. http://www.ahrq.gov/news/events/conference/2010/rosenthal/index.html