The Development of Emergency Department Patient Quality/Safety Indicators

Slide Presentation from the AHRQ 2010 Annual Conference

On September 27, 2010, Patrick S. Romano made this presentation at the 2010 Annual Conference. Select to access the PowerPoint® presentation (3.01 MB). Free PowerPoint® Viewer (Plugin Software Help).


Slide 1

The Development of Emergency Department Patient Quality/Safety Indicators

The Development of Emergency Department Patient Quality/Safety Indicators

Patrick S. Romano, MD MPH
UC Davis Center for Healthcare Policy and Research
AHRQ Annual Conference
September 27, 2010

Slide 2

Overview

Overview

  • HCUP and the AHRQ Quality Indicators
  • Goals and scope of current project
  • Literature review
  • Conceptual frameworks
  • Matrix of potential indicators
  • Specification and testing
  • Future steps

Slide 3

The HCUP Partnership: A Voluntary Federal-State-Private Collaboration

The HCUP Partnership: A Voluntary Federal-State-Private Collaboration

Map of the United States color-coded by partnership type.

White: Non-participating
Yellow: Partners Providing Inpatient Data Only
Pink: Partners Providing Inpatient & Emergency Department Data
Red: Partners Providing Inpatient, Ambulatory Surgery, & Emergency Department Data.

40+ states
90% of all discharges
24+ states submit ED encounters

Slide 4

The Making of HCUP Data

The Making of HCUP Data

Timeline showing the steps involved:

Patient enters ED/hospital
Billing record created
Hospital sends billing data and any additional data elements to Data Organizations
States store data in varying formats
AHRQ standardizes data to create uniform HCUP databases

Slide 5

Types of HCUP Databases

Types of HCUP Databases

  • State Inpatient Databases (SID)
    • Nationwide Inpatient Sample (NIS)
    • Kids' Inpatient Database; (KID)
    • Nationwide Emergency Department Sample (NEDS)
  • State Ambulatory Surgery Databases (SASD)
  • State Emergency Department Databases (SEDD)
    • Nationwide Emergency Department Sample (NEDS)

Slide 6

AHRQ Quality Indicators (QIs)

AHRQ Quality Indicators (QIs)

  • Developed through contract with UCSF-Stanford Evidence-based Practice Center & UC Davis, maintained and extended through contract with Battelle
  • Use existing HCUP (hospital discharge) data, based on readily available data elements
  • Incorporate a range of severity adjustment methods, including APR-DRGs* and comorbidity groupings
  • Disseminate software and support materials free via www.qualityindicators.ahrq.gov
  • Provide technical support to users
  • Continuous improvement through user feedback, annual coding updates, validation projects

* All Patient Refined—Diagnosis Related Groups

Slide 7

Evidence-based indicator development

Evidence-based indicator development

Timeline showing development steps:

Literature Review
User Suggestions
Initial Empirical Analyses and Definition
Panel Evaluation (Modified Delphi Process)
Further Empirical Analyses Refined Definition
Further Review? Final Definition

Slide 8

Key considerations in the evaluation of each prospective indicator

Key considerations in the evaluation of each prospective indicator

Application/experience: Is there reason to believe the indicator will be feasible and useful?

Fosters real quality improvement: Is the indicator unlikely to be gamed or cause perverse incentives?

Construct validity: Does the indicator identify quality of care problems that are suspected using other methods?

Minimum bias: Is it possible to account for differences in severity of illness & other factors that confound comparisons?

Precision: Is there substantial “true” variation at the level of provider measurement?

Face validity/consensual validity: Does the indicator capture an important and modifiable aspect of care?

Slide 9

AHRQ Quality Indicator modules

AHRQ Quality Indicator modules

  • Inpatient QIs
    • Mortality, Utilization, Volume
  • Prevention QIs
    • (Area Level) Avoidable Hospitalizations / Other Avoidable Conditions
  • Pediatric QIs
    • Neonatal QIs
  • Patient Safety QIs
    • Complications, Unexpected Death

Slide 10

Goals and Scope

Goals and Scope

  • Goals
    • Develop two sets of quality indicators that are applicable to the emergency department setting
      • Patient Safety Indicators (PSI)
      • Prevention Quality Indicators (PQI)
    • Set the stage for future incorporation into publicly available AHRQ QI software
  • Scope
    • Implement the established AHRQ QI measurement development process
    • Adapt existing AHRQ QI to ED setting when possible
    • Identify and evaluate new candidate indicators based on established measurement concepts

Slide 11

Literature review: strategy

Literature review: strategy

Search goal:

  • To find studies that introduced or used quality of care measures to assess patient safety in hospital emergency departments.

Search strategy using MESH headings in PubMed:

  • ("Quality Assurance, Health Care"[Mesh] OR "Quality Indicators, Health Care"[Mesh] OR "Quality of Health Care"[Mesh] OR "Health Care Quality, Access, and Evaluation"[Mesh] OR "United States Agency for Healthcare Research and Quality"[Mesh] OR "Outcome Assessment (Health Care)"[Mesh])
    AND "Emergency Service, Hospital"[Mesh]
    AND ("Medical Errors"[Mesh] OR "Malpractice"[Mesh] OR "Safety"[Mesh] OR "Equipment Safety"[Mesh] OR "Safety Management"[Mesh])

Validation using title and/or abstract keywords:

  • "patient safety" OR "adverse event" OR "avoidable condition"
    AND "quality"
    AND ("emergency room" OR "emergency department"
  • For the most important papers, we searched for 'all related articles'.

Slide 12

Literature review: process

Literature review: process

  • PubMed:
    • 1,050 abstracts, decreased to 687 when limited to human subjects, English language, date within 10 yrs.
    • All abstracts were reviewed for relevance (i.e., describing one or more measures of ED quality/safety).
  • National Quality Measures Clearinghouse
  • Organizations and Web sites
    • National Quality Forum
    • Federal: AHRQ and CMS/QualityNet
    • ED: ACEP and SAEM
    • AMA: Physician Consortium for Performance Improvement
    • Other developers: NCQA and The Joint Commission
    • Institute of Medicine/National Academy of Sciences
    • Canada: Institute for Clinical Evaluative Sciences, Canadian Institute for Health Information

Slide 13

Literature review: key themes

Literature review: key themes
40 journal papers, 23 documents and reports

  • Some TJC Core Measures address processes of care in ED management of pneumonia or myocardial infarction
  • Critical trauma or shock care, generally based on detailed "peer" review of medical records to assess appropriateness and timeliness of diagnostic and therapeutic interventions
  • Time-based measures, generally focused on waiting time, total LOS in the ED, ED disposition time for admitted/transferred patients
  • Appropriate prescribing and avoidance of medication errors for common conditions such as asthma, bronchiolitis, gastroenteritis, laceration
  • Appropriate use of imaging studies, laboratory, ECG
  • Appropriate assessment of pain, oxygenation, mental status/cognition
  • "Left without being seen" or "left AMA" (premature discharge from ED)
  • Other adverse consequences of crowding/boarding
  • "Missed diagnosis" identified by return within defined time window for a serious condition
  • Revisits to ED within defined time window for same or related condition

Slide 14

Conceptual framework for prioritization: American College of Emergency Physicians, 2009

Conceptual framework for prioritization: American College of Emergency Physicians, 2009

DomainExamples
Access to emergency careAccess to providers, access to treatment centers, financial barriers, hospital capacity
Quality and patient safety environmentState-supported systems, institutional barriers
Medical liability environmentLegal atmosphere, insurance availability, tort reform
Public health and injury preventionTraffic safety and drunk driving, immunization, injury control, state injury prevention efforts, health risk factors
Disaster preparednessFinancial resources, state coordination, hospital capacity, personnel

Slide 15

Conceptual framework for prioritization: Institute of Medicine, 2010

Conceptual framework for prioritization: Institute of Medicine, 2010

An image of the conceptual framework for prioritization is shown.

Slide 16

Conceptual framework for prioritization: Institute of Medicine, 2007

Conceptual framework for prioritization: Institute of Medicine, 2007

DomainApplication to the ED
SafeHigh-risk, high-stress environment “fraught with opportunities for error”... frequent interruptions and distractions, crowding, need for rapid decision-making with incomplete information, barriers to effective communication and teamwork, difficulty obtaining timely diagnostic tests
EffectiveLimited by deficiencies in pre-hospital care, unavailability of trained specialists, lack of access to patients' prior medical records, poor primary care follow-up, inability to coordinate care across settings
Patient-centeredCrowding, long wait times, boarding of admitted patients in hallways, design emphasis on visibility and monitoring rather than privacy
TimelyDesigned to provide timely care for emergent medical problems, but often overwhelmed by the demand for their services...
EfficientFrequently asked to provide care for which it is not the most efficient setting... primary care, urgent care for minor complaints, and inpatient care to admitted patients compromises efficiency
EquitableEMTALA requires EDs to treat all patients equitably... (but) variation in resources and personnel across communities may create inequities in how patients in different EDs are treated

Slide 17

Conceptual framework for prioritization: ICES/Alberta Quality Matrix for Health, 2010

Conceptual framework for prioritization: ICES/Alberta Quality Matrix for Health, 2010

DomainExamples
AcceptabilityHealth services are respectful and responsive to user needs, preferences and expectations.
AccessibilityHealth services are obtained in the most suitable setting in a reasonable time and distance.
AppropriatenessHealth services are relevant to user needs and are based on accepted or evidence-based practice.
EffectivenessHealth services are provided based on scientific knowledge to achieve desired outcomes.
EfficiencyResources are optimally used in achieving desired outcomes.
SafetyMitigate risks to avoid unintended or harmful results.
Healthy workplaceProvision of health services does not lead to an unhealthy work environment for health care staff.

Slide 18

Application of conceptual framework

Application of conceptual framework

 StructureProcessOutcome
EffectiveNurse staffing and skill mix (RN/total) in EDAspirin at arrival for AMI (TJC/CMS)Percentage of asthma encounters followed by revisit (or admission to hospital) within 3 days
Patient CenteredUse of survey data in PDSA cycles to improve patient centered care in EDPercentage of patients undergoing painful procedures who have pain score documentationPercentage of patients leaving ED without being seen by a physician (proxy outcome, LSU Health Services)
TimelyED triage policies to ensure timely evaluation of high-acuity patientsMedian time from ED arrival to ED departure for admitted ED patients (CMS)Percentage of orthopedic pain patients with 3-point reduction in pain score within 60 minutes
SafeComputerized physician order entry with decision support tools to detect medication errorsConfirmation of endo-tracheal tube placement (Cleveland Clinic Foundation)Death or disability due to air embolism from a medical device (NQF)
EfficientAvailability of laboratory and radiologic support to facilitate rapid evaluation and disposition in EDPercentage of low back pain patients with appropriate diagnostic test utilizationDollars per episode of low back pain evaluated in the ED
EquitableAvailability of adequate interpreting services in EDPercentage of non-English speaking patients for whom interpreting services are usedDisparity in any other outcome according to primary language

Slide 19

Matrix of potential indicators: Inclusion/exclusion criteria

Matrix of potential indicators
Inclusion/exclusion criteria

  • Identified from published source
    • Literature review (40 journal articles)
    • Organizations and Web sites (if a consensus-based approach and/or modified Delphi approach was used)
    • Similar review by Alessandrini et al. for PECARN
  • Address the domains of effectiveness and/or safety
    • A few measures of timeliness were included because the measure developer characterized them as having implications for safety in the ED
  • Focus on care provided within the ED (not pre-hospital care)
  • Clinical guidelines, standards of care, and ED decision rules were not included unless operationalized as indicators
  • Can be implemented in at least one HCUP partner state using available HCUP data
  • When ≥2 indicators appeared to address the same outcome, only the more recent and/or more clearly specified indicator was retained
  • Measures that were evaluated and discarded or rejected through a consensus-based expert panel process were not included

Slide 20

Matrix of potential indicators: Application of existing inpatient PSIs

Matrix of potential indicators
Application of existing inpatient PSIs

  • Foreign body left in
  • Iatrogenic pneumothorax
  • "Postoperative" hip fracture
  • "Postoperative" hemorrhage or hematoma
  • Accidental puncture or laceration
  • Transfusion reaction
  • But critical problem is timing
    • Only 5 states (GA, MA, MN, NJ, TN) have POA in SEDD; only MA and TN also have PNUM
    • In SID, POA means "present at the time the order for inpatient admission occurs" (i.e., after some period of ED treatment)
    • ED diagnoses are "lost" in SID when patient admitted to same hospital

Slide 21

Matrix of potential indicators: 35 new candidate indicators

Matrix of potential indicators
35 new candidate indicators

  • Age range
    • 12 for children only
    • 10 for adults only
    • 13 for both children and adults
  • Donabedian's typology
    • 11 process
    • 17 outcome (or proxy outcome such as revisit)
    • 6 hybrid ("missed serious diagnosis")
    • 1 patient experience or health risk behavior ("left AMA")
  • Developer(s)
    • 20 Institute for Clinical Evaluative Sciences, specified in ICD-10-CA
    • 3 ACEP and/or PCPI
    • 3 CMS
    • 4 other organizations
    • 5 researchers
  • Endorsement—6 endorsed by NQF

Slide 22

Matrix of potential indicators: 35 new candidate indicators

Matrix of potential indicators
35 new candidate indicators

  • Revisits—13
    • 4 within 24 hours (1 specified as 24 hrs or 72 hrs)
    • 3 within 48 hours (2 specified as 48 hrs or 72 hrs)
    • 6 within 72 hours (1 specified as 72 hrs or 1 week)
  • Missed serious diagnoses—7
    • 1 unanticipated death within 7 days following ED care
    • 6 admission for missed diagnosis (AMI/ACS, SAH, ectopic pregnancy, traumatic injury, appendicitis)
  • Appropriate use of diagnostic test or imaging—5
  • Acute complications of ED procedures—3
  • Time within ED awaiting definitive care—3
  • Appropriate admission for inpatient care—2
  • Appropriate use of treatment or intervention—1
  • Left "against medical advice"—1

Slide 23

Challenges in specification and testing

Challenges in specification and testing

  • Identification of patients "at risk"
    • What procedures place patients at risk for hemorrhage or accidental puncture/laceration?
  • Timing
    • Did the fall occur prior to ED arrival, in ED, or later?
  • Low frequency with "true" frequency unknown
    • Unable to choose "best" specification
  • Use of utilization flag variables to identify patients who had specific procedures
    • Ultrasound, ECG, CT scan, transfusion
  • Unable to operationalize all specifications
    • Exclusion of "planned" (or "invited") return visits to ED
    • All presenting symptoms for "missed diagnoses"

Slide 24

Future Steps

Future steps

  • Complete testing of adapted inpatient PSIs
  • Prioritize 23 candidate indicators applicable to adults to select 7-12 for full specification and testing
    • Denominator inclusion/exclusion rules
    • Numerator definition
  • Assess face validity based on empirical analyses of HCUP data from 9 states
  • Recommend 5-7 indicators for review and feedback by an external "work group" with a diverse set of stakeholders
  • Formal evaluation by expert panels through a modified Delphi panel process?
  • Release of new module of ED PSIs?

Slide 25

Acknowledgments

Acknowledgments

  • UC Davis team
    • Banafsheh Sadeghi (epidemiologist)
    • David Barnes and Aaron Bair (emergency physicians)
    • Yun Jiang and Daniel Tancredi (programming and analysis)
  • External advisors
    • Jesse Pines (GWU), Michael Phelan (Cleveland Clinic), Emily Carrier (HSC), Evaline Alessandrini (CCHMC), Astrid Guttmann (ICES), Jeremiah Schuur (Brigham & Women's)
  • AHRQ CDOM staff
    • Pamela Owens and Ryan Mutter (ED task)
    • Mamatha Pancholi and John Bott (QI program)
    • Jenny Schnaier and Carol Stocks (HCUP)
  • HCUP partners
    • Arizona, California, Florida, Hawaii, Indiana, Nebraska, South Carolina, Tennessee, Utah
Current as of December 2010
Internet Citation: The Development of Emergency Department Patient Quality/Safety Indicators: Slide Presentation from the AHRQ 2010 Annual Conference. December 2010. Agency for Healthcare Research and Quality, Rockville, MD. http://www.ahrq.gov/news/events/conference/2010/romano2/index.html