Validating the Patient Safety Indicators (PSI) in the VA: A Multi-Faceted Approach

Slide Presentation from the AHRQ 2008 Annual Conference

On September 10, 2008, Amy Rosen, Ph.D., made this presentation at the 2008 Annual Conference. Select to access the PowerPoint® presentation (800 KB; Plugin Software Help).

Slide 1

Validating the Patient Safety Indicators (PSIs) in the Department of Veterans' Affairs (VA): a Multi-Faceted Approach

• Funding: VA Health Services Research and Development (HSR&D) Service
  SDR 07-002 : 10/01/07-9/30/11
• Amy Rosen, Ph.D., Principal Investigator

Slide 2

Project Team

• Collaboration among:
  • VA's HSR&D Service
  • National Center for Patient Safety (NCPS)
  • AHRQ (Quality Indicators [QI] team and individual investigators)
• VA and non-VA clinicians, surgical experts, nurse abstractors
• National steering committee:
  • Representatives from VA Office of Quality Performance, NCPS
  • Nursing Services, Surgery, Patient Care Services
  • Selected members of the AHRQ QI team
  • Selected Patient Safety/QI Managers and other potential end-users

Slide 3

Overall Project Goal

• Develop a validated and reliable set of patient safety measures that broadly reflect the interests of key VA stakeholders, but that are generalizable beyond the VA.
• Specific Objectives:
  • Develop collaborations with key stakeholders to guide in PSI selection and validation
  • Investigate the criterion validity of the PSIs by review of the VA's Electronic Medical Record (EMR)
  • Identify processes and structures of care associated with individual PSIs
  • Revise and improve the PSIs using multiple data sources and settings of care
  • Assess the utility validity of the PSIs for QI and performance measurement

Slide 4

Goal 1: Develop Stakeholder Collaboration

• Stakeholders' meeting (Dec, 2007):
  • Approved selection of PSIs
  • Approved plan to validate AHRQ's Phase I/Phase II PSIs
  • Reviewed field consultation interview questions:
    • Recommended focus on general questions on patient safety
  • Suggested less attention on specific PSIs in field consultations:
    • Field consultations held to examine the validity of the PSIs, not to judge facilities' performance
• Contact with stakeholders subsequent to meeting:
  • Approved final interview protocols "TO/THRU" memo to sites asking them to participate

Slide 5

Goal 2: Identify False Positives—Are Cases Flagged by the AHRQ PSIs Present in the EMR?

• Obtained national access to electronic medical record (EMR): "VistaWeb"
• Hired and trained two nurse abstractors to conduct chart abstraction
• Modified AHRQ Phase I and Phase II chart abstraction tools for VA:
  • Pilot testing and clinician review
  • Five tools "ready for prime time," five almost ready, five being developed de novo
• Completed validation of pulmonary embolism/deep vein thrombosis (PE/DVT)
• Currently abstracting charts for iatrogenic pneumothorax
• Currently piloting web-based application (InfoPath) for gathering and entering chart-abstracted data

Slide 6

Formatting Features of InfoPath

• This slide shows the conditional formatting features of InfoPath; these features enable more efficient chart abstraction.
  • A1: Is there documentation that the patient had a post-operative or deep vein thrombosis during this admission?
    • Yes:
      • PE
      • DVR
    • No
    • Unable to determine
    • Record excluded
  • A2: Documentation of ascertainment of event If YES to A1, Describe documentation found in the medical record.

Slide 7

Hospital Selection

• Ran PSI software (v. 3.1a) on VA inpatient data (2003-2007):
  • Obtained rates of individual PSIs and PSI composites
• Used 12 PSIs:
  • PSIs 1-15
  • Excluded PSIs 1, 5, 8
• Population:
  • 158 VA hospitals
• Sample for chart abstraction:
  • 28 hospitals, 112 charts per PSI

Slide 8

Sample Selection Methodology

1. Stratified population by observed and expected #s of PSIs:
   • Group 1: at least 4 observed and 4 expected (n =28)
   • Group 2: at least 2 observed and 2 expected (n=33)
   • Group 3: at least 1 observed and 1 expected (n=18)
   • Total for Groups 1-3: 79 hospitals
2. Ranked 79 by AHRQ PSI composite (denominator weights):
   • Chose top 3 and bottom 3 from each group
   • Randomly selected from remaining hospitals within each group: group 1=4, group 2=4, group 3=2 to obtain 28 hospitals (10, 10, and 8, respectively)
   • Geographic distribution and ICU severity taken into account
3. Selected 6 hospitals for field consultations and ranked them based on PSI composite:
   • Geographic location and size taken into account

Slide 9

Chart Abstraction

PE/DVT

• Conducted retrospective EMR review of 112 flagged cases.
• Conducted inter-rater reliability (IRR) throughout EMR review.
  • 28 cases (25% of all charts) reviewed for IRR due to:
    • Large numbers of exclusions
    • IRR >90%
  • 89% agreement rate achieved with 1st IRR, 94% with 2nd IRR
• Issues:
  • Length of time to complete chart abstraction (1-1/2 hours for full record; 20 minutes for false positives)
  • Problems with accessing VistaWeb

Slide 10

Technical Specifications of PE/DVT

• Numerator:
  • Discharges among cases meeting the inclusion and exclusion rules for denominator.
    • ICD-9-CM codes for PE/DVT in any secondary diagnosis field
• Denominator:
  • All surgical discharges age 18 and older.
    • Defined by specific DRGs and an ICD-9-CM code for an OR procedure
• Exclusion criteria for all cases:
  • Preexisting (principal diagnosis or secondary diagnosis present on admission, if known) PE/DVT
  • Procedure for interruption of vena cava the only OR procedure
  • Procedure for interruption of vena cava occurs before or on the same day as first OR procedure
  • Medical Diagnostic Category (MDC) 14 (pregnancy, childbirth, and puerperium)

Slide 11

Post-operative PE/DVT Validation Results

• Pie chart shows:
  • True Postoperative PE/DVT: 49 cases (44%)
  • Coding-Related Inaccurate Diagnosis: 24 cases (21%)
  • Present on Admission: 16 cases (14%)
  • Pre-Procedure Diagnosis: 13 cases (12%)
  • Remote History of PE/DVT: 10 cases (9%)
• Total # of cases: 112

Slide 12

False Positives: A Comprehensive Analysis

  *  Represents coding-related inaccurate diagnosis

Slide 13

Coding-Related Inaccurate Diagnosis: Miscellaneous Category

Slide 14

PE/DVT Results: Comparison of Studies

Slide 15

Problems in Coding PE/DVT

• PE/DVT PSI designed as initial screen
• Accuracy of method to detect true positives using administrative data affected by:
  • Standards used to assign codes for "other" or secondary conditions—> based on the Uniform Hospital Discharge Data Set (UHDDS).
    • "Other" conditions: those that coexist at the time of admission, develop sequentially, affect the treatment received and/or length of stay, or affect patient care
  • Definition of PE/DVT relative to:
    • UHDDS coding standards,
    • ICD-9-CM Official Coding Guidelines for Coding and Reporting
    • Coding Clinic published by the American Hospital Association (AHA)

Slide 16

Problems in Coding PE/DVT, cont'd

• False Positive 1: chart review does not document a PE/DVT:
  1. Code was present on admission (POA) and meets UHDDS definition of "other" diagnosis.
  2. Code assigned as a current condition.
     • Should have been coded as a "history of" with a V code
     • It was still a "rule out" condition at the time of discharge
  3. Coding system issue:
     • Was miscoded (superficial vein and not deep vein) due to coding invention and ICD-9-CM alphabetic index
     • Coder did not identify the correct vein anatomically
     • Should not have been coded at all
• False Positive 2: chart review documents a PE/DVT, but it is not a postoperative PE/DVT:
  • Diagnosis of PE/DVT occurred after admission but before surgery

Slide 17

Recommendations for Improving PE/DVT

• Modify coding rules:
  • Use National Surgical Quality Improvement Program (NSQIP) definitions to influence the coding rules
  • Specify the circumstances when the PE/DVT should be coded and publish them in Coding Clinic and Official Guidelines
    • As "current conditions" or "history of"
• Begin using POA in VA
• Explore use of "997" complication code as part of the PSI algorithm to capture post-operative PSIs
• Explore expansion of POA to include a special character denoting "POA prior to surgery"
• Undertake targeted education to help coders, researchers, and healthcare professionals understand the use of coding guidelines for "PE/DVT"

Slide 18

Objective 3

Question: Do High-Performing Facilities Have Higher Rates on Structures and Processes of Care than Lower-Performing Facilities?

• Conduct two pilot field consultations locally:
  • determine feasibility and logistical problems
  • test interview questions
  • add/delete selected staff
• Conduct field consultations at 6 facilities:
  • Perform structured interviews with selected staff
  • Gather data on safety and quality
• Assess differences between sites on structures and process using qualitative methods and ratings

Slide 19

Selected Staff for Interviews

• Individual Interviews:
  • Executives
  • Service Chiefs
  • Other Middle Managers
  • Other Non-Managers
• Group Interviews:
  • Surgical Service
  • Medical Service
  • Non-Managers

Slide 20

Interview Domains

• Organization, Structure, and Culture
• Coordination of Work and Communication
• Interface within Service
• Monitoring Quality of Care
• Quality Improvement
• General Clinical Topics
• Coding
• Technology and Equipment
• Technical Competence of Staff
• Leadership
• Interface with Other Services
• Systems Issues and Human Factors
• Staffing
• Summary Evaluation of Service Overall

Slide 21

Domain: Monitoring Quality of Care/Quality Improvement

• In your facility, what are some of the initiatives related to improving patient safety that you know about?
  • On what does it focus?
  • What facilitated its implementation?
  • What were the implementation obstacles?
  • How effective do you think it is?
• What are some of the most common adverse events that you see in your day-to-day work? Please refer to the list provided.
  • What is being done now to reduce the incidence of this complication?
  • What do you think would be helpful in further reducing the incidence of this?
  • Is there anything not on the list we provided you that you believe is a concern?

Slide 22

Domain: Coding

• Who is involved in assigning ICD-9 and procedure codes to adverse events?
  • Are physicians involved in reviewing the event codes?
  • Do you think there is a concern about the accuracy of coding relating to adverse events?
  • If yes: What is the concern?
    • How is this addressed?

Slide 23

Domain: Technology and Equipment/Technical Competence of Staff

• I am curious to hear about what problems, if any, you or others have had with the technology and/or equipment on the service.
  • What problems have you had with the accessibility or availability, or both, of technology and/or equipment?
  • What problems you have had with the quality or functioning, or both, of the technology and/or equipment?
  • What problems, if any, have you or other staff had being properly trained to use the technology and/or equipment?
  • What technology and/or equipment, if any, does not exist at your hospital that would help improve patient safety?

Slide 24

Capturing Initial Impressions

• Immediately after each pilot field consultation, each interviewer summarizes her/his:
  • Impressions of each domain in a paragraph
  • Overall impressions of the site
  • —> in both cases giving specific examples
• Soon afterwards, all interviewers and other members of the PSI validation team meet to discuss the impressions.
• These discussions will be used to generate a protocol for capturing initial impressions for study's six field consultations.
  • We may rate sites, creating examples for an "ideal" site
  • We may decide to use only written impressions

Slide 25

Rating Category Possibilities

• Some numeric scale:
  • NSQIP rating (1 to 9; 1=poor and 9=excellent)
  • Other model rating (0 to 4)
• Some hierarchy scale:
  • Poor, fair, good, very good, excellent
• Some recognition scale:
  • Bronze, silver, gold

Slide 26

Example of Rating: NSQIP

Slide 27

Initial Impressions of Pilot Sites

* Our initial rating scale: Excellent, Very good, Good, Fair, Poor

Slide 28

Next Steps (1)

• Identify False Negatives
  • Use an existing "gold standard" (e.g., VA NSQIP) for 5 surgical PSIs
  • Identify risk factors by estimating logistic regression models for each of the PSIs
  • Use propensity score stratification to generate propensity class strata for each of the PSIs
  • Use AHRQ Composite Tool to review medical records of "high-risk" cases for PSIs
  • Screen EMRs of high-risk cases using keyword searches (selected "hits" will have chart review)
  • Explore machine language processing as an informatics tool to search for false negatives

Slide 29

Next Steps (2)

• Examine association between explicit processes of care and individual PSIs:
  • Match 1,680 flagged PSI cases with 1,680 controls (unflagged cases matched on demographic and clinical characteristics) to determine whether flagged cases are more likely to experience "process failures."
  • Use propensity score methodology to perform matching; chi-square tests used to examine proportion of failure rates among cases and controls.

Slide 30

Next Steps (3)

• Revise and Improve the PSIs:
  • Add additional data elements to inpatient data:
    • Present-on-admission (POA) diagnoses, do-not-resuscitate (DNR) codes, selected clinical, laboratory and pharmacy data elements
  • Link inpatient data with outpatient/inpatient data 30/60 days preceding index hospitalization (obtain POA diagnoses)
  • Link inpatient data with outpatient/inpatient data 30/60 days following index hospitalization to evaluate whether additional PSIs are detected
  • Link VA and Medicare data to examine PSI readmission in private sector
  • Improve coding by implementing coding changes
  • Modify PSI numerators and denominators on inclusion/exclusion criteria
  • Recalculate false positives and negatives

Slide 31

THANK YOU!

Slide 32

• Amy Rosen, Ph.D.
  Center for Health Quality, Outcomes & Economic Research
  (VA Center of Excellence)
• Boston University Schools of Public Health and Medicine,
  Departments of Health Policy and Management and Family Medicine

• Phone #: (781) 687-2960
  E-mail: akrosen@bu.edu