Slide Presentation from the AHRQ 2008 Annual Conference
On September 10, 2008, Amy Rosen, Ph.D., made this presentation at the 2008 Annual Conference. Select to access the PowerPoint® presentation (800 KB; Plugin Software Help).
Validating the Patient Safety Indicators (PSIs) in the Department of Veterans' Affairs (VA): a Multi-Faceted Approach
- Funding: VA Health Services Research and Development (HSR&D) Service
SDR 07-002 : 10/01/07-9/30/11
- Amy Rosen, Ph.D., Principal Investigator
- Collaboration among:
- VA's HSR&D Service
- National Center for Patient Safety (NCPS)
- AHRQ (Quality Indicators [QI] team and individual investigators)
- VA and non-VA clinicians, surgical experts, nurse abstractors
- National steering committee:
- Representatives from VA Office of Quality Performance, NCPS
- Nursing Services, Surgery, Patient Care Services
- Selected members of the AHRQ QI team
- Selected Patient Safety/QI Managers and other potential end-users
Overall Project Goal
- Develop a validated and reliable set of patient safety measures that broadly reflect the interests of key VA stakeholders, but that are generalizable beyond the VA.
- Specific Objectives:
- Develop collaborations with key stakeholders to guide in PSI selection and validation
- Investigate the criterion validity of the PSIs by review of the VA's Electronic Medical Record (EMR)
- Identify processes and structures of care associated with individual PSIs
- Revise and improve the PSIs using multiple data sources and settings of care
- Assess the utility validity of the PSIs for QI and performance measurement
Goal 1: Develop Stakeholder Collaboration
- Stakeholders' meeting (Dec, 2007):
- Approved selection of PSIs
- Approved plan to validate AHRQ's Phase I/Phase II PSIs
- Reviewed field consultation interview questions:
- Recommended focus on general questions on patient safety
- Suggested less attention on specific PSIs in field consultations:
- Field consultations held to examine the validity of the PSIs, not to judge facilities' performance
- Contact with stakeholders subsequent to meeting:
- Approved final interview protocols "TO/THRU" memo to sites asking them to participate
Goal 2: Identify False Positives—Are Cases Flagged by the AHRQ PSIs Present in the EMR?
- Obtained national access to electronic medical record (EMR): "VistaWeb"
- Hired and trained two nurse abstractors to conduct chart abstraction
- Modified AHRQ Phase I and Phase II chart abstraction tools for VA:
- Pilot testing and clinician review
- Five tools "ready for prime time," five almost ready, five being developed de novo
- Completed validation of pulmonary embolism/deep vein thrombosis (PE/DVT)
- Currently abstracting charts for iatrogenic pneumothorax
- Currently piloting web-based application (InfoPath) for gathering and entering chart-abstracted data
Formatting Features of InfoPath
- This slide shows the conditional formatting features of InfoPath; these features enable more efficient chart abstraction.
- A1: Is there documentation that the patient had a post-operative or deep vein thrombosis during this admission?
- Unable to determine
- Record excluded
- A2: Documentation of ascertainment of event If YES to A1, Describe documentation found in the medical record.
- Ran PSI software (v. 3.1a) on VA inpatient data (2003-2007):
- Obtained rates of individual PSIs and PSI composites
- Used 12 PSIs:
- PSIs 1-15
- Excluded PSIs 1, 5, 8
- Sample for chart abstraction:
- 28 hospitals, 112 charts per PSI
Sample Selection Methodology
- Stratified population by observed and expected #s of PSIs:
- Group 1: at least 4 observed and 4 expected (n =28)
- Group 2: at least 2 observed and 2 expected (n=33)
- Group 3: at least 1 observed and 1 expected (n=18)
- Total for Groups 1-3: 79 hospitals
- Ranked 79 by AHRQ PSI composite (denominator weights):
- Chose top 3 and bottom 3 from each group
- Randomly selected from remaining hospitals within each group: group 1=4, group 2=4, group 3=2 to obtain 28 hospitals (10, 10, and 8, respectively)
- Geographic distribution and ICU severity taken into account
- Selected 6 hospitals for field consultations and ranked them based on PSI composite:
- Geographic location and size taken into account
- Conducted retrospective EMR review of 112 flagged cases.
- Conducted inter-rater reliability (IRR) throughout EMR review.
- 28 cases (25% of all charts) reviewed for IRR due to:
- Large numbers of exclusions
- IRR >90%
- 89% agreement rate achieved with 1st IRR, 94% with 2nd IRR
- Length of time to complete chart abstraction (1-1/2 hours for full record; 20 minutes for false positives)
- Problems with accessing VistaWeb
Technical Specifications of PE/DVT
- Discharges among cases meeting the inclusion and exclusion rules for denominator.
- ICD-9-CM codes for PE/DVT in any secondary diagnosis field
- All surgical discharges age 18 and older.
- Defined by specific DRGs and an ICD-9-CM code for an OR procedure
- Exclusion criteria for all cases:
- Preexisting (principal diagnosis or secondary diagnosis present on admission, if known) PE/DVT
- Procedure for interruption of vena cava the only OR procedure
- Procedure for interruption of vena cava occurs before or on the same day as first OR procedure
- Medical Diagnostic Category (MDC) 14 (pregnancy, childbirth, and puerperium)
Post-operative PE/DVT Validation Results
- Pie chart shows:
- True Postoperative PE/DVT: 49 cases (44%)
- Coding-Related Inaccurate Diagnosis: 24 cases (21%)
- Present on Admission: 16 cases (14%)
- Pre-Procedure Diagnosis: 13 cases (12%)
- Remote History of PE/DVT: 10 cases (9%)
- Total # of cases: 112
False Positives: A Comprehensive Analysis
|Classification of False Positives
|DVT/PE Present on Admission (POA)
|Pre-Procedure Diagnosis of PE/DVT
|Remote History of DVT or PE (>6 months)
|Arterial (not venous) thrombosis*
|Negative PE/DVT workup*
|"Rule out PE" as cause of death*
|Superficial (not deep) thrombosis or thrombophlebitis*
* Represents coding-related inaccurate diagnosis
Coding-Related Inaccurate Diagnosis: Miscellaneous Category
|Classification of False Positives
|Vein stenosis (no thrombosis)
|PE stands for Physical Exam not Pulmonary Embolus
|Low dose Coumadin prophylactic not therapeutic
|Surgery done at outside hospital
|Cerebral embolization of arteriovenous malformation (AVM)
|Prophylactic heparin mistaken for therapeutic heparin
|Right lower extremity (RLE) U/S ordered to r/o abscess at surgical site
PE/DVT Results: Comparison of Studies
Problems in Coding PE/DVT
- PE/DVT PSI designed as initial screen
- Accuracy of method to detect true positives using administrative data affected by:
- Standards used to assign codes for "other" or secondary conditions—> based on the Uniform Hospital Discharge Data Set (UHDDS).
- "Other" conditions: those that coexist at the time of admission, develop sequentially, affect the treatment received and/or length of stay, or affect patient care
- Definition of PE/DVT relative to:
- UHDDS coding standards,
- ICD-9-CM Official Coding Guidelines for Coding and Reporting
- Coding Clinic published by the American Hospital Association (AHA)
Problems in Coding PE/DVT, cont'd
- False Positive 1: chart review does not document a PE/DVT:
- Code was present on admission (POA) and meets UHDDS definition of "other" diagnosis.
- Code assigned as a current condition.
- Should have been coded as a "history of" with a V code
- It was still a "rule out" condition at the time of discharge
- Coding system issue:
- Was miscoded (superficial vein and not deep vein) due to coding invention and ICD-9-CM alphabetic index
- Coder did not identify the correct vein anatomically
- Should not have been coded at all
- False Positive 2: chart review documents a PE/DVT, but it is not a postoperative PE/DVT:
- Diagnosis of PE/DVT occurred after admission but before surgery
Recommendations for Improving PE/DVT
- Modify coding rules:
- Use National Surgical Quality Improvement Program (NSQIP) definitions to influence the coding rules
- Specify the circumstances when the PE/DVT should be coded and publish them in Coding Clinic and Official Guidelines
- As "current conditions" or "history of"
- Begin using POA in VA
- Explore use of "997" complication code as part of the PSI algorithm to capture post-operative PSIs
- Explore expansion of POA to include a special character denoting "POA prior to surgery"
- Undertake targeted education to help coders, researchers, and healthcare professionals understand the use of coding guidelines for "PE/DVT"
Question: Do High-Performing Facilities Have Higher Rates on Structures and Processes of Care than Lower-Performing Facilities?
- Conduct two pilot field consultations locally:
- determine feasibility and logistical problems
- test interview questions
- add/delete selected staff
- Conduct field consultations at 6 facilities:
- Perform structured interviews with selected staff
- Gather data on safety and quality
- Assess differences between sites on structures and process using qualitative methods and ratings
Selected Staff for Interviews
- Individual Interviews:
- Service Chiefs
- Other Middle Managers
- Other Non-Managers
- Group Interviews:
- Surgical Service
- Medical Service
- Organization, Structure, and Culture
- Coordination of Work and Communication
- Interface within Service
- Monitoring Quality of Care
- Quality Improvement
- General Clinical Topics
- Technology and Equipment
- Technical Competence of Staff
- Interface with Other Services
- Systems Issues and Human Factors
- Summary Evaluation of Service Overall
Domain: Monitoring Quality of Care/Quality Improvement
- In your facility, what are some of the initiatives related to improving patient safety that you know about?
- On what does it focus?
- What facilitated its implementation?
- What were the implementation obstacles?
- How effective do you think it is?
- What are some of the most common adverse events that you see in your day-to-day work? Please refer to the list provided.
- What is being done now to reduce the incidence of this complication?
- What do you think would be helpful in further reducing the incidence of this?
- Is there anything not on the list we provided you that you believe is a concern?
- Who is involved in assigning ICD-9 and procedure codes to adverse events?
- Are physicians involved in reviewing the event codes?
- Do you think there is a concern about the accuracy of coding relating to adverse events?
- If yes: What is the concern?
Domain: Technology and Equipment/Technical Competence of Staff
- I am curious to hear about what problems, if any, you or others have had with the technology and/or equipment on the service.
- What problems have you had with the accessibility or availability, or both, of technology and/or equipment?
- What problems you have had with the quality or functioning, or both, of the technology and/or equipment?
- What problems, if any, have you or other staff had being properly trained to use the technology and/or equipment?
- What technology and/or equipment, if any, does not exist at your hospital that would help improve patient safety?
Capturing Initial Impressions
- Immediately after each pilot field consultation, each interviewer summarizes her/his:
- Impressions of each domain in a paragraph
- Overall impressions of the site
- —> in both cases giving specific examples
- Soon afterwards, all interviewers and other members of the PSI validation team meet to discuss the impressions.
- These discussions will be used to generate a protocol for capturing initial impressions for study's six field consultations.
- We may rate sites, creating examples for an "ideal" site
- We may decide to use only written impressions
Rating Category Possibilities
- Some numeric scale:
- NSQIP rating (1 to 9; 1=poor and 9=excellent)
- Other model rating (0 to 4)
- Some hierarchy scale:
- Poor, fair, good, very good, excellent
- Some recognition scale:
Example of Rating: NSQIP
|Interface with Other
Initial Impressions of Pilot Sites
|Monitoring Quality of Care
Questions 1, 3, 4
Questions 1, 3
Questions 2, 4
|Systems Issues and Human Factors
* Our initial rating scale: Excellent, Very good, Good, Fair, Poor
Next Steps (1)
- Identify False Negatives
- Use an existing "gold standard" (e.g., VA NSQIP) for 5 surgical PSIs
- Identify risk factors by estimating logistic regression models for each of the PSIs
- Use propensity score stratification to generate propensity class strata for each of the PSIs
- Use AHRQ Composite Tool to review medical records of "high-risk" cases for PSIs
- Screen EMRs of high-risk cases using keyword searches (selected "hits" will have chart review)
- Explore machine language processing as an informatics tool to search for false negatives
Next Steps (2)
- Examine association between explicit processes of care and individual PSIs:
- Match 1,680 flagged PSI cases with 1,680 controls (unflagged cases matched on demographic and clinical characteristics) to determine whether flagged cases are more likely to experience "process failures."
- Use propensity score methodology to perform matching; chi-square tests used to examine proportion of failure rates among cases and controls.
Next Steps (3)
- Revise and Improve the PSIs:
- Add additional data elements to inpatient data:
- Present-on-admission (POA) diagnoses, do-not-resuscitate (DNR) codes, selected clinical, laboratory and pharmacy data elements
- Link inpatient data with outpatient/inpatient data 30/60 days preceding index hospitalization (obtain POA diagnoses)
- Link inpatient data with outpatient/inpatient data 30/60 days following index hospitalization to evaluate whether additional PSIs are detected
- Link VA and Medicare data to examine PSI readmission in private sector
- Improve coding by implementing coding changes
- Modify PSI numerators and denominators on inclusion/exclusion criteria
- Recalculate false positives and negatives
- Amy Rosen, Ph.D.
Center for Health Quality, Outcomes & Economic Research
(VA Center of Excellence)
- Boston University Schools of Public Health and Medicine,
Departments of Health Policy and Management and Family Medicine
Current as of January 2009
Validating the Patient Safety Indicators (PSI) in the VA: A Multi-Faceted Approach. Slide Presentation from the AHRQ 2008 Annual Conference (Text Version). January 2009. Agency for Healthcare Research and Quality, Rockville, MD. http://www.ahrq.gov/about/annualmtg08/091008slides/Rosen.htm