Evaluating the AHRQ Inpatient Mortality Indicators for Public Reporting in California

Slide Presentation from the AHRQ 2008 Annual Conference

On September 10, 2008, Joseph Parker, Ph.D., Brian Paciotti, Ph.D., and Merry Holliday-Hanson, Ph.D., made this presentation at the 2008 Annual Conference. Select to access the PowerPoint® presentation (1.1 MB; Plugin Software Help).

Slide 1

Evaluating the AHRQ Inpatient Mortality Indicators for Public Reporting in California

AHRQ Quality Indicator (QI) Users Meeting

Joseph Parker, PhD
Brian Paciotti, PhD
Merry Holliday-Hanson, PhD
Healthcare Outcomes Center
Office of Statewide Health Planning and Development (OSHPD)

Rockville, MD
September 10, 2008

Slide 2

AHRQ Inpatient Mortality Indicators (IMIs): Background to CA Decision

• Per statute, OSHPD required to publish 9 risk-adjusted hospital outcome reports per year.
• Traditional approach to producing reports costly, time-consuming, fraught with delays.
• CA patient discharge data now available only 7 months after end of reporting year (inpatient mortality).
• State death file (necessary for 30-day mortality) not available until 15 months after end of reporting year.
• Present on Admission (POA) now incorporated in All Patient Refined Diagnosis Related Groups (APR-DRGs) risk adjustment algorithm.
• Some IMIs have undergone National Quality Forum (NQF) vetting process.

Slide 3

AHRQ Inpatient Mortality Indicators: California Plans for Public Reporting

* Endorsed by NQF May, 2008.
Indicators not planned for release are italicized.

Slide 4

Comparison of Traditional OSHPD Mortality Reports* with AHRQ IMIs

* Not including CABG report based on clinical data

Slide 5

Planned and Published OSHPD Quality Metrics: Varying Levels of Validity

* Includes two reports produced (heart attack, pneumonia) and two in progress.
** Two reports in progress (AAA repair & CHF) to be released without a formal validation study.

Slide 6

AHRQ Message on IMI Validity

• Providers, policymakers, and researchers can use with inpatient data to identify apparent variations in the quality of inpatient care.
• Although quality assessments based on administrative data cannot be definitive, they can be used to flag potential quality problems and success stories, which can then be further investigated and studied.
• Hospital associations, individual hospitals, purchasers, regulators, and policymakers at the local, State, and Federal levels can use readily available hospital administrative data to begin the assessment of quality of care.

Slide 7

OSHPD Message on IMI Validity

OSHPD views these indicators as potentially useful starting points for examining hospital quality but does not regard them as definitive measures of quality. When this information is carefully considered, with its limitations, alongside other reliable healthcare provider information, it may be helpful to patients and purchasers when making decisions about healthcare treatment choices. Healthcare providers may also benefit from using this information in quality improvement activities.

Slide 8

OSHPD Implementation of IMIs

• Used 2007 CA patient Discharge Data.
• Transformed data elements and values into formats for AHRQ software (Version 3.2).
• POA option utilized:
  • APR-DRG risk scores based only on conditions coded as pre-existing, not hospital-related complications.
• Calculated risk-adjusted rates:
  • Used APR-DRG risk model with coefficients from CA & NY.
  • Logistic regression model (with random hospital effects).
• Calculated statistical outliers:
  • No hospital case volume limit applied.
  • 95% upper and lower CIs.
  • "Better" and "Worse than Expected" labels used.

Slide 9

2007 CA Statewide California IMI Results

Slide 10

Are IMIs (Potentially) Biased in Ways That Can Be Detected Using Internal Data?

• Do certain types of hospitals (teaching, public, profit, non-profit) have worse IMI rates than other types of hospitals?
• Do hospitals with a large percentage of Do-not-resuscitate (DNR), palliative care, or skilled nursing facility (SNF) patients have worse IMI rates than other hospitals?
• Do hospitals with very poor POA coding quality (rarely coding complications) have better IMI rates than others?
• Do high transfer-intensity hospitals benefit from use of inpatient mortality compared to 30-day mortality?
• Does the AHRQ inclusion of POA improve validity of original IMI by bringing it closer to a gold standard proxy?

Slide 11

Hospital 2007 IMI Rates by Type of Hospital

¹ Includes all CA hospitals meeting minimum volume thresholds: number of actual hospitals reporting varies by IMI.
*p<.05
**p<0.01
***p<0.001 (ANOVA)

Slide 12

Effect of DNR, Palliative Care, and SNF Patients on Hospital Performance

• For each hospital, across all patients, calculated the percent of total who had the following:
  • DNR coded within 24 hours of admission.
  • ICD-9 code for palliative care (V66.7).
  • Source of admission = SNF.
• Correlated percent of patients with DNR, PC, & SNF with IMI rates.
• Determined which 10% of hospitals had highest coding rates, separately, for DNR, PC & SNF:
  • Calculated and compared the IMI rates for the top 10% group with other 90%.

Slide 13

Correlations Between Hospital DNR, Palliative Care, and SNF Admission Rates and IMI Rates

*p<.05
**p<0.01
***p<0.001

Slide 14

Do Hospitals Coding Large Numbers of DNR, Palliative Care, or SNF Admissions Have Higher IMI Rates?

Slide 15

Impact of Hospital POA Coding Quality on Performance

• Collecting POA data element does not ensure reliably coded POA information across hospitals.
• Calculated IMI rates using non-POA version and POA version of APR-DRG software.
• Produced scatter plots and calculated correlations between both versions of indicators.
• Created two POA coding quality metrics to understand impact of poor POA coding on IMI rates:
  • OSHPD metric using percent of all hospital secondary diagnoses coded POA='yes' (gross measure).
  • POA coding quality metric published by Hughes et al. (2006).

Slide 16

Craniotomy: Impact of POA (N=170, R=0.96)

Graph: Left-hand scale shows risk-adjusted mortality rates (RAMRs), % No POA Indicator, from 0 to 40; bottom scale shows RAMR, % POA Indicator, from 0 to 50; most data points are clustered at lower left, rising at roughly a 45-degree angle.

Slide 17

Stroke: Impact of POA (N=331, r=0.99)

Graph: Left-hand scale shows RAMR, % No POA Indicator, from 0 to 40; bottom scale shows RAMR, % POA Indicator, from 0 to 40; data points are clustered along a line rising at roughly a 45-degree angle.

Slide 18

GI Hemorrhage: Impact of POA (N=318, r=0.97)

Graph: Left-hand scale shows RAMR, % No POA Indicator, from 0 to 30; bottom scale shows RAMR, % POA Indicator, from 0 to 30; data points are clustered along a line rising at roughly a 45-degree angle.

Slide 19

Hip Fracture: Impact of POA (N=352, r=0.98)

Graph: Left-hand scale shows RAMR, % No POA Indicator, from 0 to 80; bottom scale shows RAMR, % POA Indicator, from 0 to 80; data points are clustered along a line rising at roughly a 45-degree angle.

Slide 20

PTCA: Impact of POA (N=140, r=0.97)

Graph: Left-hand scale shows RAMR, % No POA Indicator, from 0 to 4; bottom scale shows RAMR, % POA Indicator, from 0 to 8; data points are clustered along a line rising at roughly a 45-degree angle.

Slide 21

Carotid Endarterectomy: Impact of POA (N=219, r=0.83)

Graph: Left-hand scale shows RAMR, % No POA Indicator, from 0 to 15; bottom scale shows RAMR, % POA Indicator, from 0 to 40; data points are scattered.

Slide 22

Do Hospitals with Poor POA Coding Have Better IMI Rates Than Other Hospitals?

Slide 23

Does the Use of Inpatient Mortality vs. 30-Day Mortality Bias Hospital Results?

• 30-day mortality is the preferred measure for outcomes assessment:
  • Not impacted by hospital discharge practices.
  • May result in less timely reports.
• Transfer rates vary greatly by hospital:
  • Transfers/discharges to acute care (excluded in IMIs): 2.5% overall.
  • Transfers/discharges to SNF/IC/Other care: 14.5% overall.
• Methods:
  • Created hospital "transfer intensity" measure:
    • Rates created for top 10%, middle 80%, and bottom 10% of hospitals according to ALL patient discharges to non-acute care (SNF/Intermediate Care, Other: e.g., psych., phys. Med., rehab.).
  • Calculated crude inpatient and 30-day mortality rates for hospitals by transfer intensity group.
  • Calculated difference between hospital inpatient and 30-day crude mortality rates by IMI.
  • Calculated an "impact" measure that describes how different transfer intensity groups are impacted by use of inpatient mortality.

Slide 24

How Does Hospital Transfer Intensity Impact 30-day Crude Mortality Rates Compared to Inpatient Rates?

* Impact = Difference for High or Low group minus difference for moderate group.

Slide 25

Does Inclusion of POA Coding Improve the IMIs Relative to a Gold Standard Proxy?

• Generated 2006 risk-adjusted mortality rates (RAMRs) for hospitals using CABG gold standard proxy (Parker et al., Med Care 2006).
• For same patient cohort, generated RAMRs using AHRQ software with and without POA.
• Used inpatient mortality for both outcomes.
• Calculated correlation between AHRQ CABG IMI (with & without POA) with CABG gold standard proxy.
• Correlation with gold standard proxy improved from .87 (without POA) to .93 (with POA): a 7% improvement.

Slide 26

Summary of Findings

• Public hospitals had highest mortality rates for 5 of 6 IMIs: Both Teaching and Investor hospitals had lowest rates for 3 IMIs.
• Some patient illness severity is not explained by APR-DRG risk model (as demonstrated by DNR, PC & SNF status) but this does not strongly bias results (stroke a possible exception).
• Hospitals that do a poor job of POA coding (complications credited as pre-existing conditions) do not appear to benefit in terms of lower IMI rates.
• Hospitals that rarely transfer patients to SNF/IC/Other care perform more poorly when using inpatient mortality compared to 30-day mortality:
  • Transfer rates to SNF/IC/Other care by hospital type:
    Non-profit = 15% Investor = 20% Public= 15% Teaching = 10%
• Utilization of POA coding improves assessment of hospital quality relative to a CABG gold standard proxy.