How Good is Good Enough? A Case Study with Present on Admission (Text Version)

Slide presentation from the AHRQ 2010 conference.

On September 28, 2010, Jeff Geppert made this presentation at the 2010 Annual Conference. Select to access the PowerPoint® presentation (940 KB). Free PowerPoint® Viewer (Plugin Software Help).


Slide 1

 How Good is Good Enough? A Case Study with Present on Admission

How good is good enough? A case study with Present on Admission

AHRQ Annual 2010 Annual Conference
September , 2010
Session #45
Jeffrey Geppert, PMP, EdM, JD
Battelle Memorial Institute

Slide 2

 Acknowledgements

Acknowledgements

  • Support for Quality Indicators II (290-04-0020):
    • Mamatha Pancholi, AHRQ QI Project Officer
  • HCUP State Inpatient Databases (SID). Healthcare Cost and Utilization Project (HCUP). 2004-2008. Agency for Healthcare Research and Quality, Rockville, MD. http://www.hcup-us.ahrq.gov/sidoverview.jsp

Slide 3

Why Measurement?  

Why Measurement?

  • What is the purpose of measurement?
    • Reduce uncertainty in decisionmaking
  • What is the decision?
    • Consumers: what provider to select
    • Providers: where to allocate effort
  • What is the uncertainty?
    • Consumers: would have selected a different provider
    • Providers: would have allocated effort differently

Slide 4

 Measuring Uncertainty

Measuring Uncertainty

  • Expected Opportunity Loss (EOL):
    • Amount of loss X Probability of Loss
    • Loss relative to the best available alternative (opportunity)
  • Expected Value of Information (EVI):
    • EOLAfter measurement - EOLBefore measurement
    • EVI may be positive or negative
  • Risk aversion:
    • If EVI = 0, then the decision is about risk

Slide 5

 Measuring Uncertainty

Measuring Uncertainty

  • Measure Model:
    • Y = alpha + X*B
  • Y - the outcome of interest
    • Numerator, denominator, exclusions
  • X*B - patient characteristics
    • Moderators and mediators
  • Alpha - provider quality
    • Prior ability (related outcomes, related procedures, structure, context, past performance)
    • Current effort (signal)

Slide 6

Iatrogenic Pneumothorax  

Iatrogenic Pneumothorax

Sample of a Quality Indicator (QI)

Source: AHRQ PSI Technical Specifications, PSI #6 (Version 4.1b).

Slide 7

POA Overview: Approach  

POA Overview: Approach

  • Two sets of algorithms needed to incorporate POA information:
    1. Develop response variables and comorbidity factor covariates in the presence of POA data.
      • Less measurement error thereby more accurate and based on fewer assumptions.
    2. Develop response variables and comorbidity factor covariates in the absence of POA data.
      • Use observed POA data to estimate probability of POA for response and comorbidity factors for patients that do not have POA data.
      • Provide hospital with risk-adjusted rate that would be "most likely" had they collected POA data.
  • Present on Admission White Paper:

Slide 8

 Iatrogenic Pneumothorax

Iatrogenic Pneumothorax

  • Case study:
    • Select two hospitals (A and B) with POA data (median denominator and rate).
    • Randomly use POA data for X% of discharges and predict the missing
      • None (no POA data)
      • 10%—90%
      • All (full POA data)—"gold standard"
    • Estimate EOL based on events, deaths, inpatient days and charges (NIS 2000 analysis by Zhan & Miller, JAMA 2003;290:1868-74).

Slide 9

Expected Opportunity Loss  

Expected Opportunity Loss

Image: Line graph displays the following information and hospital rates:

SID States:
Events: 8,306
Deaths: 581
Days: 36,547
Charges: $143,697,389

Hospital A:
Events: 3
Deaths: 0
Days: 12
Charges: $48,561

Hospital B:
Events: 4
Deaths: 0
Days: 16
Charges: $63,659

Slide 10

 EVI from Measurement

EVI from Measurement

Image: Line graph displays the following information and hospital rates:

Hospital A (Prior):
Events:  3
Deaths:  0
Days:  12
Charges:   $48,561

Hospital A (Post): ;
Events:  5
Deaths:  0
Days:  24
Charges:  $94,370

Slide 11

 EVI from Measurement

EVI from Measurement

Image: Line graph displays the following information and hospital rates:

Hospital B (Prior):
Events:  4
Deaths:  0
Days:  16
Charges:  $63,659

Hospital B (Post):
Events:  5
Deaths:  0
Days:  22
Charges:  $87,472

Slide 12

 EVI from Measurement

EVI from Measurement

Image: Line graph displays the following information and hospital rates:

Hospital A (Prior):
Events: 5
Deaths: 0
Days: 24
Charges: $94,370

Hospital A (Post):
Events: 6
Deaths: 0
Days: 27
Charges: $107,226
 

Slide 13

 EVI from Measurement

EVI from Measurement

Image: Line graph displays the following information and hospital rates:

Hospital B (Prior):
Events:  5
Deaths:  0
Days:  22
Charges:  $87,472

Hospital B (Post):
Events:  6
Deaths:  0
Days:  26
Charges:  $100,306

Slide 14

EVI from Measurement  

EVI from Measurement

Image: A tables displays the following information:

Charges—Total for Hospital A and BPercent
SampleEOLEVI
Prior$112,221  
No POA data$181,843$69,62273.05%
10%$180,688-$1,155-1.21%
20$171,926-$8,762-9.19%
30$172,108$1820.19%
40$188,143$16,03516.82%
50$190,702$2,5592.68%
60$192,917$2,2152.32%
70$205,541$12,62413.25%
80$198,562-$6,980-7.32%
90$212,499$13,93714.62%
Full POA Data$207,532-$4,967-5.21%

 

Slide 15

Using Imperfect Measures  

Using Imperfect Measures

  • The use of any measure (Qm) that is positively correlated with the true measure (Qt) results in overall improvement in outcomes.
  • This is not true only if:
    • Difference between Qt and Qm
    • Variation in Qt - Qm among providers
    • Negative correlation between Qt and Qm

Slide 16

 Using Imperfect Measures

Using Imperfect Measures

Image: A graph shows infections due to Medical Care (PSI#7) for None (No POA Data) / All (Full POA Data). Most incidences (represented by dots) appear within the range of 0.03 None / 0.03 Full.

Slide 17

 Using Imperfect Measures

Using Imperfect Measures

The table shows actual reduction in adverse events in moving patients from worse to best providers based on an imperfect measure.

POA SampleNumber of Adverse EventsPercent
Best 20%Worse 20%Difference
None2835,1694,88691.7%
10%2755,2134,93892.7%
20%2675,2454,97793.4%
30%2705,2875,01794.2%
40%2595,3235,06395.1%
50%2615,3745,11496.0%
60%2445,3775,13396.4%
70%2405,4275,18797.4%
80%2295,4695,24198.4%
90%2165,5165,30099.5%
All2155,5425,327100.0%

Slide 18

Risk Aversion  

Risk Aversion

  • A consumer may rationally select a provider with worse measured quality:
    • If the consumer is risk averse and prefers a more certain outcome over the greater risk of a worse outcome.
    • If the consumer has external information about the provider that reduces uncertainty.

Slide 19

Risk Aversion  

Risk Aversion

  • A consumer may rationally select a provider with worse measured quality:
    • If the consumer is risk averse and prefers a more certain outcome over the greater risk of a worse outcome.
    • If the consumer has other information about the provider that reduces uncertainty.

Slide 20

Risk Aversion  

Risk Aversion

Image: Line graph shows Iatrogenic Pneumothorax (PSI #06) rates for Hospital B EOL vs. Hospital A EOL. Hospital A has a sharp spike, while Hospital B is an even bell curve.

Slide 21

 Conclusions

Conclusions

  • Use expected value of information (EVI) to quantify the benefit of adding enhanced data, collecting data over time.
  • Largest marginal gain in EVI may come from:
    • Initial measurement
    • Systemic relationships
  • Smallest marginal gain in EVI may come
    • Subsequent measurement
    • Provider specific measurement

Slide 22

 Conclusions

Conclusions

  • Target data collection to the measures, consumers and providers with highest marginal EVI.
  • Give consumers information about uncertainty to allow them to incorporate preferences about risk and external information.

Slide 23

Questions?  

Questions?

Image: An illustration of a microphone is shown.

Current as of December 2010
Internet Citation: How Good is Good Enough? A Case Study with Present on Admission (Text Version). December 2010. Agency for Healthcare Research and Quality, Rockville, MD. http://www.ahrq.gov/news/events/conference/2010/geppert/index.html