Current Evidence and Emerging Methods Regarding Validation of the AHRQ QIs

Select to access the PowerPoint® presentation (2.5 MB).

Slide 1

Current evidence and emerging methods regarding validation of the AHRQ QIs

Patrick S. Romano, MD MPH
UC Davis Center for Healthcare Policy and Research
AHRQ 2012 Annual Conference

Slide 2

Outline and learning objectives

• No disclosures (except support from AHRQ).
• To describe key domains of validity for health care quality measures.
• To illustrate these key domains of validity with examples based on the AHRQ Quality Indicators.
• To summarize methods and opportunities for AHRQ QI users who may wish to be involved in validation activities.

Slide 3

AHRQ Quality Indicators

Image: Colored circles depict the Quality Indicator modules:

• Inpatient QIs:
  • Mortality, Utilization, Volume.
• Pediatric / Neonatal QIs.
• Prevention QIs:
  • (Area Level) Avoidable Hospitalizations / Other Avoidable Conditions.
• Patient Safety Indicators:
  • Complications, Failure-to-rescue, Unexpected death.

Slide 4

Validity

• The degree to which a measure accurately represents the true state of the phenomenon of interest (i.e., "free of systematic error").
• Does this measure what it purports to measure (quality)?

Slide 5

Validation—A conceptual framework

• Face validity is the degree to which a measure "appears" to measure the phenomenon of interest
  • Content validity is a related concept, focused on whether the content of a measure adequately samples all relevant domains of the concept of interest (coverage).
• Criterion (concurrent) validity is the degree to which a measure generates data that agree with data from a better ("gold standard") approach.
• Predictive validity is the degree to which a measure successfully predicts an outcome of interest.
• Construct (convergent) validity is the degree to which a measure correlates with other measures, based on a construct that is grounded in prior literature or a sound conceptual framework.

Slide 6

Face validity: perspectives

• Developers.
• Expert panels.
• Users and stakeholders.

Slide 7

Face validity: NQF

• Health outcomes: "A rationale supports the relationship of the health outcome to at least one healthcare structure, process, intervention, or service."
• Intermediate outcomes: "Quantity, quality, and consistency of… evidence that the measured intermediate clinical outcome leads to a desired health outcome."
• Processes or structures: "Quantity, quality, and consistency of… evidence that the measured healthcare process leads to desired health outcomes in the target population with benefits that outweigh harms to patients."
• Patient experience: "Evidence that the measured aspects of care are those valued by patients and for which the patient is the best and/or only source (OR evidence that patient experience… is correlated with desired outcomes."

Slide 8

Face validity: NQF

• "The measure specifications are consistent with the evidence presented to support the focus of measurement… The measure is specified to capture the most inclusive target population indicated by the evidence, and exclusions are supported by the evidence."
• "Measure specifications include the target population (denominator) to whom the measure applies, identification of those from the target population who achieved the specific measure focus…, measurement time window, exclusions, risk adjustment/stratification, definitions, data source, code lists with descriptors, sampling, scoring/computation."

Slide 9

AHRQ expert panel process

• Intended to establish consensual (face) validity.
• Modified RAND/UCLA Appropriateness Method.
• Physicians of various specialties/subspecialties, nurses, other professionals (e.g., midwife, pharmacist).
• Potential PSIs were rated by 8 multispecialty panels; surgical PSIs were also rated by 3 surgical panels.
• All panelists rated all assigned indicators (1-9) on:
  • Overall usefulness.
  • Likelihood of being preventable.
  • Likelihood of being due to medical error.
  • Likelihood of being clearly charted in the medical record.
  • Susceptibility to bias due to case mix.

Slide 10

AHRQ expert panel process

• Pre-conference ratings and comments.
• Individual ratings returned to panelists with distribution of ratings and other panelists' comments.
• Telephone conference call(s) focusing on high-variability items and panelists' suggestions.
• Suggestions adopted only by consensus.
• Post-conference ratings and comments.
• Exclude indicators rated "Unclear," "Unclear-," or "Unacceptable":
  • Median score <7, OR.
  • At least 2 panelists rated the indicator in each of the extreme 3-point ranges.

Slide 11

Potential PSIs not adopted Only 18 "accepted" from original list of 48

"Experimental" PSIs

• Aspiration pneumonia.
• CABG after PTCA.
• Decubitus ulcer in high risk patients.
• In-hospital fractures possibly related to falls.
• Intraoperative nerve compression injuries.
• Malignant hyperthermia.
• Postoperative acute myocardial infarction.
• Postoperative iatrogenic complications—cardiac system.
• Postoperative iatrogenic complications—nervous system.
• Reopening of surgical site.
• Suture of laceration.
• Obstetric wound complications- cesarean.
• Obstetric wound complications- vaginal.
• Other obstetric complications.
• Postpartum urinary tract infection.
• Uterine rupture.

"Rejected" PSIs

• Dosage complications.
• Iatrogenic hypotension.
• Intestinal infection due to C. difficile.
• Postop iatrogenic complications—digestive complications.
• Postop iatrogenic complications—respiratory complications.
• Postop iatrogenic complications—urinary complications.
• Postop iatrogenic complications—vascular complications.
• Postoperative pneumonia.
• Unexpected LOS/Conditional LOS.
• Obstetric thrombosis or embolism.
• Puerperal infection.

Slide 12

Face validity varies by proposed purpose

Table 3. Overall Usefulness Ratings (Median Panel Scores)

Numbers represent median usefulness ratings, as measured on a 9-point scale (1 = Highly discourage use; 9 = Highly recommend use).
Overall usefulness ratings: major concern^-; some concern^*; majority support⁺; full support⁺⁺.
CHF indicates congestive heart failure; COPD, chronic obstructive pulmonary disease; NG, nominal group; UTI, urinary tract infection.

Slide 13

Face validity varies by method

*Numbers in parentheses are the numbers of instances in that cell where |Median (Delphi) - Median (NG)| > 1.
The median difference between groups was < 1 in all other combinations.
^†The support level can only deemed "General Support with some Concern" if statistical disagreement exists within the panel.

Slide 14

Criterion validity: perspectives

• Is the purported "gold standard" really a gold standard? How do we know?
• Administrative data perspective versus registry data perspective.
• Coding perspective versus clinical perspective (whose truth).
• Validity can change… dramatically.
• Finding false positives is easy, but what about false negatives?

Slide 15

Criterion validity: NQF

• "Empirical evidence of validity of BOTH data elements AND measure score within acceptable norms; AND.
• Identified threats to validity (lack of risk adjustment/stratification, multiple data types/methods, systematic missing or "incorrect" data) are empirically assessed and adequately addressed so that results are not biased."
• Could be satisifed by either criterion or construct testing.

Slide 16

Criterion validity:

Present on admission coding vs. chart review

Slide 17

Criterion validity of PSIs linked to NSQIP
Romano PS, et al. HSR 2009; 44(1):182
Cima RR, et al. Surg 2011; 150:943
Koch CG, et al. J Am Coll Surg 2012

VA=Veterans Affairs; C/M=Cleveland Clinic/Mayo Clinic Rochester
Sensitivity = TP/(TP+FN)—are all the real cases captured?
PPV = TP/(TP+FP)—are all the flagged cases real?
LR = Sensitivity/(100-Specificity)—how many times more likely is the event?

Slide 18

Criterion validity of PSIs linked to NSQIP
Romano PS, et al. HSR 2009; 44(1):182
Cima RR, et al. Surg 2011; 150:943
Koch CG, et al. J Am Coll Surg 2012

Slide 19

PPV of the PSIs based on chart review by nurse abstractors

Rosen, Med Care, 2012; Sadeghi, Am J Med Qual, 2010; Zrelak, J Healthc Qual, 2011;
White, Med Care, 2009; Utter, Ann Surg, 2009; Utter, J Am Coll Surg, 2010.

Slide 20

Limitations of chart abstraction for criterion validation

• Information needed to verify complication may not be available via chart review:
  • Complication was not properly evaluated or described by physicians; vicarious process.
  • Absence of evidence vs. evidence of absence.
• Time constraints limit abstractor's ability to assess some aspects of care (e.g., urinary catheter), may lead to premature termination.
• Much cheaper to look for FPs than FNs.
• Inter-hospital variation in physician documentation and nurse abstraction.
• Volunteer samples (except VA).

Slide 21

Comparing coding vs. clinical perspective for Postop DVT/PE

University HealthSystem Consortium cohort includes 505 flagged,randomly sampled surgical cases from 33 volunteer hospitals in 21 States; 450 cases were fully abstracted and submitted to UHC.

Slide 22

453.4 Acute venous embolism and thrombosis of deep vessels of lower extremity:

• 453.40 Acute venous embolism and thrombosis of unspecified deep vessels of lower extremity Deep vein thrombosis NOS.
• 453.41 Acute venous embolism and thrombosis of deep vessels of proximal lower extremity Femoral, Iliac, Popliteal, Thigh, Upper leg NOS.
• 453.42 Acute venous embolism and thrombosis of deep vessels of distal lower extremity Calf, Lower leg NOS, Peroneal, Tibial.

453.5 Chronic venous embolism and thrombosis of deep vessels of lower extremity
Excludes: personal history of venous thrombosis and embolism (V12.51)

• 453.50 Chronic venous embolism and thrombosis of unspecified deep vessels of lower extremity.
• 453.51 Chronic venous embolism and thrombosis of deep vessels of proximal lower extremity.
• 453.52 Chronic venous embolism and thrombosis of deep vessels of distal lower extremity.

453.6 Venous embolism and thrombosis of superficial vessels of lower extremity

453.7 Chronic venous embolism and thrombosis of other specified vessels
Excludes: personal history of venous thrombosis and embolism (V12.51)

• 453.71 Chronic venous embolism and thrombosis of superficial veins of upper extremity.
• 453.72 Chronic venous embolism and thrombosis of deep veins of upper extremity.
• 453.73 Chronic venous embolism and thrombosis of upper extremity, unspecified.
• 453.74 Chronic venous embolism and thrombosis of axillary veins.
• 453.75 Chronic venous embolism and thrombosis of subclavian veins.
• 453.76 Chronic venous embolism and thrombosis of internal jugular veins.
• 453.77 Chronic venous embolism and thrombosis of other thoracic veins.
• 453.79 Chronic venous embolism and thrombosis of other specified veins.

453.8 Acute venous embolism and thrombosis of other specified veins
Excludes: cerebral, coronary, intracranial sinus, nonpyogenic, mesenteric, portal, precerebral, pulmonary

• 453.81 Acute venous embolism and thrombosis of superficial veins of upper extremity.
• 453.82 Acute venous embolism and thrombosis of deep veins of upper extremity.
• 453.83 Acute venous embolism and thrombosis of upper extremity, unspecified.
• 453.84 Acute venous embolism and thrombosis of axillary veins.
• 453.85 Acute venous embolism and thrombosis of subclavian veins.
• 453.86 Acute venous embolism and thrombosis of internal jugular veins.
• 453.87 Acute venous embolism and thrombosis of other thoracic veins.
• 453.89 Acute venous embolism and thrombosis of other specified veins.

453.9 Of unspecified site (embolism of vein, thrombosis (vein))

Slide 23

Methods to reassess criterion validity of PSI 12

• Two parallel studies were conducted to update previous PPV estimates for PSI 12 and to identify actionable opportunities to improve care:
  • 7 volunteer hospitals recruited through AHRQ QI listserve, including flagged cases only.
  • 15 academic health systems recruited through UHC, including both flagged and unflagged cases with TKA surgery.
• AHRQ PSI 12 Version 4.1 software was applied to eligible cases from participating hospitals, using "present on admission" (POA) flags.
  • Hospital's own data (AHRQ) or Clinical Database (UHC).

Slide 24

Methods to reassess criterion validity of PSI 12

• Flagged cases were reviewed by trained QI nurses at each hospital, using detailed chart abstraction tool and guidelines.
• Detailed review of discrepant cases to identify possible reasons for the discrepancy:
  • Present on admission (note one hospital did not apply denominator exclusions).
  • Location of thrombosis (upper extremity and superficial thromboses are clinical FPs).
  • Chronic vs. acute embolism (based on radiographic criteria).
• Records from volunteer hospitals in AHRQ study were sampled in sequential reverse order from 6/30/2010 back to 10/1/2009, up to N=30.

Slide 25

Findings: 7 volunteer hospitals

• From a total of 171 audited charts, 15 cases were excluded post hoc (because hospital did not properly apply POA).
• 30 cases were False Positive:
  • 15 cases were POA.
  • 8 cases with upper extremity VT.
  • 1 case with SVC (central VT).
  • 3 cases with superficial VT.
  • 3 cases were chronic.
• Overall PPV = 81%.

Slide 26

Findings: AMCs with TKA patients PSI-12 flagged cases

• Positive Predictive Value
     = TP / (TP + FP)
     = 125 / (125 + 1)
     = 0.992

Image: A box contains the following text: "126 VTE flagged by PSI (+4 Readmission." A line labeled "Chart Abstraction" leads down from this box and divides into 2 lines, each connecting to one of two boxes. The first box contains the text "125 cases True Positive postop lower ext DVT or PE" and the second "1 case clinical False Positive (superficial) saphenous Vein."

Slide 27

Findings: AMCs with TKA patients PSI-12 unflagged controls

• Negative Predictive Value
     = TN / (FN + TN)
     = 458 / (458+3) = 0.993

Image: A box contains the following text: "463 Not flagged as VTE by PSI 12." A line labeled "Chart Abstraction" leads down from this box and divides into 2 lines, each connecting to one of two boxes. The first box contains the text "5 cases had VTE per UHC abstract" and the second "458 cases had no VTE (TN)." A split line leads down from the "5 cases" box to two boxes below it; the first contains the text "3 cases False Negative" and the second "2 cases superficial or upper extremity thrombosis."

Slide 28

Estimating sensitivity: Looking for a needle in a haystack

• Retrospective cross-sectional study:
  • 27 hospitals from 11 States.
  • 2006-2009 PSI-negative hospitalizations.
• Stratified sample:
  • By hospital, risk of being falsely negative.
  • Oversampled cases at risk.
• Medical records abstracted locally:
  • Trained staff, standard instrument.
• Analysis with survey statistical methods:
  • "Verification-biased sampling" approach, used model-based weights in analysis.
  • Incorporated previous estimates of PPV.

Slide 29

Suspicious records:
Could they be false negatives?

Slide 30

Sampling scheme for estimating PSI sensitivity

Slide 31

Estimated PSI sensitivity

Slide 32

Postop Hemorrhage/Hematoma

Slide 33

Predictive validity: perspectives

• Predicting what?
  • Mortality.
  • Length of stay, charges.
  • Readmission.
• Time window for prediction?
• Implied gold standard.

Slide 34

Predictive validity established:
Impact of preventing a PSI on mortality, LOS, charges
NIS 2000 analysis by Zhan & Miller, JAMA 2003;290:1868-74

Excess mortality, LOS, and charges computed from mean values for PSI cases and matched controls.

Slide 35

Predictive validity established:
Impact of preventing a PSI on mortality, LOS, charges VA PTF 2001 analysis by Rivard et al., Med Care Res Rev 65(1):67-87

Excess mortality, LOS, and charges computed from mean values for PSI cases and matched controls.

Slide 36

Predictive validity questionable based on NIS/VA
Zhan & Miller, JAMA 2003;290:1868-74
Rosen et al., Med Care 2005;43:873-84

* All differences NS for transfusion reaction and complications of anesthesia in VA/PTF.
† Mortality difference NS for foreign body in VA/PTF.

Slide 37

Comparison with England

Raleigh VS, Cooper J, Bremner SA, Scobie S, BMJ 2008, 337; a1702

Slide 38

Construct validity: perspectives

• Is the construct sound?
• Correlation with process measures:
  • Do we have the right process measures?
• Correlation with structural measures:
  • Do we have the right structural measures?
• Correlation with other outcome domains.

Slide 39

Case control study of PSI 12

• Cases (up to 20):
  • Uni-TKA or Bilat-TKA.
  • Oct 2008 to Mar 2010.
  • >40 yrs.
  • PSI-12 code for VTE within 90 days.

• Controls (up to 40):
  • Uni-TKA or Bilat-TKA.
  • Oct 2008 to Mar 2010.
  • >40 yrs.
  • NO PSI-12 code for VTE within 90 days.

• No TKA or THA within 90 days prior.
• No VTE as principal ICD-9-CM diagnosis.
• No VTE as POA.
• No pregnancy, childbirth, or puerperium.

Slide 40

Analysis of case control data

• Classified FDA-approved pharmacologic prophylaxis as receipt of the recommended dose at the recommended starting time (per package insert) before or after surgery, and continued until at least the day of discharge.
• Patients who were diagnosed with VTE on the day of surgery or the day after surgery were not included in the case control analysis.
• Other risk factors assessed included age, obesity (BMI), type of TKA, race/ethnicity, date of ambulation, personal or family history of VTE, and comorbid conditions.
• Analysis adjusted for conditional stratified sampling of controls without VTE.

Slide 41

Multivariable analyses of process factors

Multivariate adjusted odds ratios and 95% confidence intervals

• Outcome: Any VTE event diagnosed Day 2 of surgery or later.
• Excluded one hospital that screened TKA patients routinely for VTE.

Slide 42

Implications of recent validation studies

• Generalizability to nonparticipating hospitals remains unclear.
• Limited supervision/oversight of local abstractors who may have COI.
• ICD-9-CM changes and associated coder training can substantially improve criterion validity of PSIs, in the right circumstances.
• Actionable opportunities to improve care may persist despite 100% compliance with TJC process measures, but often we don't know what processes to use for construct validation.
• Estimating sensitivity or FN rate is still hard.

Slide 43

Future directions in validation

• Encourage ongoing criterion validation work using previously developed tools or new learning collaboratives.
• Continue to pursue opportunities to link registry or EHR data, especially laboratory, imaging, vital signs, etc.
• Encourage case control and intervention studies when we know what processes to measure or change.

Slide 44

Improving indicator performance

• Change indicator specifications to capture false negatives or exclude false positives.
• Change ICD-9-CM codes, coding guidelines, or Coding Clinic advice.
• Promote universal adoption of POA flag.
• Work with hospitals to improve and stan-dardize clinical documentation and coding.
• Everything will change in ICD-10-CM/PCS (10/1/2014).

Slide 45

Survey of AMC coders about postop respiratory failure

• 56 coders surved at 20 volunteer AMCs.
• When a hospital's coders reported being more likely to "query physicians when a patient appears to have experienced PRF, but the physician did not document a specific diagnosis" (by 1 level on a 5-level Likert), the hospital:
  • Had 1.8 (1.2-2.8) times higher use of 518.5.
  • Had 1.0 (0.6-1.4) times higher use of 518.8x.

Slide 46

http://qualityindicators.ahrq.gov/ValidationPilot.aspx

Image: A screenshot shows the AHRQ Quality Indicators Validation Pilot page.

Slide 47

Acknowledgments and references

• Participating hospitals and systems: Advocate Health Care, Barnes Jewish Hospital, Harborview Med Ctr, Nebraska Medical Ctr, Stanford Hospital, UC Davis Medical Ctr, Vassar Brothers Med Ctr.
• AHRQ Quality Indicators project team: Mamatha Pancholi, John Bott.
• Battelle validation support: Jeffrey Geppert, Jaime Liesmann.
• UC Davis: Pat Zrelak, Banafsheh Sadeghi, Richard White, Garth Utter.
• Sadeghi B, et al. Cases of iatrogenic pneumothorax can be identified from ICD-9-CM coded data. Am J Med Qual 2010; 25(3):211-7.
• White RH, et al. How valid is the ICD-9-CM based AHRQ Patient Safety Indicator for postoperative venous thromboembolism? Med Care 2009; 47(12):1237-43.
• White RH, et al. Evaluation of the predictive value of ICD-9-CM coded adminstrative data for venous thromboembolism in the United States. Thromb Res 2010; 126(1):61-7.
• Sadeghi B, et al. Mechanical and suboptimal pharmacologic prophylaxis and delayed mobilization but not morbid obesity are associated with venous thromboembolism after total knee arthroplasty: A case-control study. J Hosp Med 2012; in press.

Slide 48

Questions?