Overview of the Decision Guide: A Public Reporting Resource for CVEs (Text Version)

Slide Presentation from the AHRQ 2010 Annual Conference

Slide presentation from the AHRQ 2010 conference.

On September 27, 2010, Patrick S. Romano made this presentation at the 2010 Annual Conference. Select to access the PowerPoint® presentation (13.5 MB). Free PowerPoint® Viewer (Plugin Software Help).


Slide 1

Overview of the Decision Guide: A Public Reporting Resource for CVEs

Overview of the Decision Guide: A Public Reporting Resource for CVEs

Patrick S. Romano, MD MPH
Professor of Medicine and Pediatrics
University of California, Davis
June 23, 2010

Slide 2

Available from AHRQ

Available from AHRQ

Image: The cover of "Selecting Quality and Resource Use Measures: A Decision Guide for Community Quality Collaboratives" is shown.

Authors:
Patrick S. Romano, MD MPH
Dominique Ritley, MPH
David Chin, PhD student

With the help of many CVE representatives

Slide 3

Decision Guide Checklist

Decision Guide Checklist

  • Goals:
    • To provide collaboratives with frameworks and tools for selecting measures of quality and resource use (QI 101).
    • To highlight key considerations in selecting measures of quality and resource use, based on a collaborative's evolutionary stage.
  • Guide divides answers to 26 questions into five sections:
    • Introduction to Data
    • Introduction to Measures of Quality
    • Introduction to Resource Use/Efficiency Measures
    • Selecting Quality and Resource Use Measures
    • Interpreting Quality and Resource Use Measures
  • We focus today on just 6 of those questions

Slide 4

Design of public reporting programs starts with a candid self-assessment (Q25)

Design of public reporting programs starts with a candid self-assessment (Q25)

Image: Venn diagram showing the intersection of the following three items:

Resources:

  • Financial resources
  • Analytic capabilities

Environment:

  • Available data
  • Potential partners
  • Stakeholder engagement

Goals:

  • P4P
  • Public reporting
  • Performance improvement

Slide 5

Quality measurement and reporting are important issues across many industries

Quality measurement and reporting are important issues across many industries

Image: An egg farm is shown.

Slide 6

In search of a balanced set of quality measures (Q20): Iowa's inspection of Wright County Egg

In search of a balanced set of quality measures (Q20): Iowa's inspection of Wright County Egg

Image: An Iowa Department of Inspections & Appeals Egg Handler Inspection sheet is shown.

Slide 7

In search of a balanced set of quality measures (Q20): USDA

In search of a balanced set of quality measures (Q20): USDA "grader" inspection of shell egg plant

Image: A grader checklist is shown.

Slide 8

What did the USDA miss?

What did the USDA miss?

Image: A building with a wall broken is shown.

Slide 9

What else did the USDA miss?

What else did the USDA miss?

Image: A building wall is shown.

Slide 10

IOM Domains of Quality

IOM Domains of Quality

Effectiveness

  • Providing services based on scientific knowledge (avoiding overuse of inappropriate care, underuse of appropriate care)

Patient Centeredness

  • Care that is respectful of and responsive to patient preferences, needs, and values

Timeliness

  • Reducing wait times and sometimes harmful delays

Safety

  • Avoiding injuries to patients from care that is intended to help

Efficiency

  • Avoiding waste of equipment, supplies, ideas, and energy

Equity

  • Care does not vary in quality because of personal characteristics

Slide 11

In search of a balanced set of quality measures (Q20): Institute of Medicine, 2010

In search of a balanced set of quality measures (Q20): Institute of Medicine, 2010

Image: IOM's quality measures matrix is shown:

Crosscutting DimensionsComponents of Quality CareType of Care
Preventive CareAcute TreatmentChronic Condition Management
EQUITYVALUEEffectiveness   
Safety   
Timeliness   
Patient/Family-centeredness   
Access   
Efficiency   
Care Cooridination
Health Systems Infrastructure Capabilities

 

Slide 12

What types of measures should be collected and reported (HOW)?

What types of measures should be collected and reported (HOW)?

Two images of old farm buildings are shown.

Slide 13

Look inside the structure.

Look inside the structure.

Image: The inside of a farm building is shown.

Slide 14

Look for the outcomes . . .

Look for the outcomes.

Image: A dead chicken is shown.

Slide 15

Classifying types of measures

Classifying types of measures
Donabedian, 2003

  • Structure: conditions under which care is provided:
    • Material resources (facilities, equipment)
    • Human resources (ratios, qualifications, experience)
    • Organizational characteristics (size, volume, IT system)
  • Process: activities that constitute health care (adherence to guidelines):
    • Screening and diagnosis
    • Treatment and rehabilitation
    • Education and prevention
  • Outcome: changes attributable to health care:
    • Mortality, morbidity (complications, readmissions)
    • Knowledge, attitudes, and behaviors
    • Patient experiences/satisfaction

Slide 16

Framework for selecting measures (Q20)

Framework for selecting measures (Q20)

IOM DomainsStructureProcessOutcome
EffectiveCardiac nurse staffing, nursing skill mix (RN/total)Use of ACE inhibitor or ARB for patients with systolic HF30-day readmissions (or mortality) for heart failure
Patient CenteredUse of survey data to track patient-centered careHow often did you get an appointment as soon as you thought you needed?Overall rating of care
TimelyPhysician organization policy on scheduling urgent appointmentsReceived beta blocker at discharge and for 6 months after AMIPotentially avoidable hospitalizations for angina (without procedure)
SafeComputerized physician order entry with medication error detectionUse of prophylaxis for venous thromboembolism in appropriate patientsPostoperative deep vein thrombosis or pulmonary embolism
EfficientAvailability of rapid antigen testing for sore throatInappropriate use of antibiotics for sore throatDollars per episode of sore throat
EquitableAvailability of adequate interpreting servicesUse of interpreting services when appropriateDisparity in any other outcome according to primary language

Slide 17

What data sources should we use for quality measurement (Q8/Q9)?

What data sources should we use for quality measurement (Q8/Q9)?

Image: The inside of a chicken farm is shown.

Slide 18

Gold standard = Direct observation

Gold standard = Direct observation

Video recording to identify errors in pediatric trauma resuscitation:

Mean of 5.9 errors per resuscitation, 93% agreement between reviewers.

Mean of 2.2 errors in each seriously injured child, with 20% capture on medical records.

Image: Emergency medical staff are shown treating a child.

Oakley, E. et al. Pediatrics 2006;117:658-664

Slide 19

Data sources for quality measurement (Q8/Q9): Review documents and collect specimens

Data sources for quality measurement (Q8/Q9): Review documents and collect specimens

Image: The Hillandale Farms FDA inspection results form is shown.

Slide 20

Data sources for quality measurement (Q8/Q9): Review documents and collect specimens

Data sources for quality measurement (Q8/Q9): Review documents and collect specimens

  • Observe/record encounters (real or simulated)—$$$$
  • Ask patients (CAHPS® surveys):
    • Satisfaction and experiences
    • Morbidity (complications, functional status, quality-of-life)
    • Knowledge, attitudes, and behaviors
  • Ask health care providers:
    • Rate others' reputation (US News)
    • Describe material and human resources (Leapfrog survey)
    • Describe safety-related practices (Leapfrog survey)
  • Review claims/administrative data sets:
    • Mortality, morbidity (deaths, complications, readmissions)
    • Adherence to guidelines (HEDIS, PQRI)
  • Review/abstract medical records (including registries):
    • Mortality, morbidity (deaths, complications, readmissions)
    • Adherence to guidelines (HEDIS, PQRI)

Slide 21

Tremendous growth in NQF-endorsed physician measures (Q8)

Tremendous growth in NQF-endorsed physician measures (Q8)

  • National Voluntary Consensus Standards: Ambulatory Care:
    • 101 measures across 10 priority areas: asthma and respiratory illness; bone and joint conditions; diabetes; heart disease; hypertension; medication management; mental health; obesity; prenatal care; and prevention (including screening).
    • 7 instruments for patient experience.
    • 26 measures of specialty clinician care: bone and joint conditions, eye care, geriatrics, emergency care, skin care (melanoma).
  • "Additional Performance Measures 2008":
    • 67 measures for cancer care, infectious disease, perioperative care, and licensed independent practitioners.
  • "Using Clinically Enriched Administrative Data":
    • 70 measures across most original priority areas plus child health, chronic kidney disease, gastroesophageal reflux, gynecology, hepatitis, HIV/AIDS, and migraine.

Slide 22

Hospital quality measures are now available

Hospital quality measures are now available "off the shelf" (Q9)

  • CMS Medicare's HospitalCompare
  • The Joint Commission's QualityCheck
  • Commonwealth's "Why not the best?"
  • Leapfrog's voluntary survey on CPOE, ICU staffing, evidence-based hospital referral, and NQF Safe Practices implementation
  • Specialized state/regional programs (HAIs, AHRQ Quality Indicators, myhealthfinder.com, registries)
  • Other (HealthGrades, USNews, etc.)

Slide 23

Image: A screen shot of the

Image: A screen shot of the "Hospital Compare" Web site is shown.

Slide 24

Image: A screen shot of the

Image: A screen shot of the "The Joint Commission" Web site is shown.

Slide 25

Image: A screen shot of the

Image: A screen shot of the "The Commonwealth Fund" Web site is shown.

Slide 26

Image: A screen shot of the

Image: A screen shot of the "The Leap Frog Group" Web site is shown.

Slide 27

Image: A screen shot of the myHealtheFinder.com Web site is shown.

Image: A screen shot of the myHealtheFinder.com Web site is shown.

Slide 28

Hybrid data (Q2)

Hybrid data (Q2)

  • Bring together administrative (electronic claims) and medical record data to build on the strengths of each while compensating for weaknesses:
    • Increase the number of data elements for outcome ascertainment or risk adjustment.
    • Reduce the number of records that must be reviewed manually.
    • Reduce the time required to review each record.
  • At the physician level, use claims to identify patients with a relevant diagnosis or problem, and use medical records to identify specific clinical findings or treatments.
  • At the hospital level, combine ICD-9-CM coded administrative data with laboratory or other clinical data to enhance the performance of risk-adjustment models and to reduce bias in estimates of hospital performance.
    • FL, MN, VA pilot projects
    • Regulatory requirements in CA and PA (Michael Pine's work)

Slide 29

How to select measures for reporting? National Quality Forum criteria (Q22)

How to select measures for reporting? National Quality Forum criteria (Q22)

  • Importance:
    • Leverage point for improving quality.
    • Variation in quality of care or suboptimal performance (overall).
  • Scientific acceptability:
    • Well-defined and precisely specified... reliable.
    • Valid, accurately representing the concept being evaluated.
    • Discriminating between real differences in provider performance.
    • Adaptable to patient preferences and variety of settings.
    • Adequate and specified risk-adjustment strategy.
  • Usability:
    • Can be used for decision making and implementing change.
    • Differences should be meaningful practically and clinically.
  • Feasibility:
    • Benefit should be evaluated against burden.
    • Confidentiality concerns should be addressed.
    • Audit strategy should be available.

Slide 30

Who needs composite scores? (Q10)

Who needs composite scores? (Q10)

Image: the New York City Health Department new restaurant inspection grades notice is shown.

Slide 31

Trends in scores over time in LA County

Trends in scores over time in LA County

Restaurant
Grades
A (90-100)B (80-89)C (70-79)< 70Total
YearCount% of totalCount% of totalCount% of totalCount% of total 
1997-98
(6 months)
712339.9%551230.9%313917.6%209011.7%17864
1998-993579571.4%1156323.0%23354.7%4720.9%50165
1999-004120976.1%1095020.2%17153.2%2790.5%54153
2000-014426078.3%1071919.0%13582.4%1790.3%56516
2001-024978281.9%972816.0%11281.9%1270.2%60765
2002-034385975.9%1212821.0%16082.8%2060.4%57801
2003-044230678.1%1030719.0%14102.6%1680.3%54191
2004-054896781.4%993416.5%11331.9%1220.2%60156
2005-064526383.1%827315.2%8441.6%680.1%54448
2006-074471582.5%839315.5%9791.8%1100.2%54197

"A" grade was associated with 5.7% increase in revenue.
"B" grade was associated with 0.7% increase in revenue.
"C" grade was associated with 1% decrease in revenue.
Two studies showed 20% and 13% decreases in hospitalization for food-borne illness in Los Angeles County.

Source: CHOICES 2005; 20(2):97-102 (American Agricultural Economics Association)

Slide 32

Why composite measures for CVEs? (Q10)

Why composite measures for CVEs? (Q10)
(aka summary measures, roll-up measures)

  • AHRQ: "condensing multiple quality measures into a single piece of information":
    • Reduces cognitive burden for consumers, providing clearer "signal" and reducing the danger of "cognitive shortcuts".
    • Enhances reliability or ability to discriminate between higher-quality and lower-quality providers.
    • Fits well conceptually with pay-for-performance programs, which explicitly translate multiple quality measures to dollars, allowing providers to prioritize their own efforts.
  • But remember two potential concerns:
    • Difficulty achieving consensus on composite construction and scoring, perhaps due to lack of professional consensus.
    • Loss of important information if the composite combines unrelated metrics in a manner that washes out meaningful differences on individual indicators.

Slide 33

Two conceptual approaches (Q10)

Two conceptual approaches (Q10)

  • Psychometric or reflective perspective—an underlying, unmeasured factor ("quality") is the cause of what we observe; the observed data reflect this unmeasured factor:
    • Requires a correlation among the measures included in the composite, because different measures can only reflect the same latent factor (i.e., quality) if they are correlated with each other.
  • Clinometric or formative perspective—focus on guiding decision-making to optimize welfare instead of measuring an unobserved, latent factor:
    • Use clinical judgment rather than empirical analysis to select component measures.
    • Formed from or defined by specific indicators, so no correlation among component measures is required

Slide 34

Recommended approach for creating a composite score (Q10)

Recommended approach for creating a composite score (Q10)

  • Identify the purpose... and delineate the quality construct...
  • Select the individual measures and/or sub-composite measures to be combined... (may require standardization).
  • Ensure that the weighting and scoring of the components supports the goal that is articulated for the measure.
  • Combine the component scores, using a specified scoring method...
  • Testing for reliability and validity.

Slide 35

Restaurateurs' reaction to the NYC composite score

Restaurateurs' reaction to the NYC composite score

  • Purpose of composite is invalid:
    • "There is no evidence that letter grading increases the identification of risk factors for foodborne illnesses".
    • "...sophomoric, and punitive and demeaning to restaurateurs, as if they are schoolchildren who must be graded".
  • Composite is poorly constructed:
    • "How can you possibly justify including non-food safety related items? A leaky faucet, a (missing) sign, a light bulb not covered, an uncovered waste receptacle... mislead the public when it sees a B or C in the window into thinking that the food here is not safe, when the difference between an A or B grade may have nothing to do with food safety."
  • Composite is unreliable:
    • "...inconsistency from one inspector to another"
  • Composite is invalid due to unmeasured risk:
    • "Most of their buildings (in LA) are not 200 years old, and most of them are not next to empty lots with hundreds of rats. It would be nice if the city would clean up those lots."
  • Composite will have unintended consequences (gaming):
    • "...a scarlet letter that will keep people from eating out".
    • "...encourage bribery and corruption. I remember when payoffs were so commonplace that the FBI had to come in and arrest the inspectors.".
    • "...could turn back the clock on New York as the food capital of the world."

Source: New York Times, multiple articles, February-August 2010

Slide 36

New York City's

New York City's "dirtiest establishment" (now closed): Was it rats or roaches?

Image: A grocery store is shown.

Slide 37

Scoring composite measures (Q10)

Scoring composite measures (Q10)

Scoring MethodDefinitionExampleAdopter
All-or-noneThe percentage of patients for whom all indicators triggered by that patient are met.“Appropriate Care Measure” for 4 conditions (heart attack, heart failure, pneumonia, and SCIP).PHCQA Progress and Performance Report of Hospital Quality
70% StandardAll-or-none with less strict criteria (e.g., 70% not 100%).None to our knowledge 
Overall Percentage (Opportunity weighting)Percentage of all care events that were properly delivered, where each opportunity to “do the right thing” counts equally.149 hypertensive patients triggered 26 hypertension indicators 828 times. Required care was given 576 times yielding 69.9% (576/828).CMS P4P Premier Hospital Quality Incentive Demonstration
Indicator Average (Equal indicator/ event weighting)Scores are averaged across all indicators to represent the mean adherence rate.Hospital quality of care for acute myocardial infarction, congestive heart failure and pneumonia.Hospital Quality Alliance (HQA)
Patient Average (Equal patient weighting)The percentage of indicators successfully met is computed for each patient, and then averaged at the patient level.None to our knowledge 
Expert Opinion (Evidence-based)Indicators are weighted based on evidence of impact on population health and/or effort required to achieve.General Medical Services contract pays physicians more for achieving performance targets that require more time and other resources.UK National Health Service

Slide 38

Combining quality and resource use measures to highlight high-value care (Q26)

Combining quality and resource use measures to highlight high-value care (Q26)

Image: A chart displaying AMI Composite Score vs. Severity Adjusted Charge is shown.

Slide 39

Conclusion: Use available tools from AHRQ!

Conclusion: Use available tools from AHRQ!

Slide 40

AHRQ Decision Guide on Selecting Quality and Resource Use Measures

AHRQ Decision Guide on Selecting Quality and Resource Use Measures

  • Addresses 26 questions community leaders and stakeholders frequently ask about quality and resource use measurement.
  • Community quality collaborative leaders informed development.
  • Access on-line at:
    http://www.ahrq.gov/qual/perfmeasguide or to order hard copies free of charge:
Current as of December 2010
Internet Citation: Overview of the Decision Guide: A Public Reporting Resource for CVEs (Text Version): Slide Presentation from the AHRQ 2010 Annual Conference. December 2010. Agency for Healthcare Research and Quality, Rockville, MD. http://www.ahrq.gov/news/events/conference/2010/romano/index.html