Evaluation of the Use of AHRQ and Other Quality Indicators
Chapter 1. Introduction
Public recognition of health care quality issues has spiked remarkably in the past ten years, driven by a series of high-profile reports from the Institute of Medicine (IOM), the RAND Corporation, and others.6-9 These reports showed, among other facts, that preventable medical errors in hospitals result in as many as 98,000 deaths per year; preventable medication errors occur at least 1.5 million times per year; and on average, only 55% percent of recommended care is delivered.
In response, a variety of stakeholders from across the spectrum of health care delivery—including providers, professional and hospital associations, accreditation organizations, employers and business groups, insurance companies, and state and federal governments—have focused on monitoring and improving the quality of care. A primary goal has been to avoid unnecessary deaths and poor health—at a minimum, to "first do no harm" by limiting the amount of injury incurred by health care itself. Another strong motivation has been the increasing recognition that value for money is poor in the health care system. Health care costs are high and have been increasing rapidly, without the level or improvement in quality that is expected by those paying the costs.
In this environment, there is widespread agreement that "business as usual"—i.e., exclusively relying on providers to self-police the quality of care they provide without systematic measurement, monitoring, and reporting—is no longer adequate for improving the quality of care.10
Quality is increasingly recognized as a product of systems, not individuals, and attention has focused on how to analyze and redesign those systems. A consensus has emerged that although solutions will not be easy, "a key component of any solution... is the routine availability of information on performance at all levels"11—that is, "quality of care should be measured and reported routinely at both the national and provider-specific (e.g., hospital and physician) levels."12 More simply, "that which cannot be measured is difficult to improve."13
There are a variety of measurement-based approaches to improving quality. The most widespread and longest-used approach, referred to in this report as "quality improvement," is an internal effort by providers or other organizations to measure quality of care, identify areas of weakness, devise and implement changes, and monitor the effects of the changes. As results are typically not shared outside of the organization that conducts the quality improvement project, this approach can be viewed as formalization of the traditionally prevailing model of improving quality—mutual peer review. More recently, calls for external accountability of health care providers have led to two types of quality monitoring approaches. One is public reporting of quality of health care providers.b
These reports can steer patients to higher-quality care and drive providers to improve their scores in order to bolster their professional reputation. The other one is linking quality of care to financial incentives, either in the form of pay-for-performance (paying more for good performance on quality metrics) or in the form of tiered insurance products, which steer patients towards higher-quality providers by charging higher copayments for visits to providers with poorer quality scores.
These efforts all depend on reliable, valid, and agreed-upon ways to measure the quality of care. However, health care quality is difficult to measure, in particular because there are many factors contributing to outcomes of health care in addition to the actual quality of the care provided. Many of these factors—for example, patient compliance with treatment—are largely out of providers' control. Despite this and other difficulties, much progress has been made in the science of quality measurement. The underlying knowledge base has greatly expanded. The science of measuring health status has improved, as has the evidence supporting "best practices" that have been proven to lead to improvements in health status. This evidence base has allowed for the development of numerous quality indicators, which then have been tested for reliability, validity, ease of use, and usefulness for improving quality.
The quality indicators fall into two main categories: counts of inputs or actions that are known to lead to better health outcomes ("structure" and "process" measures in Donabedian's classic quality measurement framework),14 and direct measurements of the outcomes of care ("outcomes"). Unfortunately, many of the most rigorous measures are also the most difficult to implement due to their burden of data collection. The data needed for quality measurement must be collected specifically for that purpose, abstracted from (usually) paper medical records, or abstracted from existing electronic data that have been collected for other purposes, notably billing. However, as will be shown in this report, there is now a small but growing arsenal of well-tested quality indicators that can be used with existing data sources or data collection processes.
The Agency for Healthcare Research and Quality (AHRQ) was one of the pioneers of quality indicator development. As early as 1994, its Healthcare Cost and Utilization Project (HCUP) developed a publicly available set of quality indicators for hospital care based on discharge data. Congress later tasked AHRQ with developing an annual report on the quality of health care in the United States, the National Healthcare Quality Report.15 In preparation for this report, in 2001 AHRQ invested in an update of the HCUP indicators, which were renamed the AHRQ Quality Indicators (AHRQ QIs).16 Indicators of patient safety were added in 2002.17 The AHRQ QIs and associated computer code are now publicly distributed and supported by AHRQ, with regular updates.18 They are widely used by a variety of organizations for many different purposes. Meanwhile, many other organizations, both public and private, have developed and used their own sets of quality indicators.
Given the rapid growth of and robust demand for quality indicators, it is important to assess the position of the AHRQ QIs in the quality indicator "market." Who is using the AHRQ QIs today, and for what purposes? What have users' experiences been, and what are their unmet needs? Who else is developing and/or distributing indicators similar to the AHRQ QIs? Most importantly, what has been the impact of the AHRQ QIs on the quality of care delivered to patients?
To understand the answer to these and related questions, the RAND Corporation was asked to conduct a one-year evaluation to assess user experiences with the AHRQ QIs, and to identify users of other quality indicators, vendors of quality measurement products using the AHRQ QIs, and developers of quality indicators comparable to the AHRQ QIs. The results of this study are intended to inform decisions by AHRQ on future priorities for the QI program.
AHRQ has developed several types of QIs, all of which share certain core characteristics. There are currently four sets of AHRQ QIs:
- Inpatient Quality Indicators (IQIs). There are 32 IQIs reflecting the quality of care provided in hospitals. The indicators fall in four categories.
- The first category includes the in-hospital mortality rates for seven specific medical conditions. The conditions were selected because their mortality rates were shown to vary across providers and because evidence indicates that the mortality risk for those conditions may depend on quality of care. The mortality rates, adjusted for patient risk, can be compared to those of other providers or to other benchmarks, with lower rates indicating better quality.
- The second category includes the in-hospital mortality rates for eight surgical procedures. Like the condition-specific mortality rates, the procedures were chosen because their mortality risk was known to vary across providers and because evidence indicates that mortality for those procedures may be associated with inferior quality of care. The mortality rates can be benchmarked, and lower rates are better.
- The third category includes utilization rates for eleven procedures for which there is potential for overuse, underuse, or misuse. Four of the utilization rates were designed for comparisons between geographic areas (either counties or Office of Management and Budget Metro Areas), not hospitals. The rates are measured as the number of procedures performed divided by the total number of patients who were potential candidates for that procedure. Utilization rates that are much higher or lower than benchmarks may point to poor quality. The direction of the indicator—i.e., whether higher or lower rates are desirable—depends on the condition. For example, Cesarean delivery has been identified as an overused procedure, so lower rates indicate better quality (IQI 21). In contrast, vaginal delivery following a previous Cesarean delivery (VBAC) is considered an underused procedure, so higher rates indicate better quality (IQI 34).
- The fourth category includes indicators of the hospital-level volume for six complex procedures for which research suggests a positive impact of case volume on patient outcomes. The procedure volume therefore provides an indirect indication of the provider's expected outcome for the procedure. The indicators are simple counts of the number of times that the procedures are performed per year. These counts can be compared to thresholds for the minimum annual volume of procedures that providers should perform to ensure an acceptable level of quality. The volume thresholds are based on published research.
- Patient Safety Indicators (PSIs). The PSIs contain 27 indicators of potentially-avoidable complications or other adverse events related to hospital care. For example, an infection resulting from medical treatment (PSI 7) is considered an avoidable complication, and the rate of surgical instruments or other "foreign bodies" left in the patient following surgery (PSI 5) is considered a preventable adverse event. The indicators are measured as the number of complications or adverse events divided by the total number of patients at risk for those events. Twenty of the indicators were designed for comparisons between hospitals, and the remaining 7 were designed for comparisons between geographic areas (either counties or Office of Management and Budget Metro Areas).
- Prevention Quality Indicators (PQIs). The PQIs are based on the hypothesis that appropriate ambulatory care can help to prevent hospital admissions for some conditions. The indicators consist of hospital admission rates for 14 of these ambulatory care-sensitive conditions and are intended for comparisons of geographic areas. Unlike the other AHRQ QIs, they do not reflect quality of hospital care, but the quality of ambulatory care. The indicators are measured as condition-specific admission rates (e.g., the number of admissions for urinary tract infection per 100,000 people living in a Metro Area or county). Each admission to the hospital reflects a failure in the delivery of appropriate care outside the hospital, so lower rates reflect better quality.
- Pediatric Quality Indicators (PDIs). The PDIs contain 18 indicators, each of which is similar to a comparable type of IQI, PQI, or PSI, except that the PDIs in each case refer to the pediatric population. Since these indicators were released during the course of this study, they were not included in the analysis. Several of the PDIs were formerly included in the AHRQ IQI, PSI, and PQI sets. Most of the issues raised in this report would generalize to the PDIs.
There are several key characteristics that apply to all four sets of the AHRQ QIs.
- Based on administrative data.The AHRQ QIs are based on hospital discharge data that are collected using the UB-92 standardized claim reporting form and used mainly for billing purposes. The indicators can be used with existing hospital administrative databases, including the HCUP National Inpatient Sample and Medicare MedPAR data.
- Outcome-focused. Most of the AHRQ QIs reflect health care outcomes, and not the rates at which evidence-based processes of care are followed. The exceptions are the subset of IQIs which reflect utilization or volume of services.
- Hospital-focused. Most of the indicators focus on inpatient care, and all of the indicators are based on hospital data. The PQIs reflect the quality of ambulatory care, but the measurement focuses on admissions to the hospital that could have been prevented given appropriate, effective ambulatory care.
- Reviewed for scientific soundness. The AHRQ QIs were tested by the Evidence-Based Practice Center at the University of California San Francisco and Stanford University. Detailed documentation of technical information, including the methodology, evidence base, and measurement properties of the indicators, is available in the public domain.
- Available for public use. The AHRQ QIs and associated code for SAS, SPSS,c and Windows are available for public download at no cost to the user. User support is also provided free of charge.
The AHRQ QIs were originally designed to be used for research and quality improvement activities.19 However, as examples included in this report demonstrate, the QIs are now used for a variety of other purposes, including public reporting, pay-for-performance, and segmenting providers for tiered insurance products. AHRQ has evaluated and endorsed the use of the QIs for these new purposes, given certain caveats.20
This report has three main objectives:
- Provide an overview of the market for the AHRQ QIs as well as indicators and quality measurement tools developed by other organizations that are similar to the AHRQ QIs or that incorporate the AHRQ QIs.
- Provide an overview of the range of ways in which the AHRQ QIs are used by various organizations.
- Assess the market demand for the AHRQ QIs, identify unmet needs, and discuss implications for future activities by AHRQ.
The remainder of this report is divided into six sections. Section 2 outlines the methodology used in our evaluation. Section 3 assesses the market for quality indicators and AHRQ's role in that market. Section 4 evaluates the AHRQ QI program on the dimensions of importance, scientific soundness, usability, and feasibility. Section 5 presents two case studies of how the AHRQ QIs are used in particular market areas. Section 6 summarizes lessons learned, and Section 7 concludes with a discussion of the implications of the evaluation for the future of the AHRQ QI program.