Page 1 of 1

Summary

Evaluation of the Use of AHRQ and Other Quality Indicators

 

Public recognition of health care quality issues has spiked remarkably in the past ten years, driven by a series of high-profile reports from the Institute of Medicine (IOM), the RAND Corporation, and other organizations. 1-4 These reports showed, among other facts, that preventable medical errors in hospitals result in as many as 98,000 deaths per year; preventable medication errors occur at least 1.5 million times per year; and on average, only 55% percent of recommended care is delivered. In response, a variety of stakeholders from across the spectrum of health care delivery including providers, professional and hospital associations, accreditation organizations, employers and business groups, insurance companies, and state and federal governments have focused on monitoring and improving the quality of care. These efforts have focused on avoiding unnecessary deaths and poor health, while also encouraging better quality and value for health care spending. In the current environment, the quality of health care is increasingly recognized as a product of systems, not individuals, and there is widespread agreement that systematic measurement, monitoring, and reporting are needed to make meaningful advances in improving quality.

Health care quality indicators provide an important tool for measuring the quality of care. Indicators are based on evidence of "best practices" in health care that have been proven to lead to improvements in health status and thus can be used to assess, track, and monitor provider performance. While the results of quality measurement were originally not typically shared outside the provider organization conducting the quality improvement project, more recent assessments using the indicators have been included in public reports intended to steer patients toward higher-quality care and drive providers to improve their scores in order to bolster their public reputation. Indicators have also been used to link quality of care to financial incentives, either in the form of pay-for-performance (i.e., paying more for good performance on quality metrics), or in the form of tiered insurance products, which steer patients towards higher-quality providers by charging higher copayments for visits to providers with poorer quality scores.

The Agency for Healthcare Research and Quality (AHRQ) has been a pioneer in the development of health care quality indicators. In 1994 its Healthcare Cost and Utilization Project (HCUP) developed a publicly available set of quality indicators for hospital care based on discharge data. AHRQ updated the HCUP indicators in 2001, which were then renamed the AHRQ Quality Indicators (AHRQ QIs). Today, AHRQ maintains four sets of QIs:

  1. Inpatient Quality Indicators (IQIs), which reflect the quality of care provided in hospitals.
  2. Patient Safety Indicators (PSIs), which reflect potentially avoidable complications or other adverse events during hospital care.
  3. Prevention Quality Indicators (PQIs), which consist of hospital admission rates for 14 ambulatory care-sensitive conditions; and
  4. Pediatric Quality Indicators (PDIs), which combine components of the PSIs, IQIs, and PQIs, as applied to the pediatric population.

The AHRQ QIs are publicly distributed and supported by AHRQ, with regular updates. They are widely used by a variety of organizations for many different purposes. Meanwhile, many other organizations, both public and private, have developed and used their own sets of quality indicators. Given the rapid growth of and robust demand for quality indicators, it is important to assess the position of the AHRQ QIs in the quality indicator "market."

  • Who is using the AHRQ QIs today, and for what purposes?
  • What have users' experiences been, and what unmet needs do they still have?
  • Who else is developing and/or distributing indicators similar to the AHRQ QIs?
  • Most importantly, what has been the impact of the AHRQ QIs on the quality of care delivered to patients?

To answer these and related questions, the RAND Corporation was asked to conduct a one-year evaluation to assess user experiences with the AHRQ QIs and to identify users of other quality indicators, vendors of quality measurement products using the AHRQ QIs, and developers of quality indicators comparable to the AHRQ QIs. The results of this study are intended to inform decisions by AHRQ on future priorities for the QI program.

This report has three main objectives:

  1. Provide an overview of the market for the AHRQ QIs as well as indicators and quality measurement tools developed by other organizations that are similar to the AHRQ QIs or that incorporate the AHRQ QIs.
  2. Provide an overview of the range of ways in which the AHRQ QIs are used by various organizations.
  3. Assess the market demand for the AHRQ QIs, identify unmet needs, and discuss implications for future activities by AHRQ.

Return to Contents

The Market and Uses for AHRQ QIs

While AHRQ has developed four sets of QIs, all the QIs share certain key characteristics.

  • Based on administrative data.The AHRQ QIs are based on hospital discharge data and can be used with existing hospital administrative databases.
  • Outcome-focused. Most of the AHRQ QIs reflect health care outcomes, not the rates at which evidence-based processes of care are followed.
  • Hospital-focused. Most of the indicators focus on inpatient care, and all of the indicators are based on hospital data.
  • Reviewed for scientific soundness. The AHRQ QIs were tested by the Evidence-Based Practice Center at the University of California San Francisco and Stanford University, and detailed documentation of technical information is available in the public domain.
  • Available for public use. The AHRQ QIs and associated code for SAS, SPSS,a and Windows are publicly available for download at no cost to the user.

To understand the market for the AHRQ QIs, we conducted a series of interviews with users of AHRQ QIs, users of other products, developers of similar indicator sets, and vendors of quality measurement products that include AHRQ QIs. This review found that the AHRQ QI program fills a unique niche in the market for QIs since there are no other sources of hospital care quality indicators that represent both a national standard and are also publicly available, transparent, and based on administrative data. Many of our interviewees stressed that the AHRQ QIs fill an important void in this respect.

AHRQ's unique place in the market for quality indicators has led to a wide proliferation of uses for the AHRQ QIs. Our environmental scan of users of the AHRQ QIs identified 114 users of the indicators and a range of different uses, including public reporting, quality improvement/benchmarking, pay-for-performance, and research. The most common uses of the AHRQ QIs include:

  • Research. We identified 43 uses of the AHRQ QIs for research. For example, Leslie Greenwald and colleagues used the AHRQ QIs to compare the quality of care provided in physician-owned specialty hospitals and competitor hospitals.5
  • Quality improvement. We identified 23 organizations that use the AHRQ QIs as part of a quality improvement activity, including reports benchmarking performance against peers, but do not release the quality information into the public domain.
  • Public reporting. We identified 20 organizations using the AHRQ QIs for public reporting. We classified an activity as "public reporting" if a publicly available report was published that compares AHRQ QI results between hospitals (IQIs and PSIs) or geographic areas such as counties (PQIs).
  • Pay-for-Performance. We identified 4 organizations that are using the AHRQ QIs in pay-for-performance programs. Three were health plans and one was a Centers for Medicare and Medicaid Services (CMS) demonstration project.

As part of our environmental scan for users of the AHRQ QIs, we conducted a limited review of international uses. We found that the most visible current endeavor that attempts to make use of the AHRQ QIs is the Organization for Economic Cooperation and Development's (OECD) Health Care Quality Indicators (HCQI) Project. The OECD is an intergovernmental economic research institution headquartered in Paris, France, with a membership of 30 developed countries that share a commitment to democratic government and the market economy. The organization recently convened a meeting to work on the development and implementation of QIs at the international level. Preliminary discussions indicate that there is interest in using the AHRQ QIs internationally as well as sufficient data and technical capability to implement them.

Return to Contents

Key Findings

The results of our interviews and environmental scan focused on four key factors that can be used as criteria for evaluating quality indicators: importance, usability, scientific soundness, and feasibility.

Importance

Nearly all of the organizations interviewed stressed the importance of the AHRQ QI program. AHRQ was frequently mentioned as a "national leader" in measurement development and research. Many interviewees stated very strongly that they rely on AHRQ as one of the only sources for publicly available, transparent indicators based on readily available data. They stressed that without the AHRQ QIs, they would have few alternatives and would likely have to drastically change or eliminate their quality reporting and/or measurement activities. Interviewees generally felt that it was important that a federal agency like AHRQ, which is regarded as credible and respected, develop and support a quality indicator set for public use. AHRQ's credibility and the transparency of the AHRQ QI methods were considered to be key in overcoming opposition to quality measurement and reporting by stakeholders, particularly providers. Overcoming this type of opposition is particularly important for public reporting and pay-for-performance initiatives, where providers' reputations and revenues are at stake.

Although only one organization in our interviews had formally measured the impact of AHRQ QIs on the quality of care delivered to patients, many interviewees provided anecdotal evidence of the effect of the use of indicators on quality. When asked whether they had measured the impact of using the AHRQ QIs, many interviewees responded that indicator use began too recently to allow for observation of any impact. However, many interviewees reported anecdotally that their or their clients' use of the AHRQ QIs was having some type of impact on quality of care. The impacts observed usually consisted of an activity such as putting a new quality improvement process in place, rather than an improvement in outcomes.

Scientific Soundness

On the whole, our interviewees were impressed by the quality and level of detail of the AHRQ documentation on the face validity of the indicators and stated that the indicators captured important aspects of clinical care. Very rarely were indicators challenged on conceptual grounds. Users largely felt that the AHRQ QIs can be reliably constructed from hospital discharge data, but that there was a certain learning curve during which hospital coding departments had to adjust to the requirements for the QIs. Thus far, coders had mainly been trained to apply coding rules to fulfill reimbursement requirements, but now they had to understand that coding practices also had implications for quality reporting. In selected instances, we heard concerns about ambiguity in the coding rules that would not provide sufficient guidance on whether to code an indicator-relevant diagnosis.

Sample size issues (whether due to the rarity of certain procedures or the infrequency with which some procedures are conducted at certain facilities) were repeatedly mentioned as a threat to the validity of the indicators, particularly the PSIs. Most users stated that the indicators were correctly operationalized within the constraints of the underlying data source. Isolated findings of specification errors were brought to our attention, but interviewees emphasized that AHRQ was always able to address those quickly. The limitations of administrative data were frequently mentioned as a threat to validity.

Usability

Most interviewees stated that the AHRQ QIs provide a workable solution for their needs and were very appreciative of the support that the AHRQ QI team provides for implementation and ongoing use. Despite these overall favorable impressions of the usability of the QIs, three needs related to their usability for reporting were raised repeatedly: development of reporting templates, development of composite indicators, and clearer guidance on the use of the AHRQ QIs for public reporting and pay-for-performance programs.

Standard reporting format. Nine of 54 interviewees highlighted the need for a standard format for reporting AHRQ QI results as a top priority. At the simplest level, some interviewees requested AHRQ-supported, standard, basic names for the AHRQ QIs in plain language, as some of the current indicator names are difficult for non-clinical audiences to understand. Other interviewees expressed a desire for more guidance and support on other aspects of presentation.

Composite indicators. Twelve of 54 interviewees expressed a desire for an AHRQ-supported methodology for constructing a composite indicator. Forming composites would allow the results to be summarized into one statistic, which is easier to grasp and communicate, in particular for non-expert audiences. Composites would also overcome sample size limitations, as indicators could be pooled.

Guidance on using AHRQ QIs for public reporting and pay-for-performance. Interviewees who are currently using the AHRQ QIs for public reporting and pay-for-performance generally felt that they provided a workable basis for their activities. Still, interviewees stated that additional standards and guidance on the reporting of AHRQ QI results were needed. Many interviewees expressed dissatisfaction with the current AHRQ guidance on the appropriateness of the AHRQ QIs for public reporting. They felt that clearer guidance from AHRQ would help to counter opposition from those who argue that the AHRQ QIs should only be used for quality monitoring and improvement and research, but not as a public reporting or pay-for-performance tool. Taking the opposing view were several interviewees (mostly hospitals) who would like to see AHRQ make a clear statement that the AHRQ QIs are not appropriate for use in public reporting, pay-for-performance, or other reporting activities because of the limitations of the underlying administrative data.

Feasibility

We were told consistently that a major advantage of the AHRQ QIs was the feasibility of their implementation. They require only administrative data in the UB-92 format to which many users have routine access, since those data are already being used for billing and other administrative purposes and have to be collected and reported by hospitals in most states.

Interviewees emphasized that another substantial advantage of the AHRQ QIs is that the indicators have clearly defined and publicly available specifications, which helps with implementation of measurement. These specifications were regarded as of particular importance for hospitals, as the originators of the data, because the specifications enable hospitals to work with their coding departments to ensure that the required data elements were abstracted from medical records consistently and with high reliability. In addition, users who analyze data with the QIs, such as researchers, appreciated the fact that they could dissect the indicator results and relate them back to individual records. That ability helped researchers gain a better understanding of the indicator logic and distinguish data quality issues from actual quality problems.

Return to Contents

Lessons Learned for Future Activities

Interviewees' perspectives provided lessons in three areas: current, anticipated, and potential development projects involving the QIs; AHRQ's role as a measures developer and the ways in which users speculate this role could evolve; and market demand for quality indicators, in particular, user willingness to pay for QIs.

Priorities for Future Development of QIs

We asked interviewees to prioritize three categories of AHRQ development projects:

  1. Improvements in the current QI product line.
  2. Addition of new product lines.
  3. Improved support for the QI products.

Improving the current products was most frequently seen as the highest priority, followed by both the addition of new products and improvements in service, outreach, and user support for the measures. The most commonly requested improvement to the current QIs was the addition of data elements to the QI specifications, notably a flag for conditions present on hospital admission (currently in development by AHRQ), a flag for patients under do-not-resuscitate orders, and clinical data elements. The most commonly requested new product line was indicators of ambulatory care. As far as service improvements, the most frequently mentioned activities were a template and guidance for public reporting of the QIs, and guidance on next steps in quality improvement following identification of a potential quality problem using the QIs.

The Future Role of AHRQ Compared to Other Players

Our interviewees held AHRQ in very high regard. The work of the AHRQ QI team was described as technically sound, sensitive to the limitations of the underlying data, and transparent. AHRQ is regarded as an intellectual leader and "go-to" institution for health services research and the use of administrative data for hospital quality measurement. Several other organizations, especially the Joint Commission on Accreditation of Healthcare Organizations (JCAHO), CMS, the Hospital Quality Alliance (HQA), and the Leapfrog Group, are seen as prominent sources for measures, but their indictors are generally regarded as complements to the AHRQ QIs. Interviewees were quite comfortable with AHRQ having a leading role in national quality indicator development. It was generally viewed as positive that a trustworthy federal institution had defined open-source and well-documented quality measurement standards. These standards were viewed as contributing to the transparency of health care quality measurement and reducing the measurement burden for health care providers by limiting the number of measurement tools they must use to satisfy various reporting requirements.

We discussed whether it could be a viable option for AHRQ to give up parts of the current QI program in order to free up resources and set different priorities. Almost unanimously, interviewees rejected a model under which AHRQ would develop and distribute the QI software without supporting it. We received mixed reactions to a model under which AHRQ would develop and release indicators and their technical specifications, but no longer provide or support software. Interviewees were generally wary of the idea of delegating user support and/or software development and distributions to vendors, fearing that vendors would be prohibitively expensive or incapable of providing the same quality of support as the original developers.

Willingness to Pay for the AHRQ QIs

As an alternative to AHRQ realigning current funds, we asked interviewees whether AHRQ might consider financing program growth by generating additional revenues by charging users. Not unexpectedly, this proposal was not met with enthusiasm. However, almost half of interviewees (44%) were willing to pay a "reasonable fee" for access to the full QI resources.

Return to Contents

Discussion

Limitations

The majority of the interviewees in this evaluation were users of the AHRQ QIs. Non-users may have more negative opinions of the AHRQ QIs. The few non-users we did interview did not have express substantially negative opinions about the AHRQ QI program, but a larger sample of non-users may have produced different results. This study also focused on uses of the AHRQ QIs that were publicly discussed or released, so that the results likely do not fully reflect the use of AHRQ QIs for non-public uses such as confidential quality improvement activities by hospitals.

What is AHRQ's Current Market Position?

The AHRQ QIs have achieved a strong position in their market segment and no obvious alternative or competitor could be identified, although some organizations (notably JCAHO, CMS, HQA, and Leapfrog) have complimentary indicator sets. This is unlikely to change: new users have an incentive to adopt the prevailing product, because it makes their results comparable to a large number of other users. Widespread use lends legitimacy to the product, which is critical in the often-politicized debates about selecting quality indicators for public reporting and pay-for-performance.

Where are the Growth Opportunities for the AHRQ QI Program?

There are now a substantial number of users of the AHRQ QIs for public reporting and pay-for-performance programs. As the prevalence of those activities increases, we expect the number of users to increase substantially both for the programs themselves and for internal quality improvement programs and projects that will attempt to align their target measures with standards for external accountability. Our interviewees wanted expansion of the AHRQ QI program. They were largely aware and appreciative of AHRQ's current efforts to improve and expand the program, but expressed an interest in scaling up, and speeding up, those activities.

How Could Growth Be Financed?

Most interviewees stated that federal funding should be used to support future AHRQ QI activities, even though they realized that this was a difficult proposition given the pressure on public budgets in general, and on AHRQ's budget in particular. Interviewees were reluctant to see AHRQ give up software development and/or user support. As an alternative, we discussed the option of AHRQ continuing to provide specifications, software and user support but starting to charge for those services. While there was little enthusiasm for user fees, only a few stated that they would stop using the AHRQ QI product in that case. Most interviewees seemed to be willing to pay a "reasonable" amount. However, if AHRQ were to implement a charge-based model for the QIs, it would face the challenge of finding a business model that would generate sufficient revenue and still be consistent with AHRQ's mission and values as a public agency.

Return to Contents

List of Symbols

SymbolDefinition
ACOGAmerican College of Obstetricians and Gynecologists
AHRQAgency for Healthcare Research and Quality
AMIAcute myocardial infarction
APR-DRGsAll patient refined diagnosis related groups
BCBSMABlue Cross Blue Shield of Massachusetts
CMSCenters for Medicare and Medicaid Services
CHSCCenter for Studying Health System Change
DFWDallas-Fort Worth
DFWHCDallas-Fort Worth Hospital Council
DIDFWHC Data Initiative
DRGsDiagnosis related groups
DVTDeep vein thrombosis
ECEuropean Commission
ESQHEuropean Society for Quality in Healthcare
ETGsEpisode Treatment Groups
GICGroup Insurance Commission (State of Massachusetts)
HCQIHealth Care Quality Indicators Project
HCUPHealthcare Cost and Utilization Project
HQAHospital Quality Alliance
ICD-9-CMInternational Statistical Classification of Diseases and Related Health Problems - Version 9 - Clinical Modification
IHIInstitute for Healthcare Improvement
IOMInstitute of Medicine
ISQuaInternational Society of Quality in Healthcare
JCAHOJoint Commission on Accreditation of Healthcare Organizations
IQIsInpatient Quality Indicators
Mass-DACMassachusetts Data Analysis Center
MHAMassachusetts Hospital Association
NQFNational Quality Forum
OECDOrganization for Economic Cooperation and Development
PDIsPediatric Quality Indicators
PEPulmonary embolism
PQIsPrevention Quality Indicators
PSIsPatient Safety Indicators
QIQuality indicator
THCICTexas Health Care Information Collection
VBACVaginal birth after cesarean section
WHOWorld Health Organization

a. Support for SPSS will be discontinued by AHRQ in 2007.


Return to Contents
Proceed to Next Section

Current as of December 2007
Internet Citation: Summary: Evaluation of the Use of AHRQ and Other Quality Indicators. December 2007. Agency for Healthcare Research and Quality, Rockville, MD. http://www.ahrq.gov/research/findings/final-reports/qualityindicators/sum.html