Adopting a More Quantitative and Transparent Measure Selection Process
The IOM Future Directions committee recommends changes to AHRQ's measure selection process in
order to focus the outcome of the process on the central aspirations of quality improvement—improving
health, value, and equity—by closing performance gaps in health care areas likely to have the greatest
population impact, be most cost effective, and have a meaningful impact on eliminating disparities. In
order to enhance the transparency of AHRQ's process for measure selection, a Technical Advisory Subcommittee
for Measure Selection is recommended under the existing AHRQ National Advisory Council
for Healthcare Research and Quality (NAC) to advise on ranking measures for selection, inclusion in the
national healthcare reports, and retirement. As part of this process, this subcommittee should recommend
strategies for the development and acquisition of new measures and data sources.
Conceptual models of improving health care quality and eliminating disparities include measurement and
reporting as integral to achieving performance goals; performance improvement systems, in turn, depend on the
quality of data to support measures (Berwick et al., 2003; Kilbourne et al., 2006; Langley et al., 1996). Over the
past decade, growing attention to health care quality measurement has led to the generation of a large number
of quality measures now being available for use. Illustrating the magnitude of the universe of possible quality
measures, the National Quality Measures Clearinghouse inventory now contains 1,475 potential quality measures
(National Quality Measures Clearinghouse, 2009a,b). Likewise as of October 2009, the National Quality Forum
(NQF) maintained a list of 537 measures meeting its standards for endorsement (NQF, 2009b). The growth in the
number of possibilities necessitates a critical assessment of how to prioritize among existing and future measures
for use in the NHQR and NHDR. There have been calls to develop a parsimonious common set of measures to
"serve policy and frontline information needs" (McGlynn, 2003, p. I-39).
Since 2003, AHRQ has refined its measure set for the national healthcare reports and related products, and
the measure set now includes approximately 260 individual measures, including a set of 46 core measures that are
more prominently featured in the body of the 2008 NHQR and NHDR. The larger set of 260 measures is featured
in online products such as the Web-based State Snapshots, NHQRDRnet, and appendixes to the NHQR and NHDR.
The selection of measures for the national healthcare reports by AHRQ has been influenced by the availability of
national data sources internal to HHS.
ARHQ has been urged to add more performance measures to the NHQR, NHDR, and related products, and has asked the IOM Future Directions committee for guidance on prioritization among measures so that new measures could be added and highlighted in the reports while other measures could receive less emphasis or be removed entirely from AHRQ's tracking. AHRQ regards the production of the NHQR and NHDR as having reached capacity given the agency's current resources for measurement reporting, analysis, and presentation. AHRQ staff has deliberated about retiring some measures to allow for the incorporation of new measurement domains or measures, but the agency has found it difficult to retire measures because of advocacy, both internal and external to HHS,
for each of the current measures.
The Future Directions committee reviewed AHQR's existing measure selection processes and criteria to shed
light on how these processes might be improved, particularly in support of the committee's overall aim to have the
national healthcare reports focus on areas that matter most and to encourage various stakeholders to take action
on the highest impact areas for quality improvement and disparities elimination.
In this chapter, the committee describes how AHRQ's measure selection process might be enhanced by
selecting measures that support national priority areas for health care quality improvement (go to Chapter 2), by
incorporating concepts of value and equity (go to Chapter 3), and by applying more explicit quantitative techniques
in the selection process. Taking these steps would help direct attention to those performance areas with the greatest
potential impact to transform health care quality for the country and for specific populations, and identify key
areas for measure and data source development.
AHRQ'S Approach To Selecting Measures
The measure selection process for the national healthcare reports has been undertaken primarily by AHRQ
staff in consultation with an HHS Interagency Workgroup consisting of program and data experts, as well as with
some limited external feedback from AHRQ's NAC.
AHRQ's Initial Measure Selection Process and Criteria
AHRQ's initial selection approach for measures in the NHQR and NHDR began with a call for measures
involving all HHS agencies, as well as substantial input from private-sector entities that were solicited by the IOM
during the research for its 2001 Envisioning the National Healthcare Quality Report (IOM, 2001). More than 600
candidate measures were generated through the call (AHRQ, 2003a).
Subsequently, the HHS Interagency Workgroup for the NHQR/NHDR reduced the 600 candidate measures for
tracking to about 140: (1) by applying three basic criteria recommended by the IOM in 2001—importance, scientific
soundness, and feasibility (go to discussion in Box 4-1)—to each individual measure; (2) by mapping potential measures
to the elements of the earlier quality framework (effectiveness, safety, timeliness, and patient-centeredness);
and (3) by selecting clinically important conditions within effectiveness measures (AHRQ, 2003a).
During the summer of 2002, public comments were solicited from hospitals, providers, researchers, and others
via a public hearing conducted by the National Committee on Vital and Health Statistics (NCVHS) and through
a Federal Register notice (AHRQ, 2002; NCVHS, 2002). As the HHS Interagency Workgroup refined the final
package of measures for the NHQR and NHDR, input was sought from the HHS Data Council, technical and policy
experts within AHRQ, and the Quality Interagency Coordination Task Force, which spanned several federal agencies
(Veterans Affairs, Department of Defense, Federal Bureau of Prisons, and others).1 A separate review process was held for home health measures, which were not included in the initial public review cycle (AHRQ, 2003b).
As a result of this effort, the first edition of the NHQR published by AHRQ reported on 147 measures; of
these, effectiveness measures (97 measures; 65 percent of the total measures) focused on the clinical conditions chosen for Healthy People 2010 (cancer, diabetes, end-stage renal disease, heart disease, HIV/AIDS, maternal and child health, mental health, respiratory disease, and nursing home and home health care) (AHRQ, 2003a; HHS, 2009b).
|Box 4-1. The IOM 2001 Recommendations for Measure Selection Criteria for the NHQR and NHDR
In the IOM's 2001 report Envisioning the National Healthcare Quality Report, three major criteria were proposed for
- Importance of what is being measured
- Impact on health. What is the impact on health associated with this problem?
- Meaningfulness. Are policy makers and consumers concerned about this area?
- Susceptibility to being influenced by the health care system. Can the health care system meaningfully
address this aspect or problem?
- Scientific soundness of the measure
- Validity. Does the measure actually measure what it is intended to measure?
- Reliability. Does the measure provide stable results across various populations and circumstances?
- Explicitness of the evidence base. Is there scientific evidence available to support the measure?
- Feasibility of using the measure
- Existence of prototypes. Is the measure in use?
- Availability of required data across the system. Can information needed for the measure be collected in
the scale and time frame required?
- Cost or burden of measurement. How much will it cost to collect the data needed for the measure?
- Capacity of data and measure to support subgroup analyses. Can the measure be used to compare different
groups of the population?
The 2001 IOM report stipulated that it is desirable for a measure to meet all 10 elements within the three overall
criteria, but noted that it is not required that all 10 apply in order for a given measure to be considered for inclusion in the
NHQR and NHDR.
The 2001 IOM committee indicated that the three criteria, as listed above, provide a hierarchy by which measures
should be considered, with priority to be given to measures evaluated for importance and scientific soundness and then
by feasibility. For example, the committee stated:
Measures that address important areas and are scientifically sound, but are not feasible in the immediate
future, deserve potential inclusion in the data set and further consideration. However, measures that are scientifically
sound and feasible, but do not address an important problem area, would not qualify for the report
regardless of the degree of feasibility or scientific soundness.
Source: IOM, 2001, pp. 83 and 87.
AHRQ's Current Measure Selection Process and Criteria
AHRQ reduced the number of measures presented in subsequent editions in response to criticisms that the
first edition was unwieldy (Gold and Nyman, 2004). The intent was to be able to "highlight measures with indepth
analysis, rather than broad, but sparse, coverage of all 179 measures" (AHRQ, 2004).2 That basic format is
maintained by AHRQ today, with a set of approximately 46 core measures presented in the body of the reports and
more detailed tables available online for a larger set of measures. To select the 46 core measures for the NHQR
and NHDR, AHRQ staff and the HHS Interagency Workgroup prioritized measures by the three original IOM
criteria and several additional ones. Usability was added as a new primary criterion—one that is also articulated by NQF in considering the suitability of any measure as a voluntary consensus standard.3 AHRQ's current criteria and principles for prioritizing measures in the NHQR and NHDR are summarized in Box 4-2. AHRQ gives greater weight to "primary criteria" than to "secondary criteria," and the "balancing principles" were also added to ensure that the final set of core measures covered a variety of conditions and sites of care.
AHRQ also emphasizes health care process measures over health outcome measures due to the fact that the
focus of the reports is health care delivery and that outcome measures are often too distal or rare (e.g., mortality)
to be linked to the delivery of a particular service. Whenever a close relationship is deemed to exist (e.g.,
use of colorectal cancer screening to presentation with more advanced colorectal cancer), then AHRQ has tried
to present paired process and outcome measures. The Future Directions committee recognizes the limitations of
process measures, as does AHRQ, and encourages AHRQ to continue to report paired measures whenever possible.
Additionally, the committee encourages AHRQ to develop or adopt outcome measures as they hold great
interest for policy makers, particularly outcomes associated with the implementation of specific programs. For
example, AHRQ already reports on receipt of care for heart attack and inpatient mortality, but could also report
related information on outcomes such as: "Since the beginning of public reporting on readmission rates for AMI
by the Centers for Medicare and Medicaid Services [CMS], the readmission rates have been reduced X percent,
yielding a potential savings to the federal Medicare budget of $Y."
Assessing Importance of Topic Areas for Inclusion
Over time, AHRQ has taken stock of which health conditions or intervention topic areas warranted consideration
within the NHQR and NHDR to determine if there should be measurement additions or deletions. AHRQ
provided the Future Directions committee with a side-by-side comparison of the specific factors considered in
identifying important topics for national reporting (Appendix E). These factors include: the leading causes of death,
disability, or activity limitation; principal hospital diagnoses; costly conditions in general and among hospitalizations
specifically; areas with notable Black-White racial and educational-level disparities measured in life years
lost; other significant racial and ethnic disparities; and priority areas named in several advisory reports from the
IOM and HHS (e.g., HHS strategic plans; the 2003 IOM report Priority Areas for National Action: Transforming
Health Care Quality4 (IOM, 2003). From the sources, AHRQ has identified nationally relevant topics not yet
reported in the NHQR, NHDR, or related products. For example, AHRQ added measures, such as obesity and substance abuse measures5 to the 2008 reports.
AHRQ's NAC provides advice on content. The NAC and an existing subcommittee consisting of a few NAC
members with an interest in the NHQR and NHDR serve as a sounding board for AHRQ staff and provide input
to the AHRQ report development process (e.g., recommendations to improve dissemination and to pay increased
attention to child health measures; the need to close measurement gaps and set priorities; the need to address
cost, waste, and value issues). Thus, the selection of new measures appears to be driven primarily by the need to
address new topic areas based on expert opinion (e.g., IOM, NAC, HHS Interagency Workgroup), some general
quantitative information about the overall burden of a condition on society and individuals, and the availability of
data to report on a topic. In 2008, the NAC observed that the caliber of the NHQR and NHDR has improved with
each updating (AHRQ, 2008a), and the Future Directions committee agrees.
|Box 4-2. AHRQ's Current Criteria and Principles for Prioritizing Measures
- Impact on health (e.g., clinical significance, prevalence).
- Susceptibility to being influenced by the health system (e.g., high utility for directing public policy, and sensitive
- Scientific Soundness (assumed because AHRQ only uses consensus-based endorsed measures).
- Capacity of data and measure for subgroup analysis (e.g., the ability to track multiple groups and at multiple
levels so a number of comparisons are possible).
- Cost or burden of measurement.
- Availability of required data for national and subgroup analysis.
- Measure prototype in use.
- Usability: easy to interpret and understand (methodological simplicity).
- Type of Measure: evidence-based health care process measures favored over health outcome measures because
most outcome measures were too distal to an identified intervention.
- Applicable to general population rather than unique to select population.
- Data available regularly/data available recently.
- Linkable to established indicator sets (i.e., Healthy People 2010 targets).
- Data source supports multivariate modeling (e.g., socioeconomic status, race, and ethnicity).
- Balance across health conditions.
- Balance across sites of care.
- At least some state data.
- At least some multivariate models.
Source: AHRQ, 2005.
Improving Measure Selectiom
The Future Directions committee concludes that for the NHQR and NHDR to be more strategic and address
the most important opportunities for concerted national action, AHRQ's approach to measure selection needs to
be modified. The Future Directions committee recommends broadening the range of input that AHRQ currently
receives, making the process transparent, and incorporating a more systematic and quantitative process for ranking
measures. The proposed selection process more closely looks at the gap between current and desired performance
levels and the relative value of bridging that gap while also taking equity into account. This is a somewhat different
approach for AHRQ, one that focuses on closing the quality gap rather than simply selecting conditions and
measures based on the highest prevalence and costs.
Focusing on High-Impact Areas
The committee's definition of high-impact areas for quality improvement builds on previous IOM and NQF guidance on determining what constitutes the criteria of importance in measure selection and endorsement (IOM, 2001, p. 83; NQF, 2009a). Specifically, the committee's definition refocuses how AHRQ evaluates "impact on health" for the purposes of selecting measures for the NHQR and NHDR.
High impact areas for health care quality improvement: Ideally, "high impact" quality improvement and
disparity reduction areas would be assessed by quantitatively ranking the population health impact of closing
the gap between current performance and desired levels of performance (such as 100 percent of persons in
need achieving guideline recommended care). These could be assessed for the entire population of the nation
and/or for specific priority populations when data allow.
The committee's advice should not be construed to mean that an area would be considered a high impact area
solely based on how large the gap is between current performance and desired performance levels (e.g., a spread
of 25 percentage points is not automatically more befitting of attention than one that has a spread of 10 percentage
points); closure of a smaller gap could be ranked higher than a larger gap if its closure would yield a greater
health outcome for the nation's population. While the committee members' emphasis is on quantitative assessment,
they are cognizant that data limitations will at times require expert opinion to qualitatively rank measures,
particularly in the absence of detailed data to allow assessment of equity considerations for different population
groups. In these cases, a qualitative assessment of the impact of the intervention targeted by the measure would
be combined with a quantitative assessment of the size of the gap or the disparity in order to rank the relative
importance of the measure.
The NAC has observed that health care quality measurement in the United States has been "incremental and
evolutionary," unfolding in the absence of a unified performance measurement strategy backed by a plan to obtain
data to support key measures. The Future Directions committee hopes that an additional outcome of its proposed
measure selection process would be the identification of measure and data needs and the formulation of a strategy
for their development.
For the reasons just cited and discussed further below, the committee recommends that AHRQ establish a new
Technical Advisory Subcommittee on Measure Selection that can advise the NAC and AHRQ on performance
Recommendation 3: AHRQ should appoint a Technical Advisory Subcommittee for Measure Selection
to the National Advisory Council for Healthcare Research and Quality (NAC). The technical
advisory subcommittee should conduct its evaluation of measure selection, prioritization, inclusion,
and retirement through a transparent process that incorporates stakeholder input and provides
public documentation of decision-making. This subcommittee should:
- Identify health care quality measures for the NHQR and NHDR that reflect and will help
measure progress in the national priority areas for improving the quality of health care and
eliminating disparities while providing balance across the IOM Future Directions committee's
revised health care quality framework.
- Prioritize existing and future health care quality measures based on their potential to improve
value and equity.
- Recommend the retirement of health care quality measures from the NHQR and NHDR for
reasons including but not limited to the evolution of national priorities, new evidence on the
quality of the measure, or the attainment of national goals.
- Recommend a health care quality measure and data source development strategy for national
reporting based on potential high-impact areas for inclusion in AHRQ's national quality
The committee's rationale for the establishment of the proposed NAC Technical Advisory Subcommittee
for Measure Selection is discussed below. Subsequent sections of this chapter discuss desirable attributes of
transparency in AHRQ's process for selecting performance measures, a stepwise process to applying qualitative
and quantitative criteria in prioritizing measures, and quantitative methods that have potential applicability to the
process for assessing value and equity.
Establishing an Entity for Measure Selection
The Future Directions committee considered several organizational alternatives to take on the responsibility
of measure selection, but ultimately recommended the formation of the NAC Technical Advisory Subcommittee
for Measure Selection.
Retaining the Status Quo
Retaining the status quo, with responsibility resting with AHRQ staff and HHS Interagency Workgroup
members, is considered less desirable, even after possibly supplementing the current process with opportunities
for public input and comment, because the process would likely retain its current limitations. The status quo did
not appear tenable because AHRQ and HHS Interagency Workgroup members have already acknowledged the
difficulty of being able to prioritize and eliminate health care quality measures through the current process (other
than plans to semi-retire from the 2009 reports process measures that have a greater than 95 percent achievement
rate [AHRQ, 2008a]).6 Furthermore, a critical parallel can be drawn to the lessons learned from Healthy People
2010. While Healthy People 2010 contains too many "primarily disease-oriented" objectives, it is nonetheless a
"challenge to move away from a biomedical model because it is easier to create specific and measurable health
targets that are disease specific," "funding for many of the possible interventions is disease-specific," and there
are "strong constituencies," both internal and external, for featuring those diseases (Fielding, 2009). Currently, the
NHQR and NHDR are heavily weighted to the clinical conditions in Healthy People 2010, and a Future Directions
committee concern is that some of the NHQR and NHDR content may be a product of this same history.
Ultimately, the committee felt strongly that the decision-making process about measures needed to be a
public one rather than internal to the HHS Interagency Workgroup and AHRQ staff so that decisions are more
transparent and justified to those who advocate for the inclusion or exclusion of specific measures. AHRQ could
improve the transparency of its existing practices by (1) publicizing on its Web site the documentation supporting
decisions behind the agency's selection of measures and (2) establishing a public comment period on those
decisions. However, the Future Directions committee also believes that AHRQ needs more focused external support
to make difficult decisions when ranking among measures, particularly as the selection process may result
in a substantial change in the portfolio of measures over time. Furthermore, they need the technical, quantitative
expertise to evaluate candidate measures.
1 Go to http://www.quic.gov (accessed November 28, 2009) for a full list of member agencies. The HHS Data Council coordinates all health
and non-health data collection and analysis activities of HHS, including an integrated health data collection strategy, coordination of health data
standards and health information and privacy activities. The HHS Data Council consists of senior level officials designated by their agency or
staff office heads, the HHS Privacy Advocate, and the Secretary's senior advisor on health statistics. It is co-chaired by the Assistant Secretary
for Planning and Evaluation and a rotating Operating Division (OpDiv) head; AHRQ is the current OpDiv co-chair. For more information, go to
http://aspe.hhs.gov/datacncl/ (accessed May 14, 2010).
2 Additional measures were added to the initial full measure set.
3 The measure evaluation criteria used by NQF for measure endorsement are available at http://www.qualityforum.org/uploadedFiles/Quality_Forum/Measuring_Performance/Consensus_Development_Process%E2%80%99s_Principle/EvalCriteria2008-08-28Final.pdf?n=4701 [Plugin Software Help]
(accessed March 26, 2009). There is substantial overlap in the criteria for measure endorsement and selection to date, whether past IOM
recommendations or current AHRQ processes for selection.
4 Similarly, NQF uses factors such as "affects large numbers, leading cause of morbidity/mortality, high resource use (current and/or future),
severity of illness, and patient/societal consequences of poor quality" in determining the importance of a measure for endorsement (NQF,
5 Obesity-related measures include ones addressing whether adults with obesity ever received advice from a health provider to exercise more,
or whether children received advice from a health provider about healthful eating or being physically active. Substance abuse measurement
relates to the number of persons age 12 years and over who needed treatment for illicit drug use and received such treatment at a specialty facility
in the past 12 months.
6 Personal communication, Ernest Moy, Agency for Healthcare Research and Quality, October 9, 2009.
Return to Contents
Proceed to Next Section