Future Directions for the National Healthcare Quality and Disparities Reports

Chapter 4: Adopting a More Quantitative and Transparent Measure Select (pt. 3)

Finalizing the List of Measures for the National Healthcare Reports

By quantitatively evaluating measures based on their potential value and equity, a hierarchical list should emerge. The relative ranking of measures within each of the framework's eight components of quality care (go to Chapter 3) and across these components will help guide the number of measures chosen within each component and overall. The committee is neither recommending the specific number of measures that should be included in the NHQR, NHDR, or related products (or a number for each of the eight quality of care components of the framework) nor establishing a specific threshold of how large the impact must be for a measure to be adopted by AHRQ.

Nevertheless, it is possible that more measures could emerge (i.e., affirmative answers to each of Criteria D, E, and F) than AHRQ resources can manage. In that instance, the Technical Advisory Subcommittee's expert opinion would need to be employed in setting a threshold within the quantitative rankings to determine which measures should be included just as it would need to be engaged in deciding which measures to include when the answer is affirmative to one or two but not all of the impact criteria (value and equity).

The committee further acknowledges that quantitative techniques do not uniformly apply to all elements of the updated framework. Given that, the Future Directions committee still encourages some representation of each component. However, over time better data may allow more even application of the quantitative methods across the components and it could turn out that some components will have a greater quality improvement impact than others and thus should become areas of greater focus within the national healthcare reports.

Relationship Among Priority Areas, Framework, and Measure Selection

Priority setting, use of the framework, and measure ranking are sequential steps in determining the final selection of measures to be reported in the NHQR and NHDR. First, AHRQ and the Technical Advisory Subcommittee should identify metrics that are relevant to priority areas. They should explicitly explore each component of the framework when looking for measures related to each priority area. Then, within each component of quality, they should assess the value and equity contribution of closing the gap between current performance and desired levels for each measure. Finally, they should select measures for reporting that have the highest relative impact on value and equity.

To further clarify the relationships, Figure 4-3 illustrates how specific measures might be aligned with individual national priority areas and categorized by the framework components. Although the names of some priority areas and components are similar or the same, the roles for priority areas and framework components in this model are distinct.

Figure 4-3a depicts a scenario in which three potential measures are identified from the priority area of improving safety; each of these measures is also categorized into the framework component of safety. Each safety measure then undergoes the rigors of the evaluation process outlined above; safety measure number 1 is chosen for inclusion, whereas the fate of the second and third measures depends on the relative ranking of each compared to other safety measures. In addition to the example in the figure, AHRQ and the Technical Advisory Subcommittee should explore whether there are safety measures in other framework components (in addition to the safety component) (e.g., are there measures of timeliness that are relevant to the priority area of safety?). In this way, the framework serves as a check on the comprehensiveness and robustness of the measures being proposed for the identified priority area.

Figure 4-3b depicts a scenario in which measures relate to the priority area of improving population health through different components of the framework. Population health measure number 1 is an access measure and should be compared against other access measures for initial ranking. Similarly, population health measures numbers 2 and 3 should be assessed against other measures categorized as fitting the framework components, patient/family-centeredness and timeliness, respectively. When sufficient quantitative data (e.g., CPB, net health benefit) are available to compare the highest ranking measures from each component with other components' measures, then another ranking step could be taken.

Quantitative Tools For Prioritizing Measures

Techniques for assessing and describing relative degree of value and equity/inequity among performance measures and their utilization in prioritizing performance measures for inclusion in the NHQR and NHDR are considered in the discussion that follows. Similarly, the Phase 1 report for Healthy People 2020 indicated that as communities seek to prioritize their own local objectives for improvement, they should try to use more quantitative techniques in the process (Secretary's Advisory Committee on National Health Promotion and Disease Prevention Objectives for 2020, 2008a). Box 4-3 defines types of quantification approaches that can be applied to prioritizing measures.

Box 4-3. Healthy People 2020: An Explanation of the Prioritization Criteria Used for Sorting Healthy People Objectives

Overall burden. The burden of a disease is a numerical description of the health impact of disease and injury at the population level. Burden can be measured in terms of the number of deaths in a population, or the number of existing cases in a population. A summary measure, or index, of population health can also be used. The quality-adjusted life year (QALY) is a summary measure that is commonly used to describe burden. It is a measure of years of life lived (or years of life gained due to an intervention), which has been statistically adjusted to take quality of life into account.

Preventable or reducible burden. This is an estimate, based on best available evidence, of the degree to which a particular disease and its overall burden can be prevented. Decision makers at multiple levels can use this information to decide which clinical preventive services matter the most, so that they can prioritize their actions. For example, preventable clinical burden can be calculated to include the cumulative effect of delivering a service multiple times at recommended intervals over a recommended age range, instead of delivering the service at a single point in time to one large sample of individuals (Barclay and Lie, 2006).

A variety of approaches can be considered to determine the preventability of disease burden. For example, one could look at the burden of death and disability that can be avoided through means such as: vaccination, early diagnosis, timely and adequate medical treatment, application of hygienic measures, environmental sanitation, implementation of policy change (e.g., increased tax on alcohol products), or health education usually coupled with other actions.

Cost-effectiveness. Cost-effectiveness analysis is used to evaluate the outcomes and costs of interventions that are designed to improve health. It has been used to compare costs and years of life gained for interventions such as screening for breast cancer and vaccinating against pneumococcal pneumonia (Russell et al., 1996). The outcomes are usually not assigned monetary values, as is the case in cost-benefit analysis (Sarvela and McDermott, 1993). Instead, results are typically summarized in terms of ratios that show the cost of achieving a unit of health outcome (such as the cost per year of life, QALY gained) for different types of patients or populations and different types of interventions (Russell et al., 1996). The purpose of analyzing the cost-effectiveness of interventions is to examine the tradeoffs, or "opportunity costs," of making various choices.

Several concerns have been raised about use of cost-effectiveness analysis for setting priorities. These include the difficulties of: measuring quality of life; developing valid summary measures of population health over the life course; generalizing results to different settings; accounting for the fact that programs work synergistically (thereby making it difficult to isolate the effects of one intervention); and addressing "uncertainty" and lack of information about the cost-effectiveness of many potential interventions (Russell et al., 1996).

Effectiveness measures compose about 80 percent of the measures in the 2008 quality report; others have similarly found the measurement domain of effectiveness the most "metric-saturated" (Romano, 2009). This is the type of measure to which the quantitative evaluation measures might be most easily applied. The committee emphasizes that performance measures for which these quantitative evaluation techniques cannot be applied should not necessarily be removed from consideration or reporting in the NHQR and NHDR.

Tools for Assessing Value

Prioritizing among performance measures involves assessing the relative value of a measure's associated intervention to other interventions on the basis of evidence and data that can provide a quantitative ranking. A number of metrics measure different aspects of value (e.g., cost, quality, or impact on health outcomes). The following discussion reviews these available metrics for their use in prioritizing among measures and presents two different strategies that use these tools to highlight health care interventions that may yield the greatest impact for quality improvement.

Despite the validity of these concerns, they need not prevent the use of cost-effectiveness analysis to inform decision making. For example, uncertainty about the cost-effectiveness of an intervention does not necessarily mean that the intervention should not be implemented. Information about the probable costs of an intervention, as well as the likelihood that it will be effective can be taken into consideration in calculating an estimate of its expected cost-effectiveness.

To help users make decisions based on the best information available, Healthy People 2020 should provide data on the degree of confidence concerning these key factors. For example, in the case of burden, Healthy People 2020 should provide quantitative estimates of uncertainty (i.e., information about the reliability of the estimate based on current evidence), as well as qualitative information that could influence uncertainty, (e.g., factors such as the estimate of current burden).

In the face of substantial uncertainty, users will need to make decisions based on incomplete information. Presenting the best available information can permit informed decision-making. In some cases, effects can be quantified by drawing on statistical, epidemiological, economic, or other quantitative methods. Sensitivity analysis (a technique for assessing the extent to which changed assumptions or inputs will affect the ranking of alternatives) may be used (HHS, 2009a) (e.g., how the life expectancy gains of cancer surgery change as the rate of surgical mortality changes).

Value of information (VOI) analysis could also be used to determine when collecting more information on uncertain factors could be worth the cost of generating that information. In other cases, more qualitative approaches to decision-making under uncertainty will need to be used.

Net health benefit. A program's net health benefit is the difference between the health benefit achieved by a program, and the amount of health gain that would be needed to justify the program's costs. If resources are spent on one program instead of another one that would create a higher net health benefit, an opportunity for greater net gains in health is lost. The difference between the net health benefit of two different interventions is the cost of choosing to spend resources on the "wrong" program. Thus, net health benefit is different from cost-effectiveness in that it looks more explicitly at the "opportunity costs" of investing in programs of lesser net value (Hauck et al., 2004).

Reduced health inequities. Some have noted that health inequities can be reduced by diminishing the health status of those who are better off. Healthy People 2020 should be explicit about the need to focus on improving the health status of those who are worse off. Because minority populations in the United States often have worse health status than the general population, this principle specifies the need to improve the health of these groups.

It must also be acknowledged that data-based criteria for priorities could disadvantage population groups with limited data or limited tests of interventions. Lack of complete data about these population sub-groups should not justify a lack of action aimed at reducing disparities. Improving the data on the needs of these groups and intervention effectiveness for these groups should be a priority.

Selecting Measures with the Potential for the Greatest Health Impact

Quality-adjusted life years (QALYs) are the most widely used metric for quantifying the impact on health of health care interventions. QALYS can play a role in identifying areas where quality improvement interventions could have the greatest health impact. The use of QALYs as a value metric is rooted in the assumption that people value additional years of life spent in better health than they otherwise would have enjoyed without the application of some clinical intervention. QALYs have been derived for many clinical preventive services and for some commonly used diagnostic tests and therapeutic procedures. They have been identified as the best standardized measures of health effectiveness because of their "widespread use, flexibility, and relative simplicity" (IOM, 2006, p. 10). Life years can be estimated based on absolute risk reduction from clinical trials, and QALYS can be obtained directly from participants in clinical trials or estimated based on published quality-of-life data for various conditions. A similar construct, disability-adjusted life years (DALYs), is often used by the World Health Organization in international studies (Gold et al., 2002).

When beneficial clinical interventions are applied to medically affected populations, the result in health benefit (measured as the total QALYs saved based on the number of persons affected by the intervention) is referred to as CPB (Maciosek et al., 2009). CPB is the health burden that is prevented or averted by a clinical intervention; it represents the absolute risk reduction from the intervention that can then be generalized to the relevant population (e.g., nation as a whole).

Conceptually, it does not matter whether CPB results from improved use of a proven intervention (e.g., influenza vaccination, which is one of AHRQ's effectiveness measures) or from reduction in harm to patients through improvement care processes (e.g., reduction of adverse drug events, which is one of AHRQ's safety measures). In either case, an improvement in health, measurable in QALYs saved, has been achieved.

CPB is relevant to prioritizing quality measures based on its ability to quantify the health impact of a measure's associated clinical intervention. Therefore, CPB provides a means for comparisons across different clinical interventions (e.g., mammography versus maintenance-phase medications for depression), facilitating prioritization of measures of those clinical interventions. Additionally, estimates of health impact can be used to compare measures either for the overall population or for subpopulations (in the context of assessing disparities).

Selecting Measures That Target the Most Effective Use of Health Care Resources

The high (and growing) cost of health care in the United States is pushing cost considerations to the forefront of the political agenda (Davis, 2008; Fisher et al., 2009). Cost-effectiveness analysis (CEA) is perhaps the most widely used method for considering cost in the context of health gain from medical care (Gold et al., 1996). In its most complete form, CEA "measures net cost per QALY saved [using a clinical intervention], for which net costs equal the cost of the intervention minus any downstream financial savings" (Maciosek et al., 2009, p. 350). CEA facilitates comparisons across interventions by providing a common metric for comparing costs across different interventions or activities, thus informing allocation decisions designed to maximize health (measured by QALYs) within confined resources (Gold et al., 1996; Neumann et al., 2008; Wong et al., 2009).

There have been calls for explicit consideration of CEA in the prioritization of quality measures and health care policy (Maciosek et al., 2009; Neumann et al., 2008; Siu et al., 1992; Wong et al., 2009; Woolf, 2009). These recommendations are supported by a burgeoning literature on the cost-effectiveness of several clinical preventive services and certain diagnostic testing and therapies (e.g., surgical and other procedures, devices, drugs and behavioral interventions), including the establishment of a searchable registry for CEA (Center for the Evaluation of Value and Risk in Health, 2009; NIHR Centre for Reviews and Dissemination, 2009). Most of the preventive and diagnostic services or interventions for which CEA data may be available fall within AHRQ's framework component of effectiveness measures (Bentley et al., 2008; Hurley et al., 2009); less is known about the costeffectiveness of clinical interventions in the safety or timeliness components, but there are some examples (Barlow et al., 2007; Furuno et al., 2008; Rothberg et al., 2005; van Hulst et al., 2002). Data permitting, CEA could play a role in selecting and prioritizing quality measures for a number of framework components.

The committee recognizes that there has been some resistance to using CEA for health care improvement. One criticism relates to the potential for bias in the conduct of CEA. For example, CEAs conducted by industry (e.g., health plans, pharmaceutical companies) frequently provide quite favorable results (Bell et al., 2006). Too often, CEA data follow rather than precede release of an intervention or technology into practice, limiting their usefulness at the time of its implementation (Greenberg et al., 2004). Furthermore, few CEAs report actual costs of implementing the intervention into routine care (Neumann et al., 2008), but instead focus largely on the cost of the intervention itself. Finally, ethical questions have been raised in terms of the impact of CEA on different populations, such as the elderly or disabled. Strict application of CEA to interventions designed to improve quality of life among the dying might yield results suggesting that minimal additional QALYs might not outweigh the costs.

These issues are potentially addressable (Neumann et al., 2008). For example, CEAs could employ standard and transparent methods, which may require some public financing so that they are not solely conducted by entities with a business interest in the result. Further, ethical considerations can be accounted for by incorporating balance and equity into policy decisions in conjunction with CEA, which is consistent with this committee's broader definition of health care value (go to Chapter 3).

CEA represents one approach to formal, evidence-based comparisons of interventions that account for tradeoffs in costs and health benefits. These analyses could help track an important aspect of health care value and target the selection of measures that promote optimal health outcomes (e.g., QALYs, mortality rates, life expectancy).

Prioritizing Measures with High Health Impact and Effective Resource Use

To identify measures with the greatest potential value, particularly related to clinical effectiveness measures, the committee examined two strategies that employ health impact analysis and cost-effectiveness analysis. Without endorsing any specific strategy or methodology, the committee believes that the discussion below provides examples of ways in which AHRQ could select high-value, prioritized measures for performance reporting.

An Approach with Separate and Combined Clinically Preventable Burden and Cost-Effectiveness Rankings

Measurement of health impact in terms of both CPB and cost-effectiveness (CE) can be used to determine which among a given list of preventive measures has the greatest potential for quality improvement. In one example of this approach, Maciosek and colleagues examined a list of measures based on health care services interventions recommended by the U.S. Preventive Services Task Force (USPSTF). (Detailed methods for these calculations and additional information on the results are published elsewhere [Maciosek et al., 2006a,b]). CE and CPB calculations were used as the criteria to assess the relative value of each service. CPB was defined as "the total QALYS that could be gained if the clinical preventive service was delivered at recommended intervals" to a designated cohort; that is, total QALYs were compared between 100 percent of patients being advised to use or consider the intervention, and no use at all. CE was defined as "the average net cost per QALY gained in typical practice by offering the clinical preventive service at recommended intervals to a U.S. birth cohort over the recommended age range" (Maciosek et al., 2006a, pp. 53-54) (i.e., net cost of the intervention divided by the QALYs saved).

Once calculations for health impact and CE were completed for each service, analysts ranked the calculations by scoring them on a scale of 1 to 5, with 5 being the best score (i.e., the highest estimates for health impact, and the lowest cost-effectiveness ratio for CE). This quintile scale was created to rank the calculated estimates of CPB and CE without overstating the precision of the individual estimates. An overall score was then derived by adding the CPB and CE scores together, conveying services of greatest value within a given set. Table 4-1 depicts these individual and combined scores with the ultimate ranking of clinical preventive services.

Although the calculations for CE in the study by Maciosek and colleagues effectively included CPB (as the denominator of the equation), presenting CE and CPB separately allows decision-makers to consider both criteria either simultaneously or in isolation. This separation of factors may be useful when a measure's associated intervention ranks low in cost-effectiveness yet has a significantly high health impact, which decision-makers may value more and thus give the measurement area a higher priority. Measures and associated interventions that rank lower in a prioritization scheme should be assumed to retain value to some stakeholders or regions who may want to continue to invest in tracking or improvement activities in those areas. Although the Maciosek study was specific to preventive services, the same methods can be applied to rank the value of other types of health care services (i.e., acute treatment, chronic condition management) as long as there is enough information to perform the calculations.

A Net Health Benefit Approach

Another approach to prioritizing measures is based on the concept of net health benefits (Stinnett and Mullahy, 1998). This approach is used to quantify the potential value of quality improvement for a given measure by estimating the incremental health benefit gained by a clinical standard of care net of its incremental costs: "the difference between the health benefit achieved by a program, and the amount of health gain that would be needed to justify the program's cost" (Hauck et al., 2004, p. 85; Secretary's Advisory Committee on National Health Promotion and Disease Prevention Objectives for 2020, 2008a).

This approach assumes that measures are defined with reference to some standard of care, that the benefits of implementation are measureable in terms of QALYs (or a similar metric of health benefit) on the basis of clinical evidence or consensus, and that the standard of care pertains to clinical quality, patient safety, organizational characteristics, utilization, or aspects of patient-provider relationships. The logic is as follows�if the costs and health benefits of standard-concordant care are known, and the costs and health benefits of non-standard-concordant care are also known, then the net health benefit (NHB) of the standard (the measure) can be calculated�the result being the population health benefits net of cost. As a result, different clinical interventions can be compared to see which are most productive.

Tengs and Graham (1996) illustrate how spending could be directed to clinical interventions with the potential for the greatest return. They examined the costs and benefits of 185 interventions, finding that the United States spent about $21.4 billion on these lifesaving interventions, averting 56,700 premature deaths and, in doing so, saving 592,000 life years. However, a smaller amount of funds could have been better allocated to minimize premature deaths and maximize life years to save an additional 595,000 life years.

Although cost-effectiveness estimates (measured in QALYs) are used in this method, they are only a part of the total calculation. In addition to comparing the costs and effectiveness of a standard of care, the net health benefit for a standard of care takes into account society's willingness to pay for an additional unit of health benefit (as measured by QALYs). Knowing the societal cost-effectiveness threshold allows for the calculation of opportunity costs for achieving the desired standard of care. Thus, a net health benefit calculation derives the actual costs and opportunity cost of accomplishing a standard of care if an intervention were fully implemented to maximize its benefit. This, in turn, allows one to calculate the expected population value of improving the performance rate of a measure for a given clinical intervention to 100 percent.

In Appendix F, a commissioned paper by David Meltzer and Jeanette Chung provides an illustrative analysis of Pap smears and estimates that 405,999 life years would be gained if every 18-year-old female received triennial screening (while current actual rates of screening yield 293,351 life years). Thus, the value of quality improvement would be the difference between perfect and actual implementation: 112,648 life years lost.

Meltzer and Chung's paper explores the net health benefit methods and their theoretical applicability to 14 NHQR measures that span different framework components. The strategy can be used to estimate the potential value of improving performance on existing quality measures, which can then be used to prioritize measures for reporting. Meltzer and Chung examine the applicability of these techniques for process measures with an associated standard of care, composite process of care measures, and incidence rates of complications (e.g., foreign body left in during a procedure per 1,000 hospital discharges). While the technique is well suited to analyze process measures, it is difficult to use for composite process measures or for most outcome measures because no specific treatment or intervention is defined. The issues with each of these measure types are discussed in more depth in their paper.

Limitations of These Strategies

While both of the approaches discussed above are useful for informing decision-makers of where to invest resources to improve health care, they have important limitations. First, these methods for prioritization do not include any equity or disparities considerations for specific priority population groups. It is conceivable, however, that CPB and CE estimates could be calculated for specific population groups if the necessary data were available; a few studies on the economic impact of disparities have recently been released (LaVeist et al., 2009; Waidmann, 2009). Second, the information necessary to compute CE and health impact calculations may not be readily available; it is rarely the case that analysts have all of the necessary information to do these estimates and must consequently make assumptions. These assumptions should be clearly identified, and sensitivity analyses should be used to examine the effect of assumptions on results. In the absence of data from the peer-reviewed literature, the assumptions should be guided by expert opinion and the gray literature.

A third limitation, and an important one given the multidimensional aspect of health care value, is that the above-discussed approaches for prioritization are not readily applicable to all measures given that the calculation rests on quantifiable standards of information (e.g., financial cost, QALYs). The approaches apply primarily to clinical effectiveness measures and, to some extent, to safety and efficiency measures when a health care service or intervention has been identified to improve health outcomes with known costs. Yet there are measures reported in the NHQR and NHDR�some access, timeliness, and patient-centeredness measures�for which underlying interventions or processes are not easily tied to monetary or life duration factors. For example, the health impact of patient perceptions of care that promotes informed patient decision-making or alleviates suffering at the endof- life is not easily translated to QALYs. Measures without an easily quantifiable impact arguably represent important and desirable ends in themselves, apart from any demonstrable effect on health. For these measures, alternative means are needed to weigh the relative impact of gaps or disparities. This might be achieved through formal assessment of the relative value, or ranking of the health care processes captured in qualitative dimensions by consumers. Such rankings could facilitate prioritization if coupled with consideration of the gap or disparity in performance and the size of the population affected by the gap. Although this approach would not allow direct comparison with CPB or net health benefit, it would help facilitate prioritization among measures falling within a particular quality component of the framework.

The framework components of care coordination and health systems infrastructure capabilities were not assessed using these strategies because measures for these components were not presented in the latest edition of the national healthcare reports. Chapter 3 referenced some studies that indicated potential cost-effectiveness using care coordination and implementing Health IT. However, the evidence base for such interventions on improving the quality of care would need to be further examined to evaluate the applicability of these prioritization strategies to them.

Finally, the resources required to discover, collect, and collate the data needed for these prioritization approaches, along with the human capital to perform the computation and analysis are substantial. Depending on the data available, a thorough search of the literature and calculations for a single measure will require a considerable amount of dedicated time. If the NAC Technical Advisory Subcommittee for Measure Selection and AHRQ were to use such prioritization approaches, which this committee strongly recommends, appropriate resources to support this effort would be required. The Phase 1 report on Healthy People 2020 suggests that communities use similar techniques to prioritize their objectives and that support be given to communities in terms of technical support materials to make this possible. There would be synergy in AHRQ and CDC partnering to advance these more quantitative approaches to prioritization as well as partnerships with other public or private entities utilizing these techniques.

Return to Contents
Proceed to Next Section

Page last reviewed October 2014
Page originally created September 2012
Internet Citation: Chapter 4: Adopting a More Quantitative and Transparent Measure Select (pt. 3). Content last reviewed October 2014. Agency for Healthcare Research and Quality, Rockville, MD. http://www.ahrq.gov/research/findings/final-reports/iomqrdrreport/futureqrdr4b.html