Page 1 of 1

Appendix F: The Expected Population Value of Quality Indicator Reporti

Future Directions for the National Healthcare Quality and Disparities

Appendix F. The Expected Population Value of Quality Indicator Reporting (EPV-QIR): A Framework for Prioritizing Healthcare Performance Measurement

Appendix A: Incorporating Inappropriate Use into the EPV-QIR Framework

Incorporating inappropriate use or overuse into the EPV-QIR framework requires adjustments to the framework.

If overuse is defined as excessive use of an intervention in a measure population, the primary adjustment is that the NHB of current use can be negative compared to no use. Therefore, the population value of current implementation (PVCI), needs to be calculated over the parts of the measure population receiving appropriate and excessive levels of use, and can be negative if the extent and negative net health benefits of overuse exceed the positive net health benefits produced by overuse. Because of the potential to increase overuse, the maximum population value of quality improvement (MaxPVQI) with overuse will exceed that if overuse did not exist.

Example: Cervical cancer screening provides a good example of how overuse might be addressed in this framework. The NHQR process measure for assessing the effectiveness of cervical cancer screening is defined as "the percent of women age 18 and older who report they had a Papaniculaou (Pap) smear within the past 3 years." However, women often receive Pap smears annually rather than triennially although some estimates suggest that more frequent screening is not cost-effective. Below, we demonstrate our calculations of the maximum value of quality improvement that may be attained for a cohort of 18-year-old women.

According to data from the 2005 Behavioral Risk Factor Surveillance System (BRFSS), 94% of women reported ever having a Pap smear. Of those women, 64% reported having a Pap smear less than one year ago; 16.3% reported having a Pap smear between 1 and 2 years ago; 5.9% reported a Pap smear between 2 and 3 years ago; and 13.6% reported having a Pap smear 3 or more years ago. We estimate the rate of women having a Pap smear within 3 years as (0.94) * (0.862) = 0.81.

According to the U.S. Census Bureau, the projected population of women in the 18-24 year age group in the U.S. in 2010 is 15,037,000. We will use 2,148,143 as a rough estimate of the number of women in the 18-year-old cohort in 2010.

Meltzer (unpublished research) has previously estimated that triennial screening increases life expectancy by 70 days or 0.19178 life years (LY) compared to no screening, at a cost of $500. Biennial screening increases life expectancy by 71 days or adds an additional 0.1945LY compared to no screening at an incremental cost of $750. Annual screening increases life expectancy by 0.1954LY compared to no screening at an incremental cost of $1,500. Assuming λ = $50,000, the net health benefit of triennial screening vs. alternative standards are:

NHB3vsNo = 0.19LY - $500/$50,000 = 0.189LY

NHB2vsNo = 0.1945LY - $750/$50,000 = 0.179LY

NHB1vsNo = 0.1954LY - $1,500/$50,000 = 0.1654LY

Value of Perfect Implementation. If all 18-year-olds received triennial screening, VPI3 = 2,148,143 X 0.189LY = 405,999LY could be gained.

Using screening rates above, we estimate that 60% of 18-year-olds will receive annual screening, 15% will receive biennial screening, 5.5% will receive triennial screening, and we will assume the rest (20%) will be treated as not screened. Thus the value of current implementation (VCI) consists of the sum of value over these three groups:

VCI1 = 1,288,886 X 0.1654LY = 213,182LY

VCI2 = 322,221 X 0.1795LY = 57,839LY

VCI3 = 118,148 X 0.189LY = 22,330LY

The total value of current implementation is the sum of these values: 293,351LY.

The maximum value of quality improvement in this cohort is equal to 405,999LY � 293,351LY = 112,648LY. Note that this is more than simply implementing triennial screening among the 20% of the cohort who are not screened. By reducing underuse among 20%, and moving that subgroup to triennial screening, one would achieve a maximum of (429,629 women) * (0.189LY) = 81, 200 LY. Adding this number to current implementation would equal 374,551LY, which is still 31,448LY short of the maximum potential value achievable in this cohort. The reason for this is that screening more frequently than every 3 years results in negative net health benefits, and such frequent screening constitutes overuse. In this example, if all women in the cohort receiving annual screening received triennial screening instead, 30,418LY could be saved. Similarly, if all women receiving biennial screening received triennial screening instead, 3,061LY could be saved. This example demonstrates the flexibility of our general framework in being able to address quality improvement that reduces underuse, as well as quality improvement that reduces overuse.

Inappropriate Use. If inappropriate use is defined as use in a population in which the NHB is negative, then the EPV-QIR framework needs to add in the population value of the harms arising from that inappropriate use, which would be calculated by multiplying the number of persons in the population experiencing inappropriate use and multiplying it times the per-capita negative NHB in that population. This is easily generalized to multiple populations in which inappropriate use can occur, each of which might have different levels of negative NHB. As with overuse, the total benefit of perfect implementation will exceed the benefits of perfect implementation if overuse did not occur. As a result, measures such as the MaxPVQI based only the measure population cannot be used to place upper bounds on the benefits of better use in the entire population, and more detailed calculations that reflect effects in the entire population are needed to establish bounds. Because of the net health benefit framework, an intervention can produce negative net health benefits either because it harms health or because its costs exceed the value of the health benefits it produces. As a result, bounds developed would need to reflect potential effects on both health outcomes and costs.

Return to Appendix F

Appendix B: Sample EPV-QIR Calculations

Calculation 1. A Basic Example. Percent of Individuals Age 65+ Who Ever Received a Pneumococcal Vaccination [NHQR CAPVACC65EVER]

Standard of Care. The standard of care in this measure is receipt of at least one pneumococcal vaccination among individuals age 65+. This measure essentially implies at least a one-time vaccination at age 65.

Number of Individuals Receiving Standard of Care and Non-Standard Care. We used data from the U.S. Census Bureau to obtain estimates of the U.S. population age 65+ in 2008. Table A.1.1 provides the distribution of individuals age 65+ in the U.S. by receipt of standard/non-standard care.

Table A.1.1 Number of Individuals Receiving Standard and Nonstandard Care: CAPVACC65EVER

ParameterEstimateSource
Base Population38,869,716U.S. Census Bureau
Current Implementation Rate57%NHQR 2008 (2006)
N Receiving Standard of Care22,272,347 
N Not Receiving Standard of Care16,597,369 

Source: [Base Population] Table 1: Annual Estimates of the Resident Population by Sex and Five-Year Age Groups for the United States: April 1, 2000 to July 1, 2008 (NCEST2008-01). Source: Population Division, U.S. Census Bureau. Release Date: May 14, 2009.

Calculation of Net Health Benefit. Table A.1.2 presents the inputs into the calculation of net health benefit (NHB) associated with one-time vaccination at age 65, and no vaccination. Estimates of the costs and effectiveness of one-time pneumococcal vaccination at age 65 come from Sisk et al. (1997). This study was chosen because it was conducted in the U.S. population age 65+, and because it evaluated one-time vaccination at age 65. This study was also chosen because estimates of costs and QALYs were provided in the published article for both the standard of care and the comparator (no vaccination). Column 1 shows the vaccination category. Column 2 shows the cost per person in 2009 U.S. dollars, and Column 3 shows the QALYs gained per person under no vaccination and under one-time vaccination at age 65. Columns 4 and 5 show the incremental cost and incremental effectiveness of one-time vaccination at age 65 compared to no vaccination. Column 6 shows the cost-effectiveness threshold value used in this analysis, which is $100,000 per QALY. Column 7 shows the final NHB associated with one-time vaccination at age 65.

Table A.1.2 Calculation of Net Health Benefits: CAPVACC65EVER

Care TypeCost per Person
in 2009 $USD
Outcomes
(QALYs per Person)
Incremental Cost Effective-
ness in QALYs
λ ($/QALY)Net Health Benefit in QALYs
No Vaccination1477.88752  100,000 
Vaccination at Age 651377.89157-100.00405100,0000.00415

Incremental cost and effectiveness are relative to baseline care type of no vaccination.
Source: Sisk JE, Moskowitz AJ, Whang W, Lin JD, Fedson DS, McBean AM, Plouffe JF, Cetron MS, Butler JC. Cost-effectiveness of vaccination against pneumococcal bacteremia among elderly people. JAMA. 1997;278:1333-39.

EVQI Calculations. Table A.1.3 displays our calculations for the population value of perfect implementation, population value of current implementation, and maximum population value of quality improvement for the CAPVACC65EVER measure. Columns 2-4 display the NHB associated with the standard of care and non-standard care (no vaccination), the NHB associated with perfect implementation, and the number of persons age 65+ in the U.S. who currently receive non-standard care and the number receiving the standard of care. The VPI is given in Column 5, and is calculated by multiplying the NHB in Column 2 by the number of persons in Column 4. The sum of PVPI ("Total") is equal to the total number of QALYs that can be achieved if all individuals age 65+ received one-time pneumococcal vaccination at age 65. The VPI for CAPVACC65EVER is 529,704 QALYs. The PVCI is calculated in Column 6 by multiplying the NHB associated with perfect implementation (Column 3) by the number of individuals in Column 4. The total PVCI is the total number of QALYs that is currently achieved given current patterns of vaccination in this population. The PVCI of CAPVACC65EVER is 241,545 QALYs. The gap between perfect implementation and current implementation is quantified by the maximum value of quality improvement (VQI) in Column 7. The MaxPVQI is simply the difference between PVPI and PVCI, and represents the upper bound on the number of additional QALYs that can be gained by improving quality of care with respect to CAPVACC65EVER. If an intervention that was 100% effective in moving provider behavior toward consistent performance of the standard of care could be costlessly implemented, 288,159 QALYs could be gained.

Table A.1.3 The Value of Perfect and Current Implementation, and Quality Improvement: CAPVACC65EVER

Care TypeNet Health Benefit in QALYsNHB Under Perfect Implementation QALYsN PersonsPopulation Value of Perfect Implementation
(VPI) QALYs
Population Value of Current Implementation
(VCI) QALYs
Maximum Population
Value of Quality
Improvement QALYs
No Vaccination0.000000.0041516,597,36968,871068,871
Vaccination at Age 650.004150.0041522,272,34792,42092,4200
Total  38,869,716161,29192,42068,871

In the case of CAPVACC65EVER, 54% of individuals age 65+ report ever having received pneumococcal vaccination. However, this rate of implementation has yielded only a 45.6% net health benefit achievement rate.

Limitations. In our calculations for CAPVACC65EVER, we assumed that individuals would be compliant with the measure if they ever had a pneumococcal vaccination from age 65 onwards. As currently defined, however, individuals aged 65+ who received vaccination before the age of 65 could be considered to have received treatment compliant with the measure. This measure may benefit from additional clarification of the numerator and denominator definitions.

Calculation 2. The Multiple Standard/Multiple Comparator (MSMC) Problem. Percent of Women Age 50+ Who Report They Had a Mammogram within the Past 2 Years [NHQR BRCA2YRMAMM]

Standard of Care. The standard of care in this measure is lifetime annual or biennial screening mammography among women beginning at age 40.

Multiple Standards. In the case of BRCA2YRMAMM, two screening strategies can be considered to meet the standard of care: annual mammography and biennial mammography. Separate estimates of NHB should be used for annual mammography and biennial mammography because the total costs of annual and biennial screening over a woman's lifetime (past age 40) must surely be different, even if there may be less difference in clinical effectiveness.

Multiple Comparators. Women who do not receive annual or biennial mammography beginning at age 40 cannot easily be grouped into a single "no screening within 2 years" category. Women who receive non-standard care may never receive screening. Or, they may be screened at intervals other than every one or two years; they may begin screening at an age other than 40 years; and/or they may terminate screening at some point in their lives. All of these alternative screening strategies, including the absence of screening, are non-compliant with the standard of care in the BRCA2YRMAMM measure.

Simplifying Assumptions. For our calculations, we assumed that women age 40+ could be classified into the following mutually exclusive screening categories:

  1. Never received mammography.
  2. Annual mammography.
  3. Biennial mammography.
  4. Triennial mammography.
  5. Mammography at 4-year or greater intervals.

As specified, women receiving annual mammography or biennial mammography are said to be in compliance with the standard of care in the NHQR measure.

Number of Women Receiving Standard of Care and Non-Standard Care. We used data from the 2005 Behavioral Risk Factor Surveillance Survey (BRFSS) to obtain weighted estimates of the number of women age 40+ in the U.S. population who received a mammogram within the past 2 years.3 Table A.2.1 provides the distribution of women age 40+ in the U.S. by screening interval.

Table A.2.1 Numbers of Women Receiving Standard and Non-standard Care, and Screening Strategy Used for NHB Estimates: BRCA2YRMAMM

Care TypeNumber of Women
Receiving Standard
of Care (NS)
Number of Women
NOT Receiving
Standard of Care
(NSN)
Corresponding Screening
Strategy from
Stout et al. (2006)
Annual Mammogram5,838,335 Begin age 40, end age 80, 1 y interval
Biennial Mammogram1,586,308 Begin age 40, end age 80, 2 y interval
Triennial Mammogram 585,818Begin age 45, end age 75, 3 y interval
Mammogram Interval between 3 and 5 yrs 378,626Begin age 55, end age 70, 5 y interval
Mammogram Interval 5 yrs or greater 538,696Begin age 55, end age 70, 5 y interval
Don't Know 61,398No screening
Refused 1,340No screening
Total7,424,6431,565,878 

Sources: 2006 BRFSS.
Stout NK, Rosenberg MA, Trentham-Dietz A, Smith MA, Robinson SM, Fryback DG. Retrospective cost-effectiveness analysis of screening mammography. J Natl Cancer Inst 2006;98(11):774-82.

Net Health Benefits. We obtained estimates of costs and effectiveness for each of the five screening strategies from Stout et al. (2006). This study was chosen because of its relative recency, the availability of cost and effectiveness estimates in dollars and QALYs, and because both standards of care (annual and biennial mammography) were evaluated. However, screening strategies in Stout et al. (2006) were not perfectly congruent with implied standards in BRCA2YRMAMM. Whereas BRCA2YRMAMM suggests annual or biennial mammography beginning at age 40 and continuing for the rest of a woman's lifetime, Stout et al. evaluated annual and biennial mammography beginning at age 40 and ending at age 80. Estimates of costs and effectiveness for comparator screening strategies can also be obtained from Stout et al. For triennial mammography, we use estimates of costs and effectiveness associated with triennial screening beginning at age 45 and ending at age 75. For intervals greater than 3 years, we use estimates of costs and effectiveness associated with screening every 5 years beginning at age 55 and ending at age 70. Table A.2.1 provides a crosswalk between observed screening intervals in the BRFSS data, and the screening strategies corresponding to the NHB estimates from Stout et al. (2006) that we use in our calculations.

Calculation of Net Health Benefit. Table A.2.2 shows data elements and end-calculation of net health benefits for each screening interval or "screening policy." The first column shows the cost per woman in 2009 U.S. dollars for each screening standard and comparator.4 The second column shows the effectiveness of each screening standard and comparator in QALYs. The third and fourth columns present the incremental cost and incremental effectiveness of each screening modality relative to a baseline of "no screening." The fifth column contains the cost-effectiveness threshold value that we used in our analysis, which we assumed to be $100,000. This value is roughly equal to the inflation-adjusted 2009 dollar value corresponding to $50,000 in 1982 $USD5. The sixth column presents the net health benefit calculated for each screening modality.

As evident in Column 6, the NHBs across screening modalities can differ substantially. In our example, the NHB of mammography ranges from 0.005 QALYs per woman following a 5-year screening strategy, to 0.02039 QALYs per woman following an annual screening strategy. Indeed, the NHB accruing to annual screening is 1.25 times larger than the NHB accruing to biennial screening, and both annual and biennial screening are considered to be compliant with BRCA2YRMAMM. This illustrates why it can be problematic to use a single estimate of NHB for multiple standards and multiple comparators.

Table A.2.2 Calculation of Net Health Benefits: BRCA2YRMAMM

Care TypeCost per Woman
in 2009 $USD
Outcomes
(QALYs per Person)
Incremental Cost Effective-
ness in QALYs
λ ($/QALY)Net Health Benefit in QALYs
Annual Screening*
(begin age 40, end age 80)
3,3169.99581,9610.04000100,0000.02039
Biennial Screening*
(begin age 40, end age 80)
2,4619.98321,1050.02737100,0000.01632
Triennial Screening
(begin age 45, end age 75)
1,9879.97266320.01684100,0000.01053
5-year Screening
(begin age 55, end age 70)
1,5929.96322370.00737100,0000.00500
No screening1,3559.9558  100,000 

Incremental cost and effectiveness are relative to baseline care type of no screening.
* Care Type compliant with quality measure.
Stout NK, Rosenberg MA, Trentham-Dietz A, Smith MA, Robinson SM, Fryback DG. Retrospective cost-effectiveness analysis of screening mammography. J Natl Cancer Inst. 2006;98(11):774-82.

EPV-QIR Calculations. Table A.2.3 presents our population value of perfect implementation, population value of current implementation, and (maximum) population value of quality improvement calculations. Column 1 presents each screening standard and comparator under consideration; Column 2 presents the NHB associated with each standard/comparator as computed in Table A.2.2, Column 3 presents the NHB associated with the acceptable standards of care (annual, biennial, or an average of the two), and Column 4 presents the number of women currently in each screening category.

Table A.2.3 The Value of Perfect and Current Implementation, and Quality Improvement: BRCA2YRMAMM

Care TypeNet Health Benefit in QALYsNHB Under Perfect Implementation QALYsN Women (40+) in Screening
Category
Population Value of Perfect Implementation
(VPI) QALYs
Population Value of Current Implementation
(VCI) QALYs
Maximum Population
Value of Quality
Improvement QALYs
Annual Screening*
(begin age 40, end age 80)
0.020390.0203939,244,638800,384800,3840
Biennial Screening*
(begin age 40, end age 80)
0.016320.0163210,662,986173,975173,9750
Triennial Screening
(begin age 45, end age 75)
0.010530.018363,937,80372,27941,45130,829
5-year Screening
(begin age 55, end age 70)
0.005000.018366,166,136113,18130,83182,350
No screening0.000000.01836416,9907,65407,654
Total  60,428,5541,167,4741,046,640120,833

* Care Type compliant with quality measure.

As shown in Column 5, the value of perfect implementation (VPI) is the total possible number of QALYs that can be gained by moving all women who are not currently receiving the standards of care, to biennial or annual screening. We take the average of the NHB associated with annual screening and the NHB associated with biennial screening, and then multiply this average by the number of women in each of the comparator screening categories. For women receiving annual or biennial screening, we multiply the number of women in these categories by the NHB associated with annual screening and the NHB associated with biennial screening, respectively. The VPI for the BRCA2YRMAMM measure is the sum of VPI across all screening modalities, and represents the total number of QALYs that could be achieved if all women received mammograms annual or biennially.

Column 6 presents the value of current implementation (VCI), which is the number of QALYs currently achieved given current patterns of mammography. The VCI for each screening strategy is calculated by multiplying the number of women in each screening category by the NHB associated with that screening strategy (go to Table A.2.2). The PVCI for the BRCA2YRMAMM is the sum of PVCI across all screening modalities and represents the total number of QALYs currently achieved under existing practice.

Column 7 shows the maximum potential value of quality improvement, which is the difference PVPI, PVCI, and PVQI. As discussed in Part I, the maximum potential value of quality improvement represents an upper bound on the QALYs that can be achieved from improving quality of care if an intervention that was 100% effective in changing provider behavior to comply with standards of care were implemented costlessly.

For the BRCA2YRMAMM measure, the maximum potential value of quality improvement (MaxPVQI) is 120,833 QALYs. A total of 1,167,474 QALYs can be achieved if all women age 40+ received annual or biennial screening beginning at age 40 and continuing for the rest of their lives. Given present patterns of screening mammography, 1,046,640 QALYs are being achieved. Although 82% of eligible women are receiving annual or biennial mammography, roughly 90% of total possible QALYs are being achieved.


3 We used the Web-enabled Analysis Tool (WEAT) available at www.cdc.gov/BRFSS/ to cross-tabulate the number of years since a respondent's last mammogram by respondent's age.
4 The original cost data in Stout et al. (2006) were presented for the aggregate cohort of 95 million women studied, and were measured in 2000 $USD. We divided the aggregate cost by 95 million to obtain an estimate of the cost per woman for each screening modality. We then used the Consumer Price Index to adjust for inflation.
5 A common cost-effectiveness threshold is $50,000. However, this value was first used in 1982, and rarely adjusted in studies to account for inflation.


Return to Appendix F
Proceed to Next Section

 

Current as of December 2010
Internet Citation: Appendix F: The Expected Population Value of Quality Indicator Reporti: Future Directions for the National Healthcare Quality and Disparities . December 2010. Agency for Healthcare Research and Quality, Rockville, MD. http://www.ahrq.gov/research/findings/final-reports/iomqrdrreport/futureqrdrapfapab.html