Explaining Comparisons of Health Care Quality Scores

It is best to keep your actual data displays as uncluttered and simple as possible. However, evidence from cognitive testing of the Agency for Healthcare Research and Quality (AHRQ) Quality Indicators (QIs) suggests that there is some information that needs to accompany the data display.[1] Key topics include the following:

  • Explaining why averages are presented.
  • Specifying whether a high or low score is more desirable.
  • Warning against putting too much emphasis on small differences.

Explaining Why Averages Are Presented

Typically, horizontal bar graphs present the average score either on the top of the bars or in the middle of the bars. In some cases, both a State and National average are presented. But many consumers do not immediately understand why a data display includes an indication of an average score. You can help readers by saying explicitly why you're showing the average score and how they can use that information.


EXAMPLE: Explaining Averages in the Context of a Bar Graph

Title: Model Report for AHRQ Quality Indicators
Sponsor: Agency for Healthcare Research and Quality
URL: http://qualityindicators.ahrq.gov/Downloads/Modules/QI_Reporting/Model_Report_Composite.pdf

The developers of a model report for the AHRQ Quality Indicators did cognitive testing of a data display that included a note regarding averages right underneath the bar graph. They found that, when this text was added, people understood why the average was there and how to use it.

Death rate for heart attack patients

This graph shows you the percent of patients admitted to each hospital after having a heart attack (called an acute myocardial infarction), who died during their hospital stay. This information is for patients admitted during [insert year].

When you are choosing a hospital, you should look for the hospital that has a lower number of deaths. A lower number is shown by a shorter bar on the graph below.

Sample bar chart showing the death rate for heart attack patients in 4 hospitals: Hospital A through D. The average death rate of hospitals across the state is 9 out of 100. Hospital A has a rate of 2 per 100; Hospital D 4 per 100, Hospital B 9 per 100, and Hospital C 12 per 100.

Average of hospitals across the state: The average rate of patients who died after having a heart attack, in hospitals across your state. This number is included so you have:

  • a better idea of what is typical for your state.
  • a basis for comparing individual hospitals’ performance.
Source: Agency for Healthcare Research and Quality. Hospital Quality Model Report: Composites. 2009. Available at http://qualityindicators.ahrq.gov/modules/Default.aspx.

Specifying Whether a High or Low Score Is More Desirable

Some measurement sets include items where a high score is more desirable as well as items for which a low score is more desirable. Ideally, all your measures would be in the same direction, but sometimes that is just not possible. When you cannot show all measures in the same direction, you need to indicate clearly whether to look for a high or low score to find the best performer.

The example above shows how that can be done with two simple sentences which have been subjected to rigorous cognitive testing:

When you are choosing a hospital, you should look for the hospital that has a lower number of deaths. A lower number is shown by a shorter bar on the graph below.

Warning Against Putting Too Much Emphasis on Small Differences in Scores

It is important to warn consumers not to pay too much attention to very small differences. For example, if one medical group provided timely mammograms to 83 percent of its patients and another provided them to 85 percent, that’s not much of a difference.

When providing absolute data only with no interpretation (e.g., showing the organizations sorted into “relative” categories), it is important to have a caveat in the data displays or offer an overall caveat for the whole report.

There are two options for the caveat:

  • The first is to simply say “Do not pay attention to small differences in scores.” The problem is that what is “small” to me may be “large” to you.
  • The other option, and one which is also an alternative way to signal “statistically significant” differences, is to specify “Look for differences of n points or more. Smaller differences may not be meaningful.”

 

[1] This testing was done initially to explore how to add data from the CAHPS Hospital Survey to the Center for Medicare & Medicaid Services’ (CMS) Hospital Compare Web site. Consumers were confused by the average bars in the display that CMS used at that time. The approach shown on this page was adopted for the cognitive testing of the AHRQ Quality Indicator Model Reports and eliminated the confusion for almost all those tested.

Also in "Choosing a Point of Comparison"

Page last reviewed March 2016
Page originally created February 2015
Internet Citation: Explaining Comparisons of Health Care Quality Scores. Content last reviewed March 2016. Agency for Healthcare Research and Quality, Rockville, MD. https://www.ahrq.gov/talkingquality/translate/compare/explain.html
Back To Top