Model Public Report Elements: A Sampler

B. Presentation of Measure Ratings

An effective presentation of measure ratings is based on the following principles:

  • Consumers do not define or understand quality in the same way that it is typically measured and reported.
  • A framework can help consumers understand the larger concept of quality, as well as better comprehend the individual quality indicators.
  • A framework may enable consumers to more effectively participate in the quality discussion.
  • If a consistent framework is used in all reports, consumers will expect to see performance ratings in all categories of the framework.

A few existing quality frameworks might be considered as an organizing concept for individual measures within a quality report:

  • The Institute of Medicine (IOM) developed a framework consisting of six categories: Effective, safe, patient centered, timely, equitable, and efficient.1
  • The IOM also developed these three categories: overuse, underuse, and misuse.2
  • Donabedian developed a framework consisting of three categories: structure, process, and outcome.3

A framework to communicate about health care quality should:

  • Indicate clearly what quality of care is.
  • Contain no more than three or four categories.
  • Be consistent with how quality is measured and conceptualized within the policy and industry arenas (that is, the major categories that are used to define quality by experts and the industry should be reflected in some way in the framework provided to consumers).

The following examples use a framework in communicating what quality of care means.

Tool: Health Matters

Sponsor: Colorado Business Group on Health

URL:  http://www.cbghealth.org/cbgh/index.cfm?LinkServID=E0FEA580-D721-5B0E-DA115CC5582742B9&showMeta=0 PDF File, page 2; Plugin Software Help

Excerpt from Health Matters defining quality health care as effective, efficient, safe, timely, focused on the individual, and equitable.

 

Tool: Community Checkup Report

Sponsor: Puget Sound Health Alliance

URL: http://www.wacommunitycheckup.org/editable/files/CommunityCheckup_Nov2008/FAQs_AboutQualityHealthCare_Nov2008.pdf (PDF File; Plugin Software Help)

 

 Excerpt from Frequently Asked Questions About the Community Checkup that defines quality health care and describes quality of care in the Puget Sound region. The checkup focuses mostly on effective care.

This next example, which is not from an existing Web report, shows an abbreviated IOM framework for communicating what quality is, which has the advantages of using only three categories and having undergone cognitive testing. With this kind of framework, quality indicators can be displayed within each of the three categories of performance. This approach can make each quality indicator easier to understand while also conveying that "quality" incorporates three different but important dimensions.

Source: Study funded by the Robert Wood Johnson Foundation on Giving Consumers a Framework for Understanding Quality.

Reference: Hibbard JH, Greene J, Daniel D. What is quality anyway?: Performance reports which clearly communicate the meaning of quality of care. Med Care Res Rev 2010; 67:275; originally published online January 21, 2010.

Excerpt from upcoming journal article that defines the best quality medical care as effective, safe, and patient focused.

As noted above, the most effective way to use a framework is to actually show the quality indicators arrayed under the main categories of the framework, as in the example below. The framework categories tell the user what the individual indicators mean.

Source: Study funded by the Robert Wood Johnson Foundation on Giving Consumers a Framework for Understanding Quality.

Reference: Hibbard JH, Greene J, Daniel D. What is quality anyway? Performance reports which clearly communicate the meaning of quality of care. Med Care Res Rev 2010; 67:275; originally published online January 21, 2010.

How does one select quality indicators for each element of a framework? The decision to use a framework implies making some early choices about the types of performance measures to include in your report. The table below shows examples of indicators to include if using the Structure, Process, and Outcome framework; the IOM framework of six domains; or the Hibbard-abbreviated IOM framework.

Reference: Romano PS, Hussey P, Ritley D. Selecting Quality and Resource Use Measures: A Decision Guide for Community Quality Collaboratives. Rockville, MD: Agency for Healthcare Research and Quality; May 2010. AHRQ Publication No. 09(10)-0073.

Matrix of Quality Measure Typologies With Examples

IOM DomainsStructureProcessOutcome
EffectiveCardiac nurse staffing, nursing skill mix (RN/total)Use of angiotensin-converting enzyme (ACE) inhibitor or angiotensin receptor blocker (ARB) for patients with systolic heart failure30-day readmissions (or mortality) for heart failure
Patient centeredUse of survey data to improve patient-centered careDid the nurses treat you with courtesy and respect?Overall rating of care
TimelyPhysician organization policy on scheduling urgent appointmentsReceived beta blocker at discharge and for 6 months after acute myocardial infarctionPotentially avoidable hospitalizations for angina (without procedure)
SafeComputerized physician order entry with medication error detectionUse of prophylaxis for venous thromboembolism in appropriate patientsPostoperative deep vein thrombosis or pulmonary embolism
EfficientAvailability of rapid antigen testing for sore throatInappropriate use of antibiotics for sore throatDollars per episode of sore throat
EquitableAvailability of adequate interpreting servicesUse of interpreting services when appropriateDisparity in any other outcome according to primary language

1. Displaying measures so that consumers can understand them

Using comparative data to make a choice can be cognitively burdensome. People are more likely to use data if the information is easy to understand and process. Making it easier actually increases the individual's motivation to use the data. There are multiple ways to make the data more evaluable (that is, easier for the user to quickly see better and worse options without much effort). 

Strategies to make comparative data more evaluable and less cognitively burdensome include:

  • Use data display approaches that do not require users to hold information in their mind as they look from page to page or at different points on the same page:
    • Make relevant choices visible on one screen.
    • Use symbols instead of numbers (many people have limited numeric skills).
    • Use symbols that do not require a legend to understand what they mean.
    • Use language and terms that consumers understand (avoid language that requires the user to look up the definition).
    • Do not assume people will understand advanced statistical concepts, such as using confidence intervals in data displays.  Consumers do not understand them, and they introduce ambiguity that can result in consumers being less likely to rely on or trust the data.
  • Do some of the cognitive work for the user:
    • Summarize information. Providing an overall summary measure of all the other measures helps the end users by doing some of the work for them.
    • Order by performance (or have a function that allows this).  This makes it easy to quickly see top and bottom performers.
    • Use affective labels (e.g., excellent, good, fair, poor) that interpret the information for the user.
    • Highlight high performers.

Here is an example of using symbols instead of numbers. These symbols use color and have words embedded in the symbol to make them easy to use without consulting a legend.

Tool: CalHospitalCompare

Sponsors: California Hospital Assessment and Reporting Taskforce, California HealthCare Foundation, and University of California, San Francisco

URL: http://www.calhospitalcompare.org/comparison.aspx?c=&h=210&h=218&h=155&h=146&h=256  

Sample ratings Web page showing ratings for ICU mortality rate and patient safety. Ratings are color-coded symbols with words. The rating shown are superior in dark green, average in tan, and above average in light green.

Here is another example that uses symbols instead of numbers. In this example, the symbols use a color, a shape, and a word embedded in the symbol.  Average scores are faded so that the higher and lower performers "pop" out more. Thus, users can discern a pattern in the data.  The use of colors and shapes helps the user easily see patterns in the data. The embedded words in the symbols mean that the user does not need to rely as much on a legend. The embedded words also help the user by interpreting the information for them (e.g., better, worse). 

Source: Study funded by the Robert Wood Johnson Foundation on Giving Consumers a Framework for Understanding Quality.

Reference: Hibbard JH, Greene J, Daniel D. What is quality anyway?  Performance reports which clearly communicate the meaning of quality of care. Med Care Res Rev 2010; 67:275; originally published online January 21, 2010.

Sample rating chart for several hospitals on measures showing whether the hospital is effective, safe, and patient focused. Arrows point to examples of the rating symbols. Below average is an inverted blue triangle with the word below on it. Average is the gray word average. Better than average is a yellow oval with the word better on it.

The next example shows a report format that helps the user by providing two overall summary measures: one based on clinical standards and the other based on patient ratings.  This approach greatly reduces the cognitive burden of using multiple data points to compare multiple options.

Tool: Medical Group Ratings

Source: California Office of the Patient Advocate

URL: http://www.opa.ca.gov/report_card/medicalgrouprating.aspx?County=KERN  

Screenshot of California Office of the Patient Advocate Web page with medical group ratings. Arrows point to the column headings of Meeting National Standards of Care and Patients Rate Medical Groups. A star system is used, with one star equal to poor, two equal to fair, three equal to good, and four equal to excellent.

Here is an example of a report that uses symbols that have inherent meaning.  Words are embedded in the symbol to make it less necessary to use a legend.

Tool: Partner for Quality Care—Quality of Care Ratings

Sponsor: Oregon Health Care Quality Corp

URL: http://www.partnerforqualitycare.org/selection.php  

Screenshot of Partner for Quality Care diabetes care rating Web page for several medical groups. An arrow points to the rating descriptions. Better is a yellow circle with the word Better on it. Average is a gray square with the word Average on it. Below average is an inverted blue triangle with the word Below on it.

The next example uses a summary measure and orders hospitals by performance, with highest quality providers listed first.  Ordering is a powerful way of helping the user quickly discern better and worse options.

Tool: Partner for Quality Care—Quality of Care Ratings

Sponsor: Oregon Health Care Quality Corp

URL: http://www.partnerforqualitycare.org/care.php?region_name=portland_metro  

Screenshot of hospital ratings in descending order from scores of 95 percent to 93 percent.

2. Using consumer-friendly language for measures

Using everyday language and terms familiar to consumers is key to making public reports understandable and usable. Using technical terms and expecting consumers to look them up (even if the definition is only a click away) will discourage use. Writing at a 6th grade level will make the information accessible to a wider audience. 

Here are some examples of translating technical labels into plain language.

Source: AHRQ Model Quality Reports (available at http://www.qualityindicators.ahrq.gov/Downloads/Modules_Non_Software/QI%20Reporting/Model%20Report%20Health%20Topics.pdf; Plugin Software Help)

Excerpt from model quality care with the heading 'Compare hospital scores on operations for cancer of the esophagus and pancreas.' The user selects the indicator name for detailed results on how each hospital performed. A sample chart is shown with ratings for four hospitals on death rate for operations to remove part or all of the esophagus and death rate for operations to remove part or all of the pancreas. Ratings are shown in color-coded words. Better than average is green, Average is gray, and Worse than average is purple.

In the next example, the original "untranslated" technical version of the ambulatory care quality performance indicators is shown, as well as the plain language translation of those technical labels.

Source: Study funded by the Robert Wood Johnson Foundation on Giving Consumers a Framework for Understanding Quality.

Reference: Hibbard JH, Greene J, Daniel D. What is quality anyway? Performance reports which clearly communicate the meaning of quality of care. Med Care Res Rev 2010; 67:275; originally published online January 21, 2010.

UntranslatedPercent of patients with diabetes who had A1c testPercent of women receiving breast cancer screeningProvider uses electronic prescribing to prevent medication errors
TranslatedDiabetes patients receive recommended testsWomen receive recommended cancer screeningsHas procedures to prevent medication errors

3. Displaying resource use measures

Consumers have had very little access to comparative information on costs and resource use, and until recently, we have known very little about how to present this information effectively. A 2011 AHRQ-funded study led by Judith Hibbard of the University of Oregon and Shoshanna Sofaer of Baruch College used focus groups, cognitive tests, and a randomized laboratory study to identify:

  • Cost/resource use measures that resonated with the public.
  • Displays of such measures that were most accurately understood, led to "high value" choices, and gave people the most confidence in their choices.

An overarching recommendation of this study is to present cost information in displays that include a strong quality signal (easy to understand and use quality information). In addition, displays work better if they help consumers understand the meaning of the measures by using labels and other strategies that interpret the information for consumers. We present three sample displays with specific recommended features.

Presenting Comparative Costs for Doctor's Office Visit

Sample data table compares costs for doctors' office visits. Red arrows point out key features: Average costs for office visit; whether a given doctor's use of treatments prove to get results is better, average, or below average.

Recommended features:

  • Present cost data in the same display as quality data.
  • Present exact cost information.
  • Use a "word icon" for quality, which is highly "evaluable" and thus represents a strong quality signal.
Presenting Comparative Hospital Cost, Quality, and Value Information

Sample data table compares hospital cost, quality, and value. Red arrows point out key features: Average costs; high value (high quality and low cost); whether a given hospital's quality of care is better, average, or below average

Recommended features:

  • Combine information on quality, cost, and value in the same display.
  • Call out value using a familiar signal (checkmark), which is present only for those with high quality and low cost.
  • Use a "word icon" for quality, which is highly "evaluable" and thus represents a strong quality signal.
  • Use dollar signs to represent comparative costs.
Presenting Comparative Information About Resource Use (Imaging)

Sample data table compares doctors resource use, including weekend hours, driving distance, and same-day office visits. A red arrow points out the key feature: use of MRIs and CAT scans only when appropriate.

Recommended features:

  • Use labels and symbols to characterize use of imaging, rather than giving numeric levels.
  • Use labels that interpret numeric imaging scores so that consumers do not need to determine for themselves which provider is providing more appropriate care.
  • Whenever possible, merge resource use and quality into one highly interpretable score.

4. Encouraging providers to report accurate and complete data

In some cases, significant effort is required by providers to collect, clean, and submit the required data. In these instances, collaboratives may choose to recognize that effort specifically.

In the underlined text and blue box below, Colorado Business Group on Health helps the visitor recognize the commitment made by participating hospitals.

Tool: Health Matters 2009

Sponsor: Colorado Business Group on Health

URL: http://www.cbghealth.org/cbgh/index.cfm?LinkServID=E049392D-D33F-F20A-D2ED70858A72E914&showMeta=0 (PDF File; Plugin Software Help)

Excerpt from Colorado Rural Hospitals report. The introduction gives the number of rural hospitals in Colorado and mentions the Leapfrog Group survey they were asked to complete. The question Does your hospital report its results to the Leapfrog Group is underlined in red. A blue text box thanks the hospitals that participated in the survey.

In most cases, those providers who choose to participate in public reporting will have to invest significant resources just in collecting and sharing the data and making sure it is accurate; they will have made a significant commitment to transparency. However, if the program is voluntary, other providers may elect not to provide the data. To recognize the additional contributions of those providers who share accurate data about their performance, some community collaboratives have instituted penalties for nonparticipation or incomplete or inaccurate data submission. 

The following examples illustrate ways to denote or call attention to a lack of transparency, effort, or completeness. The report below specifically notes the facility's refusal to sign off on the accuracy of the data submitted.

Tool: Hospital Performance Report

Sponsor:  State of New Jersey Department of Health and Senior Services

URL:  Sample screen shown no longer available online. 

Screenshot of report on cardiac surgery mortality. An arrow points to the facility designation Medical Center, which has a footnote stating that the facility refused to sign off on its data.

This report describes levels of participation both in terms of the data submitted and actual level of performance.

Tool:  Clinical Outcomes Assessment Program

Sponsor: Foundation for Health Care Quality

URL:  Sample screen shown no longer available online.

Screenshot of Clinical Outcomes Assessment Program Levels of Participation page. The text describes two levels of participation, full compliance with community quality standards and partial compliance.

In this report, "NR" (measure not reported) is distinguished from nonparticipation as a whole.

Tool: Focus on Hospitals

Sponsor:  Missouri Hospital Association

URL: No longer available online.

Screenshot of data for heart attack treatment. Some hospitals have numeric scores, others have NR for not reported, and others have a note that the hospital did not participate in the report.

In this report, a notation is made when data are presented for only some patients with a given condition, but not all.

Tool: New York State Hospital Profile

Sponsor:  New York State Department of Health

URL: http://hospitals.nyhealth.gov/index.php  

Screenshot of New York State Department of Health Compare Hospitals Web page. The measure shown is Pneumonia patients assessed and given influenza vaccination. The first hospital has a footnote for number of cases that indicates that the hospital's submission was based on a sample of its relevant discharges.

5. Noting in a report that "sample sizes are too small"

Sometimes a provider has too few patients with a particular condition or undergoing a specific procedure to calculate a quality measure. In such cases, a performance label should not be assigned. It is important, however, to let the consumer know that the lack of a measure or performance label does not reflect poorly on the provider's quality of care.

Some sites simply indicate that there are "too few cases."

In the example below, the site provides a warning to the reader, "* Too few patients (<160) with condition of interest to meaningfully report for this clinic."

Tool: Health Alliance Community Checkup

Sponsor: Puget Sound Health Alliance

URL:  http://www.wacommunitycheckup.org/index.php?p=viewreports&orgname=all&county=All+Counties  

Screenshot showing heart disease treatment ratings for several medicala groups. Two groups have asterisks for their rating, which indicates too few patients with the condition to meaningfully report for the clinic.

6. Using consumer-friendly phrases to replace clinical and technical terms

It is better to use consumer-friendly language (that has undergone cognitive testing), rather than technical terms even if they are linked to a glossary.

The following glossaries may serve as a starting place for report sponsors who seek lay definitions of technical terms. Use of terms that are found in one or more glossaries is not a substitute for cognitive testing.

Current as of November 2011
Internet Citation: Model Public Report Elements: A Sampler. November 2011. Agency for Healthcare Research and Quality, Rockville, MD. http://www.ahrq.gov/professionals/quality-patient-safety/quality-resources/value/pubrptsampler/pubrptsampl2b.html