Understanding of Public Comparative Quality Reports: Effective Use of Explanatory Information
The purpose of the Best Practices in Public Reporting series is to provide practical approaches to designing public reports that make health care performance information clear, meaningful, and usable by consumers. Report 2 focuses on focuses on the explanatory information in public reports, beyond the performance data itself, that helps to accurately communicate quality ratings to consumers and motivate them to use the ratings in making informed health care decisions.
Select for Report 1 (on presentation of comparative health care performance data) or Report 3 (on maximizing public awareness).
Select for print version (PDF File, 250 KB; Plugin Software Help).
Prepared by Judith Hibbard, Dr.P.H., and Shoshanna Sofaer, Dr.P.H., Center for Health Improvement.
Prepared for the Agency for Healthcare Research and Quality, Contract No: HHSA290200710022T.
Value of Effective Public Reports
Information Makes a Report More Effective
Recommendation No. 1: Engage and motivate consumers to
explore and use reports
Recommendation No. 2: Deepen consumers' understanding
of health care quality and quality measures
Recommendation No. 3: Legitimize the report's sponsor
and the report's credibility
Recommendation No. 4: Provide information about the
importance, meaning, and interpretation of specific measures
Recommendation No. 5: Help consumers understand the
implications of resource use information
Recommendation No. 6: Help consumers avoid common
pitfalls that lead to misinterpretation of quality data
Recommendation No. 7: Provide consumers guidance and
support in using the information
Recommendation No. 8: Provide consumers appropriate
access to more detailed technical information
Recommendation No. 9: Test the report with consumers
before going live
A set of provider performance
measures and ratings is not enough to create an effective public report. The
purpose of this guide is to identify critical explanatory information needed to
accurately communicate quality ratings to consumers and motivate them to use
the ratings in making informed health care decisions. The guide includes nine
evidence-based recommendations and examples to highlight the recommendations. The
intended audiences include Chartered Value Exchanges (CVEs) and other community
collaboratives. The guide also may be of interest to
States, health plans, and purchasers involved in the design, production,
dissemination, and promotion of comparative health care quality and cost
information for consumers.
Return to Contents
of Effective Public Reports
To date, relatively few consumers
have seen comparative performance reports or used them to make health care
choices, such as selecting a physician or hospital.1
Consumer use of reports, however, has the potential to influence quality in at
least three ways:
performance information makes it more likely that consumers will understand the
dimensions of quality and tap this growing understanding in obtaining
high-quality health care for themselves and their family members.
- The collective
effect of many informed consumer choices may stimulate quality improvement
among providers as they perceive that performance data can affect their market
- Public reports
that affect providers' public image by clearly identifying them as high-quality
or low-quality providers may encourage them to improve the quality of care they
provide, to protect or enhance their reputations.
ways to make public reports relevant and useful to consumers is part of an
overall strategy to improve health care. This report, which is organized around
a set of nine recommendations, is intended to help report card sponsors, including
CVEs, achieve this goal.
Return to Contents
Explanatory Information Makes a Report More Effective
Most Americans have some experience using data to compare
products and services, but public reports of health care quality are not yet in
the mainstream. Many people are unaware of their availability, and a large
number of those reports have not been easy to understand and use. Evidence and
experience make it clear, however, that report designers need to provide
explanatory information, in addition to the actual comparative data, to
maximize a report's use and impact. This report addresses this need. Whether
information is presented online or in print, the right kind of explanatory
- Engage and motivate consumers to explore and use
consumers' understanding of health care quality and quality measures.
- Legitimize the report's
sponsor and the report's credibility.
information about the importance, meaning, and interpretation of specific
consumers understand the implications of "resource use" information.
- Help consumers avoid
common pitfalls that lead to misinterpreting quality data.
- Provide consumers
guidance and support in using the information.
- Provide consumers
appropriate access to more detailed technical information.
To help achieve these eight aims, report card sponsors are
the report before going live.
The rest of this report examines these nine recommendations
and offers guidance on how to develop and present appropriate explanatory information.
Return to Contents
Recommendation No. 1:
Engage and motivate consumers to explore and use reports
Americans are inundated with information. In the same way
that advertisers need to work to get their messages heard above the clutter,2
sponsors of health care quality reports must give the public compelling reasons
to pay attention to their data. Although many consumers have expressed (in
focus groups, for example) considerable interest in quality data, that needs to
be taken with a grain of salt. When people hear about a new source of
information in a focus group, they are likely to find it interesting. But then
life happens. People get busy, they are not reminded of the benefits of quality
data, and the information loses salience. This does not mean that it is impossible
to get people interested in comparative quality information. It does mean that
their interest can never be taken for granted, but rather has to be developed
The first page of a report is critical. Once
a user arrives at the first page—whether it is printed or online—there are just
a few seconds to motivate that person to keep looking rather than moving on to
something else or "surfing away," in the case of an online report. A few
concisely stated key messages, emphasizing why this information is important
and relevant to the consumer, can effectively capture a consumer's attention.
Some examples are:
- The quality of the care
among (insert facility type and community) can vary widely.
- Consumers who do not look
at provider ratings are more likely to make a poor choice.
- A poor choice may have
serious consequences for a person's health and finances.
- The information in the
report can be useful in selecting a high-quality or high-value provider.
- The information also can
be used in other ways (e.g., to help loved ones make a choice, to start a
conversation with their doctor or another health professional).
The first page of the "Model Report" shown in Figure 1 is an
example from an evidence-based comparative quality report that has been
extensively tested and reviewed by experts. It was developed to report on the
AHRQ Quality Indicators, a set of measures of hospital quality based on
administrative data collected by most States.
Since this is not a "real" report, it does not have a graphic
theme or "look." Its intent is to convey the content and language of a report. The
key points include:
- A brief definition of quality in consumer-oriented
- The reasons for publishing comparative data on
- Several reasons an individual should look at
- A brief summary of the information in the report.
Subheads and bullets
break up blocks of text. A link at the bottom of the page immediately takes the
user to the data. The rest of the report contains a lot more explanatory
information, which is either wrapped around the data presentations or positioned
at the end of the report.
Return to Contents
Recommendation No. 2: Deepen consumers' understanding of health
care quality and quality measures
Consumers and health professionals understand quality issues
related to health care differently. Health professionals typically see health
care quality as multifaceted; some aspects viewed as critical by professionals
are not, initially, seen the same way by consumers. For example, professionals
see clinical process measures (e.g., providing a prescription for a beta
blocker to patients discharged from the hospital after a heart attack) as
critical components of quality and need little in the way of explanatory
information. But consumers generally are not familiar with the evidence that
links clinical processes to patient outcomes, which consumers care deeply about
(e.g., not having another heart attack).
What is to be done about this difference in perspective? First,
it is important to recognize that consumers will only find a report useful if
it connects to what they already care about. Beyond that, public reports also
offer an opportunity to deepen consumers' understanding of health care quality
in a way that brings them closer to understanding it as health care
professionals do. We need to engage consumers by using explanatory information
to make a connection between what they already care about and the more sophisticated
elements of quality they could easily come to understand and care about.
Provide a framework to help consumers understand
One way to increase consumer understanding is to provide a
simple, easy-to-understand but comprehensive quality definition or framework
that builds on what is known about how consumers and patients define quality. An
excellent quality framework that is generally accepted among health
professionals already exists in the form of the "six aims" of health care
articulated in the Institute of Medicine's report Crossing the Quality Chasm.3 Care should be: (1) safe, (2)
effective, (3) timely, (4) patient centered, (5) efficient, and (6) equitable.
A simplified version of this framework was tested with
consumers and seems to work well.4 The concept of safe care is described as
"care that protects patients from errors and harm"; the concept of effective
care is captured in the phrase "care that is proven to work"; and the phrase
"care that is responsive to a patient's needs and preferences" reflects much of
what is implied by "patient-centered," "timely," and even "equitable."
If reporting a full complement of measures, a report sponsor
may opt to organize provider ratings into these three categories:
on "care that protects patients from medical errors and does not cause
harm," which would include measures such as surgical infection rates or injuries
on "care that is proven to work," which would include measures such as
percentage of diabetes patients who receive all five recommended tests
on "care that is responsive to a patient's needs and preferences," which
would include measures such as patient experience.
For more information about this framework, refer to Best Practices in Public Reporting No. 1.
Make sure reports state clearly which aspects of
quality they cover
need to make clear the aspects of quality they cover. Using everyday language
and listing the main categories of quality data included help to orient the
reader. For example, a lead-in to a report could spell out that the report
contains measures on patient experiences and effectiveness: "In this report,
physicians are compared on 'how responsive they are to patients' needs and
preferences' and on 'whether they use practices that are proven to work.'"
The following report
example (Figure 2, 155 KB) is from Minnesota Community Measurement, a member of
Minnesota's CVE. The report's home page
offers succinct explanations of the report's purpose and value, as well as of
the measures presented and why they were chosen. This particular online report
focuses narrowly on physician quality of care for people with diabetes.
Return to Contents
Recommendation No. 3: Legitimize
the report's sponsor and the report's credibility
Focus group research on public reporting consistently finds
that the public mistrusts quality information if they think the source of that
information has an interest, especially a financial interest, in the ratings.
Advertising, including provider and health plan advertising, is so ubiquitous
that people are naturally wary that quality reports may simply be marketing
ploys. Others may be concerned that the report has been developed by purchasers
and heath plans to reduce coverage costs. The public needs to know that the
data in reports are objective and that the report is sponsored and supported by
trustworthy, expert sources.
The most important contributors to a report's credibility
are the credibility of the sponsors and how clearly the sponsorship is
presented. If a sponsor is well known and trusted by the consumer audience (not
just among health professionals), little needs to be done beyond clearly identifying
that sponsor. When sponsors are not well known to consumers, it is important to
include a brief description or mission statement up front (in as few words as
possible, such as the organization's "tag line") with a link to additional
information in the back of the report. (Note in Figure 2 that a link to a
description of the sponsor is at the bottom of the home page.) Explaining why the sponsor is issuing the
report, and that it has no financial interest in the impact of the report on
providers, can help reinforce its credibility to consumers.
If the report is sponsored or published by a CVE, the
organization or name may not be familiar to consumers. Even if a CVE is not
well known, some of its members likely are well known, and their explicit endorsement
of the report can be valuable. Provide clearly labeled links to letters of
endorsements from well-known organizations in the State or community, for
example, via a special tab on the left-hand side of the home page. Make sure,
however, that the full range of partners is shown as endorsing the report. If
only one group (e.g., the health plan association, the medical society, a major
employer) is shown, people might think it is "their" report and, perhaps,
biased in "their" direction.
Establish credibility by demonstrating fairness
The public wants to know that reports are fair to those
being rated. Focus group researchers have heard repeatedly that a specific
measure is not really the sole responsibility of the entity or individual being
assessed. Some consumers say that the responsibility is broader—the patient's, or another health professional's,
or shared by multiple professionals.
For example, when older women were asked about the inclusion
of a mammography rate in an early HEDIS (Healthcare Effectiveness Data and Information Set) measurement set
used to compare health plans, they thought that either the woman herself or her
doctor was responsible for whether she had a mammogram, not the health plan.5
In response to this feedback, explanatory information was added to the
presentation of these data in an early version of Medicare Compare. This information
acknowledged that patients and physicians affect a health plan's mammography
rate but also specified exactly how
health plans can act to ensure that more women get needed mammograms.
Another way to demonstrate fairness is by describing key
aspects of the sponsor's interactions with the providers who are rated. While
it is a good idea to make clear that you are at arm's length with providers, it
also can help to: (1) conduct a dry run of data collection and aggregation,
which is reported only to providers, prior to the actual public report; and (2)
give providers an opportunity to comment on the findings. If you take these
steps, tell the public about them, briefly and in plain language.
Provide the right level of detail to ensure
Many report designers believe that for the public to trust a
report, they need to know a lot of the technical details about how the data
were collected and how the scores were generated. In particular, report
designers think people need to know the extent to which differences between
those rated are statistically or substantially significant. Because of this
concern, some designers address statistical significance by including details (e.g.,
confidence intervals in graphs or highly technical presentations of data) in
the main body of the report.
Such complex data presentations are unlikely to be either
read or understood. In fact, consumers may see information about adjustments to
the data as a sign that someone is "messing with the information." Therefore,
it is important to find the right balance between technical details and summary
This challenge can be addressed by providing technical details
in a special section toward the back of the report (after the measures or
ratings). Links to this information should be provided early, however, to
signal that the details are available to anyone who wants them. It is
appropriate to have links like this throughout the report (e.g., via a tab at
the left or on the top of the screen, for an online report) to reinforce the
continuous availability of this information, as it is hard to predict when a
given individual may want to look at it. Realistically, health professionals
are more likely to look at this information than consumers are, but it must be
made available to all. Most important, it must be written as clearly and simply
as possible; unfamiliar terms need to be defined as they are used.
Technical details provided with the data display should
include information about the time period covered by the data and data sources,
including mention of whether data provided by the providers or health plans
being rated are validated or audited in some way. When survey information is
reported, people want to know that the sample was random and reasonably large.
They also want to know that the surveys were conducted and scores generated by
an independent entity.
Explain how scores were generated
Scoring can make a big difference in the effect comparative
data have on consumers' understanding of quality information. For example, when
Hospital CAHPS® data are presented on the Hospital Compare Web site, graphs
show the percentage of patients in each hospital who gave the best possible
rating for a given experience. However, when a composite of several measures is
reported (such as, for example, communication between patients and their
nurses) the graphs show the percentage of patients in each hospital who gave the
best possible report on ALL items in
From a consumer-engagement perspective, it is a good thing
if the information shows variation, because it reinforces the idea that there
is real variation in quality. On a more fundamental level, it makes people more
interested in the information and more likely to think it can help them make a
good choice. However, these scoring decisions need to be made clear, both in
the individual data presentations and in the technical details.
Another aspect of scoring is risk adjustment or, in some
cases, "smoothing" of data through hierarchical modeling when some of the
entities being rated are substantially smaller than others.6,7 These strategies involve complex statistical
techniques, and they cannot be presented in public reports using language one
might use in a graduate (or even undergraduate) course. The language must be as
simple as possible but not so simplistic that the steps taken to ensure
accuracy and fairness of the data are unclear.
The authors have found through research that people tend to
understand adjustments based on age or severity of illness but react negatively
to adjustments based on social factors such as education level. To the extent
that a report sponsor plans to stratify data by race, ethnicity, or income
level, the authors caution that this will have to be done carefully so that consumers
do not see it as a manipulation of the "real" data or discrimination against
racial, ethnic, or income groups.
Return to Contents
Recommendation No. 4: Provide information about the importance,
meaning, and interpretation of specific measures
In addition to providing a broad framework that defines
different aspects of quality, reports need to offer simply stated explanations
around their graphic presentations of data. They need to describe how the
measures relate to quality and, sometimes, how to interpret the graphic.
Use terms consumers understand
Many reports justify using technical terms by including a
glossary. People rarely use glossaries, however, and are not likely to examine
information they do not understand. If technical terms are used, they must be
defined immediately in everyday language that will be understood by individuals
at an eighth-grade reading level or lower. An even better strategy is to use a common
term (e.g., breast cancer screening test), with the technical term (e.g.,
mammography) in parentheses.
Ideally, the measures reported will have been vetted
previously with consumers to see if they find them important, relevant, and
appropriate to the providers or health plans being rated. If a measure has not
been formally vetted, it may be necessary to conduct focus groups to obtain
input on how to present it. Focus groups were conducted before finalizing the
Hospital CAHPS survey, for example. They helped determine which items stayed in
the survey and helped inform decisions about the contextual information needed.8 For more information about testing, refer to
"Recommendation No. 9: Test the report with consumers before going live" later
in this report.
Explain different types of measures
The explanatory information needed depends on the type of
measure, because consumers' interest in and level of understanding of the
different types of measures will vary. When developing such information for
public reports, consider the following key points about each type of measure:
- Patient experience measures, such as
those derived from CAHPS® surveys: People seem to naturally understand this
kind of measure. Most, but not all, people value and will use rating information
from other patients and consumers.
- Outcome measures: These measures are just beginning to
be included in reports. Early testing on these measures with consumers
indicates a range of responses to them:
- Patient safety measures, such as
measures of the frequency of infections, falls, and other negative
consequences of care delivery: Once explained in plain language, these
measures seem to resonate with many consumers. It appears important when
presenting safety measures to emphasize that specific
bad outcomes can be prevented by providers.
- Mortality measures: These elicit a
wide range of responses from consumers. Some say they do not want to read
or think about the potential of death when they seek medical care. Others
believe it is important to have this information, however, particularly
for serious illnesses or high-risk procedures. Still others think that simply
not dying does not equate to high-quality care. Their definition of
quality goes far beyond that to issues of being cured, improving quality
of life, or having a positive experience as a patient. Finally, people's
interest in a disease- or procedure-specific mortality measure will
depend on whether they care about that disease or procedure. It makes
sense to include outcome measures in public reports; however, some may
require explanatory information, and all must be described in lay terms.
- Clinical process measures: It is
almost always necessary to explain these kinds of measures. Clinical
processes are not familiar to many consumers, and they rarely know the
evidence of how a particular process supports a desirable outcome. The report
must use plain language to describe the process, so that the importance is
clear (e.g., patient given right medication at right time). The label
should help people make the connection between the process and the
For example, the HEDIS assessments
of health plans include a mammography measure as a key effectiveness indicator.
When this measure was introduced, mammography rates were considerably lower
than they are now. At that time, many women did not know the benefit of getting
a mammogram. They typically thought of it as a test they had to take (and pass)
to make sure they did not have breast cancer. However, a much more important
benefit of getting a mammogram is that the earlier breast cancer is detected,
the greater likelihood that it can be treated successfully, often using less
invasive treatments. Explaining this in plain language links mammography rates
to outcomes that are important to all women: being cured and having less
When these data were first
presented, the Centers for Medicare & Medicaid Services (CMS) included an
explanation of these benefits in a section of the report called "Why Is This Important?"
Further, because women often did not think health plans could do much about a
mammography rate, CMS included another section titled "What Does This Measure Show?"
that discussed how health plans influence mammography rates.
Provide guidance on how to read graphs and
Often people need to be told what to look for in a graph.
For example, they need guidance to understand if a high score or a low score
indicates higher performance. A bar on a graph to indicate the average or some
other benchmark requires an explanation, as shown in Figure 3.
Figure 3. A Plain Language
Explanation Helps Consumers Interpret Data Quickly
Return to Contents
Recommendation No. 5: Help consumers understand the implications
of resource use information
"Resource use" is a broad term, which is
intended to include measures of price and efficiency and capture potential
inefficiencies such as hospital readmissions. Report sponsors are just
beginning to include resource use information in public reports; there are few
well-developed measures in this arena. Resource use measures need to be
relevant to consumers, but as one might expect, there is not much research yet
on what is relevant to them and how the measures should be presented so that they
will not be misunderstood. The term "resource use" has not been tested with
consumers and is likely to be poorly understood.
common consumer beliefs are likely barriers to using cost and efficiency
measures effectively. One is the widespread belief that "more care is better."
Consumers are much more concerned about not getting something they need than
they are about getting care they do not need.
second common belief is that price, as with most consumer goods, is a
reasonable proxy for quality. That is, when consumers are not getting a clear
message about quality, they are likely to use cost as a proxy for quality. This
can result in counterproductive choices.
to address these misconceptions in public reports is a first step in
communicating about resource use issues. Keeping these misconceptions in mind
as sponsors create approaches for reporting on resource use measures will be
Return to Contents
6: Help consumers avoid common pitfalls that lead to
misinterpretation of quality data
the authors do not recommend providing extensive statistical detail within data
presentations, it is important to include caveats about certain measures that
could otherwise be misleading. As shown in the examples below, reports must
communicate to users:
- That they should use caution in
comparing providers on certain measures.
- That they cannot assess a provider's
overall performance by extrapolating from a limited set of measures that reflect
only some of the services they provide.
patient safety measures address extremely rare events, including "never
events." The problem with measures of rare events is that it is difficult to
make valid comparisons among providers or health plans. For example, Hospital A
reports two deaths in a given year among hip replacement patients, while Hospital
B (with about the same number of surgeries and mix of patients) reports one
death. Consumers might falsely conclude that Hospital B's performance is twice
as good as Hospital A. Since deaths among hip-replacement patients are
extremely rare, the difference in rates in a given year may be random—more
noise than signal.
are some options for addressing this kind of measure:
- Exclude such measures from your report.
- Combine data for multiple years so that the
noise diminishes. This means changing the measure from deaths in a single year
to deaths over the last 3 years or last 5 years. Even then the rates may be
- Aggregate data from one of these measures, such
as deaths among hip-replacement patients, with deaths from other low-risk
surgeries or procedures.
- Present data on the measure, but add a statement
with the data, pointing out that the event is rare and that users should be
cautious in comparing facilities or providers on the basis of this measure
you do report such measures, consider using counts rather than rates. Because
so many Americans have limited numeracy, it is difficult for them to grapple
with a rate such as 1 death per 10,000 patients or .0001 percent.
common misunderstanding is that the measures included in an online or printed
report tell the whole story of a given provider's or plan's performance.
Consumers may sometimes think they have the whole story and can judge a
hospital, for example, based solely on how it performs on measures related to
cardiovascular care, even when they are going to the hospital for cancer
surgery. This kind of caveat is better presented in a section on how to use the
report than near any single data presentation, since it really speaks to the
report as a whole.
Return to Contents
Recommendation No. 7:
Provide consumers guidance and support in using the information
We have reviewed
how the home page of the report presented on a Web site or the first page of a
printed report is a good place to quickly indicate the benefits of a report
(i.e., how it can be used by consumers). Additional "decision support
information" can be helpful in providing guidance and motivation for those who
are not familiar with or are less confident in using quality information for
themselves or their loved ones.
In many cases, the
desired primary use of comparative quality information is to influence consumer
choice. Simply providing information, however, is not sufficient. Many believe
that "decision support" must be included in reports.9 For example, reports often provide
information on a wide range of measures but do not address how to put all the
information together to make a decision. The cognitive burden of this task is
extremely high for almost anyone.10
Efforts have been
made to provide formal decision support within online reports by asking
individual users what is most important to them and then using internal
functionality to calculate the available options that best meet their
preferences. This is a complex undertaking, made more difficult by the fact
that many health care consumers find it hard to identify the aspects of quality
that are most important to them.
Following are five
simple, practical approaches to providing decision support for consumers:
- Provide a list of what consumers should think about when
they make a decision of this kind. This list can include issues related to
information covered in the report, but it may include issues other than
quality, such as convenient location or languages spoken by staff. For
example, the report could list several things to consider in choosing a
primary care physician, such as how well each communicates with patients,
how easy it is to make an appointment and whether the physician provides
needed preventive services on time, and whether the physician delivers the
right care to patients with specific conditions.
beyond typical quality measures, the list might also include whether the
physician's office is located conveniently for the person, whether the
physician is included in the network of the patient's health plan or accepts
the patient's insurance, and whether the physician speaks a language other than
English. If consumers are urged or required to gather information from outside
sources, the report should contain links or other directions on how to access
that information. This kind of simple decision support has been shown to be
highly valued by many consumers.11
- Identify, through a label or a visual such as a checkmark,
which of the choices is the "best scoring" across the board or the "best
value" if cost data also are included. Consumer
Reports has used this approach for many years. This kind of label
operates like a summary score.
scores, common in survey-based measures, can be helpful in reducing the total
number of data points in a report, but they do not help as much as summary or
"roll-up" scores. Unlike composite scores, summary or "roll-up" scores combine
a large number of specific measures that may or may not be highly related to
one another statistically. They relate to a single provider or facility.
Consumers often say they want summary or "roll-up" scores, which can make it
easier for consumers to evaluate choices and make decisions.
issues must be addressed in developing and reporting summary or "roll-up"
scores, however. First, it is important not to wash out any variation across
providers—something that will make it harder for consumers to make a decision.
Summary scores that reveal that some providers and facilities are better across
the board can be extremely helpful to consumers.
care must be taken in weighting items in summary scores (i.e., giving more
weight to some measures than to others). For example, it might seem obvious on
the face of it to assign greater weight to a measure of the number of patients
who die from central-line infections versus another measure of how quiet a
hospital is at night. It is not clear, however, what the right weighting would
be for any number of measures, and it is inevitable that different consumers will
weight a set of measures differently.
Call out key differences in performance, (i.e., pointing
out places where differences in scores are particularly large).
Provide examples of specific ways consumers can use
information, not just for making personal health care choices, but also to
learn more about what kind of care is high quality, to help loved ones
make a decision, or to begin a conversation with their physician or other
provider. Stories and testimonials can illustrate how information
can be applied (e.g., including first-person statements by consumers about
how using the report made a difference in their choices, health, or
- Make explicit what
actions consumers can take to protect themselves from poor-quality care.
The most obvious step consumers can take is to avoid choosing and using
poor-quality providers. Sometimes, as we know, consumers have little or no
choice of health plans or hospital. One step a consumer might take in this
case would be to talk about the issue of poor hospital quality with his or
In recent research, physicians were
asked how they would respond if patients expressed concerns about hospital quality
information they had seen in a public report.12
While many physicians said they would try to reassure their patient about the
quality of the hospital, many also said they would alert the involved specialist
about the patient's concerns, be vigilant about specific concerns while on
rounds, or speak to the nursing staff or, in some cases, hospital executives. A
few physicians said they would encourage the patient and family to be proactive
in monitoring the patient's care and to make it clear when they thought
something was not going as it should. This research implies that consumers can
and should talk to their doctors about quality concerns.
Return to Contents
Recommendation No. 8: Provide
consumers appropriate access to more detailed technical information
People can become overwhelmed and give up on a report if
they cannot find what they are looking for. Report design should use layering
and navigation aids so that users are not overwhelmed with information or
required to look at information that is of lesser interest to them. In general,
this means report designers need to balance the consumer's need for brevity and
ease of access to what they care about most with the sponsor's need to provide
full descriptions, caveats, and disclosures about the data. Using the Web to
disseminate reports makes creating this balance much easier than in print
A good Web site report incorporates several features to help
people know where they are and get where they want to go:
- A set of tabs at the top
to help users get to the main sections from anywhere in the report.
- A set of tabs at the side
to help users navigate within report sections.
- Internal links from one
part of the report to another.
The amount of explanatory information provided up front
should be limited, unless there is a special reason for breaking this rule.
(For an example of such a case, go to Figure 4.)
4. Example of a Case When More Up-Front
Information Is Useful13
We recently created and tested a Model Report on Hospice Quality. Our
initial focus group research confirmed that the public does not understand
what a hospice is or what it does. Indeed, they have serious misconceptions
that might make it impossible for them to understand available quality
measures about hospice. For example, to many people hospice is a place, a
facility where people go. In fact, the vast majority of people use hospice
services in their own homes, where they are visited by a wide range of
hospice team members.
In response to this feedback, our Model Report included extensive
information about the nature of hospice, both up front and at the back.
Even so, we gave users the option on the report home page to explore this
information if they chose or to go right to the data. Keep in mind that
users may start with the data and then return to the more educational
components of the site, especially if they have the right navigational cues
wherever they are.
Explanatory information that is
best located toward the back of a report includes:
- Technical details of how
data are collected and analyzed.
- Methods to provide
feedback or ask questions about the report.
- Letters of endorsement.
- More detailed
information about how to use the report such as described in
Recommendation No. 7.
- Things to keep in mind
while using the data (this is where the caveats go).
- Links to other reports
on quality that cover aspects not covered in the current report.
- Additional resources and
links on topics relevant to the report (e.g., information about diseases
or conditions if the report is organized that way).
- Contact information for
rated providers and health plans, if the number is small.
Although every report does not need to include all of the
above, the first and second items are the most important to include. Even if
consumers do not look at the technical details, they like to know they are
there. Their advocates, as well as other stakeholders, will look at these
Last, but definitely not least, offering a vehicle for
feedback reinforces the idea that the sponsor is open to improvement and
provides a quick and inexpensive way to get information to evaluate the
effectiveness of the report. Allowing users to ask questions may be helpful in
situations where consumers need clarification about different reports on the
same provider or plan.
Return to Contents
Recommendation No. 9: Test the report with consumers before
Report format, language, and structure should be tested with
consumers. This sounds daunting, but it does not have to be. The best way to
test is through cognitive interviews.14
This technique was initially created to assess survey questionnaires to make
sure potential respondents would consistently understand questions and possible
response options as the survey developers intended. Cognitive interviews are
the gold standard in survey testing. For more than a dozen years, cognitive
interviews have been used to test public reports to ensure that consumers
understand them as intended, that they can navigate their way around them, and
that they are likely to respond positively to the content and how it is
Conduct cognitive tests
Use a draft that looks as much as possible like the
potential Web site or printed report. Then develop a protocol for cognitive
testing and recruit 10 to 15 subjects who reflect a cross-section of the
intended audience. Each test should take about 90 minutes and involve three
persons—a subject, an interviewer, and a note taker—where possible.
Tape-recording each interview is recommended. Watch for parts of a page where
the respondent spends more time or less time, when they seem confused, etc. It
is helpful to offer a financial incentive to subjects for their participation,
ranging from $75 to $125, depending on the local cost of living. Saturation
(where little more can be learned) is reached after 10 to 15 individual
The key to a good cognitive test is to guide people, step by
step, through the report, get their reactions to it, and ask questions to find
out if they understand both the text and the graphics. Do not ask people if
they understand; rather, ask them a knowledge question and see if they respond
with the correct information. For example, ask them to identify the highest and
lowest performers, to characterize the main point of a section of text, to
discuss why an average is included, or to describe where the data come from.
Ask if anything in the report makes them trust or distrust it, where too much
or too little has been said.
Do not be surprised if people are brutally honest and if the
process turns out to be somewhat painful, especially for those who have
agonized over just the right wording and graphics. Refine reports in response
to the results of these tests. Schedule two or three rounds
of tests, revising the draft report between rounds, if time and money allow.
Disciplined testing and use of evidence-based practices will
help improve the likelihood that the report will be understood, appreciated,
and most importantly, used.
Return to Contents
Purpose of Report Series
The purpose of this three-part series of reports is to provide practical
approaches to designing public reports that make health care performance
information clear, meaningful, and usable by consumers.
The goal is to help sponsors present information so that a wide variety of
people can understand and apply it easily to key decisions, even if they do not
want to spend a lot of time on details and have limited technical knowledge of
Together the three reports cover the wide range of issues and challenges
faced by report sponsors:
Return to Contents
- 2008 update on consumers' views of patient safety and
quality information. Menlo Park, CA: Kaiser Family Foundation; October 2008.
- Shaller Consulting. Consumers in health care: creating decision-support
tools that work. Oakland: California HealthCare Foundation; June 2006.
of Medicine Committee on Quality Health Care in America. Crossing the quality chasm: a new health system for the 21st
century. Washington, DC: National Academy Press; 2001. p. 5-6.
- Hibbard JH, Pawlson LG. Why not give consumers a framework for
understanding quality? Jt Comm J
Qual Saf 2004 June;30(6):347-51.
- McGee J, Sofaer S, Kreling B. Findings from
focus groups conducted for the National Committee for Quality Assurance (NCQA)
Medicare and Medicaid Consumer Information Projects. Washington, DC: National
Committee for Quality Assurance; July 1996.
- Iezzoni LI. Risk adjusting rehabilitation outcomes: an
overview of methodological issues. Am J Phys Med Rehabil
- Shahian DM,
Normand ST. Comparison of "risk-adjusted" hospital outcomes. Circulation 2008 Apr;117:1955-63.
- Sofaer S, Firminger K. Patient perceptions
of the quality of health services. Annu Rev Public Health 2005 Apr;26:513-59.
- Spranca M. RAND report summary: consumer use of information
when making treatment decisions. Chicago: BlueCross
BlueShield Association; 2005.
- Peters EM,
Hibbard H, Slovic P, et al. Numeracy skill and the
communication, comprehension, and use of risk and benefit information. Health Aff 2007;26(3):741-8.
LA, Garfinkel SA, Hibbard JH, et al. Health insurance
knowledge among Medicare beneficiaries. Health
Serv Res 2002;37(1):43-63.
- Barr JK,
Bernard SL, Sofaer S, et al. Physicians' views on public reporting of hospital
quality data. Med Care Res Rev 2008 Dec;65(6):655-73; Epub 2008 July 2.
- Sofaer S, Hopper SS, Firminger K, et al. Addressing the need for public reporting of
comparative hospice quality: a focus
group study. Jt Comm J Qual Saf 2009;35(8):422-9.
- Harris-Kojetin LD, Fowler FJ, Brown JA, et al. The use of cognitive testing
to develop and evaluate CAHPS® 1.0 core survey items. Consumer
Assessment of Health Plans Study. Med Care 1999;37:MS10-21.
Return to Contents
The authors wish to
thank the following people for reviewing this document: Katherine Crosson, M.P.H., Agency for Healthcare Research and Quality; Irene Fraser, Ph.D., Agency for Healthcare Research and
Quality; Linda Greenberg, Ph.D., Agency
for Healthcare Research and Quality; Jan De La Mare, M.P.Aff., Agency for Healthcare Research and Quality;
Lisette Lejnieks, M.B.A.,
independent consultant; Peggy
M.S.P.H., Agency for Healthcare Research and Quality; Chris Queram, Wisconsin Collaborative
for Healthcare Quality; Dale Shaller, M.P.A., Shaller
Consulting; Karen Shore, Ph.D., Center for Health Improvement; Nancy Brands Ward, Center for Health Improvement; and Jan Whitehouse, Greater Detroit Area
We consider our
Learning Network tools to be works in progress and always welcome your
comments. Please forward suggestions to Peggy
McNamara at email@example.com.
AHRQ Publication No. 10-0082-1-EF
Current as of May 2010
Hibbard J, Sofaer S. Best Practices in Public Reporting No. 2: Maximizing Consumer Understanding of Public Comparative Quality Reports: Effective Use of Explanatory Information. AHRQ Publication No. 10-0082-EF, May 2010, Agency for Healthcare Research and Quality, Rockville, MD. http://www.ahrq.gov/qual/pubrptguide2.htm