Best Practices in Public Reporting No. 2: Maximizing Consumer Understanding of Public Comparative Quality Reports: Effective Use of Explanatory Information

The purpose of the Best Practices in Public Reporting series is to provide practical approaches to designing public reports that make health care performance information clear, meaningful, and usable by consumers. This report, Report 2, focuses on focuses on the explanatory information in public reports, beyond the performance data itself, that helps to accurately communicate quality ratings to consumers and motivate them to use the ratings in making informed health care decisions.

Prepared by Judith Hibbard, Dr.P.H., and Shoshanna Sofaer, Dr.P.H., Center for Health Improvement.

Prepared for the Agency for Healthcare Research and Quality, Contract No: HHSA290200710022T.

Contents

Purpose
Value of Effective Public Reports
How Explanatory Information Makes a Report More Effective
   Recommendation No. 1: Engage and Motivate Consumers To Explore and Use Reports
   Recommendation No. 2: Deepen Consumers' Understanding of Health Care Quality and Quality Measures
   Recommendation No. 3: Legitimize the Report's Sponsor and the Report's Credibility
   Recommendation No. 4: Provide Information About the Importance, Meaning, and Interpretation of Specific Measures
   Recommendation No. 5: Help Consumers Understand the Implications of Resource Use Information
   Recommendation No. 6: Help Consumers Avoid Common Pitfalls That Lead to Misinterpretation of Quality Data
   Recommendation No. 7: Provide Consumers Guidance and Support in Using the Information
   Recommendation No. 8: Provide consumers appropriate access to more detailed technical information
   Recommendation No. 9: Test the Report With Consumers Before Going Live
Purpose of Report Series

Purpose

A set of provider performance measures and ratings is not enough to create an effective public report. The purpose of this guide is to identify critical explanatory information needed to accurately communicate quality ratings to consumers and motivate them to use the ratings in making informed health care decisions. The guide includes nine evidence-based recommendations and examples to highlight the recommendations. The intended audiences include Chartered Value Exchanges (CVEs) and other community collaboratives. The guide also may be of interest to States, health plans, and purchasers involved in the design, production, dissemination, and promotion of comparative health care quality and cost information for consumers.

Return to Contents

Value of Effective Public Reports

To date, relatively few consumers have seen comparative performance reports or used them to make health care choices, such as selecting a physician or hospital.¹ Consumer use of reports, however, has the potential to influence quality in at least three ways:

1. Provider performance information makes it more likely that consumers will understand the dimensions of quality and tap this growing understanding in obtaining high-quality health care for themselves and their family members.
2. The collective effect of many informed consumer choices may stimulate quality improvement among providers as they perceive that performance data can affect their market shares.
3. Public reports that affect providers' public image by clearly identifying them as high-quality or low-quality providers may encourage them to improve the quality of care they provide, to protect or enhance their reputations.

Finding ways to make public reports relevant and useful to consumers is part of an overall strategy to improve health care. This report, which is organized around a set of nine recommendations, is intended to help report card sponsors, including CVEs, achieve this goal.

Return to Contents

How Explanatory Information Makes a Report More Effective

Most Americans have some experience using data to compare products and services, but public reports of health care quality are not yet in the mainstream. Many people are unaware of their availability, and a large number of those reports have not been easy to understand and use. Evidence and experience make it clear, however, that report designers need to provide explanatory information, in addition to the actual comparative data, to maximize a report's use and impact. This report addresses this need. Whether information is presented online or in print, the right kind of explanatory information will:

1. Engage and motivate consumers to explore and use reports.
2. Deepen consumers' understanding of health care quality and quality measures.
3. Legitimize the report's sponsor and the report's credibility.
4. Provide information about the importance, meaning, and interpretation of specific measures.
5. Help consumers understand the implications of "resource use" information.
6. Help consumers avoid common pitfalls that lead to misinterpreting quality data.
7. Provide consumers guidance and support in using the information.
8. Provide consumers appropriate access to more detailed technical information.

To help achieve these eight aims, report card sponsors are encouraged to:

9. Test the report before going live.

The rest of this report examines these nine recommendations and offers guidance on how to develop and present appropriate explanatory information.

Recommendation No. 1: Engage and Motivate Consumers To Explore and Use Reports

Americans are inundated with information. In the same way that advertisers need to work to get their messages heard above the clutter,² sponsors of health care quality reports must give the public compelling reasons to pay attention to their data. Although many consumers have expressed (in focus groups, for example) considerable interest in quality data, that needs to be taken with a grain of salt. When people hear about a new source of information in a focus group, they are likely to find it interesting. But then life happens. People get busy, they are not reminded of the benefits of quality data, and the information loses salience. This does not mean that it is impossible to get people interested in comparative quality information. It does mean that their interest can never be taken for granted, but rather has to be developed and encouraged.

The first page of a report is critical. Once a user arrives at the first page—whether it is printed or online—there are just a few seconds to motivate that person to keep looking rather than moving on to something else or "surfing away," in the case of an online report. A few concisely stated key messages, emphasizing why this information is important and relevant to the consumer, can effectively capture a consumer's attention. Some examples are:

• The quality of the care among (insert facility type and community) can vary widely.
• Consumers who do not look at provider ratings are more likely to make a poor choice.
• A poor choice may have serious consequences for a person's health and finances.
• The information in the report can be useful in selecting a high-quality or high-value provider.
• The information also can be used in other ways (e.g., to help loved ones make a choice, to start a conversation with their doctor or another health professional).

The first page of the "Model Report" shown in Figure 1 is an example from an evidence-based comparative quality report that has been extensively tested and reviewed by experts. It was developed to report on the AHRQ Quality Indicators, a set of measures of hospital quality based on administrative data collected by most States.

Since this is not a "real" report, it does not have a graphic theme or "look." Its intent is to convey the content and language of a report. The key points include:

• A brief definition of quality in consumer-oriented language.
• The reasons for publishing comparative data on hospital quality.
• Several reasons an individual should look at this information.
• A brief summary of the information in the report.

Subheads and bullets break up blocks of text. A link at the bottom of the page immediately takes the user to the data. The rest of the report contains a lot more explanatory information, which is either wrapped around the data presentations or positioned at the end of the report.

Figure 1. Model Report on AHRQ Quality Indicators

Recommendation No. 2: Deepen Consumers' Understanding of Health Care Quality and Quality Measures

Consumers and health professionals understand quality issues related to health care differently. Health professionals typically see health care quality as multifaceted; some aspects viewed as critical by professionals are not, initially, seen the same way by consumers. For example, professionals see clinical process measures (e.g., providing a prescription for a beta blocker to patients discharged from the hospital after a heart attack) as critical components of quality and need little in the way of explanatory information. But consumers generally are not familiar with the evidence that links clinical processes to patient outcomes, which consumers care deeply about (e.g., not having another heart attack).

What is to be done about this difference in perspective? First, it is important to recognize that consumers will only find a report useful if it connects to what they already care about. Beyond that, public reports also offer an opportunity to deepen consumers' understanding of health care quality in a way that brings them closer to understanding it as health care professionals do. We need to engage consumers by using explanatory information to make a connection between what they already care about and the more sophisticated elements of quality they could easily come to understand and care about.

Provide a Framework To Help Consumers Understand Quality

One way to increase consumer understanding is to provide a simple, easy-to-understand but comprehensive quality definition or framework that builds on what is known about how consumers and patients define quality. An excellent quality framework that is generally accepted among health professionals already exists in the form of the "six aims" of health care articulated in the Institute of Medicine's report Crossing the Quality Chasm.³ Care should be: (1) safe, (2) effective, (3) timely, (4) patient centered, (5) efficient, and (6) equitable.

A simplified version of this framework was tested with consumers and seems to work well.⁴ The concept of safe care is described as "care that protects patients from errors and harm;" the concept of effective care is captured in the phrase "care that is proven to work;" and the phrase "care that is responsive to a patient's needs and preferences" reflects much of what is implied by "patient-centered," "timely," and even "equitable."

If reporting a full complement of measures, a report sponsor may opt to organize provider ratings into these three categories:

1. Section on "care that protects patients from medical errors and does not cause harm," which would include measures such as surgical infection rates or injuries from falls.
2. Section on "care that is proven to work," which would include measures such as percentage of diabetes patients who receive all five recommended tests regularly.
3. Section on "care that is responsive to a patient's needs and preferences," which would include measures such as patient experience.

For more information about this framework, refer to Best Practices in Public Reporting No. 1.

Make Sure Reports State Clearly Which Aspects of Quality They Cover

Reports need to make clear the aspects of quality they cover. Using everyday language and listing the main categories of quality data included help to orient the reader. For example, a lead-in to a report could spell out that the report contains measures on patient experiences and effectiveness: "In this report, physicians are compared on 'how responsive they are to patients' needs and preferences' and on 'whether they use practices that are proven to work.'"

The following report example (Figure 2) is from Minnesota Community Measurement, a member of Minnesota's CVE. The report's home page offers succinct explanations of the report's purpose and value, as well as of the measures presented and why they were chosen. This particular online report focuses narrowly on physician quality of care for people with diabetes.

Figure 2. The D5 Web Site Offers a Clear Explanation of Its Purpose and Value

The five goals for living well with diabetes (D5): controlling blood pressure, controlling cholesterol, controlling blood sugar, being tobacco free, and taking aspirin daily.

Recommendation No. 3: Legitimize the Report's Sponsor and the Report's Credibility

Focus group research on public reporting consistently finds that the public mistrusts quality information if they think the source of that information has an interest, especially a financial interest, in the ratings. Advertising, including provider and health plan advertising, is so ubiquitous that people are naturally wary that quality reports may simply be marketing ploys. Others may be concerned that the report has been developed by purchasers and heath plans to reduce coverage costs. The public needs to know that the data in reports are objective and that the report is sponsored and supported by trustworthy, expert sources.

The most important contributors to a report's credibility are the credibility of the sponsors and how clearly the sponsorship is presented. If a sponsor is well known and trusted by the consumer audience (not just among health professionals), little needs to be done beyond clearly identifying that sponsor. When sponsors are not well known to consumers, it is important to include a brief description or mission statement up front (in as few words as possible, such as the organization's "tag line") with a link to additional information in the back of the report. (Note in Figure 2 that a link to a description of the sponsor is at the bottom of the home page.) Explaining why the sponsor is issuing the report, and that it has no financial interest in the impact of the report on providers, can help reinforce its credibility to consumers.

If the report is sponsored or published by a CVE, the organization or name may not be familiar to consumers. Even if a CVE is not well known, some of its members likely are well known, and their explicit endorsement of the report can be valuable. Provide clearly labeled links to letters of endorsements from well-known organizations in the State or community, for example, via a special tab on the left-hand side of the home page. Make sure, however, that the full range of partners is shown as endorsing the report. If only one group (e.g., the health plan association, the medical society, a major employer) is shown, people might think it is "their" report and, perhaps, biased in "their" direction.

Establish Credibility by Demonstrating Fairness

The public wants to know that reports are fair to those being rated. Focus group researchers have heard repeatedly that a specific measure is not really the sole responsibility of the entity or individual being assessed. Some consumers say that the responsibility is broader—the patient's, or another health professional's, or shared by multiple professionals.

For example, when older women were asked about the inclusion of a mammography rate in an early HEDIS (Healthcare Effectiveness Data and Information Set) measurement set used to compare health plans, they thought that either the woman herself or her doctor was responsible for whether she had a mammogram, not the health plan.⁵ In response to this feedback, explanatory information was added to the presentation of these data in an early version of Medicare Compare. This information acknowledged that patients and physicians affect a health plan's mammography rate but also specified exactly how health plans can act to ensure that more women get needed mammograms.

Another way to demonstrate fairness is by describing key aspects of the sponsor's interactions with the providers who are rated. While it is a good idea to make clear that you are at arm's length with providers, it also can help to: (1) conduct a dry run of data collection and aggregation, which is reported only to providers, prior to the actual public report; and (2) give providers an opportunity to comment on the findings. If you take these steps, tell the public about them, briefly and in plain language.

Provide the Right Level of Detail To Ensure Credibility

Many report designers believe that for the public to trust a report, they need to know a lot of the technical details about how the data were collected and how the scores were generated. In particular, report designers think people need to know the extent to which differences between those rated are statistically or substantially significant. Because of this concern, some designers address statistical significance by including details (e.g., confidence intervals in graphs or highly technical presentations of data) in the main body of the report.

Such complex data presentations are unlikely to be either read or understood. In fact, consumers may see information about adjustments to the data as a sign that someone is "messing with the information." Therefore, it is important to find the right balance between technical details and summary information.

This challenge can be addressed by providing technical details in a special section toward the back of the report (after the measures or ratings). Links to this information should be provided early, however, to signal that the details are available to anyone who wants them. It is appropriate to have links like this throughout the report (e.g., via a tab at the left or on the top of the screen, for an online report) to reinforce the continuous availability of this information, as it is hard to predict when a given individual may want to look at it. Realistically, health professionals are more likely to look at this information than consumers are, but it must be made available to all. Most important, it must be written as clearly and simply as possible; unfamiliar terms need to be defined as they are used.

Technical details provided with the data display should include information about the time period covered by the data and data sources, including mention of whether data provided by the providers or health plans being rated are validated or audited in some way. When survey information is reported, people want to know that the sample was random and reasonably large. They also want to know that the surveys were conducted and scores generated by an independent entity.

Explain How Scores Were Generated

Scoring can make a big difference in the effect comparative data have on consumers' understanding of quality information. For example, when Hospital CAHPS^® data are presented on the Hospital Compare Web site, graphs show the percentage of patients in each hospital who gave the best possible rating for a given experience. However, when a composite of several measures is reported (such as, for example, communication between patients and their nurses) the graphs show the percentage of patients in each hospital who gave the best possible report on ALL items in the composite.

From a consumer-engagement perspective, it is a good thing if the information shows variation, because it reinforces the idea that there is real variation in quality. On a more fundamental level, it makes people more interested in the information and more likely to think it can help them make a good choice. However, these scoring decisions need to be made clear, both in the individual data presentations and in the technical details.

Another aspect of scoring is risk adjustment or, in some cases, "smoothing" of data through hierarchical modeling when some of the entities being rated are substantially smaller than others.^6,7 These strategies involve complex statistical techniques, and they cannot be presented in public reports using language one might use in a graduate (or even undergraduate) course. The language must be as simple as possible but not so simplistic that the steps taken to ensure accuracy and fairness of the data are unclear.

The authors have found through research that people tend to understand adjustments based on age or severity of illness but react negatively to adjustments based on social factors such as education level. To the extent that a report sponsor plans to stratify data by race, ethnicity, or income level, the authors caution that this will have to be done carefully so that consumers do not see it as a manipulation of the "real" data or discrimination against racial, ethnic, or income groups.

Recommendation No. 4: Provide Information About the Importance, Meaning, and Interpretation of Specific Measures

In addition to providing a broad framework that defines different aspects of quality, reports need to offer simply stated explanations around their graphic presentations of data. They need to describe how the measures relate to quality and, sometimes, how to interpret the graphic.

Use Terms Consumers Understand

Many reports justify using technical terms by including a glossary. People rarely use glossaries, however, and are not likely to examine information they do not understand. If technical terms are used, they must be defined immediately in everyday language that will be understood by individuals at an eighth-grade reading level or lower. An even better strategy is to use a common term (e.g., breast cancer screening test), with the technical term (e.g., mammography) in parentheses.

Ideally, the measures reported will have been vetted previously with consumers to see if they find them important, relevant, and appropriate to the providers or health plans being rated. If a measure has not been formally vetted, it may be necessary to conduct focus groups to obtain input on how to present it. Focus groups were conducted before finalizing the Hospital CAHPS survey, for example. They helped determine which items stayed in the survey and helped inform decisions about the contextual information needed.⁸ For more information about testing, refer to "Recommendation No. 9: Test the Report With Consumers Before Going Live" later in this report.

Explain Different Types of Measures

The explanatory information needed depends on the type of measure, because consumers' interest in and level of understanding of the different types of measures will vary. When developing such information for public reports, consider the following key points about each type of measure:

• Patient experience measures, such as those derived from CAHPS^® surveys: People seem to naturally understand this kind of measure. Most, but not all, people value and will use rating information from other patients and consumers.
• Outcome measures: These measures are just beginning to be included in reports. Early testing on these measures with consumers indicates a range of responses to them:
  • Patient safety measures, such as measures of the frequency of infections, falls, and other negative consequences of care delivery: Once explained in plain language, these measures seem to resonate with many consumers. It appears important when presenting safety measures to emphasize that specific bad outcomes can be prevented by providers.
  • Mortality measures: These elicit a wide range of responses from consumers. Some say they do not want to read or think about the potential of death when they seek medical care. Others believe it is important to have this information, however, particularly for serious illnesses or high-risk procedures. Still others think that simply not dying does not equate to high-quality care. Their definition of quality goes far beyond that to issues of being cured, improving quality of life, or having a positive experience as a patient. Finally, people's interest in a disease- or procedure-specific mortality measure will depend on whether they care about that disease or procedure. It makes sense to include outcome measures in public reports; however, some may require explanatory information, and all must be described in lay terms.
• Clinical process measures: It is almost always necessary to explain these kinds of measures. Clinical processes are not familiar to many consumers, and they rarely know the evidence of how a particular process supports a desirable outcome. The report must use plain language to describe the process, so that the importance is clear (e.g., patient given right medication at right time). The label should help people make the connection between the process and the outcome.
  For example, the HEDIS assessments of health plans include a mammography measure as a key effectiveness indicator. When this measure was introduced, mammography rates were considerably lower than they are now. At that time, many women did not know the benefit of getting a mammogram. They typically thought of it as a test they had to take (and pass) to make sure they did not have breast cancer. However, a much more important benefit of getting a mammogram is that the earlier breast cancer is detected, the greater likelihood that it can be treated successfully, often using less invasive treatments. Explaining this in plain language links mammography rates to outcomes that are important to all women: being cured and having less physical disfigurement.
  When these data were first presented, the Centers for Medicare & Medicaid Services (CMS) included an explanation of these benefits in a section of the report called "Why Is This Important?" Further, because women often did not think health plans could do much about a mammography rate, CMS included another section titled "What Does This Measure Show?" that discussed how health plans influence mammography rates.

Provide Guidance on How To Read Graphs and Understand Measures

Often people need to be told what to look for in a graph. For example, they need guidance to understand if a high score or a low score indicates higher performance. A bar on a graph to indicate the average or some other benchmark requires an explanation, as shown in Figure 3.

Figure 3. A Plain Language Explanation Helps Consumers Interpret Data Quickly

Recommendation No. 5: Help Consumers Understand the Implications of Resource Use Information

"Resource use" is a broad term, which is intended to include measures of price and efficiency and capture potential inefficiencies such as hospital re-admissions. Report sponsors are just beginning to include resource use information in public reports; there are few well-developed measures in this arena. Resource use measures need to be relevant to consumers, but as one might expect, there is not much research yet on what is relevant to them and how the measures should be presented so that they will not be misunderstood. The term "resource use" has not been tested with consumers and is likely to be poorly understood.

Two common consumer beliefs are likely barriers to using cost and efficiency measures effectively. One is the widespread belief that "more care is better." Consumers are much more concerned about not getting something they need than they are about getting care they do not need.

The second common belief is that price, as with most consumer goods, is a reasonable proxy for quality. That is, when consumers are not getting a clear message about quality, they are likely to use cost as a proxy for quality. This can result in counterproductive choices.

Beginning to address these misconceptions in public reports is a first step in communicating about resource use issues. Keeping these misconceptions in mind as sponsors create approaches for reporting on resource use measures will be essential.

Recommendation No. 6: Help Consumers Avoid Common Pitfalls That Lead to Misinterpretation of Quality Data

While the authors do not recommend providing extensive statistical detail within data presentations, it is important to include caveats about certain measures that could otherwise be misleading. As shown in the examples below, reports must communicate to users:

That they should use caution in comparing providers on certain measures.
That they cannot assess a provider's overall performance by extrapolating from a limited set of measures that reflect only some of the services they provide.

Some patient safety measures address extremely rare events, including "never events." The problem with measures of rare events is that it is difficult to make valid comparisons among providers or health plans. For example, Hospital A reports two deaths in a given year among hip replacement patients, while Hospital B (with about the same number of surgeries and mix of patients) reports one death. Consumers might falsely conclude that Hospital B's performance is twice as good as Hospital A. Since deaths among hip-replacement patients are extremely rare, the difference in rates in a given year may be random—more noise than signal.

Here are some options for addressing this kind of measure:

• Exclude such measures from your report.
• Combine data for multiple years so that the noise diminishes. This means changing the measure from deaths in a single year to deaths over the last 3 years or last 5 years. Even then the rates may be extremely low.
• Aggregate data from one of these measures, such as deaths among hip-replacement patients, with deaths from other low-risk surgeries or procedures.
• Present data on the measure, but add a statement with the data, pointing out that the event is rare and that users should be cautious in comparing facilities or providers on the basis of this measure alone.

If you do report such measures, consider using counts rather than rates. Because so many Americans have limited numeracy, it is difficult for them to grapple with a rate such as 1 death per 10,000 patients or .0001 percent.

Another common misunderstanding is that the measures included in an online or printed report tell the whole story of a given provider's or plan's performance. Consumers may sometimes think they have the whole story and can judge a hospital, for example, based solely on how it performs on measures related to cardiovascular care, even when they are going to the hospital for cancer surgery. This kind of caveat is better presented in a section on how to use the report than near any single data presentation, since it really speaks to the report as a whole.

Recommendation No. 7: Provide Consumers Guidance and Support in Using the Information

We have reviewed how the home page of the report presented on a Web site or the first page of a printed report is a good place to quickly indicate the benefits of a report (i.e., how it can be used by consumers). Additional "decision support information" can be helpful in providing guidance and motivation for those who are not familiar with or are less confident in using quality information for themselves or their loved ones.

In many cases, the desired primary use of comparative quality information is to influence consumer choice. Simply providing information, however, is not sufficient. Many believe that "decision support" must be included in reports.⁹ For example, reports often provide information on a wide range of measures but do not address how to put all the information together to make a decision. The cognitive burden of this task is extremely high for almost anyone.¹⁰

Efforts have been made to provide formal decision support within online reports by asking individual users what is most important to them and then using internal functionality to calculate the available options that best meet their preferences. This is a complex undertaking, made more difficult by the fact that many health care consumers find it hard to identify the aspects of quality that are most important to them.

Following are five simple, practical approaches to providing decision support for consumers:

1. Provide a list of what consumers should think about when they make a decision of this kind. This list can include issues related to information covered in the report, but it may include issues other than quality, such as convenient location or languages spoken by staff. For example, the report could list several things to consider in choosing a primary care physician, such as how well each communicates with patients, how easy it is to make an appointment and whether the physician provides needed preventive services on time, and whether the physician delivers the right care to patients with specific conditions.
   Going beyond typical quality measures, the list might also include whether the physician's office is located conveniently for the person, whether the physician is included in the network of the patient's health plan or accepts the patient's insurance, and whether the physician speaks a language other than English. If consumers are urged or required to gather information from outside sources, the report should contain links or other directions on how to access that information. This kind of simple decision support has been shown to be highly valued by many consumers.¹¹
2. Identify, through a label or a visual such as a checkmark, which of the choices is the "best scoring" across the board or the "best value" if cost data also are included. Consumer Reports has used this approach for many years. This kind of label operates like a summary score.
   Composite scores, common in survey-based measures, can be helpful in reducing the total number of data points in a report, but they do not help as much as summary or "roll-up" scores. Unlike composite scores, summary or "roll-up" scores combine a large number of specific measures that may or may not be highly related to one another statistically. They relate to a single provider or facility. Consumers often say they want summary or "roll-up" scores, which can make it easier for consumers to evaluate choices and make decisions.
   Two issues must be addressed in developing and reporting summary or "roll-up" scores, however. First, it is important not to wash out any variation across providers—something that will make it harder for consumers to make a decision. Summary scores that reveal that some providers and facilities are better across the board can be extremely helpful to consumers.
   Second, care must be taken in weighting items in summary scores (i.e., giving more weight to some measures than to others). For example, it might seem obvious on the face of it to assign greater weight to a measure of the number of patients who die from central-line infections versus another measure of how quiet a hospital is at night. It is not clear, however, what the right weighting would be for any number of measures, and it is inevitable that different consumers will weight a set of measures differently.
3. Call out key differences in performance, (i.e., pointing out places where differences in scores are particularly large).
4. Provide examples of specific ways consumers can use information, not just for making personal health care choices, but also to learn more about what kind of care is high quality, to help loved ones make a decision, or to begin a conversation with their physician or other provider. Stories and testimonials can illustrate how information can be applied (e.g., including first-person statements by consumers about how using the report made a difference in their choices, health, or finances).
5. Make explicit what actions consumers can take to protect themselves from poor-quality care. The most obvious step consumers can take is to avoid choosing and using poor-quality providers. Sometimes, as we know, consumers have little or no choice of health plans or hospital. One step a consumer might take in this case would be to talk about the issue of poor hospital quality with his or her physician.
   In recent research, physicians were asked how they would respond if patients expressed concerns about hospital quality information they had seen in a public report.¹² While many physicians said they would try to reassure their patient about the quality of the hospital, many also said they would alert the involved specialist about the patient's concerns, be vigilant about specific concerns while on rounds, or speak to the nursing staff or, in some cases, hospital executives. A few physicians said they would encourage the patient and family to be proactive in monitoring the patient's care and to make it clear when they thought something was not going as it should. This research implies that consumers can and should talk to their doctors about quality concerns.

Recommendation No. 8: Provide Consumers Appropriate Access to More Detailed Technical Information

People can become overwhelmed and give up on a report if they cannot find what they are looking for. Report design should use layering and navigation aids so that users are not overwhelmed with information or required to look at information that is of lesser interest to them. In general, this means report designers need to balance the consumer's need for brevity and ease of access to what they care about most with the sponsor's need to provide full descriptions, caveats, and disclosures about the data. Using the Web to disseminate reports makes creating this balance much easier than in print reports.

A good Web site report incorporates several features to help people know where they are and get where they want to go:

• A set of tabs at the top to help users get to the main sections from anywhere in the report.
• A set of tabs at the side to help users navigate within report sections.
• Internal links from one part of the report to another.

The amount of explanatory information provided up front should be limited, unless there is a special reason for breaking this rule. (For an example of such a case, go to Figure 4.)

Figure 4. Example of a Case When More Up-Front Information Is Useful¹³

Explanatory information that is best located toward the back of a report includes:

1. Technical details of how data are collected and analyzed.
2. Methods to provide feedback or ask questions about the report.
3. Letters of endorsement.
4. More detailed information about how to use the report such as described in Recommendation No. 7.
5. Things to keep in mind while using the data (this is where the caveats go).
6. Links to other reports on quality that cover aspects not covered in the current report.
7. Additional resources and links on topics relevant to the report (e.g., information about diseases or conditions if the report is organized that way).
8. Contact information for rated providers and health plans, if the number is small.

Although every report does not need to include all of the above, the first and second items are the most important to include. Even if consumers do not look at the technical details, they like to know they are there. Their advocates, as well as other stakeholders, will look at these details carefully.

Last, but definitely not least, offering a vehicle for feedback reinforces the idea that the sponsor is open to improvement and provides a quick and inexpensive way to get information to evaluate the effectiveness of the report. Allowing users to ask questions may be helpful in situations where consumers need clarification about different reports on the same provider or plan.

Recommendation No. 9: Test the Report With Consumers Before Going Live

Report format, language, and structure should be tested with consumers. This sounds daunting, but it does not have to be. The best way to test is through cognitive interviews.¹⁴ This technique was initially created to assess survey questionnaires to make sure potential respondents would consistently understand questions and possible response options as the survey developers intended. Cognitive interviews are the gold standard in survey testing. For more than a dozen years, cognitive interviews have been used to test public reports to ensure that consumers understand them as intended, that they can navigate their way around them, and that they are likely to respond positively to the content and how it is presented.

Conduct Cognitive Tests

Use a draft that looks as much as possible like the potential Web site or printed report. Then develop a protocol for cognitive testing and recruit 10 to 15 subjects who reflect a cross-section of the intended audience. Each test should take about 90 minutes and involve three persons—a subject, an interviewer, and a note taker—where possible. Tape-recording each interview is recommended. Watch for parts of a page where the respondent spends more time or less time, when they seem confused, etc. It is helpful to offer a financial incentive to subjects for their participation, ranging from $75 to $125, depending on the local cost of living. Saturation (where little more can be learned) is reached after 10 to 15 individual cognitive tests.

The key to a good cognitive test is to guide people, step by step, through the report, get their reactions to it, and ask questions to find out if they understand both the text and the graphics. Do not ask people if they understand; rather, ask them a knowledge question and see if they respond with the correct information. For example, ask them to identify the highest and lowest performers, to characterize the main point of a section of text, to discuss why an average is included, or to describe where the data come from. Ask if anything in the report makes them trust or distrust it, where too much or too little has been said.

Do not be surprised if people are brutally honest and if the process turns out to be somewhat painful, especially for those who have agonized over just the right wording and graphics. Refine reports in response to the results of these tests. Schedule two or three rounds of tests, revising the draft report between rounds, if time and money allow.

Disciplined testing and use of evidence-based practices will help improve the likelihood that the report will be understood, appreciated, and most importantly, used.

Return to Contents

Purpose of Report Series

The purpose of this three-part series of reports is to provide practical approaches to designing public reports that make health care performance information clear, meaningful, and usable by consumers. The goal is to help sponsors present information so that a wide variety of people can understand and apply it easily to key decisions, even if they do not want to spend a lot of time on details and have limited technical knowledge of the subject.

Together the three reports cover the wide range of issues and challenges faced by report sponsors:

• Best Practices in Public Reporting No. 1: How To Effectively Present Health Care Performance Data to Consumers focuses on the challenges involved in designing a public report card so that the performance information is easily understood by consumers and on strategies to make it easier for consumers to understand and use comparative health care quality reports.
• Best Practices in Public Reporting No. 2: Maximizing Consumer Understanding of Public Comparative Quality Reports: Effective Use of Explanatory Information focuses on the explanatory information in public reports, beyond the performance data itself, that helps to accurately communicate quality ratings to consumers and motivate them to use the ratings in making informed health care decisions.
• Best Practices in Public Reporting No. 3: How To Maximize Public Awareness and Use of Comparative Quality Reports Through Effective Promotion and Dissemination Strategies applies social marketing and other principles to explore how to target reports to specific audiences, develop messages to promote the report with key audiences, engage consumer advocacy and community groups in promoting reports and helping people use them, disseminate reports through trusted channels, and ensure that consumers see and use comparative quality reports.

Return to Contents

References

1. 2008 update on consumers' views of patient safety and quality information. Menlo Park, CA: Kaiser Family Foundation; October 2008.
2. Shaller Consulting. Consumers in health care: creating decision-support tools that work. Oakland: California HealthCare Foundation; June 2006.
3. Institute of Medicine Committee on Quality Health Care in America. Crossing the quality chasm: a new health system for the 21st century. Washington, DC: National Academy Press; 2001. p. 5-6.
4. Hibbard JH, Pawlson LG. Why not give consumers a framework for understanding quality? Jt Comm J Qual Saf 2004 June;30(6):347-51.
5. McGee J, Sofaer S, Kreling B. Findings from focus groups conducted for the National Committee for Quality Assurance (NCQA) Medicare and Medicaid Consumer Information Projects. Washington, DC: National Committee for Quality Assurance; July 1996.
6. Iezzoni LI. Risk adjusting rehabilitation outcomes: an overview of methodological issues. Am J Phys Med Rehabil 2004 Apr;83(4):316-26.
7. Shahian DM, Normand ST. Comparison of "risk-adjusted" hospital outcomes. Circulation 2008 Apr;117:1955-63.
8. Sofaer S, Firminger K. Patient perceptions of the quality of health services. Annu Rev Public Health 2005 Apr;26:513-59.
9. Spranca M. RAND report summary: consumer use of information when making treatment decisions. Chicago: BlueCross BlueShield Association; 2005.
10. Peters EM, Hibbard H, Slovic P, et al. Numeracy skill and the communication, comprehension, and use of risk and benefit information. Health Aff 2007;26(3):741-8.
11. McCormack LA, Garfinkel SA, Hibbard JH, et al. Health insurance knowledge among Medicare beneficiaries. Health Serv Res 2002;37(1):43-63.
12. Barr JK, Bernard SL, Sofaer S, et al. Physicians' views on public reporting of hospital quality data. Med Care Res Rev 2008 Dec;65(6):655-73; Epub 2008 July 2.
13. Sofaer S, Hopper SS, Firminger K, et al. Addressing the need for public reporting of comparative hospice quality: a focus group study. Jt Comm J Qual Saf 2009;35(8):422-9.
14. Harris-Kojetin LD, Fowler FJ, Brown JA, et al. The use of cognitive testing to develop and evaluate CAHPS^® 1.0 core survey items. Consumer Assessment of Health Plans Study. Med Care 1999;37:MS10-21.

Return to Contents

Acknowledgments

The authors wish to thank the following people for reviewing this document: Katherine Crosson, M.P.H., Agency for Healthcare Research and Quality; Irene Fraser, Ph.D., Agency for Healthcare Research and Quality; Linda Greenberg, Ph.D., Agency for Healthcare Research and Quality; Jan De La Mare, M.P.Aff., Agency for Healthcare Research and Quality; Lisette Lejnieks, M.B.A., independent consultant; Peggy McNamara, M.S.P.H., Agency for Healthcare Research and Quality; Chris Queram, Wisconsin Collaborative for Healthcare Quality; Dale Shaller, M.P.A., Shaller Consulting; Karen Shore, Ph.D., Center for Health Improvement; Nancy Brands Ward, Center for Health Improvement; and Jan Whitehouse, Greater Detroit Area Health Council.

We consider our Learning Network tools to be works in progress and always welcome your comments. Please forward suggestions to Peggy McNamara at peggy.mcnamara@ahrq.hhs.gov.