Future Directions for the National Healthcare Quality and Disparities Reports
Chapter 6: Improving Presentation of Information (pt. 3)
Refining the Presentation of Data
The success of the NHQR and NHDR in reaching various audiences and spurring action depends on the presentation of information. The products developed by AHRQ have the potential to tell a more complete quality improvement story, provided the products are accessible, engaging, and informative.
Improving the Presentation of Graphic Displays
Graphic displays in the NHQR and NHDR document historical trends, present geographic variation using maps, and stratify measures by demographic characteristics. Communicating information through the simultaneous presentation of words, numbers, and, in some cases, pictures, requires that the displays be as effective as possible (Tufte, 1983). Therefore, visual design problems can undermine the usefulness of data being presented (Few, 2006). To assess the quality of graphic displays in the two reports, the committee commissioned input from Howard Wainer, an expert on data display.11 The suggestions presented to AHRQ are only one way of enhancing data displays, but they represent well-regarded, theory-based practice.
Documenting Historical Trend Data
As noted earlier, the committee finds there should be less reliance in the NHQR and NHDR on trended data displays unless the trends inform future activities rather than solely document the past. Currently, most trend data take up substantial space in the documents without being particularly informative other than to reinforce a repeated message—that the pace of change is slow. These trend graphs are often visually cluttered with overlapping lines and many numbers over-written on a graph (Box 6-3).
Box 6-3. A Suggested Approach to Improving Data Displays
Figure A (both before and after) presents data on patients with tuberculosis who completed treatment, but the original Figure A (below left) is visually cluttered: the graph contains a multitude of data points, a y-axis that is not descriptive, a caption that does not convey findings, and labels that are far from the data points. The reader is less readily able to discern which age groups performed best, which performed worst, and if any age groups had improved in the percentage of patients that received the recommended care.
To more clearly convey information, Figure A (below right) was revised so that the reader can more readily gain a sense of the component data without having to append the "visual noise of numerical values." These improvements include:
Additionally, instead of a heading that defines the measure specifications, the heading was changed to an informative caption that conveys the graph's key finding: "Although rates of completion of tuberculosis treatment have been increasing overall, adults are 10% less likely than children to complete treatment."
When AHRQ determines the scales of the x- and y-axes, the purpose of the graph should be taken into consideration. For example, in the "after" figure below, the compressed y-axis scale may exaggerate differences between age groups. However, the compression allows readers to more easily determine the best and worst performing groups. AHRQ should weigh these considerations and consider the absolute level of performance when choosing axes scales.
Source for the "Before" Graph: AHRQ, 2009e, p. 83.
Captioning and Labeling
Captions for each display should be informative and focused. A good graph can be made even stronger by having an informative and interpretive caption or figure heading. Captions that explicitly relay the principal point of the display have benefits: the reader can discover the point of the display more easily and less helpful displays are eliminated. Additionally, strong graphical displays avoid legends whenever possible because legends require the viewer to learn the legend and apply it to the display. This requires two moments of perception and makes the viewer read the display rather than see it (Green, 1998). Go to Boxes 6-4 and 6-5 for examples.
Box 6-4. A Suggested Approach to Improving the Labeling of Graphic Data Approach to Improving Data Displays
Figure B (both before and after) presents data on rates of pressure sores among nursing home residents. The original figure (below left) could better convey its key finding (that although rates of pressure sores have been declining, Black residents still have higher rates compared to all other racial groups) with modifications:
Terminal year data should be included in text if AHRQ decides not to include it in the graph.
Source for the "Before" Graph: AHRQ, 2009d, p. 84.
Box 6-5. An Example of a Complex Data Display
An informative heading or caption should explain what constitutes better performance. The measure captions currently used as headings in the NHQR and NHDR do not always indicate whether better performance is associated with a positive percentage change or a negative percentage change. As shown in Figure C, below left, without reading the supporting text for this figure, a reader might not readily grasp that being on a transplant waiting list for a dialysis patient is a positive thing and that a high percentage is desirable.
Source for the "Before" Graph: AHRQ, 2009e, p. 50.
The choice of x- and y-axis scales can influence the readability and interpretability of a graph. The x- and y-axis should place observed differences on a scale that acknowledges the range of possible clinically important differences. In a series of experiments conducted at AT&T Bell Laboratories, Cleveland and colleagues determined that scales should be chosen so that the data fill as much of the scale-line rectangle as possible (Cleveland, 1994a,b). A separate issue that must be considered is the choice of the ratio of the x- and y-axis scales. Altering the ratio of the scale can modify a person's perception of the data (Cooper et al., 2003; Schriger and Cooper, 2001). The committee encourages AHRQ to choose an aspect ratio that appropriately conveys the data.
Alternative Data Displays
In addition to displaying trend data in graphs, AHRQ might consider utilizing alternate visual displays. For example, Figure 6-2, which was created by the CDC, succinctly presents information to readers, including readers who may not be data experts. The figure could be further improved by specifying whether the symbols represent absolute numbers of infected people or a rate ratio. In creating the figure, CDC likely meant for the figure to represent a rate ratio; however, readers may draw the conclusion that seven times more African Americans are infected than Whites (an absolute count). The display could be made clearer by including 100 small symbols for each group, and coloring in 7 for the African American population, 2.5 for the Hispanic population, and 1 for the White population.
The committee recognizes that there are benefits to readers in using a small number of graphic formats with the same type of display from page-to-page, so that readers do not have to learn to interpret a new type of graph, but finds that some diversity of presentation can enhance a report. Alternative displays might be particularly useful for the Highlights section and fact sheets.
The legends on the maps in the NHQR and NHDR are often uninformative as they are simply above average, average, below average, and no data. For ease of comprehension, the labels might, at a minimum, contain numeric values (averages and ranges). Additionally, ordering by performance level achieved makes a coherent visual impact and suggests an implicit underlying structure. For example, gradations of a single color would better show performance levels on maps (go to Figure 6-3). A sequence of "increasing darkness" of a single color can assist the reader in identifying increasing or decreasing rates, as utilized by Pickle and colleagues over five gradients (Pickle et al., 1996). Additionally, colors should be chosen to avoid common color vision deficiencies and so that no single color visually dominates (Pickle et al., 1996). In the NHQR and NHDR maps, the color black represents better performance and AHRQ's use of two other colors (green and blue) does not have the visual impact of a single color gradient. Examples of such visual displays abound, and the committee believes that AHRQ may benefit from additional professional consultation on how to better present its data.
Enhancing the Supporting Text for Data Displays
The text supporting a data display should convey information gleaned from data analysis, such as analysis not captured in the figure and implications of significant findings. Currently, supporting text for displays in the NHQR and NHDR describes what the graph depicts. The text refrains from providing additional analyses and provides minimal direction on methods that could be used to improve quality or eliminate disparities.
Refining the Presentation of Summarized Information
Summary and composite measures are useful tools for conveying information about complex constructs, such as the multiple elements of appropriate care for a stage of life (e.g., end-of-life care) or a condition that is inadequately portrayed through a single measure (e.g., diabetes). To be consistent with AHRQ's use of the terms composite and summary measures, this report defines composite measures as the bundling of two or more measures that look at different aspects of care for a specific clinical condition (AHRQ, 2008b).12 As an example, the composite measure on diabetes care is the percentage of adults age 40 and over with diabetes who received all three recommended services (hemoglobin A1c measurement, dilated eye examination, foot examination).
Summary measures bundle a number of conceptually similar specific measures of health care services or outcomes across multiple conditions or health care settings in order to present a single metric for a given aspect of health care delivery (e.g., combining performance rates for all prevention measures). AHRQ's State Snapshots present such summary measures to report the performance of single and combined states on measures for different types of care (i.e., preventive, acute treatment, chronic care) and settings of care (i.e., home health, hospital, nursing home, ambulatory) (AHRQ, 2009a). Similarly, AHRQ summarizes measures in the Highlights section by core measure totals, types of settings, and types of clinical measures (including some clinical conditions across composite and individual measures).
The committee's purpose is not to recommend specific composite or summary measures for inclusion in the national healthcare reports; rather, the committee considers desirable properties that AHRQ may consider when evaluating the way in which such measures are reported. A principal consideration in the use of a composite or summary measure is the quality of the individual measures being inputted and the relationship of these measures to one another (Murray et al., 2000). The weight of the measures that comprise the composite or summary measure may need to be considered. AHRQ does not use differential weights in its composite and summary measures; rather, it weighs every component measure equally. Implicit in choosing weights are subjective judgments about the relative clinical significance and prioritization of the component measures. AHRQ should clearly denote that composite and summary measures use equal weights and provide the denominators for each component measure (in an appendix, for instance) so that users of the data can perform their own analyses using differential weights, if they so choose (Martinez-Vidal and Brodt, 2006).
Presenting the Methodology of Summary and Composite Measures
Several standard pieces of information should accompany any composite or summary measure. While such information need not be displayed in the main body of a report, it should appear at least as an appendix, including:
- The methodological considerations taken into account when creating a composite or summary measure (e.g., how the measure is weighted).
- A description of the individual constituent measures that make up the composite or summary measure, their data source, and the distribution (e.g., means, standard deviations, ranges, floor and ceiling effects).
- A summary description of the psychometric properties of the composite measure, including how the component measures relate to each other (i.e., the pair-wise correlation coefficients of the individual quality measures or a coefficient alpha).
- The standard error of the composite measure, in addition to the estimated composite measure.
The general methodology for the composite measures presented in the 2008 NHQR is discussed in the print report (AHRQ, 2009e, p. 20), and some measure specifications for composites included in the NHQR and NHDR are provided via online appendixes.13 However, the appendixes do not contain all of the information outlined above. For instance, data on the individual constituent measures for the reported composite measures are sometimes unavailable or not easily accessible. Likewise, some methodological information is provided online for the summary measures used in the State Snapshots14, but much of the above information is also missing for those measures.
For example, the first figure presented in the 2008 NHQR (AHRQ, 2009e, p. 3) pools trend data from quality measures to quantify the overall change in quality for the health care system according to the measures AHRQ has chosen to profile. The median annual rate of change from baseline to most recent data year is reported as 1.4 percent. The NHQR does not, however, report the distribution of the underlying rates of change across measures, including the distribution and variability of the underlying rates. While it is important to know how many indicators are getting better and how many are getting worse, standard errors and correlations in rates of change are essential to identifying which measures tend to improve or worsen together.
The committee recognizes the benefits of using composite measures and summarization techniques, and AHRQ staff should continue to identify measurement areas that can benefit from such presentation. However, the committee finds that AHRQ needs to be more transparent in its methods. Methodological information may be presented in the print and online reports, although such detail may be more appropriate for appendixes where researchers who need such facts can obtain them.
Enhancing the Summary Dashboards of the State Snapshots
Dashboards are a valuable tool for efficiently and effectively communicating summarized information (Few, 2006). AHRQ utilizes this technique to provide a picture of how a state is performing relative to other states on "overall health care quality" and for 12 topics across types of care (i.e., prevention, acute, and chronic), settings of care, and clinical conditions.15 Despite the intended purpose of simplistically conveying information, the state dashboards may confuse users. For instance, Montana appears to be doing worse today than in the baseline year, although performance may or may not be better than in the past. The Montana dashboard does not say the arrows on the meters are reflecting relative performance, nor does it have a statement such as, "Montana's overall performance is better in the most recent data year than its baseline performance, but other states have improved more, so its overall performance ranks lower than previously."
In the State Snapshots, Arkansas is positively rated for having a low disparity rate. 16 This rating, however, may not reflect better outcomes. The low disparity rate is principally because the performance metrics of Arkansas's White population are lower than the corresponding data for the White population in other states. Meanwhile, the quality data attributed to Arkansas's Black or African American population are in line with the corresponding national measures. Thus, lower quality metrics associated with both White and Black individuals in Arkansas results in a smaller difference between the two populations (and thus a smaller disparity). 17
Statistical Quality of Data Reporting
Given the volume and numerous sources of reported measures, there are challenges in providing clear and useful information to readers. However, clearly stating the analytic methodology for the reports and making this methodology more readily available is important for the researchers, as they may seek to manipulate the data for their own purposes, or look to replicate such measurement reporting. Providing such methodological information also enhances the transparency of the NHQR and NHDR.
For three sections of the NHQR or NHDR, the committee assessed (1) measurement properties and definitions of quality indicators, (2) the description and use of analytical adjustments, (3) methods of summarization, (4) selection and use of benchmarks, and (5) use of prediction rules. (Go to Appendix H for additional information.) The committee's review indicated that, when possible, AHRQ should make available online the following supplementary information to inform the research community:
- Data quality. Information regarding who collected the data, the reliability and validity of collected data, limitations of the data, and the extent of missing data should be reported. While this information may be difficult to gather, the quality of the NHQR and NHDR hinge on the quality of the data. A standard template could be constructed and populated, and when information cannot be determined, at a minimum, this fact could be stated.
- Description and use of analytical adjustments. Key features of analytical adjustment are required for readers to understand and correctly interpret findings. These features include a clear definition of the outcome (including the units of measurement); the observed covariates and definitions used in adjustment; justification for adjustment and how the adjustment was made; the sample sizes or weights used in the analysis; the reference population used; and how well the statistical model performed (fit) for adjustment.
- Summary measures. The choice and definition of methods of summarization should be made explicit. For example, if the summary measure is a change in performance from one time period to the next, the time periods need to be stated; the estimate should be defined (regression-based coefficient or difference in means); and the statistical significance or other metric for displaying uncertainty in the estimate should be provided.
- Prediction rules. In some instances, prediction inferences for when a particular goal will be achieved are made. In such instances, the statistical model used for the prediction should be stated, its fit assessed relative to reasonable competitor models, and the statistical uncertainty surrounding the prediction should be reported. One prediction would be the number of years to reach a particular benchmark at the current rate of change.
Conclusions on Data Presentation
Taking advantage of the full power of data displays and concise summarization will be critical for AHRQ to continue to streamline a vast amount of information. To strengthen data presentation in the reports, the committee recommends:
Recommendation 8: AHRQ should engage experts in communications and in presentation of statistical and graphical information to ensure that more actionable messages are clearly communicated to intended audiences, summarization methods and the use of graphics are meaningful and easily understood, and statistical methods are available for researchers using data.
The data presented in the NHQR, NHDR, and their related products need to provide clear and coherent messages about the state of health care and the level of quality that has been achieved. The reports should strive to promote actionability by relaying realistic benchmarks and leading users to resources that illuminate methods of quality improvement and disparities elimination. As discussed, AHRQ can explore various dissemination strategies to ensure the messages are effectively conveyed to relevant audiences. By employing the messaging and presentation strategies discussed in this chapter, the NHQR and NHDR may be more valuable to a wider spectrum of users while still presenting data and methods useful to researchers in the field.
11 Howard Wainer's paper, "Commentaries on the 2008 National Healthcare Quality Report, the 2008 National Healthcare Disparities Report and State Snapshots," was provided directly to AHRQ staff and archived in the IOM public access file for the Future Directions project.
12 Ten out of the 12 reported composite measures in the reports involve the bundling of process measures, while the other 2 involve outcome measures for surgical procedures.
13 The 2008 measure specifications are accessible at (accessed January 15, 2010).
14 Accessible by visiting the Methods section of the State Snapshots at http://statesnapshots.ahrq.gov/snaps08/Methods.jsp?menuId=58&state=#stateSummary (accessed January 15, 2010).
15 Go to the Montana dashboard at http://statesnapshots.ahrq.gov/snaps08/dashboard.jsp?menuId=4&state=MT&level=0 (accessed December 8, 2009).
16 Go to Arkansas: Focus on Disparities at http://statesnapshots.ahrq.gov/snaps08/disparities.jsp?menuId=47&state=AR&level=83 (accessed December 20, 2009).
17 Personal communication, William Golden, University of Arkansas for Medical Sciences and Arkansas Medicaid, Department of Human Services, December 8, 2009.