Section 4. Visitors' Overall Ratings of the Web Sites

To determine how satisfied consumers and health care professionals are with the Web sites, we analyzed the bounce rate from Web analytics along with results of the following five survey questions:

• How would you rate your experience using the Web site? (0=Poor; 10=Excellent).
• How easy was it to use the Web site? (for example, not too many clicks, easy to search the Web site) (0=Very difficult; 10=Very easy).
• What was the primary purpose of your visit to the Web site today?.
• Were you able to accomplish the primary purpose of your visit? (Definitely, Mostly, Only partially, Not at all)
• How likely are you to use the Web site information now to choose a hospital or change hospitals? (Definitely, Mostly, Only partially, Not at all).

One key indicator was the average rating for overall satisfaction, as measured in the "Experience" question above. A second indicator was the average rating for overall usability, as measured in the "Usability" question above. The third indicator combined the next two questions. It was calculated as the percentage of respondents able to accomplish their primary purpose, among those whose primary purpose was to choose or compare hospitals or see the performance of a specific hospital.

Last, the likelihood of using the information to choose a hospital or change hospitals is an indicator of consumer engagement with site content. For exact text of the survey questions, response options, and data about answers by respondent type (patient, friend or family member, or health care professional), go to Appendix B.

Key Findings

Overall satisfaction varied considerably among the sites. Among the 1,034 respondents from all Web sites, the mean overall satisfaction score, rated on a scale of 0-10, was 6.84 (the median was 7). Overall scores for individual Web sites (among those with at least 25 survey responses) ranged from a mean of 4.8 to 7.43.

Satisfaction was lower among consumer visitors than among health care professionals. The mean satisfaction score among consumers (n=438) was 6.38, with a range among individual Web sites with at least 20 consumer respondents of 5.67 to 7.36. The mean for health care professionals (n=294) was 7.35, with a range for individual Web sites with at least 15 health care professional respondents of 7.23 to 8.5.

Overall usability scores varied considerably among the sites. On a scale of 0-10, all respondents across all Web sites (n=1,034) gave the sites a mean usability score (that is, how easy it was to use the Web site) of 7.21 (the median was 8). Mean overall ratings for individual Web sites ranged from 5.24 to 8.0 among Web sites with at least 25 survey responses.

Usability scores were lower among consumer visitors than among health care professionals. Consumers gave lower usability scores (mean of 6.83; range among sites with at least 20 consumer respondents: 5.9-7.5; go to Table 7) than health care professionals (mean of 7.67; range among sites with at least 15 health care professional respondents: 7.4-8.6; go to Table 8).

Some consumers are using the data for hospital selection. Although the satisfaction and usability scores suggest that consumers can find the data hard to use, 44 percent of consumers (n=128) reported that they were "likely" or "very likely" to use the data to choose a hospital or change hospitals. Of the consumer respondents who had a primary purpose of choosing or comparing hospitals who were also asked whether they were likely to use the information to choose a hospital or change hospitals (n=36), 75 percent reported that they were likely or very likely to do so.

Implications

• Web sites are currently serving health care professional respondents better than consumer respondents on several key metrics: overall satisfaction ratings, usability ratings, and ability to accomplish the primary purpose of a visit among those whose primary purpose was to choose a hospital, compare hospitals, or see the performance of a specific hospital.
• The wide range of consumer and health care professional satisfaction and usability scores indicates that these groups find some Web sites easier to use than others, implying that Web sites with lower scores may be able to improve. Section 7 includes further discussion of how to potentially improve the Web sites for all visitors, and specifically for consumers.
• Most published research has suggested that public reports have more impact on health care professional behavior than consumer behavior.ⁱ However, our findings suggest that the Web sites are succeeding in serving some consumers who are seeking quality information for the purpose of hospital selection.

^i. Fung CH, Lim YW, Mattke S, et al. Systematic review: the evidence that publishing patient care performance data improves quality of care. Ann Intern Med 2008;148:111-23.