Chapter 3. Description of Methods

Designing Consumer Reporting Systems for Patient Safety Events

Technical Expert Panel (TEP), Delphi Surveys, and First TEP Meeting

The creation of recommendations for design features of consumer reporting systems was driven by extensive input and review from a Technical Expert Panel (TEP). The TEP included experts in patient safety, patient safety event reporting systems, health care delivery and quality improvement, patient-centered care, and patient advocacy. The project team chose the TEP members for their knowledge and reputation in their respective fields, and for their proven ability to think creatively and work collaboratively. The TEP was tasked with considering design features of potential consumer reporting systems and making recommendations on preferable system attributes.

Draft TEP recommendations for consumer reporting system design features were generated using the Delphi Method and three rounds of consensus-building meetings. The Delphi Method is an anonymous process for developing consensus on topics that require input from individuals with a wide range of expertise and background. Through an iterative series of three questionnaires (i.e., a three-round Delphi process), the TEP described the ideal attributes of a consumer reporting system without full specification. After the completion of the Delphi Method, the TEP met to discuss and clarify their recommendations and provide greater specificity.

Additional Project Activities

The work of the TEP was enhanced and supplemented by four sets of activities:

  • Initial round of consumer focus groups to discuss ideal system design features.
  • Stakeholder interviews.
  • Environmental scan and literature review.
  • Second round of consumer focus groups to provide feedback on draft recommendations.

I. Initial Round of Consumer Focus Groups

Methods and recruitment process. We conducted six consumer focus groups in three geographic areas: two in Denver, two in Houston, and two in Boston during June through September 2009. Focus groups were conducted with individuals (or their family members) who had experienced patient safety events, to discuss design features for consumer reporting systems. We recruited participants by working closely with consumer advocacy organizations in each location as well as with CAPS community contacts. All participants had experienced a patient safety event (either personally, through a family member, or through someone for whom they care) in the last 10 years. Recruitment efforts focused on obtaining participants who experienced patient safety events in three different settings of care:

  • Hospital or home immediately following discharge from a hospital
  • Ambulatory or outpatient care settings (such as health centers and clinics, doctor's offices, hospital outpatient centers for diagnostic tests, same-day surgery, cancer treatment, and rehabilitation care)
  • Long-term care facilities (nursing homes, assisted living, or other residential care facilities such as group homes)

The focus groups explored the research questions developed by AHRQ that were presented above.

Characteristics of participants. Three groups included 9 participants, one group included 8 participants, one group included 7 participants, and one group included 6 people, for a total of 48 consumers across all six groups. The participants were predominantly female (79 percent), while 65 percent were aged 50 and older, 77 percent were white, and 77 percent had a college degree or higher. One third of the participants had experienced a patient safety event themselves; over three quarters of the participants (77 percent) had a family member who had experienced a patient safety event. Some participants also worked in the patient safety field, many were members of consumer networks, and some were active in advocacy activities. About 88 percent of the patient safety events occurred in a hospital setting. One quarter of patient safety events occurred in an ambulatory setting and 10 percent in a long-term care facility. (Respondents could report more than one person who experienced a patient safety event and more than one setting, so these numbers do not add up to 100 percent)

Findings. Focus group participants were able to generate many key design features of ideal consumer reporting systems. Many of these features emerged in multiple focus groups. Participants reported universal agreement on several key design features relating to accessibility, levels of operations, reporting modalities, and maximization of reporting. Participants strongly supported the value of consumer reporting systems to contribute to improvements in health care quality and safety, strengthen accountability, support patient empowerment, and aid learning. They said that consumers can report extensive information about patient safety events, including details about the sequence and timing of events, the individuals involved and their roles in the event, and communication or coordination issues that contributed to the event. Participants stressed that the consumer's perspective provides critical insight into what happened and why. Frequently, consumers can provide information that clinicians miss or may not consider salient. Other frequently mentioned types of information that consumers can provide were the consequences (emotional and physical) of the event; information about near misses; and information about hygiene and sanitation, staff attitudes and behaviors, and other issues.

 Table 1. Key features of ideal consumer reporting systems from focus groups summarizes key findings related to participants' views about the scope and range of possible consumer reporting systems, operational approach, infrastructure and design features, purpose and goals of a system, linkages to quality improvement, and ways to maximize utilization of the system.

Accessibility of consumer reporting system(s). Participants thought it was important for patients, family members, and others to be able to report a patient safety event from the time the event occurred ("real time") to well after the event, once they have had time to heal, grieve, care for people harmed, and gather information for reporting. In addition, participants said it was important that consumers be able to review and update their report as new information becomes available. They recommended designing the system to facilitate access for consumers who may face language, literacy, or cultural barriers to reporting and making it available for health care workers and witnesses, not just patients and families.

Anonymity and confidentiality. The ability to self-identify was another important concern for participants. However, they also thought it was critical that reporters have options to choose anonymity or different levels of privacy. Patients, family members, and others who submit reports need to feel confident that they are not jeopardizing their care. Several thought whistle-blower protection for health care professionals was an important feature.

Levels of reporting. Most of the participants envisioned an integrated multilevel reporting system, or "one big system," rather than the current fragmented situation. As one participant said, "Our biggest problem now is fragmentation of places to report. Ideally, this system should fix that." Participants perceived advantages to reporting at the local level (generally referring to reporting at the point of care) in terms of convenience for the consumer and rapid feedback to the health care institution.

However, some participants also questioned whether they could trust a local reporting system. Participants generally thought of local-level reporting as linked to a State, regional, national, or even an international system.

Organizations suitable to operate a consumer reporting system(s). Independence was a top consideration for participants in determining what type of organization(s) would be best suited to operate a consumer reporting system. In their view, organizations should not have any conflicts of interest raised by association with health care institutions, providers, or insurance companies. Although many participants favored private, independent, nonprofit organizations because of their independence, participants also recognized the value of a government agency's authority. Even among participants who opposed government operation of a consumer reporting system (e.g., because it would be too bureaucratic), participants often recognized that the government would need to be involved in some way for the system to be effective. It was critical in participants' views that the organization(s) have the authority to hold providers and health care institutions accountable for responding to reports or remedying the risks that led to patient safety events.

Operational approach. Participants thought that a consumer reporting system should have staff who could help individuals file a report, act as their advocates, and provide a "human touch." Assigning a staff member to follow through on a report would offer continuity to the reporter.

Infrastructure and design preferences. Participants offered opinions about various infrastructure and design elements. They thought the system should offer multiple reporting modalities, including face-to-face, Internet, phone, and mail, to accommodate the preferences of different consumers. Many participants also thought the system should allow for both structured and unstructured (e.g., narrative) reports or use some combination. One participant advised that the system be "evolvable" to keep pace with developing platforms for electronic communication.

Feedback to reporters was voiced as an essential element of an ideal consumer reporting system, beginning with acknowledgement of the report. Participants said that consumers should be informed about the results of any investigation, why the event happened, and what actions have been taken to prevent recurrence (e.g., policy and procedure changes). Without this feedback, consumers have no way of knowing that their report made a difference. Participants contrasted this ideal feedback loop to their experience with the Joint Commission, which provides no feedback beyond an acknowledgement. Participants' interest in identifying themselves was tied to their interest in feedback. As one stated, "If I'm going to report, I am going to put my name on it. How else will I know if something was done?"

Participants thought the consumer-reported data should be linked to other organizations and systems. They see the value of such linkages as facilitating change and improvement of patient safety; regulating and enforcing accountability, including tracing provider and organizational performance over time; and improving coordination of care across providers and facilities.

Purpose and goals for consumer reporting system. Participants offered that the primary purpose of a consumer-focused reporting system was to improve the quality of health care, particularly patient safety; patients and family members are highly motivated to share their experiences "to make sure it never happens again." In participants' view, another primary goal is accountability for providers and health care institutions. Specifically, participants mentioned linking consumer reporting to accrediting and licensure. Also important to participants is that the systems provide information about providers and health care organizations (e.g., patient safety track records, "report cards") that the public can use to make informed choices about sources of care. Education of health care professionals and consumers was mentioned several times as a goal of consumer reporting systems.

Linking consumer reporting system to quality improvement. To link the consumer reporting system to quality improvement, participants advised that consumer-reported information should be publicly available to raise awareness of providers' performance and highlight improvements. In addition, they said the system should provide timely feedback to providers and provider organizations to facilitate corrective actions.

Maximizing reporting. The success of a consumer reporting system depends on consumers' awareness of and their willingness and motivation to use the system. Participants suggested launching a public information campaign for the community at large to raise awareness of the system and of patient safety more generally. They also favored disseminating information within health care settings (e.g., on hospital TV channels, through print materials, at intake). The message needs to be clear and convincing that consumer-reported information is used and makes a difference.

Limitations. Although we attempted to identify participants who had experienced patient safety events in three different settings of care (hospitals, ambulatory care settings, and long-term care facilities), the majority of the focus group participants (88 percent) had experiences in the hospital setting. A different mix of participants (e.g., higher proportion of those who had experienced patient safety events in ambulatory care settings or long-term care facilities) may have generated different ideas about ideal reporting systems.

The focus groups included participants who had experienced a range of patient safety events, from near misses to those that caused serious harm or death of a family member. This may have created a different dynamic than if the group composition had been more uniform with regard to the severity of the patient safety event.

II. Stakeholder Interviews

Methods and recruitment process. The project team conducted 25 telephone interviews with stakeholders during August through October 2009. We conducted the interviews using a semistructured interview guide that focused on AHRQ's key research questions. These questions provided a consistent framework for gathering information from stakeholders and other groups across different aspects of the project. Stakeholders were chosen to represent a variety of groups, including consumer advocacy, public and private health care systems, risk management, and patient safety reporting, and others.

Findings Most of the stakeholders strongly supported the value of consumer reporting systems to aid in learning, contribute to improvements in health care quality and patient safety, and support consumer empowerment. They recognized that patients and family members offer a unique perspective and can provide valuable insights into patient safety events over the continuum of care, such as problems that occur during transitions of care or after discharge. The consumer perspective also offers insight into the consequences of the event for the individual harmed and family members, whether and how the event was disclosed, cultural and environmental issues in the health care setting that may set the stage for patient safety events, and many other factors that contribute to patient safety events.

Stakeholders discussed a wide range of issues related to the scope, design, infrastructure, operation, and purpose of ideal consumer reporting systems. Table 2. Key features of ideal consumer reporting systems from stakeholder interviews , summarizes the primary themes and ideas that emerged across the interviews.

Accessibility of consumer reporting systems. Stakeholders stressed the importance of designing a reporting system that is accessible and easy to use for a wide range of consumers, including those with limited health literacy and for whom English is not their first language. Other access considerations included offering multiple modalities for reporting, ensuring availability in multiple settings of care, building on existing systems/platforms that are familiar to consumers, not limiting what can be reported, and providing a safe environment for reporting.

Timing of reporting. Stakeholders supported real-time reporting, with some discussing it in conjunction with rapid response teams, positing that real-time reporting would allow for real-time response, remediation, or investigation. They also supported reporting after the fact, including after extended periods of time. This approach would allow consumers time to come to terms with their experience emotionally, in addition to allowing time for the full consequences of an event to be experienced.

Anonymity and confidentiality. Most stakeholders favoredallowing consumers to choose whether to reveal their identity or remain anonymous. The consumer advocates advised that most patients want to provide their name or identity, but some fear that their care may be compromised if providers learn that they have reported. However, a sizable minority of stakeholders (none of whom was a consumer advocate) thought anonymous reporting should not be permitted. Their concern was that anonymous reporting would not allow follow-up with the reporter to conduct a thorough investigation, make improvements, or reconcile the patient's perception with the provider's perception.

Who can report. Stakeholders generally offered the view that consumer reporting systems should receive reports from patients, family members, caregivers, and others who witness a patient safety event. However, a minority opinion was that only patients or their family members/caregivers should be allowed to report because others might not have sufficient information about the event.

Voluntary vs. mandatory reporting. There was consensus that consumer reporting systems should be voluntary.

Levels of operation. Stakeholders' views about the appropriate level or levels of operation were closely linked to views about the purpose and goals of a reporting system. Most stakeholders envisioned a multilevel system in which reports "roll up" from the local level (i.e., generally referring to the health care institution) to State, regional, and national levels. Many thought that local-level reporting would be appropriate if the goal was to feed back information to the health care institution and implement rapid changes. Stakeholders from provider organizations particularly favored this approach. Several of the consumer advocates expressed the opposite viewpoint and felt strongly that consumer reporting systems should be outside the health care institution and implemented at higher levels (i.e., State, regional, national). Stakeholders thought that a State-level system would be appropriate if the goal was accountability. A national-level system was essential if the goal is system-wide learning. There was little discussion about the possibility of international-level consumer reporting systems.

Organizations suitable to operate consumer reporting systems. Federal government agencies and private, independent, nonprofit organizations were mentioned most frequently, followed by hospitals and other health care institutions. However, for each of these organization types, opinions were divided with some stakeholders also opposed.

The primary arguments for a Federal government agency were neutrality and authority. Stakeholders who opposed a Federal government role expressed concern that the system would become politicized. In addition, some viewed government agencies as inefficient and thought the consumer reporting agency should be operated by a "lean" organization.

Private, independent, nonprofit organizations were preferred by many of the stakeholders because of their neutrality. Specifically, stakeholders said the organization should be independent of government, health care systems, providers, malpractice insurers, and other interest groups.

A number of stakeholders thought hospitals and other health care institutions should be responsible for consumer reporting systems. This view was particularly marked among stakeholders from health care organizations. They thought that health care institutions were well positioned because they already have experience collecting patient safety data and are trusted by health care providers; in addition, the system would not be perceived as outside interference. Other stakeholders, particularly consumer advocates, opposed this approach because it does not provide a safe and neutral place for consumers to report.

There was limited support for consumer advocacy organizations, although a number of stakeholders said that consumers need to be at the table to design consumer reporting systems. Some stakeholders were concerned that a reporting system operated by a consumer advocacy organization would not be accepted or trusted by health care professionals.

A number of stakeholders named organizations they thought would not be suitableto operate consumer reporting systems, including the Joint Commission and patient safety organizations (PSOs). The primary objections to the Joint Commission were perceptions that it does not provide meaningful feedback to consumers and it does not "do anything with this data. " Objections to the PSOs related to the confidentiality provisions and lack of accessibility of the data. Some stakeholders also thought that the enforcement and licensing agencies would not be suitable because they receive a high volume of complaints and only follow-up on a small percentage.

Several stakeholders suggested building the reporting system on an established organization with a relevant mission and experience. Stakeholders named the following factors as important to consider in identifying organizations suitable to operate consumer reporting systems (in descending order of frequency of mention): independence, neutrality, transparency; consumer involvement; authority, ability to investigate; efficiency; and health care professional involvement in designing the system.

Operational approach. Stakeholders said that system support staff would be needed to respond to consumer reports, assist individuals with reporting, serve as patient advocates and provide support, triage reports based on agreed-upon criteria, and conduct investigations. In addition, analytic staff would be needed for coding and data analysis.

Stakeholders' views about financing consumer reporting systems were divided; several stakeholders thought the system should be federally funded because patient safety is a national concern and responsibility; also, Federal funding would increase the perception of neutrality. Others thought health care organizations are obligated to fund the system.

Infrastructure and design. Most of the stakeholders thought that consumer reporting systems should offer multiple modalities for consumer reporting; they mentioned telephone most frequently, followed by Web-based (e.g., E-mail, Web portals), in person, and regular mail. Most stakeholders also thought it would be important to incorporate both structured and unstructured reporting formats. Narrative reports would provide rich information and would likely be preferred by consumers. However, analysis is time intensive, expensive, and challenging. For structured reporting formats, stakeholders suggested surveys (e.g., outreach surveys, Hospital Consumer Assessment of Healthcare Providers and Systems [HCAHPS]®), algorithm-based questionnaires, and other standardized data collection instruments. Structured formats were identified as a low-cost approach that could build on existing organizational mechanisms.

Almost all of the stakeholders thought that feedback to consumers was critical. Their suggestions about the nature and content of feedback included acknowledging the report and thanking consumers for reporting, providing information about what action has been/will be taken, informing the consumer about the impact of the report on identifying a problem and making improvements, providing a time frame for when the reporter will receive additional information, and sharing information about the number of similar reports received.

Goals and purpose. Stakeholders voiced strong support for learning about and improving health care quality and patient safety as closely linked goals. They viewed learning as a goal at both the organizational and system-wide levels. Several stakeholders also considered patient empowerment as a goal. However, there was little discussion of exactly how the system would empower consumers beyond giving them "a voice."

Fewer stakeholders expressed support for accountability as a goal. They argued that there was no reason to duplicate existing accountability organizations and mechanisms (e.g., Joint Commission, medical boards, legal system). In addition, stakeholders pointed out that the punishment of individuals would not result in systemic improvement. Several stakeholders believed that focusing on accountability would threaten the collaboration and trust needed for the reporting system to be effective. Ultimately, a focus on punishment would undermine the system because providers and health care institutions would discourage reporting.

The stakeholders who believed accountability was an important goal discussed it in terms of measurement, follow-up, and changed behavior rather than punishment. For example, one stakeholder described consumer reporting as a "measurement tool" that could hold people accountable. Some stakeholders said that an ideal reporting system would not have regulatory authority itself but would have linkages to regulators (e.g., to licensing boards).

Linkages to quality improvement. Stakeholders generally supported the idea of linking consumer reporting systems to existing efforts to improve quality of care and patient safety, although they offered few specific suggestions about how such linkages could be accomplished. Stakeholders did not suggest linking to programs that offer financial incentives for quality (pay-for-performance models) or providing feedback to health care facilities or providers. Ideas about linking consumer reporting to other quality and patient safety improvements appeared to be shaped by the stakeholder's views about system goals and levels of reporting. According to one stakeholder, if consumer reporting systems operated locally (i.e., at the level of the health care facility) with a goal to improve quality of care, then linkages are not needed, and concerns about interoperability would be misplaced; this suggests that consumer reporting systems could function as an independent effort to improve quality and patient safety. However, if consumer reporting systems operate at a State or national level, some see linkage to the local level as essential.

The issue of public reporting drew both support and skepticism. Supporters invoked the goal of transparency and described public reporting as a tool for quality improvement. Skeptics were concerned about the quality of data and the lack of a robust denominator, making comparisons difficult.

Maximizing reporting. Almost all of the stakeholders agreed that communications and marketing efforts were needed to create awareness of consumer reporting systems and encourage reporting. They suggested communication strategies within health care institutions and public campaigns in the community at large. Specific suggestions for communications campaigns included orientation and literature given to patients when they first enter the hospital/health care facility and also at discharge, signage and materials (e.g., newsletter, brochures, waiting room videos) in health care facilities, information on hospital advertisements and Web sites, TV and radio advertising, use of social media (e.g., YouTube, Facebook, viral marketing), and working through trusted community organizations. Stakeholders advised that communication strategies need to focus on consumer empowerment and stress the critical importance of consumer reporting to improve patient safety. Several stakeholders suggested that consumer testimonials could be powerful.

Some stakeholders thought that to maximize reporting the system needed to be proactive and actively reach out to patients. A few stakeholders suggested post-discharge surveys or exit interviews to solicit information about patient safety events.

Limitations. Because constraints on the numbers of stakeholders we could interview, we were unable to include individuals from all sectors or perspectives relevant to consumer reporting of patient safety events (e.g., did not include individuals from pharmaceutical and medical device companies or from the malpractice field). The small numbers of stakeholders from different categories precluded analysis of findings by category. Furthermore, many of the stakeholders brought multiple perspectives to bear on the issues discussed and thus cannot be placed in a single category. In some cases, where there appeared to be marked differences, we note these apparent trends, but they should be interpreted with caution.

III. Environmental Scan and Literature Review

Methods. To inform the development of recommendations of key design features of ideal consumer reporting systems for patient safety events, we identified a broad group of existing reporting systems and collected information on their characteristics. Three methods were used to collect information for this report:

  1. An environmental scan that reviewed information available on the Internet describing patient safety event reporting systems.
  2. A literature review that abstracted information mainly from articles published in peer-reviewed journals on reporting systems. To a lesser extent, reports and other literature that were not published in peer-reviewed journals were also included.
  3. Nine key informant interviews with representatives from important patient safety event reporting systems.

The environmental scan and literature review largely focused on the characteristics and features of systems, providing information on the design and operations of these systems. The key informant interviews also provided some of this information, but generally went beyond this to include more qualitative and explanatory information. That is, although the environmental scan and literature review provided information on the "what" associated with systems, the interviews helped address the "how" and "why" for the systems.

From the environmental scan, literature review, and key informant interviews, we collected information on the relevant system characteristics and design features consistent with those identified by the TEP during the three rounds of Delphi questionnaires and the first TEP meeting:

  • Scope and range of patient safety event reporting systems:
    • Level of operations (local, regional, national, or international).
    • System ownership (public, private, or mixed).
    • Degree of focus on consumer reporting.
  • System goals, purpose, or mission.
  • System reporting characteristics:
    • Voluntary versus mandatory reporting.
    • Confidentiality/anonymous reporting and user feedback.
    • Availability of staff to assist with reporting.
  • System Infrastructure:
    • System staff.
    • System funding.
    • Links to other systems.
  • System Design:
    • Methods of reporting permitted.
    • Format of reports.
    • Types of events that can be reported.
    • Investigations of reported events.
    • Analysis of reports.
  • Information Dissemination and Responses to Reports:
    • Dissemination of reported information.
    • Responses to reported information.
  • Strategies for Maximizing Reporting:
    • Publicity/outreach/marketing.
    • Direct involvement of those who report.

We also collected general comments from the environmental scan, literature review, and key informant interviews related to desired or ideal characteristics and features of patient safety event reporting systems.

Findings. A total of 74 patient safety event reporting systems were identified. For an illustrative subset of 13 systems, we were able to gather detailed information from the environmental scan, literature review, or key informant interviews; these 13 systems are discussed in more detail in the report.

The identified systems range from local operations (i.e., at a single hospital) to national systems (in the United States and other countries) and one international system. Most systems currently operate at local/regional, State, or national levels. Few systems appear to operate at multiple levels, which suggest that targeting new consumer reporting systems to a single level may be advantageous.

There is a mix of government-run and private-sector reporting systems. However, even privately-run systems rely on government funding, at least for development and initial periods of operation. Based on the collected information, only a minority of systems (20.3 percent) permitted consumer/patient reporting. Most systems involved reporting by health care providers or systems.

The most common goal among the identified reporting systems is learning about patterns and causes of patient safety events. Other common goals include monitoring the types and occurrences of patient safety events (surveillance), improving patient health and safety, and improving the quality of medical care.

Most systems, particularly those attempting to elicit consumer input, generally allow for a mix of anonymous and confidential reporting. Most identified systems involve voluntary reporting. However, mandatory reporting is seen among a majority of State systems, where the system owners have regulatory authority over the potential reporters. Mandatory systems do not (in general) permit anonymous or confidential reporting, although voluntary systems almost always allow these reporting options. Few reporting systems have links to other systems or databases.

Little information was collected on system funding or on systems staff, including whether staff are available to assist with reporting. Key informant interviews indicated that staff based in call centers will likely require health care backgrounds and/or special training in eliciting needed information to collect complete reports. In addition, systems will require staff skilled in designing and maintaining reporting systems (i.e., information technologies); entering and coding data; analyzing and investigating reports; disseminating information; and preparing recommendations and strategies to address patient safety events.

The identified systems include a range of reporting methods, with online submission being the most common. Although computer-based submission requires specific activities to ensure anonymity or confidentiality of the reporter, this method permits rapid evaluation and analysis of reports from multiple locations. A majority of systems accept reports using a combination of narrative and standardized forms. A small number of systems use more advanced methods for prompting users for needed information (such as the hierarchical classification system of AIMS) to obtain more complete reports.

Reporting systems generally permit (or encourage) reports on a diverse range of events. The type of event that can be reported is often unspecified or described only in general terms such as incidents or complaints. Only a minority of systems indicate that they investigate reported events. Details on the types of analyses of reported events are often unspecified, although a number of systems do perform root cause analysis.

Many systems do not specify how (or whether) reported information is disseminated. Full public dissemination of reported information appears to be uncommon. Beyond dissemination, a variety of activities are undertaken using information from reports, including establishment of priorities, development of guidelines or recommendations, and development of educational interventions. Few systems undertake activities to encourage or maximize event reporting, and almost no information was collected on subsequent direct involvement of individuals providing reports.

Overall, there is not a single set of characteristics or design features representative of the majority of patient safety event reporting systems. Rather, there is a broad range of characteristics and features that reflect the diversity of system goals. These goals likely determine many system features, such as levels of operation, reporting specifications, and dissemination activities. This suggests that it is important to define the goals, purpose, and mission for a consumer reporting system before attempting to specify detailed system characteristics.

Limitations. The environmental scan and literature review is not meant to be a complete listing of all characteristics and features of all existing patient safety event reporting systems. The summary report is also not meant to be a complete listing of all characteristics and features of the identified systems; that would not be possible without more detailed research and interviews with individuals at every system.

IV. Second Round of Consumer Focus Groups

Methods. We conducted four consumer focus groups in May 2010: two groups in Chicago and two in San Francisco. We recruited participants by working closely with consumer advocacy organizations in each location, as well as with CAPS community contacts. All participants had experienced a patient safety event (either personally or regarding a family member or someone for whom they care) in the past 10 years.

A total of 33 patients and family members participated in the focus groups. Most of the participants were female (79 percent), aged 50 or older (58 percent), White (57 percent), and had a college degree or higher (79 percent). Over half of the participants (55 percent) had experienced a patient safety event themselves and 70 percent had a family member who had experienced a patient safety event. Among the participants, about 91 percent reported that the event occurred in a hospital setting, 21 percent in an ambulatory care setting, and 18 percent in a long-term care setting.1

Recruitment focused on individuals who reported or tried to report the patient safety event. Across all four focus groups, 67 percent of participants reported the event and 12 percent tried to report the event.

Findings. The participants agreed with many of the draft recommendations proposed by the TEP. However, they had mixed reactions to some recommendations and in some cases agreed with elements of a recommendation but thought that clarifications, modifications, or additions were needed. Table 3. Overview of participant responses to draft recommendations for key design features of consumer reporting systems , provides an overview of participants' responses to all of the draft recommendations.

Types of information and sources of reports. The participants agreed that all types of patient safety events, including no-harm and near-miss events, should be reportable. They also supported a system design that captures the subjective experience of the consumer (i.e., their story) as well as objective information about the event. They agreed that the system should be available to all potential reporters, including health care providers and bystanders.

Purpose and goals for consumer reporting system. Across all four focus groups, the participants supported the notion that the consumer reporting system should be a learning system, oriented to identifying and addressing problems in patient safety. The participants also voiced strong support for accountability as a system goal, although their interpretations of accountability varied. Some thought accountability meant making changes based on what was learned from consumer reports, through training, and by modifying policies or procedures. Others viewed accountability as a matter of "consequences" or "punishment," such as warning, reprimanding, or firing providers.

Level of operation. The participants generally agreed with the recommendation for a multilevel system and with the alternative model that includes analysis at the local level, although there was less extensive discussion about this alternative. They felt that personalized and timely feedback would be more readily achieved locally. However, they indicated that national-level reporting is essential to identify trends in patient safety events and to raise awareness of patient safety issues and the importance of reporting.

Access, reporting modalities, format, and accessibility. The participants agreed with the recommendations that the consumer reporting system allow for multiple reporting modalities as well as a mix of structured and unstructured reporting, and that it be designed to facilitate access for diverse populations. They felt that system success depends to a large extent on designing the system to be easy to use by a wide range of consumers.

Anonymity and confidentiality. The participants agreed that consumers should be able to submit confidential reports. There was some difference of opinion about allowing anonymous reporting because of concerns about the inability to follow-up about reports and about "bogging down the system" because consumers could potentially be less thoughtful and selective about reporting. Also, anonymity could open the system up to false reports (e.g., for vindictive purposes). Some participants supported anonymous reporting for health care providers.

Type of organization. The participants were supportive of the guiding principles and characteristics of an operating organization, as proposed by the TEP. They supported strong consumer involvement with respect to oversight, but also believed that health care professionals and other experts need to be involved to lend credibility to the system and provide the necessary expertise. Many participants thought the system should be connected with the government in some way; however, they were concerned with how ties to government could affect system independence, neutrality, and efficiency. Some participants also were concerned with potential tension between financing and independence.

Feedback to consumers. The participants agreed that the system should respond to consumers in a timely manner and provide meaningful feedback. They stressed that acknowledgement of receipt of a report does not constitute meaningful feedback. The system should make it clear to consumers when they can expect different types of feedback. Some participants were disappointed that the recommendations did not guarantee a response from the health care setting where the patient safety event occurred; others thought this limitation was reasonable and to be expected.

Linkages. The participants agreed that the consumer reporting system should have linkages with a wide range of organizations that could help the system to improve health care and use the reported information. They were clear that a consumer must be able to choose whether his or her report is shared outside of the consumer reporting system, specifically back to the facility where the patient safety event took place.

Analytic functionality. There were mixed reactions to the analytic functionality recommendations set out by the TEP. Some participants cautioned that if consumers understand that causal analysis would be conducted at the aggregate level rather than at an individual level, they may perceive the system as less useful and be less willing to report. Others thought it was appropriate that the system be focused on improvements in the larger health care system, not individual cases.

Public reporting. Most participants strongly supported public reporting, believing that it would make the system more visible and foster legitimacy. They suggested that public reports could demonstrate how consumer reports lead to change and improvement in the health care system. However, one group of participants questioned whether the public reporting would truly be effective in improving health care outcomes.

Maximizing reporting. The participants agreed with the TEP recommendation to maximize reporting and suggested various methods for raising awareness and encouraging use of the system. Discussion about maximizing reporting focused on specific ways to disseminate information about the system within health care settings and in the community at large. Some participants suggested actively soliciting patient safety experiences, for example through hospital exit interviews or patient surveys.

Limitations. We recruited individuals who had experienced a patient safety event, either reported or attempted to report the event, and considered the topic of reporting patient safety events to be important. Because of these recruiting requirements, which we considered essential to receiving thoughtful feedback, the opinions of and feedback from the focus groups participants may not be representative of all patients and family members.

Although we sought to recruit a mix of participants in terms of race, ethnicity, and other sociodemographic characteristics, the participants were generally well educated (79 percent had earned a college degree or higher) and White (57 percent). Recruiters were able to increase the percentage of African American participants in the Round 2 focus groups compared with the Round 1 focus groups (34 percent and 21 percent, respectively), likely the result of working with an African American community contact. However, we were less successful in recruiting Hispanic participants, as several Hispanic participants initially agreed to participate, but changed their minds prior to the focus group. Consequently, it would be useful to obtain additional feedback from groups that were underrepresented in these focus groups, particularly Hispanics and individuals with low education levels.

Given the time constraints for conducting the focus groups, it was challenging to fully explain and explore the details of each recommendation—some of which are quite complex—and to ensure a common understanding. It was particularly challenging for participants to fully grasp the elements of the recommendations related to accountability, public reporting, and levels of operations. Consequently, interpretations of these recommendations differed somewhat among participants.

Second, Third, and Fourth TEP Meetings

After receiving reports on the first TEP meeting, consumer focus groups, key stakeholder interviews, and the environmental scan and literature review (all of which are included in Appendices to this report), the TEP convened for a second meeting. At this meeting, the TEP further crystallized their recommendations with an eye toward feasibility and considering barriers to an ideal system.

A third TEP meeting was held to resolve unanswered questions from the previous two meetings. The third TEP meeting was held by Webinar on February 23, 2010. The goal of the meeting was to develop draft recommendations for consumer reporting systems for patient safety events, transitioning from initial discussions of TWIS to the complete specification of TWIS. This included finalizing consensus points, discussing unresolved issues from the second TEP meeting to develop consensus on key design features, and discussing new design features (not previously discussed) if necessary.

Following the third TEP meeting, a draft final report was prepared presenting a summary of the activities to-date in this project, the draft design feature recommendations, a series of graphics illustrating how these design features may be organized within a consumer reporting system, and a discussion of limitations and additional issues. The draft report was reviewed by a group of external peer-reviewers, whose names and affiliations are presented in Appendix B. A summary of the draft recommendations was also presented at a second set of consumer focus groups.

Comments from the consumer focus group participants, the external reviewers, and members of the TEP were presented at a fourth TEP meeting held in October 2010. The goals of this final TEP meeting were to discuss the feedback received on the draft final report and reach consensus regarding materials for the final report. The results of this fourth meeting are the recommendations found in this final report.


1 Participants could report more than one person who experienced a patient safety event (i.e., self and family) and more than one setting. Consequently, percentages sum to more than 100 percent.

Current as of July 2011
Internet Citation: Chapter 3. Description of Methods: Designing Consumer Reporting Systems for Patient Safety Events. July 2011. Agency for Healthcare Research and Quality, Rockville, MD. http://www.ahrq.gov/professionals/quality-patient-safety/patient-safety-resources/resources/consumer-experience/reporting/chapter3.html