Executive Summary

Designing Consumer Reporting Systems for Patient Safety Events

The Agency for Healthcare Research and Quality (AHRQ) funded the Designing Consumer Reporting Systems for Patient Safety Events project to develop recommendations for ideal reporting systems that consumers would use to report experiences with patient safety events. The iterative process for developing these recommendations involves extensive support from a Technical Expert Panel (TEP), input from consumer focus groups and stakeholder interviews, and an environmental scan and literature review. The ultimate outcome of this project is to outline the key design specifications for the development of consumer reporting systems for patient safety events. This report presents recommendations developed by the TEP following the fourth expert panel meetings. These recommendations incorporate input from the focus groups, stakeholder interviews, and environmental scan report as well as from external peer reviewers.

RTI International, a nonprofit research organization, is carrying out this research contract for AHRQ in collaboration with Consumers Advancing Patient Safety (CAPS), a nonprofit, consumer-led organization dedicated to creating new pathways for consumers and providers to work collaboratively to achieve health care that is safe, compassionate, and just.

Methods

The recommendations contained in this report were created using the IDEALS framework, in which a recommended system evolves through three stages:

  1. A theoretical system capturing a vision of what an ideal system would be (even if, realistically, that cannot be attained).
  2. An ultimate ideal system that is built on the theoretical ideal system, but contains achievable operational and practical goals.
  3. A technologically workable ideal system (TWIS).

As a result of this process, recommended systems were specified in terms of currently available technologies and components while meeting the specifications of the prior steps of the process (theoretical ideal, ultimate ideal, and technologically workable systems).

The implementation of the IDEALS concept has been extensively supported by the TEP at every stage. The TEP includes experts in patient safety, patient safety event reporting systems, health care delivery and quality improvement, patient-centered care, and patient advocacy. TEP members were chosen not only for their knowledge and reputation in their respective fields, but also for their ability to think creatively and work collaboratively with colleagues. Appendix A lists the names, affiliations, and components of the project in which they participated for each of the TEP members.

Members of the TEP initially participated in three rounds of questionnaires using the Delphi Method. These iterative questionnaires were designed to elicit information related to key design features for consumer reporting systems in response to a set of six research questions developed by AHRQ:

  1. What type of information can consumers provide concerning their health care experience with patient safety events that may be useful and/or actionable in a patient safety event reporting system?
  2. What are the scope and range of options for consumer reporting mechanisms? How would these options differ at the national, regional, State, or local level?
  3. What type of infrastructure is needed to enable effective, actionable consumer reporting of patient safety events?
  4. What is the most effective operational approach for consumers to report patient safety event information?
  5. How would consumer reporting of patient safety events be linked to quality and/or patient safety improvement efforts?
  6. How can a reporting system maximize the willingness and ability of consumers to report on patient safety events?

Following completion of the three rounds of Delphi questionnaires, the TEP met for the first time in June 2009. Using the Delphi results as a basis, potential recommendations for consumer reporting system design features were elicited from the TEP members using the nominal group technique (NGT). Using this technique, design features related to a particular area of consumer reporting systems were elicited from the panel members. The list of design features was then discussed and features were added, removed, or combined. TEP members then voted on the features they felt were most important, and the panel moved on to the next topic.

A second meeting was held in December 2009. At this meeting, the TEP reviewed the results of the first TEP meeting, consumer focus groups, key stakeholder interviews, and the environmental scan and literature review. A half day of this meeting was used to discuss the results of the aforementioned activities, and a subsequent full day was used to further develop TEP recommendations for design features of TWIS for consumer reporting of patient safety events.

The TEP convened for a third time in February 2010 to resolve unanswered questions and address additional issues remaining from the previous two meetings. The goal of the meeting was to develop draft recommendations regarding consumer reporting systems for patient safety events, to transition from initial discussions of TWIS to the complete specification of TWIS. This included finalizing consensus points, discussing unresolved issues from the second TEP meeting to develop recommendations on key design features, and discussing new (that is, previously undiscussed) design features, if necessary.

Following the third TEP meeting, a draft final report was prepared presenting a summary of the activities to date in this project, the draft design feature recommendations, a series of graphics illustrating how these design features may be organized within a consumer reporting system, and a discussion of limitations and additional issues. The draft report was reviewed by a group of external peer-reviewers, whose names and afiliations are presented in Appendix B. A summary of the draft recommendations was also presented at a second set of consumer focus groups to obtain a consumer perspective on the emerging system design.

Comments from the consumer focus group participants, the external reviewers, and members of the TEP were presented at a fourth TEP meeting held in October 2010. The goals of this final TEP meeting were to discuss the feedback received on the draft final report and reach consensus regarding materials for the final report. The results of this fourth meeting are the recommendations found in this final report.

Findings

During the first TEP meeting, the TEP established assumptions that underlie the resulting recommendations. Specifically, the TEP agreed that consumers have valuable information about the health care system and their experiences within it, and that health care systems and providers may not have access to this information. Based on this, the TEP confirmed that there is great value in obtaining information from consumers.

The TEP reached consensus on many recommendations for design features of consumer reporting systems. Table ES-1 summarizes these recommendations made by the TEP. TEP members stressed that, while reporting of patient safety events may often be associated with hospital-based incidents, these recommendations are applicable to patient safety events in all health care settings. There were also some disagreements among TEP members regarding specific design features, which led to development of recommendations for alternative models of reporting systems. For example, TEP members' opinions diverged on whether consumer reporting systems should perform their own Root Cause Analysis (RCA) for selected patient safety events. Further, for certain other recommendations, the TEP noted the importance of specific design features but chose not to specify details of these features because of their dynamic nature. For example, the TEP commented on the importance of public reporting, but elected not to specify details for public reporting as this area was rapidly changing. In these cases, the TEP generally agreed that detailed recommendations would need to be made at system implementation.

Table ES-1. Recommendations for Key Features of Ideal Consumer Reporting Systems from the Technical Expert Panel

1. What type of information can consumers provide concerning their health care experience with patient safety events that may be useful and/or actionable in a patient safety event reporting system?
Recommendation 1.1: Types of Information. The systems should collect information on all types of events, ranging from near-miss and no-harm events to adverse events. The systems should capture both objective information about what occurred and more subjective information, based on the consumer's unique perspective. Information collected from consumers should include where a patient safety event occurred; what contributed to the event; whether or to whom an event was reported; what happened when an event was reported; and the impacts or consequences of the event.
Recommendation 1.2: Sources of Reports. The systems should allow for reporting by any individual, but the emphasis is on obtaining the consumer perspective.
2. What are the scope and range of options for consumer reporting mechanisms? How would these options differ at the international, national, regional, State, or local levels?
Recommendation 2.1: Purpose and Goals. The dual purposes of a consumer reporting system are to learn and to be accountable to consumers providing reports. To learn means obtaining the consumer perspective and experience to identify, mitigate, and prevent risks, hazards, and harms; improve outcomes; and advance patient safety. To be accountable to consumers providing reports means that reported information is actively used to design meaningful improvements in patient safety.
Recommendation 2.2: Level of Operation. Reports should be collected locally and communicated to a centralized (national) level that can aggregate and analyze data and triage or distribute information to State and local levels for action. Reporting systems will need to be flexible regarding analysis and other activities occurring at local levels, based on needs, capabilities, and funding/resources for these local activities.
3. What type of infrastructure is needed to enable effective, actionable consumer reporting of patient safety events?
Recommendation 3.1: Linkages. Systems should have linkages to a broad range of organizations that can change health care practices and demonstrate that reported information was used. Linkages should be formed for the purpose of encouraging consumer reporting, improving analysis, sharing results, and changing delivery for quality improvement. Linkages will also ensure timely information sharing. Because linkages are dynamic and rapidly changing, their exact nature and specifications will be more fully specified at implementation.
Recommendation 3.2: Analytic Functionality. Systems will need decision rules for what kinds of events receive different levels or types of analysis. Systems should collect information and conduct aggregate causal analyses. Systems should also gather responses of organizations to reports and evaluate their feedback.
4. What is the most effective operational approach for consumers to report patient safety event information?
Recommendation 4.1: Type of Organization. Guiding principles and characteristics that should be sought for organizations that own or operate consumer reporting systems are the following: independent entity with a steady stream of sustainable funding, where "independent" is defined as an entity that is completely separate in ownership, governance, and affiliation from entities that provide health care and whose members, employees, or affiliate entities may be the subjects of reports about adverse events; governing body members' fiduciary responsibility is to represent the public; neutral oversight body with consumer representation; transparency of goals, process, and results; consumer involvement in organizational governance and operations; and dedication to analyzing incoming information to identify threats to patient safety and feeding it back to systems that may be able to act on it.
Recommendation 4.2: Access at Different Points in Time. Systems should allow reporting at any point in time.
Recommendation 4.3: Reporting Modalities. To maximize reporting, systems should allow multiple routes or modalities for reporting.
Recommendation 4.4: Reporting Format. Systems should allow a mix of structured and unstructured reporting.
Recommendation 4.5: Anonymity. The system will allow anonymous reporting, but the system should be designed to discourage anonymous reporting by ensuring and encouraging well-designed confidential reporting. The system could allow reporters to opt out of confidentiality to increase the report's efficacy in certain situations.
5. How would consumer reporting of patient safety events be linked to quality and/or patient safety improvement efforts?
Recommendation 5.1: Linking to Quality and Patient Safety Improvement Efforts. Systems should be linked to efforts to improve quality and patient safety. If the reporter allows, his or her reports to the consumer reporting system will be automatically forwarded to appropriate existing reporting systems at the local or facility level.
Recommendation 5.2: Public Reporting. Public reporting should be used to hold systems accountable to their own goals. Systems should:
  • Publish information such as how much a system was used.
  • Publish information on what was learned.
  • Publish information about what recommendations and changes were made as a result of the system.
  • To the extent determinable, publish information about the responsiveness of institutions to patient safety issues.
Because this is an evolving and dynamic issue, the exact specifications will be developed at implementation and will be determined over time as the issue develops.
6. How can a reporting system maximize the willingness and ability of consumers to report on patient safety events?
Recommendation 6.1: Maximizing Reporting. System design should facilitate reporting to ensure maximum use; that is, maximize the ease/ability of consumers to submit reports. This will include public awareness campaigns or other outreach/marketing activities and getting "buy-in" from appropriate individuals and organizations as part of implementation.
Recommendation 6.2: Accessibility. Systems should be designed to facilitate access for diverse populations (e.g., age, race/ethnicity, education, language, disability).
Recommendation 6.3: Feedback. Systems should provide meaningful and timely feedback to reporters. Feedback will include public reporting, awareness campaigns, and meaningful acknowledgement of the receipt of a report. Systems will not be able to assure reporters that they will receive meaningful and timely feedback from the health care facility/system where a patient safety event took place.

Discussions by the TEP provided clarification and elaboration on these draft recommendations and recommendations for alternative models for consumer reporting systems. Specific discussion points include the following:

  1. What type of information can consumers provide concerning their health care experience with patient safety events that may be useful and/or actionable in a patient safety event reporting system?
    • The TEP recommendations focused on collecting broad and diverse input from consumers for patient safety event reporting systems. This includes collection of all types of information, including both objective findings and those from the consumer's unique perspective.
  2. What are the scope and range of options for consumer reporting mechanisms? How would these options differ at the international, national, regional, State, or local levels?
    • The TEP described dual purposes of a consumer reporting system: to learn and to be accountable to consumers providing reports. In this context, accountability is defined as consumer reporting systems being accountable to those who submit reports, where systems have a responsibility to actively use reported information in the pursuit of meaningful improvements to patient safety.
    • The TEP described two system models with respect to level of operations. The first model (Recommendation 2.2) separated data collection/reporting capabilities at the local level from data aggregation, analysis, and distribution activities at a centralized (potentially national) level. A second model for consumer reporting systems included additional capabilities at the local level. These additional local capabilities could include data analysis and direct interactions with health care providers or facilities. The TEP did not make recommendations about international systems.
  3. What type of infrastructure is needed to enable effective, actionable consumer reporting of patient safety events?
    • The TEP recognized that system linkages represent a dynamic, rapidly changing area. As such, the TEP chose not to specify how such linkages would work operationally. The exact nature and specifications of linkages can be specified more fully based on pilot testing and at system implementation. For Recommendation 3.1, TEP members did not indicate specific entities that comprise the "broad range of organizations" to which consumer reporting systems should link. However, discussion by the TEP indicated that these organizations should include State and Federal regulatory and financing organizations (e.g., the Food and Drug Administration [FDA]); health care facilities and systems; accrediting bodies; insurers; employer health care groups; user groups or communities of interests; and organizations that can provide assistance to consumers submitting reports beyond that available directly from consumer reporting systems.
    • In discussions related to Recommendation 3.2, the TEP identified three system models regarding analytic functionality. In one model, systems do not expend resources to conduct RCAs, but request RCAs when performed by other organizations and analyze the collected RCA information. In an alternative model, systems are able to perform RCAs on selected events. In this model, public decision rules are used to determine which events warrant an RCA and the performance of RCAs are subject to financial constraint. A third alternative model discussed by the TEP was collection of data using a standardized form for focused initiatives. This would involve development of a data collection instrument to gather information on a particular type of patient safety event. The instrument would be submitted to relevant health care facilities (presumably facilities where events of the specified type occurred based on reports to the system) and used to collect information related to the event. Related to this discussion, external reviewers and TEP members recognized that there are only limited numbers of individuals trained to perform RCAs at hospitals—and even fewer such individuals in other health care settings. This could affect the quality of RCAs produced
  4. What is the most effective operational approach for consumers to report patient safety event information?
    • The TEP explicitly chose not to specify the types of organizations to operate consumer reporting systems, electing instead to recommend characteristics of such organizations. However, the TEP agreed that these could be either public or private organizations. Four potential business models were discussed: commission model, Patient Safety Organization (PSO) model, quality improvement organization (QIO) model, or a subscription/co-op/consumer-driven model (similar to Consumers Union, where members pay dues to support the organization). The TEP did not reach consensus on the type or types of business models appropriate for operating consumer reporting systems. Some TEP members indicated that consumer reporting systems should not be owned or operated by accreditation or regulatory agencies such as the Joint Commission, the Centers for Medicare & Medicaid Services (CMS), or State Departments of Health; however, there was not universal agreement on this point. The TEP did not develop specific recommendations regarding financing of consumer reporting systems. However, this was described as a crucial issue because it could influence consumer perceptions regarding the reporting system. TEP members agreed that there is a need to align the scope and activities of a consumer reporting system with the available funding. Members also discussed that because multiple stakeholders may benefit from consumer reporting systems, it would be desirable to have a funding model that included all entities that benefit. Funding of the Pennsylvania State reporting system was described positively in that it is based on a fixed State government budget item (funded by hospitals) and not subject to annual legislative appropriations
    • Regarding other design features, TEP members indicated that both structured and unstructured (narrative) responses in reports are useful for conveying a consumer's unique perspective, and each type of information has different uses in analysis. The TEP therefore recommended that systems collect both types of responses.
    • The TEP endorsed confidential (as opposed to anonymous) reporting, to allow systems to provide feedback to consumers as well as the opportunity to collect additional information from individuals who submitted reports. However, the TEP recognized that some reporters will prefer to report anonymously, and therefore recommended that anonymous reporting be allowed, although confidential reporting was to be encouraged. There was also concern regarding the ability of consumer reporting systems to keep information confidential when involved in legal proceedings (e.g., if served with a subpoena). An alternative model suggest by several TEP members was for a system to pursue legislative protection from releasing any confidential information, thereby assuring people that their information will not be used against them.
  5. How would consumer reporting of patient safety events be linked to quality and/or patient safety improvement efforts?
    • The TEP emphasized that new consumer reporting systems should not replace existing systems and efforts to improve quality or patient safety, but should link to existing QI, RCA, or reporting systems. Systems could give reporters the option of having their report automatically forwarded to the appropriate local facility, system, or organization related to the reported events and to appropriate State or national systems. Further specification of this design feature may have to wait until implementation because the technology required for such interoperability is likely to change rapidly.
    • TEP members broadly agreed that public reporting (that is, allowing public access to nonconfidential information from patient safety events reports) was a key approach to linking consumer reporting of patient safety events to patient safety improvements. However, the TEP also recognized that public reporting is a dynamic area that will become more important and more accepted over time. As such, although the TEP endorsed public reporting as an important system component and stated that systems should have the capability to engage in public reporting, the TEP did not include specific types of public reporting in the consensus recommendation (5.2). Rather, the TEP explicitly recommended that specifics regarding public reporting be developed based on pilot testing and at system implementation.
  6. How can a reporting system maximize the willingness and ability of consumers to report on patient safety events?
    • TEP recommendations in this area focused on systems being highly inclusive and responsive. TEP members recognized that beyond development of consumer reporting systems, additional activities will be required to inform the public about such systems and encourage patient safety event reporting. TEP members emphasized that a main feature (perhaps the main feature) with respect to maximizing consumer reporting is timely feedback. Systems will also need to interact with consumers to assess their own performance. The TEP acknowledged that meaningful consumer feedback is an area that will need to be explored as a system develops; that is, it will not be known what feedback is meaningful to consumers until the system starts.

Discussion

A variety of policy considerations arise from the TEP recommendations. This report highlights considerations that can broadly be classified in three groups: issues of rapidly changing technology use and practices, collaboration and coordination among groups and agencies involved in patient safety, and ownership/operation of consumer reporting systems. It is likely that new legislation, regulations, and policies will be needed to address these issues.

Throughout the course of this project, there has been an expected tension between the goal to develop design features for ideal consumer reporting systems and the knowledge of real-world barriers and limitations in the design and operation of such systems. To some extent, this has reflected an interest by TEP members in issues related to implementation of the consumer reporting systems that they were designing. Implementation of consumer reporting systems and related activities that go beyond specifying recommendations for system design features are outside of the scope of this project. However, for future work, it will be critical to consider these issues. TEP members stressed the need for pilot testing in multiple local and regional settings prior to broad-scale implementation of a consumer reporting system. The TEP also recognized that while all of the recommended design features are important for the final consumer reporting system, only a subset of the recommendations need to be specified at initial system implementation; other recommendations can be specified at a later time.

Although reporting of patient safety events is often associated with hospital-based incidents, these recommendations are applicable to patient safety events in all health care settings. TEP members indicated that the consumer perspective may be particularly important for patient safety events that occur during health care transitions (i.e., when care for a patient transfers from one provider or health care organization to another).

External reviewers also discussed a broader question regarding expectations for what can be accomplished by consumer reporting systems. Multiple reporting systems have been developed and implemented over the past decade; as such, consumer reporting systems may not be able to contribute substantially to information on the incidence or types of events. However, the dual purpose recommendations for consumer reporting systems (to learn with the intention of improving patient safety and to be accountable to those who submit reports) highlights a unique and critical role for consumer reporting systems.

The idea of consumer reporting systems for patient safety events holds great promise. Turning that idea into a reality will require significant political will, policy coordination, and resource investment. The design features presented in this report provide an actionable foundation for the necessary next steps of implementation and operation.

Limitations

Resource constraints—as well as attempts to minimize the already considerable time that members of the TEP graciously devoted to this project—limited our ability to explore all possible areas for recommended consumer-reporting-system design features. In addition, although the TEP consisted of a diverse group of individuals with experience in a range of relevant areas, it is not possible to include individuals with all types of appropriate backgrounds and expertise on a single panel. This limitation was addressed by providing TEP members additional information from consumer focus groups, stakeholder interviews, and an environmental scan and literature review. Information from the consumer focus groups in particular provided additional perspectives and viewpoints for TEP members to take into account in developing recommendations for key design features.

There are several areas in which the TEP chose not to develop explicit recommendations. For example, although TEP members discussed the types of organizations that would be most appropriate to operate consumer reporting systems, members elected not to recommend specific organizations or organization types, and instead specified characteristics of such organizations. Similarly, although TEP members discussed several potential financial models for consumer reporting systems, the TEP chose not to present a recommended approach to finance such systems.

Other recommendations recognize the importance of certain key design features, but do not specify the operational aspects of those features. For example, Recommendation 3.1 emphasizes the importance of linkages for consumer reporting systems and states that systems should have linkages with a broad range of organizations. However, recognizing that linkages are a dynamic and rapidly changing area, the TEP chose to specify neither how such linkages should operate nor the exact types of organizations with which to establish linkages. The operational details of such recommendations will need to be specified based on pilot testing and at implementation of consumer reporting systems.

There were numerous discussions and comments by TEP members regarding implementation issues for consumer reporting systems. However, this project focused on recommendations for system key design features and did not develop recommendations specific to implementation issues. Implementation challenges described by TEP members are presented in the Discussion section of this report.

Current as of July 2011
Internet Citation: Executive Summary: Designing Consumer Reporting Systems for Patient Safety Events. July 2011. Agency for Healthcare Research and Quality, Rockville, MD. http://www.ahrq.gov/professionals/quality-patient-safety/patient-safety-resources/resources/consumer-experience/reporting/executive-summary.html