Designing Consumer Reporting Systems for Patient Safety Events
Chapter 4. Results and Key Findings
The objective of this study is to provide recommendations from the iterative, consensus development process on key design features for consumer reporting systems for patient safety events. The six research questions listed in the Chapter 1. Background section of this report provided the framework for developing these recommendations.
Through the Delphi questionnaires and Technical Expert Panel (TEP) meetings, with input from the focus groups, stakeholder interviews, and environmental scan/literature review, the TEP has developed recommendations for the key design features. The TEP and external reviewers emphasized that these recommendations are applicable to consumer reporting systems based not only in hospitals but also in a variety of other health care environments. Below, we present the consensus recommendations related to each of the six research questions from the third TEP meeting. Following this, the Analysis section then combines these recommendations to present potential holistic frameworks for the design of consumer reporting systems. Finally, the Discussion and Policy Implications section concludes this report by highlighting possible next steps in the development of consumer reporting systems, focusing on implementation issues that go beyond the scope of the current project.
1. What type of information can consumers provide concerning their health care experience with patient safety events that may be useful and/or actionable in a patient safety event reporting system?
- What happened?
- Was the problem reported? To whom?
- What happened when the problem was reported?
- What caused the patient safety event to happen?
- Where did the patient safety event happen?
- What impact did the patient safety event have?
- What were the consequences of the patient safety event?
Recommendation 1.1: Types of information. The systems should collect information on all types of events, ranging from near-miss and no-harm events to adverse events. The systems should capture both objective information about what occurred and more subjective information based on the consumer's unique perspective. Information collected from consumers should include where a patient safety event occurred; what contributed to the event; whether or to whom an event was reported; what happened when an event was reported; and the impacts or consequences of the event.
The TEP recommendations focused on collecting broad and diverse input from consumers for patient safety event reporting systems. However, the TEP expressed concerns about asking consumers what "caused" an event, and felt it was more appropriate to request information on what "contributed" to an event from the consumer's unique perspective. TEP members and an external reviewer also cautioned that consumers are likely to identify and report incidents that don't represent patient safety events or errors. The TEP emphasized that consumer reporting systems will not provide an immediate or urgent intervention/response. Systems will, if possible, provide general guidance on the types of individual or resource to contact when an urgent intervention is desired.
2. What are the scope and range of options for consumer reporting mechanisms? How would these options differ at the international, national, regional, State, or local levels?
Recommendation 2.1: Purpose and goals. The dual purposes of a consumer reporting system are to learn and to be accountable to consumers providing reports. To learn means obtaining the consumer perspective and experience to identify, mitigate, and prevent risks, hazards, and harms; improve outcomes; and advance patient safety. To be accountable to consumers providing reports means that reported information is actively used to design meaningful improvements in patient safety.
After extensive discussions, the TEP recommended that consumer reporting systems have two purposes. The first is focused on learning by obtaining the consumer perspective, improving outcomes, and advancing patient safety. The second purpose focuses on accountability, which is specifically defined as consumer reporting systems being accountable to those who submit reports. In this context, systems have a responsibility to actively use reported information in the pursuit of meaningful improvements to patient safety. These two purposes are linked, as use of reported information to design meaningful changes in patient safety is predicated by learning from these reports. In general, it will be an aggregate of reports (rather than a single, individual report) that leads to learning and resultant activities to pursue improvements in safety; as such, systems need to consider accountability to all those who have submitted reports.
The TEP acknowledged that others, particularly in consumer groups, advocate a definition of accountability for consumer reporting systems that involves "consequences" for providers and medical entities responsible for a patient safety event. However, a majority agreed that accountability related to this recommendation most appropriately refers to the system being accountable to consumers, not systems holding others (e.g., health care providers or medical care entities) accountable. TEP members cautioned that a more punitive definition of accountability could triggers concern and hesitation with many potential stakeholders and could widen gaps between providers and patients, resulting in systems being less effective. An external reviewer provided similar comments, emphasizing that without buy-in from health professionals and hospitals, a consumer reporting system could increase division between providers and consumers. The report by the Institute of Medicine, Patient Safety: Achieving a New Standard for Care, discussed a continuum of applications, ranging from accountability (which is generally the focus of public-sector legal and regulatory bodies) to learning (both for professionals and for organizations). The two related purposes recommend by the TEP may reflect this continuum of potential purposes and goals for consumer reporting systems.
Recommendation 2.2: Level of operation. Reports should be collected locally and communicated to a centralized (national) level that can aggregate and analyze data and triage or distribute information to State and local levels for action. Reporting systems will need to be flexible regarding analysis and other activities occurring at local levels, based on needs, capabilities, and funding/resources for these local activities.
The TEP separated data collection/reporting capabilities at the local level from data aggregation, analysis, and distribution activities at a centralized (potentially national) level. TEP members indicated that certain reporting system functions need to occur at a national (or central) level, such as aggregation of data from multiple sources. However, the TEP was reluctant to specify reporting system functions and activities that should or should not occur at local levels; they emphasized that there needs to be flexibility regarding local level activities based on the need, capabilities, and funding available for these activities. Members discussed the substantial resources needed for system development and stressed the importance of system pilot testing at different levels and in multiple localities. Decisions regarding potential system activities at local levels can be informed by pilot testing during system implementation. Similarly, the TEP indicated that potential differences in their recommendations at the international, national, regional, State, or local levels would need to be assessed during implementation. The TEP did discuss potential advantages of State-based reporting systems, which include the ability to assess care at multiple sites (that is, not only hospitals). The Pennsylvania system was used an example of a State-based system. While issues and examples from State-based reporting systems were provided by certain TEP members during meeting discussions, the TEP did not provide any design feature recommendations specific to State-based systems.
The TEP did not address issues related to interoperability among consumer reporting systems that operate at different levels. However, in an issue related to interoperability, the TEP chose not to specify how linkages between organizations would function in Recommendation 3.1.
3. What type of infrastructure is needed to enable effective, actionable consumer reporting of patient safety events?
Recommendation 3.1: Linkages. Systems should have linkages to a broad range of organizations that can change health care practices and demonstrate that reported information was used. Linkages should be formed for the purpose of encouraging consumer reporting, improving analysis, sharing results, and changing delivery for quality improvement. Linkages will also ensure timely information sharing. Because linkages are dynamic and rapidly changing, their exact nature and specifications will be more fully specified at implementation.
Although the TEP recommendations focused on the importance of linkages for consumer reporting systems, the TEP also recognized that system linkages represent a dynamic, rapidly changing area. As such, the TEP chose not to specify how such linkages would work operationally. The exact nature and specifications of linkages can be specified more fully at system implementation. Regardless of the exact nature of linkages, the TEP also emphasized the importance of considering the purpose of linkages, and evaluating whether current linkages achieve these purposes in a timely manner. The "broad range of organizations" specified in Recommendation 3.1 should include all stakeholders who are part of the current construct of how health care services are delivered or financed. Many types of organizations are involved (directly or indirectly) in health care and can play important roles in improving patient safety, either by changing provider practices and/or by changing their own practices. Such organizations should include State and Federal regulatory and financing organizations (e.g., the Food and Drug Administration [FDA], the Centers for Medicare & Medicaid Services [CMS]); health care facilities and systems; accrediting bodies; Patient Safety Organizations [PSOs]; insurers; employer health care groups; user groups or communities of interests, where individuals who experienced patient safety events can share their stories; and organizations that can provide assistance to consumers submitting reports beyond that available directly from consumer reporting systems. TEP members also indicated that consumer reporting systems should share information with other patient safety event reporting systems, such as FDA's MedWatch system, but did not provide details as to how interactions with other reporting systems would occur. Sharing of information with other organizations would be subject to confidentiality constraints specified by the reporter and stated by the consumer reporting system.
Recommendation 3.2: Analytic functionality. Systems will need decision rules for what kinds of events receive different levels or types of analysis. Systems should collect information and conduct aggregate causal analyses. Systems should also gather responses of organizations to reports and evaluate their feedback.
In discussions related to recommendation 3.2, the TEP identified three system models regarding analytic functionality. In one model, systems do not expend resources to conduct Root Cause Analyses (RCAs), but request RCAs when performed by other organizations and analyze collected RCA information. A barrier to this approach may be confidentiality rules prohibiting sharing of RCAs, particularly those conducted by PSOs. In an alternative model, systems are able to perform RCAs on selected events. In this model, public decision rules are used to determine which events warrant an RCA and the performance of RCAs are subject to financial constraint. When RCAs are conducted, patients will be consulted during the RCA process. For this alternative model, TEP members commented that there could be barriers to accessing needed information (from health care organizations in particular) to perform RCAs, in addition to associated costs and the need for trained personnel. The TEP did discuss that many hospitals do not have personnel trained to perform RCAs correctly. This is even a greater difficulty in nonhospital environments, and thus may limit the availability of high-quality RCA from multiple health care settings. A third alternative model discussed by the TEP was collection of data using a standardized form for focused initiatives. This would involve development of a data collection instrument to gather information on a particular type of patient safety event. The instrument would be submitted to relevant health care facilities (presumably facilities where events of the specified type occurred based on reports to the system) and used to collect information related to the event. Information from the standardized form would be aggregated and analyzed by consumer reporting systems and disseminated as part of feedback and reports. This model was described as a means for consumer reporting systems to actively gather information related to patient safety events but involving less cost than performing RCAs.
4. What is the most effective operational approach for consumers to report patient safety event information?
- In what kind of organization (e.g., public�private partnership, public, private) should a consumer reporting system be housed?
- How should a consumer reporting system for patient safety events be financed?
- Independent entity with a steady stream of sustainable funding, where "independent" is defined as an entity that is completely separate in ownership, governance, and affiliation from entities that provide health care and whose members, employees, or affiliate entities may be the subjects of reports about adverse events.
- Governing body members' fiduciary responsibility is to represent the public.
- Neutral oversight body with consumer representation.
- Transparency of goals, process, and results.
- Consumer involvement in organizational governance and operations.
- Dedication to analyzing incoming information to identify threats to patient safety and feeding it back to systems that may be able to act on it.
The TEP explicitly chose not to specify the types of organizations to operate consumer reporting systems, electing instead to recommend characteristics of such organizations. However, the TEP indicated that multiple stakeholders should be involved in the operation of consumer reporting systems. Consumer focus group participants also supported the involvement of multiple stakeholders, indicating that systems should not be operated exclusively by consumers. In discussing the potential types of organizations or business models that could operate consumer reporting systems, the TEP agreed that these could be either public or private organizations. Four potential business models were discussed: commission model, PSO model, quality improvement organization (QIO) model, or a subscription/co-op/consumer-driven model (similar to Consumers Union, where members pay dues to support the organization). The TEP did not reach consensus on the type or types of business models appropriate for operating consumer reporting systems, nor did the TEP indicate that these four were the only possible business models for operating such systems. The TEP also indicated that ownership of consumer reporting systems should be credible to multiple stakeholders. Some TEP members indicated that consumer reporting systems should not be owned or operated by accreditation or regulatory agency such as the Joint Commission, CMS, or State Departments of Health; however, there was not universal agreement on this comment. TEP members also discussed whether independent coalitions or partnerships between organizations operating at different levels (e.g., between regional and national organizations) could operate consumer reporting systems. The TEP did not reach consensus on this point.
TEP members discussed in detail whether Patient Safety Organizations (PSOs) may be relevant models for consumer reporting systems. Members identified characteristics of PSOs that are consistent with the recommended design features for consumer reporting systems, including having a goal of learning; being devoted to analyzing reported information to identify threats to patient safety and feed information back to systems that may be able to act on it; and having statutory confidentiality protections. However, the confidentiality rules for PSOs vary among States, which was viewed as a negative characteristic by the TEP; that is, for consumer reporting systems, confidentiality protections should be consistent across all regions in which a system operates. Other features of PSOs were also felt not to be a good "fit" for consumer reporting systems. The TEP concluded that PSOs are not the right model for consumer reporting systems, but that there are many areas in which consumer reporting systems can learn from PSOs.
Financing of Consumer Reporting Systems
The TEP chose not to develop specific recommendations regarding financing of consumer reporting systems. However, this issue was discussed by TEP members. Key points in this discussion included:
- There is a need to align the scope and activities of a consumer reporting system with the available funding. Systems should not "overpromise" on what they can deliver, and should make their expectations consistent with their available funding.
- Consumer reporting systems may be viewed as a "public good". Thus, there may be a role for granting and nonprofit organizations in funding such systems.
- Multiple stakeholders may benefit from consumer reporting systems; it would be desirable to have a funding model that included all entities that benefit.
- The source of financing could influence consumer perceptions regarding the reporting system. The Joint Commission was cited as an example of possible difficulties with consumer perception because it is funded by facilities that it accredits and surveys. However, funding of the Pennsylvania State-reporting system was described positively in that it is based on a fixed State government budget item (funded by hospitals) and not subject to annual appropriations. TEP members also discussed the funding model used by ECRI, which is funded through subscription services; findings from research conducted by ECRI are distributed to paying subscribers.
Other Design Features of Consumer Reporting Systems
Recommendation 4.3: Reporting modalities. To maximize reporting, systems should allow multiple routes or modalities for reporting.
Recommendation 4.4: Reporting format. Systems should allow a mix of structured and unstructured reporting.
The TEP recommended that consumer reporting systems be broad with respect to access to the system at different points in time and reporting modalities. TEP members indicated that both structured and unstructured (narrative) responses in reports are useful for conveying a consumer's unique perspective, and each type of information has different uses in analysis. The TEP therefore recommended that systems collect both types of responses.
Recommendation 4.5: Anonymity. The system will allow anonymous reporting, but the system should be designed to discourage anonymous reporting by ensuring and encouraging well-designed confidential reporting. The system could allow reporters to opt out of confidentiality to increase the report's efficacy in certain situations.
The TEP endorsed confidential (as opposed to anonymous) reporting, to allow systems to provide feedback to consumers and the opportunity to collect additional information from individuals who submitted reports. However, the TEP recognized that some reporters will prefer to report anonymously, and therefore recommended that anonymous reporting be allowed, although confidential reporting was to be encouraged. The characteristics of system confidentiality (called "well-designed confidentiality" by one TEP member) specified in the TEP recommendation were not fully specified; some members described it as being similar to confidentiality used for the aviation industry reporting system, where links to a reporter's identity are discarded after a period of time. TEP members did express concerns that, in some cases, providing information about a patient safety event could lead to the identification of a reporter. There was also concern regarding the ability of consumer reporting systems to keep information confidential when involved in legal proceedings (e.g., if served with a subpoena). An alternative model suggested by several TEP members was for consumer reporting systems to pursue legislative protection from releasing any confidential information, thereby assuring people that their information will not be used against them.
5. How would consumer reporting of patient safety events be linked to quality and/or patient safety improvement efforts?
Recommendation 5.1: Linking to quality and patient safety improvement efforts. Systems should be linked to efforts to improve quality and patient safety. If the reporter allows, his or her reports to the consumer reporting system will be automatically forwarded to appropriate existing reporting systems at the local or facility level.
The TEP emphasized that new consumer reporting systems should not replace other system, but should link to existing QI, RCA, or reporting systems. System should encourage patients to register their report directly to existing systems at the facility where an event took place; one TEP member commented that a well-intentioned facility will be able to do more to correct errors than a national system, although there was not universal agreement on this statement. Systems could give reporters the option of having their report automatically forwarded (presumably by the hub or central operations facility of a consumer reporting system, although this was not explicitly stated) to the appropriate local facility, system, or organization related to the reported events and to appropriate State or national systems. To this end, one TEP member commented that consumer reporting systems should understand the type and format of reports or other information that can best be understood and acted on at health care facilities. This would allow information shared by consumer reporting systems to "interdigitate" with materials from health care facilities (that is, provide information that is understandable by personnel in health care facilities and supplements their own information, leading to greater opportunities to address patient safety issues). Consumer reporting systems will need to ensure that their reports can be used by "downstream" users (e.g., health care facilities and systems) so they can address problems. Further specification of this design feature may have to wait until implementation because the technology required for such interoperability is likely to change rapidly.
- Publish information, such as how much a system was used.
- Publish information on what was learned.
- Publish information about what recommendations and changes were made as a result of the system.
- To the extent determinable, publish information about the responsiveness of institutions to patient safety issues.
However, because this is an evolving and dynamic issue, the exact specifications will be developed at implementation and will be determined over time as the issue develops.
TEP members broadly agreed that public reporting (that is, allowing public access to nonconfidential information from patient safety events reports) was a key approach to linking consumer reporting of patient safety events to patient safety improvements. TEP members discussed that public reporting will inform consumers about rates of patient safety events, although this is subject to multiple caveats including incomplete or unknown denominators, small sample sizes, and rare events. Furthermore, public reporting will allow consumers to assess the responsiveness of consumer reporting systems. However, the TEP also recognized that public reporting is a dynamic area that will become more important and more accepted over time. One TEP member emphasized the lack of data linking public reporting to actual improvements. As such, although the TEP endorsed public reporting as an important system activity for the purposes listed above and stated that systems should have the capability to engage in public reporting, the TEP did not include specific types of public reporting in the consensus recommendation (5.2). Rather, the TEP explicitly recommended that specifics regarding public reporting be developed as part of pilot testing and at system implementation. TEP members discussed several types of public reporting, but did not include these as part of the recommendation. However, the TEP did agree that more often an aggregate of reports (rather than a single, individual report) leads to learning and resultant improvements in safety (as discussed under Recommendation 2.1). Therefore, information coming from reporting systems about what it has accomplished would be targeted to all those who have submitted reports, rather than to each individual who submitted a specific report. The TEP also indicated that they did not specify all circumstances, caveats, and limitations for when public reporting should or should not occur.
6. How can a reporting system maximize the willingness and ability of consumers to report on patient safety events?
Recommendation 6.1: Maximizing reporting. System design should facilitate reporting to ensure maximum use, that is, maximize the ease/ability of consumers to submit reports. This will include public awareness campaigns or other outreach/marketing activities and getting "buy-in" from appropriate individuals and organizations as part of implementation.
TEP recommendations in response to this AHRQ research question focused on systems being highly inclusive and responsive. TEP members recognized that, beyond development of consumer reporting systems, additional activities will be required to inform the public about such system and encourage patient safety event reporting. TEP members also recommended that systems be aware of the diversity of consumers who experience patient safety events and include features to facilitate access for diverse and underserved populations. However, the TEP did not recommend specific design features to facilitate access; these are likely to evolve over time as technology evolves.
Recommendation 6.3: Feedback. Systems should provide meaningful and timely feedback to reporters. Feedback will include public reporting, awareness campaigns, and meaningful acknowledgement of the receipt of a report. Systems will not be able to assure reporters that they will receive meaningful and timely feedback from the health care facility/system where a patient safety event took place.
TEP members emphasized that a main feature (perhaps the main feature) with respect to maximizing consumer reporting is meaningful and timely feedback. This will include acknowledgements when reports are initially submitted and additional updates as more information is obtained or actions are taken. Systems will also need to interact with consumers to assess their own performance. The TEP acknowledged that meaningful consumer feedback is an area that will need to be explored as part of pilot testing and as a system develops; that is, we will not know what feedback is meaningful to consumers until the system starts. The type and extent of feedback provided will also need to be tailored to the available funding. Further, TEP members recognized that although systems should provide feedback on their own activities, they may not be able to provide feedback on what other entities (e.g., health care facilities) do with shared information. As part of this recommendation, the TEP also specified that public reporting should be included as part of feedback, although public reporting is addressed separately in Recommendation 5.2.