Chapter 5. Analysis—Potential Consumer Reporting System Design
In broad overview, consumer reporting systems for patient safety events involve several design components (Exhibit 1). First, an interface is available to facilitate submission of reports when consumers experience or notice actual or potential patient safety events. The system will provide multiple options for report submissions, including submitting directly to a national "hub" (that is, a single physical or virtual location where activities of a consumer reporting system are centralized) and potentially reporting locally through face-to-face interactions with system representatives.
After submission, reported information is transmitted to the system's central hub. The central hub has two purposes. First, system staff determines how to categorize, triage, and respond to an individual report. Second, the hub acts as a multilevel analytic engine—it collects report information, performs aggregate causal analysis, and identifies problems at involved health care facilities/systems. In one model, the system hub will collect information from the health care facility where an event took place, including Root Cause Analysis (RCA) data, but will not perform its own RCAs. RCA data from health care facilities will be combined with other report information (both information submitted in a report and that subsequently collected following a report submission) for use in subsequent analyses. As discussed by one Technical Expert Panel (TEP) member, additional legislation/regulations may be required to facilitate sharing of information by health care facilities. The RCA data may also be part of public reporting (providing broad public access to nonconfidential information from reports and other data sources) by systems. A caveat discussed by the TEP is the limited number of individuals trained to correctly perform RCAs, which may affect the usefulness of available RCAs. In an alternative model, systems will have the capability to perform RCAs on selected events. In this model, public decision rules are used to determine which events warrant an RCA and the performance of RCAs are subject to financial constraint. When RCAs are conducted, patients will be consulted during the analysis process. In this alternative model, there could be barriers to accessing needed information (from health care organizations in particular) to perform RCAs, in addition to associated costs and the need for trained personnel (go to Exhibit 5 for more detail).
The TEP recommended that reporting systems for patient safety events have a governing body to ensure that actions and results align with goals. In this context, a governing body is a group of individuals who oversee operations of a consumer reporting system and are responsible for strategic decision making. The governing body must be neutral and independent of the health care delivery system.
The system central hub will have direct links to three types of external entities. First, after receiving and processing consumer report information, the system will provide timely and meaningful feedback to consumers. This will include notices of receipt of reports (i.e., personalized acknowledgements when reports are first transmitted to the system) as well as later communications from the system when analyses are performed, additional information is received, or actions are taken related to reports. Systems will also engage in public reporting (as defined above) to demonstrate the system's functions and operations, increase transparency, and improve the quality and safety of health care delivery. The exact nature of public reporting and extent of such activities is not specified in the system design features, and will be determined during pilot testing and at system implementation. Reporting systems may also operate campaigns to increase awareness of the systems and their goals among the general public.
Second, the system hub will interact with the health care facility where an event took place. If the consumer is willing, the hub will immediately forward all information about a specific event to the facility in question. This will provide the facility with the opportunity to respond to its consumers and engage in meaningful quality improvement. The system will also provide feedback to the facility regarding system findings. This feedback will be screened to protect the identity of the reporting consumer.
Third, reporting systems will link to other external systems and organizations when those linkages could encourage reporting, improve analysis, positively change care delivery, and improve care quality. Links can include State, national, or international organizations. Links can also include local organizations as part of the local effector arm, a reporting system capability that permits direct interactions with health care providers or facilities. Reporting systems would need to demonstrate that these linkages add value to the systems and increase the effectiveness of actions to improve patient safety.
The consumer reporting system recommended by the TEP can be broken into four main components (Exhibit 2). To establish an effective system with high user satisfaction, each of these stages must function properly.
The first stage is initiated when a health care consumer who has experienced a patient safety event (which occurs as a result of an interaction between the consumer and a health care facility or provider) seeks a place to report. Once a consumer has committed to making a report, the system will assist him or her in providing all necessary information. Report preparation will also involve multiple design features that improve accessibility and consumer trust (e.g., allowing consumers to decide between confidential or anonymous reports and with whom a report can be shared).
Inputting the report to the system is the second component. After consumers select from an array of transmission modes for inputting the report, information from the report is standardized and categorized for subsequent analyses. Information related to the event may also be provided by health care providers or facilities after the report is processed (e.g., RCAs performed at a health care facility).
In the third component, the system's central hub will aggregate and analyze reported information. Reports will be reviewed by well-qualified staff who understand patient safety issues and are trained to extract and triage information effectively. The system will analyze trends and conduct aggregate causal analysis, but may also evaluate selected individual incidents.
After data have been processed, the system must communicate results to a variety of stakeholders, at the local level (including reporters and health care facilities) and possibly at the State level (e.g., licensing and accreditation bodies) and national level (e.g., regulatory and financing agencies). Communications may also take place at the international level (e.g., with other reporting systems), to further a system's goal of learning. These communications are the final, and most visible, components in a consumer reporting system. First and foremost, the system will provide the consumer with timely acknowledgement of receipt of the report and meaningful subsequent updates regarding actions taken. The system will also provide feedback to health care facilities and providers with the goals of fostering health care system delivery change and quality and safety improvement. To this end, information provided by consumer reporting systems should "interdigitate" with systems at health care facilities; that is, consumer reporting systems should understand the type and format of reports or other information that can best be understood and acted on at health care facilities, and provide feedback in this manner. All information released to the care facility will be de-identified (unless the consumer explicitly indicates otherwise), and every precaution will be taken to protect the identity of the reporter.
The system may also share information with entities (including organizations, government agencies, for-profit and nonprofit enterprises, or individuals) not involved in the event. Depending on the nature of the report and whether the reporter permits, systems and organizations outside of the consumer reporting system (for example, other existing reporting systems, State regulatory agencies, or the Food and Drug Administration [FDA]) may be consulted and provided with report data. This may require decisions regarding de-identification of providers or facilities in reports, beyond de-identification of the reporter.
Finally, aggregate system information will be released publicly. Public reporting (providing broad public access to nonconfidential information from reports and other data sources) will serve two main types of functions. First, it will hold the system accountable to its own goals. This will include providing information on how often the system was used, what the system learned from reports, and what recommendations or changes were made as a result of the system. Second, to the extent determinable, it will provide information on the responses of health care institutions to patient safety issues highlighted in reports. As public reporting is an evolving and dynamic issue, the exact specifications for this reporting will be developed during pilot testing and at implementation, and will be reassessed over time. Important considerations in the development of any public reporting program will include decisions regarding reporting of individuals incidents versus aggregate information; identification of health care facilities and/or providers; ensuring confidentiality for individuals who submitted reports; and legal protections for system information that could be subject to subpoenas or discovery during legal proceedings. Considerable attention is needed regarding the circumstances, caveats, and limitations for when public reporting should or should not occur.
The reporting system will collect reports from several categories of individuals who experienced or witnessed patient safety events (Exhibit 3). The primary users of the system will be consumers—the patients receiving care and their friends and family. The system will also accept reports from medical care providers (although there was not universal agreement among TEP members on this) and from bystanders (individuals who witnessed an event, but were not directly involved with the individual receiving care).
Reporters will be asked to complete a standardized reporting form. As described in the report To Err is Human, Building a Safer Health System (IOM, 1999), a standardized reporting form can permit data to be combined and tracked over time; lessen the burden on health care organizations that operate in multiple states or are subject to regulation by multiple organizations; and facilitate communications about patient safety. As discussed in the report Patient Safety: Achieving a New Standard for Care (IOM, 2004), domain areas for a common patient safety reporting format include information on:
- The discovery: who discovered/reported the event (roles, not names) and how it was discovered.
- The event itself: the type of event; where (in the process of care) and when it occurred; who was involved (functions, not names); why it occurred (most dominant cause); and risk assessment (including event severity, preventability, and likelihood of recurrence).
- Narrative of event, including contributory factors.
- Ancillary information, including medical product information and patient demographic and clinical information.
The form will guarantee user confidentiality but will also offer users the option of reporting anonymously. The form will discourage users from reporting anonymously by noting the limitations of this type of reporting (e.g., being able to contact the individual making the report subsequently to gather additional information or provide feedback). The form will assure users that confidential information protection is a top priority for the system.
The standardized reporting form will allow both structured and unstructured reporting. Structured reporting—possibly in the form of multiple choice questions—ensures that the system will capture standardized information necessary to analyze and classify reports. These prompts will be designed to capture the type of event (e.g., harm event, no harm event, close call/near miss), the location of the event, the people involved, and basic information regarding what happened. This information will allow the system to route the report appropriately and make quick initial judgments as to how the report will be used. Standardized questions will also be helpful in collecting system-level aggregate information. The reporting form may use an existing taxonomy such as the WHO-ICPS structure to facilitate standardization and aggregation.
Reporters will also be given the opportunity to provide a narrative account of the patient safety event that they experienced or witnessed. This unstructured report allows the reporter to identify the most salient issues from his or her perspective and ensures that valuable information is not lost when it does not fit into a standardized format. By eliciting specific information with standardized questions and allowing unstructured narratives, the standardized reporting form is balanced and calibrated to provide the system with rich and usable information. Pilot testing will likely be important for finalizing the reporting form.
Individuals will be able to submit reports at any time during or after the event. The systems could flag reports that occur significantly after the event, which may be used differently from more concurrent information. Systems will also caution reporters that this is not a substitute for calling 911, and other approaches should be used for emergencies requiring rapid responses. Systems may provide general guidance regarding the types of individuals or resources to be contacted if an immediate or urgent response is desired. Users will also be able to update their reports over time, providing additional information as they learn more or have continued experiences related to their patient safety event.
After consumer reports are submitted, they are transmitted to the reporting system's central hub (Exhibit 4). To ensure effective use of reported information at the central hub, the TEP identified several structural and organizational characteristics of consumer reporting systems as important. These characteristics emphasize the central role of consumers as the focus of the system.
TEP members discussed two main themes regarding the system's organizational characteristics. First, the structure and governance of the system's central hub will ensure that the system is answerable to consumers. In support of this consumer focus, the system will be independent with a steady stream of sustainable funding. Second, the system will be structured and organized to inspire consumer trust and demonstrate credibility. To this end, the system will be governed by a neutral oversight body that strives for transparency of goals, processes and results, and obtains consumer involvement in organizational governance and operations. In keeping with the purposes of the system (Recommendation 2.1), the oversight body will assist the system to be accountable to those who submit reports.
The reporting system will also link to other systems or organizations that have capabilities outside of the scope of a patient safety event reporting system and that add value to the system. Attempts should be made to link to organizations that can use system results to change health care practices, demonstrate that reported information was used, encourage reporting, improve analyses, and address those needs of reporters that cannot be addressed internally. These other organizations may function at local, regional, State, national, or international levels.
A reporting system for patient safety events will have multiple entry points (i.e., methods for submitting reports) to increase consumer ease and accessibility and maximize reporting (Exhibit 5). Reporters will have the option to input their information directly to the system or through a system representative. If they choose the former option, the reporter can make his or her submission by mail, fax, E-mail, or Internet. A "smart form" would be used for each of these options, allowing submitted information to be entered into the reporting system with standardized data fields. If the reporter prefers a human interaction for report submission, a toll-free telephone line will be available and staffed with trained representatives who will gather relevant information and submit standardized reports.
TEP members disagreed on the need (or desirability) of having local offices collect reports. As an alternative system model, local offices with in-person representatives (indicated in italic to reflect lack of consensus) may make systems more user friendly and increase reporting. However, TEP members indicated that it is not clear that systems would have the resources to accommodate the infrastructure necessary to operate local offices.
Regardless of the input channel, all information received from consumers will be entered into systems using common data standards and processed by multilevel analytic engines. Structured inputs (e.g., drop-down menus, multiple choice questions) are immediately useable; narrative descriptions can be classified using natural language processing. After this processing, structured and unstructured information collected during the reporting phase will be functional data that can be used to aggregate information, identify trends, and perform aggregate causal analysis. A standard taxonomy may be used to guide data aggregation. A number of such taxonomies exist; AHRQ has developed common data formats for the Patient Safety Organization (PSO) program, and one TEP member advocated use of the WHO-ICPS taxonomy structure.
Although TEP members agreed that the system should collect RCA when it has been conducted, they also agreed with concerns expressed by external reviewers about the dearth of individuals trained to perform RCAs. There are limited numbers of trained individuals present at most hospitals, and even fewer individuals based in other health care settings. This may have substantial impacts on the quality and usefulness of available RCAs. TEP members disagreed about the depth at which consumer reporting systems should investigate individual reports. As an alternative model, some experts thought that the system should develop public decision rules that indicated when the system should perform RCA on select events, subject to financial restraints (presented in italics to indicate an area of lack of consensus among the TEP). This relates to work by Tjerk van der Schaaf regarding the PRISM system, as described in the report Patient Safety: Achieving a New Standard for Care (IOM, 2004). Others felt that the system should not expend resources to conduct costly RCAs. The split in expert opinion suggests the development of two separate system models.
Return to Contents
Proceed to Next Section