Chapter 41. Human Factors and Medical Devices
Harvey J. Murff, M.D.
Harvard Medical School
John W. Gosbee, M.D., M.S.
Department of Veterans Affairs National Center for Patient Safety
David W. Bates, M.D., M.Sc.
Harvard Medical School
Introduction
Human factors engineering (HFE), also known as usability engineering or ergonomics, is the study of how humans interact with machines and complex systems.1 Through the merging of cognitive psychology, engineering and other disciplines, human factors researchers have detailed numerous principles concerning device and software program designs that allow for optimal usage.2 When these principles are violated, improper use of a machine is more likely to result.3,4 Specific examples have been detailed during observations of user errors with electronic infusion devices.5
Medical device misuse is an important cause of medical error,6,7 and therefore, incorporating human factors methodology into the design of medical devices has assumed an important role in ensuring patient safety.2,8 This chapter first describes the use of HFE principles as a safety practice in the design of medical devices and their evaluation both prior to and after institutional purchase. Next, medical device alarms and the contribution of HFE to alarm improvements will be evaluated. Finally, the chapter reviews the use of preoperative checklist procedures to reduce anesthesia device failures (see also Chapter 23).
Subchapter 41.1. The Use of Human Factors in Reducing Device-related Medical Errors
Background
Human factors engineering is a powerful component in the design of usable, safe medical devices.8 HFE principles can be incorporated as safety practices that occur at various points during device development and usage. Industry can use HFE principles at multiple times in the design and developmental cycle of medical devices and software packages.3 Healthcare institutions can consider results of HFE evaluations when deciding which products to purchase. Finally, HFE principles can also be incorporated into the ongoing evaluation of devices that have already been purchased and are in use. While these practices have high face validity, there has been little formal study of their effectiveness in reducing medical error. They are presented here because they may hold promise if scrutinized rigorously, and to familiarize readers with their potential to reduce medical error.
Design and Developmental Phase
Data collected by the United States Food and Drug Administration (FDA) in the late 1980s demonstrated that almost half of all medical device recalls resulted from design flaws.9 In 1990, Congress passed the Safe Medical Devices Act, giving the FDA the ability to mandate good manufacturing practices (GMP). These GMP involve design controls for manufacturers that help ensure the use of HFE within medical device design.9 As described in the Good Manufacturing Practice Regulation, Design Control subsection (Title 21-Section 820.30), these include the use of iterative design and testing during the developmental phase. The Act requires that designs be "appropriate and address the intended use of the device, including the needs of the user and patient."10 Multiple human factors techniques, such as user studies, prototype tests, and task/function analysis, are utilized in the development and design process.
Manufacturers are required not only to use human factors principles to repeatedly test the product in all phases of design, but also to validate the ultimate device design. Validation entails testing the device, either in an actual clinical situation or a simulation, and documenting that the device conforms to the individual user's needs. Thus, manufacturers are required to apply HFE methods through the multiple phases of device design and development cycles.10
Human factors engineering practices for medical device design and evaluation have been well described. In 1993 the Association for the Advancement of Medical Instrumentation and the American National Standards Institute established guidelines for the incorporation of HFE principles into medical device design.11 This comprehensive document helped direct attention to the problem of poor medical device design and helped establish the design standards necessary to ensure safe medical equipment and thus should stand as the safety benchmark for industry.
Only limited data are available concerning the application of HFE principles to medical device design, and most are not published. Nonetheless, the application of human factors principles during a device's design phase has been demonstrated to reduce user error. Patient controlled analgesia (PCA) pumps are a case in point of how HFE principles in product design reduce user error. User errors associated with poor interface design have been described with PCA pumps.12,13 Lin and colleagues investigated whether applying human factors engineering principles to the design of the user interface of a PCA pump could result in fewer dosage errors as well as less time spent programming the device.14 Information on device usage was obtained through cognitive task analysis. This involved observing and interviewing nurses operating PCA pumps both in the laboratory setting and in the field. Utilizing this feedback, as well as other human factors design principles, a "new" PCA pump interface was designed. Twelve recovery room nurses were required to complete specific tasks with both the standard PCA user interface and the newly designed interface. There were 29 programming errors on the traditional interface and 13 on the redesigned interface (an error reduction of 55%, p<0.01). Furthermore, users were able to program in the necessary orders in 18% less time.13
Another example involves the design of an ultrasound machine. In this study, Aucella and colleagues15 interviewed sonographers, videotaped the ultrasound device being used, and performed usability testing through simulation to collect information regarding the operator-machine interface of the ultrasound machine. After their extensive investigations they implemented over 100 design changes to the console and control panel. Although errors with the machine were not measured, comments collected by the authors from the beta operators of the newly designed device suggested that the resulting machine was much easier to use.
There are enormous numbers of medical devices and software being designed and developed. Thus the FDA has initiated several regulatory mechanisms to ensure compliance with these guidelines. Some of the mechanisms include site inspections of manufacturers, review and approval of medical devices before marketing, and review of medical device incident reports.16 Despite the tremendous amount of effort put forth by the FDA to ensure compliance with the Good Manufacturing Practices Regulation, individual institutions should critically analyze whether a device they intend to purchase meets HFE principles for user-centered design.
Device Evaluation Prior to Purchase
Adhering to HFE principles during initial design stages of a medical device is essential. However, human factors analysis should also be incorporated into the institutional decision to acquire a new medical device or software program.3 Device purchasers should strongly consider institution-specific human factors testing. Usability testing at the institutional level establishes built-in redundancies to capture any design problems missed by manufacturers. Furthermore, the users and environments at individual institutions will differ, possibly in important ways, from the users and environments in which the device or program was initially designed and tested. It is important for an institution to be aware of who the intended users of the device or software will be, as well as where and when they plan to use the device. The information for such evaluations may be obtained from vendors, from an in-house analysis, or from independent organizations.
Vendors must be able to prove to the FDA that the user will be able to operate the medical device in the way in which it was intended.10 As companies are required to collect human factors analysis data, it is important that institutions wishing to purchase a new medical device or software receive and carefully review this information. Gosbee provides a list of questions to ask a vendor before a purchase, which include: "How long does it take to learn to operate the system? How long does it take to complete typical set-up tasks? What are the types and frequency of errors that could happen, and the systems to thwart them?"3
It is also important to consider the environment in which a device will be used. Idiosyncratic features of the environment, such as excessive noise or poor lighting, and differences in user skill or acuity due to fatigue or otherwise, may affect safety and the device's in-house usability.
Some institutions have developed in-house usability labs, in order to rigorously test any device before purchasing. The Mayo Clinic uses simulations to test the usability of medical software before purchasing.17 By carefully measuring user performance with the software they are able to uncover latent errors in the design. The usability lab is also able to measure the time necessary to learn to use the new software. This important information can help predict the device's or software's influence on workflow as well as its predilection for operator misuse.
Even without sophisticated usability laboratories, an institution can use basic human factors techniques to evaluate a product before purchase.3 Powerful techniques such as cognitive walk-through can be easily utilized at any institution. This involves observing the end-users of a product interact with the product. As they attempt to use the device, they are instructed to "think out loud." Careful observation of the user's actions and comments can identify potential design flaws that might make it difficult to utilize the device or software.
Independent organizations are another potential source of information on device safety. Unfortunately, most independent sources do not make clear to what degree HFE principles were used in product evaluations, although they do provide some assessment of safety. One such organization is ECRI (formerly the Emergency Care Research Institute), a nonprofit international health services research agency. Another is the Institute of Safe Medical Practices (ISMP). Both release newsletters and publications regarding product safety. By searching these and similar databases, institutions can gather additional information concerning product safety prior to purchasing a device. ERCI also publishes articles specifically geared to the institutions that might wish to purchase a medical device or software.
Regardless of the level of pre-procurement testing, some unsafe designs will not be detected until after the product is in use.3 Therefore, it is important for institutions to continuously evaluate these products to ensure safety.
Ongoing Device Evaluation
Devices and software at greatest risk for user error should be systematically evaluated. This is particularly important in areas where multiple devices are used with different interfaces, such as the operating room or the intensive care units.3 Furthermore, areas where multiple medications are stored together should be scrutinized for potential latent errors within device or software user interfaces prior to user errors occurring.
Resources are available that can help direct an institution's search. Through publications from the FDA, ECRI, ISMP and similar organizations, medical device problems identified at other institutions can be targeted. Thus an important safety practice may be using this published information to search for latent errors within already purchased medical devices and applying this information toward a directed product evaluation at the local institution.
Another potential safety practice is to educate practitioners about HFE principles to increase awareness of medical device user error.3 Several groups, including the American Nurses' Credentialing Center and the American Society of Health-Systems Pharmacists, recommend incorporating HFE training within healthcare curricula as a means to reduce error.18
To create a culture of safety within medicine, practitioners must couple the ability to identify potential design weaknesses with a change in the prevailing culture of silence surrounding medical errors. Educational programs directed at healthcare providers in training should address both of these important concerns. Curricula for teaching medical student and medical residents HFE principles have been described18 and will likely be adopted at other institutions. Casarett and Helms caution that an unintended result of error curriculum19 may be that residents become too willing to attribute an error to system causes. Their concern is that the resident will ignore any possible individual contribution to the adverse medical event and not learn from analyses of the event. This concern has been discounted by Gosbee, stating that any error-in-medicine curriculum should aim to "teach residents to see when errors are due to inadequate skills and knowledge versus when they are due to inherent cognitive limitations and biases."18
Subchapter 41.2. Refining the Performance of Medical Device Alarms
Background
Numerous aspects of patient care compete for providers' attention and can reduce their vigilance in monitoring medical devices. Alarms can alert providers to urgent situations that might have been missed due to other distractions and have become a necessary part of patient monitoring. In a study looking at critical incidents within a neonatal intensive care unit, 10% were detected through alarms.20
However, fundamental flaws in the design of current alarm systems likely decrease their impact.21 There are reports documenting some alarm failings in the medical literature,22 but few data address interventions to improve alarm system effectiveness. For an alarm to be effective it requires that a medical problem trigger the alarm, that personnel identify the source and reason for the alarm, and that the medical problem be corrected prior to patient injury. This section reviews 2 aspects of alarm safety: (1) the use of HFE principles in the redesign of medical alarms to improve identification of the source and reason for alarm, and (2) practices in both device design and programming that may improve safety by decreasing false positive alarms.
Identification of Alarm Source and Reason
The recognition accuracy of alarms within the operating room is quite low. When presented with alarm sounds and asked to identify the source, anesthesiologists, operating room technicians, and operating room nurses correctly identify the device producing the alarm only 33 to 53.8% of the time.23-25 Furthermore, experiments suggest that humans have difficulty reliably recognizing more than 6 alarms at one time.26 The sheer number of different medical devices with alarms can make it difficult to discern one alarm from another and studies within the human factors literature have documented the inability of medical providers to discern between high priority and low priority alarms.27 While this is a known problem in operating rooms and intensive care units, how well alarms are recognized in other settings has not been described.
Some effort has been made to improve alarm systems through redesign.28 One non-medical study examined ways to improve the recognition of auditory alarms by comparing abstract alarm sounds with specially designed alarms using speech and auditory icons.29 Other studies within the human factors literature have revealed certain acoustical properties that are more likely to result in a higher sense of perceived urgency by the operator.
In a series of experiments, Edworthy required subjects to rank the level of urgency associated with different alarms.30 The acoustical properties of the alarms were altered for the different subjects. Level of urgency was then correlated with a specific alarm sound. After ranking a set of acoustic parameters based on perceived urgency, the experimenters predicted what urgency ranking the alarm would receive and played the alarms for a new set of subjects. The correlation between the subjects' urgency rating and the investigators' predicted ratings was 93% (p<0.0001). Acoustical properties such as fundamental frequency, harmonic series, and delayed harmonics all affected the users perceived urgency.
Another study looked at the redesign of an alarm to improve detectability within the operating room.31 An alarm that was spectrally rich, frequency-modulated, and contained small amounts of interpolated silence was detectable with at least 93% accuracy over background operating room noise. However, both of these alarm experiments have only been done in laboratory settings. In addition, Burt and colleagues found that when subjects were required to urgently perform a task, the prior acoustically manipulated perception of urgency was ignored in order to attend to the situational urgency of the task.32 Furthermore, with both alarms and clinical tasks competing for an operator's attention, the newly designed alarm might not be as discernible. It has continued to be a challenge to create the best auditory alarm sound to indicate an emergency.
Visual Interfaces for Alarms
Alarms can also be visual. Some research has been done to improve hemodynamic monitoring device displays. Responses to abnormal values are delayed when workload for the anesthesiologist is high,33 prompting interest in improving current visual displays. Furthermore, the clinical decision process often rests on the practitioner's interpretation of a patient's hemodynamic parameters. Thus, it is important that this information be presented in a way that assists with decision making and minimizes errors of interpretation.
Two observational studies have compared different visual displays of data to traditional visual monitors.34,35 Each evaluated errors in performing a designated task as well as response time to completion. One measured how quickly subjects recognized a change in a parameter34 and the other measured how long it took for anesthesiologist to manipulate a set of abnormal parameters to a stable set.34,35 Both studies used computerized simulations of anesthesiology cases, with subjects serving as their own controls. In one study, subjects were required to identify when changes in physiologic parameters occurred using different visual formats.34 Response time and accuracy to the simulated cases was compared among a histogram, polygon, and numerical display. Subject responses were more accurate with the histogram and polygon displays (p=0.01).
In the other study, 20 anesthesiologists with an average working experience of 5 years were required to perform specific tasks on an anesthesia simulator35 (see Chapter 45). The tasks consisted of returning a set of abnormal hemodynamic parameters to normal using intravenous medications. A specific time for the completion was determined and this time was compared among 3 different visual interfaces. Trial time was significantly shorter with the traditional display (p<0.01), yet there were fewer failed trials using the other monitor displays (26% with the profilogram display, 11% with the ecological display, and 42% with the traditional display). The slower time with the non-traditional displays could have resulted from the subject's lack of experience with such screens. Nevertheless, the newer interfaces produced fewer failed attempts at arriving at the appropriate hemodynamic parameters on the simulator, suggesting that these displays might improve the clinical decision process.
None of the studies comparing traditional auditory alarms and visual monitor displays reported any adverse event associated with the newer technology. However these studies are limited by the artificial nature of the experiments.29,34,35 Anesthesiologists have many tasks to perform during anesthesia, often amidst great distraction. Attending to monitors is only one aspect of their workload. Because these laboratory experiments do not include all of the different "real world" problems and diversions that an anesthesiologist might face, it is difficult to generalize them to the workplace. Also, because this experimental task might be taken out of the context of caring for a patient in the operating room, the subject might simply focus on the completion of the experimental task and not consider other tasks that the anesthesiologist would be required to perform in a real situation.
Decreasing the Frequency of Alarms
Poorly designed device alarms can create not only problems with alarm recognition but also frequent false positive alarms. Two observational studies found that from 72 to 75% of alarms during routine general anesthesia did not require corrective action.36,37 Another study showed that only 3% of all auditory alarms during routine anesthesia monitoring represented a patient risk.38 Providers frequently must interrupt clinical tasks to silence these false positive alarms. More concerning is the fact that when alarms are unreliable, they tend to be ignored.21,39 This "cry-wolf" effect is a significant detriment to the optimal performance of alarm systems and may result in dire consequences when "true alarms" are ignored.
False alarms can be managed in two ways. Devices can be designed so that they identify and eliminate false alarms before triggering or users can manipulate alarm parameters to reduce false alarms. User manipulation can range from adjusting alarm thresholds40 to even turning the alarms off.22 There are no data describing how often operators reset alarm parameters to reduce false positive rates.
Some research has focused on the identification of alarm parameters that improve or optimize alarm accuracy (i.e., to improve the ratio of true positives to false positives—the "signal-to-noise" ratio). For example, Rheineck-Leyssius and Kalkman studied how altering an alarm parameter on a pulse oximeter would affect the incidence of hypoxemia.40 Consecutive patients admitted to the recovery room of a regional hospital in the Netherlands after general or regional anesthesia were randomized to either a lower limit of SpO2 90% or SpO2 85%. The 2 groups were comparable at baseline. The outcomes measured were hypoxemia, defined by a pulse oximeter reading less than or equal to 90% or 85%. The authors were also required to judge if they believed a signal to be artifact versus a true positive. The authors were blinded as to which group the subject was randomized to during artifact assessment and data analysis. The relative risk of having a hypoxic episode (Sp02£85%) in the group with the lower alarm limit set at 85% (as compared with those with the lower alarm limit set at 90%) was 3.10 (95% CI: 1.32-7.28, p<0.001). One weakness of this study was the lack of a bedside observer to verify the validity of the measurement, so that it is unclear to what degree measurement bias could have affected the results. The pulse oximeter was considered the "gold standard" for measuring hypoxia and thus false positives were calculated based on alarm artifact rates (outliers, loss of signal). Keeping the lower alarm limit for a pulse oximeter at 90% did reduce the number of patients with hypoxemia, however it also increased the false positive rate (33% versus 28%). A higher false positive rate on an alarm could make it more likely that an operator might disregard the alarm. The majority of alarms were transient and lasting less than 20 seconds. The authors also noted a 60% reduction in the number of triggered alarms in the Sp02 90% group by introducing a "theoretical delay" of 15 seconds between crossing the alarm threshold and actually triggering the alarm. Other investigators have documented a 26% reduction in mean alarm rate by increasing the alarm delay from 5 to 10 seconds.41
Overall, only modest evidence supports the practice of not lowering pulse oximeter lower alarms limit settings below 90%. This intervention could reduce hypoxemic events with little added cost. However, there would be an increased number of false positive alarms, which might affect attendance to the device. Fortunately, newer technological advances in oximetry appear to reduce false positives rates and may make this less of a problem. In a study in the Netherlands, a conventional pulse oximeter was compared with a "third generation" pulse oximeter equipped with a signal processing technique designed to reduce false positives.42 This "smart" pulse oximeter applied signal quality tests, differentially amplified the input signal, and applied motion-detection testing to the identified pulse. The "smart" pulse-oximeter only triggered one false positive (an alarm that did not coincide with hypoxia) and had a relative risk of 0.09 (95% CI: 0.02-0.48) for generating a false positive alarm when compared with conventional pulse oximetry with a 21-second delay.
Comment
Observational studies have suggested that current alarm systems could be improved, but future laboratory studies are needed to determine which properties of alarm systems are most effective to alert operators. These tests must be followed by field studies and ultimately with trials looking at actual patient outcomes to determine the best designs. Information concerning the cost and feasibility of implementing these changes should also be gathered.
False positive alarms remain a significant problem. Few data exist on the incidence of resetting alarm parameters or at what parameter values alarm accuracy is optimized. Advances in alarm technology aimed at reducing false positives appear a promising alternative to resetting parameters.
5600 Fishers Lane Rockville, MD 20857