Chapter 45. Simulator-Based Training and Patient Safety

Ashish K. Jha, M.D.
University of California, San Francisco School of Medicine

Bradford W. Duncan, M.D.
Stanford University School of Medicine

David W. Bates, M.D., M.Sc.
Harvard Medical School

Background

For a number of years, simulators have been used in aviation, nuclear power, military flight operations and other industries as a training tool and method to assess performance. Their use is nearly universal in high reliability organizations.¹ Recently the use of simulation in medicine has increased markedly, in part due to greater awareness of the importance of patient safety.

Defined broadly, a simulator replicates a task environment with sufficient realism to serve a desired purpose.¹ In medical training, simulators can substitute for actual patients and can be as simple as utilizing pigs' feet to practice suturing, or as complex as virtual reality machines and re-creations of actual clinical environments for surgeons, radiologists and anesthesiologists. In a general sense, they improve patient safety by allowing physicians to become better trained without putting patients at risk. For example, in a randomized controlled trial, Peugnet and colleagues used a virtual reality simulator to train physicians to perform retinal photocoagulation and found that surgeons who trained with the simulator performed the procedure as well as those who trained with patients.² Gaba lists several other advantages of simulation, among them³:

• Presentation of uncommon but critical scenarios in which a rapid response is needed (e.g., malignant hyperthermia, which occurs once in every 40,000 anesthesia cases).⁴ To conduct systematic training about managing such critical events there is little alternative but to use simulation.
• Errors can be allowed to occur and reach their conclusion—in real life a more capable clinician would have to intervene—so participants can see the results of their decisions and actions.
• With mannequin-based simulators clinicians can use actual medical equipment, exposing limitations in the human-machine interface.
• Complete interpersonal interactions with other clinical staff can be explored and training in teamwork, leadership, and communication provided. In a number of medical fields, simulation has been used in crew resource management (CRM) training¹ (see Chapter 44), where the focus is on behavioral skills such as inter-team communication during critical incidents.

Human error is a leading cause of adverse anesthesia events that lead to poor patient outcomes⁵ and may play an important role in surgical errors as well.⁶ However, it may be difficult to demonstrate improved patient outcomes from simulation because adverse events are unusual and there are an extreme number of potential confounders. Given these difficulties, performance has been used as a surrogate outcome, despite concerns that it may be a less than perfect measure. It is yet unclear which attributes of performance matter most to patient outcomes. Additionally, the methods (e.g., reliability and consistency) and timing (e.g., duration of follow-up) by which performance is measured also remain ill-defined.¹ Studies of the effectiveness of simulators often are limited in that they measure performance using the same training simulator, which may favor those who have trained on the simulator itself. In other words, seemingly improved performance may not translate to actual patient care. Of the studies that have extended laboratory simulation to patient care, few have evaluated the impact on medical error or established a clear link between simulator training and patient outcomes.

This chapter reviews the evidence regarding the use of simulators in the training and ongoing education of healthcare providers as a patient safety measure. Not included in this review is the use of simulators in the planning or preparation of diagnostic or therapeutic interventions in specific patients (e.g., calculation of radiotherapy doses using computer simulations, planning a surgical procedure for a specific patient using 3-D simulations based on the anatomical data from the actual patient). While these kinds of simulation clearly decrease the morbidity and mortality associated with their particular procedures, they are omitted here because they focus on unique characteristics of individual patients that may not be generalizable to other patients. Other uses of simulators beyond the scope of this discussion include simulators to measure worker proficiency,^7-10 to identify areas for educational intervention, and to target improvements in training.^11,12

Simulators in Anesthesia

Patient simulators have been most widely studied in anesthesia where human error may account for over 80% of critical incidents.⁵ Simulators range from simple mannequins to high-fidelity simulators that recreate the operating room experience. According to one study, 71% of medical schools in Canada, the United Kingdom and other western nations used mannequins or some other form of simulator to teach anesthesia to medical students.¹³ There is a growing body of literature on the different roles that simulators can play in anesthesia training.¹⁴

Studies have found that simulators can effectively identify errors and appropriateness of decision making in anesthesia. For example, during 19 comprehensive anesthesia simulations, DeAnda et al documented 132 unplanned incidents, of which 87 (66%) were due to human error and 32 (27%) were considered critical incidents.¹⁵ Schwid and O'Donnell have also used simulators to document the type of errors that anesthesiologists make in critical incidents, finding errors in monitor usage (37%), airway management (17%), ventilator management (13%), and drug administration (10%).¹⁶ Gaba and colleagues studied both the appropriateness of decisions and response time of anesthesia trainees to simulated critical incidents.⁸ They found great individual variability as well as variability by incident in the accuracy and timeliness of response. Some simulated incidents, such as cardiac arrest, had major errors in management a majority of the time. Based on these studies, patient simulators can be used to identify areas for further education or training of anesthesia providers.

We identified 2 studies of the effect of simulators on anesthetists' performance. Schwid and colleagues studied the impact of a computer screen-based anesthesia simulator in a randomized, controlled trial of 31 first-year anesthesia residents.¹⁷ Residents that had trained on the simulator with individualized debriefing responded better to critical events on a mannequin-based simulator than those who received standard training without the simulator. Using a randomized, controlled design, Chopra and colleagues studied management of simulated critical situations by 28 resident or staff anesthesiologists.¹⁸ The performance of subjects who trained on the simulator was superior to that of subjects who did not receive that training.

Another setting where simulators may play an important role is in crew resource management (CRM). Though establishing the effectiveness of simulation in CRM training may be difficult,¹⁹ initial work has been done on reliably and consistently rating performance.¹⁰ CRM is discussed further in Chapter 44.

Proficiency on a simulator does not ensure proficiency in clinical settings. Simulator fidelity (i.e., how accurately the simulator replicates reality) is imperfect. It is much more difficult to "re-create" a human being than to do so for, say, an airplane. This limitation is illustrated by a study conducted by Sayre and colleagues. They studied emergency medical technicians (EMT) who learned intubation techniques on anesthesia mannequins.²⁰ After successfully intubating the mannequins 10 times, they were permitted to intubate patients in the field, where their proficiency was only 53%. Other factors can inhibit optimum learning using simulation or the applicability of learning to real practice. Some participants may be more vigilant than usual during simulator sessions. Others may be unable to "suspend disbelief," may treat the simulation only as a game, or act in a cavalier fashion, knowing that the simulator is not a real patient.²¹ Refinement of simulators to make them more sophisticated and life-like may help to improve the quality of the training that simulators can provide. Appropriate construction of curricula and debriefings can also minimize the potential problems of simulation training.

Simulators in Radiology

The number of radiologic examinations that require sedation, analgesia, or contrast media has increased rapidly in recent years.²² Despite their rarity, serious medication reactions do occur and require prompt, appropriate management. Some evidence of suboptimal management²³ has prompted the creation of computer-based simulators to improve training in these areas.²⁴ Simulators have also been used to measure the effectiveness of strategies to teach trainees about critical incidents, but studies have not reported the effectiveness of simulators as adjuncts to training. Sica and colleagues developed a computer-based simulator that was used to study the effectiveness of a lecture and videotape-based intervention on critical incidents for radiology housestaff.²⁵ Those residents who underwent the intervention scored better on the simulator than those that had only received basic, standard training. The authors concluded that the simulator was an effective way of assessing the utility of the educational course.

Simulators in Surgery

As surgical technique and expertise has changed drastically over recent decades, some methods used to train surgeons have evolved as well. Simulators in the surgical setting are aimed at improving surgeons' technical skills and dexterity. Training on simulators and virtual reality machines, though still in its nascent stages, is becoming increasingly accepted.^26,27 Surgical simulators have been developed for a variety of procedures: endovascular repair of abdominal aortic aneurysms,²⁸ sinus surgery^29,30 gynecologic surgery,³¹ orthopedic surgery,³² prostatic surgery,³³ amniocentesis procedures,³⁴ and oral surgery.³⁵ Nonetheless, many of these have yet to be formally evaluated in terms of efficacy in improving physician performance in patient care.

We identified several studies that evaluated physician performance after training on a surgical simulator. Derossis evaluated surgical residents and attendings in a randomized study and found that those trained with a simulator had greater proficiency in suturing, transferring, and mesh placement, when tested on the simulator, than did the control group.³⁶ They subsequently found that when tested in vivo in pigs, surgeons (both attendings and residents) who had been randomized to the simulator arm were more proficient at the same skills.³⁷ Scott and colleagues studied the impact of a video-trainer on laparoscopic cholecystectomy skills.³⁸ They randomized surgical residents to training on a video-trainer versus no formal training over a 30-day period. At the end of the period, when tested on pre-specified tasks on the video trainer, those who trained on the video trainer did uniformly better.

Intuitively, improved technical skills should lead to fewer complications during surgery. Wallwiener and colleagues developed a surgical trainer that, when used in conjunction with other improvements in their training program, led to lower rates of hysteroscopic complications.^39,40 However, for most simulators the link between improvements in technical skills and dexterity from simulator training and prevention of adverse events has yet to be established and deserves formal investigation. Further, problem-based surgical simulation (e.g., avoiding inadvertent ligation of the ureter during hysterectomy) may improve patient safety not only by improving skills, but also by training surgeons to better anticipate and avoid complications and to manage them should they occur.

Simulators in Gastroenterology

Simulators have been developed to train physicians in the technical skills required in endoscopy.^41-43 We identified one study that evaluated the effect of simulator training on physician performance. In a small randomized controlled trial enrolling 10 residents, Tuggy and colleagues found residents who trained for flexible sigmoidoscopy using a virtual reality simulator were faster, visualized a greater portion of the colon, and made fewer directional errors in actual patients.⁴⁴

Simulators in Cardiology

Cardiology training has long used a variety of simulators from audiocassettes of heart tones to full patient simulators. Two simulators have been evaluated. Champagne and colleagues demonstrated that a heart sound simulator could increase medical students' recognition of pathologic heart sounds.⁴⁵ Ewy and colleagues studied the efficacy of "Harvey," a cardiology patient simulator.⁴⁶ In a study enrolling 208 senior medical students at 5 medical schools, participants were randomized to receive training on Harvey versus a standard cardiology curriculum during their cardiology elective. Students who had been trained with Harvey performed skills better both when tested with the simulator and when tested with real patients. Some physicians have expressed concern that training on simulators may decrease professionalism. In this study, there was no difference in the way patients perceived the professionalism of the students trained on Harvey compared with students who received standard training.

Systems that simulate the cardiovascular anatomy and physiology have also been developed. Swanson and colleagues created a cardiovascular simulator to train physicians on the workings of mechanical valves and balloon assist devices, and to recognized diseased vessels.⁴⁷ As cardiac procedures have become more invasive, simulators to train cardiologist in these procedures have become more common,^{48, 49} but their effect on resident performance has not been evaluated formally.

Comment

Although simulators have been used for many years in a variety of settings, data on their efficacy are still emerging. While there is currently no evidence that simulation-based training leads to improved patient outcome, it may prove difficult to conduct such studies. These would require large cohorts of patients to be followed during and after care by clinicians who were randomized to have undergone different cumulative amounts of simulation training. Because adverse events are uncommon, and there are a large number of patient-based and system-based factors that contribute to negative outcomes, any such study would have to be massive and prolonged. Instead, provider performance, with its known limitations, has been and will continue to be used as a surrogate outcome. Nonetheless, as Gaba has asserted "...no industry in which human lives depend on skilled performance has waited for unequivocal proof of the benefits of simulation before embracing it."⁵⁰ Certain benefits are clear. In training for procedures, simulators have high face validity because they ease trainees' transition to actual patients, which seems inherently beneficial as a means to avoid adverse events. Further, procedural success is related to the experience of the operator, known as the volume-outcome relationship^{51, 52} (see Chapter 19). As simulators become more advanced, they may be reasonable substitutes to improve proficiency of both trainees and low volume physician operators. This increase in proficiency may have an important impact in patient outcomes. Future studies of the link between simulator-based training and performance on actual patients will improve our ability to better assess the appropriate role of simulators in training and patient safety.

The costs of simulators vary widely and need to be considered. "Home-made" or simple trainers are far less expensive than complex simulators or full-scale simulation centers. The average cost of high-fidelity patient simulators is on the order of $200,000. Medium fidelity simulators may be as little as $25,000. Establishing a dedicated simulation center can cost up to $1,000,000 (including the simulator) depending on the amount of space, the type of clinical equipment to be used, the extent of renovations needed, and the sophistication of the audio-visual equipment. However, such capital costs are amortized over a long period of time, and such centers typically are used for a wide variety of training curricula for diverse target populations. Further, for most simulation training the dominant cost is that of instructor time. Another indirect cost is that of removing clinical personnel from revenue producing work to undergo training. The healthcare industry currently does not fully embed time or costs of training into the system, but instead often leaves these costs for the individual clinicians to bear.

There are potential risks to simulation-based training. Where the simulator cannot properly replicate the tasks or task environment of caring for patients, there is a risk that clinicians might acquire inappropriate behaviors (negative training) or develop a false sense of security in their skills that could theoretically lead to harm. Although there are no data to suggest that this currently happens, such risks will have to be weighed and evaluated as simulators become more commonly used.

In summary, although there is currently little evidence that simulation training improves patient care, the experience with simulation in other industries and the high face validity of their applications in healthcare has led many institutions to adopt the technology. It is likely that simulators will continue to be used and their role in training of medical personnel will grow. Definitive experiments to improve our understanding of their effects on training will allow them to be used more intelligently to improve provider performance, reduce errors and ultimately, promote patient safety. Although such experiments will be difficult and costly, they may be justified to determine how this technology can best be applied.

References

1. Bushell E, Gaba, D.M. Anesthesia Simulation and Patient Safety. Problems in Anesthesia 2001;In Press.

2. Peugnet F, Dubois P, Rouland JF. Virtual reality versus conventional training in retinal photocoagulation: a first clinical assessment. Comput Aided Surg 1998;3:20-26.

3. Gaba DM, DeAnda A. A comprehensive anesthesia simulation environment: re-creating the operating room for research and training. Anesthesiology. 1988;69:387-394.

4. Dunn D. Malignant hyperthermia. AORN J 1997;65:728,731,passim.

5. Cooper JB, Newbower RS, Long CD, McPeek B. Preventable anesthesia mishaps: a study of human factors. Anesthesiology 1978;49:399-406.

6. Spencer FC. Human error in hospitals and industrial accidents: current concepts. J Am Coll Surg 2000;191:410-418.

7. Keyser EJ, Derossis AM, Antoniuk M, Sigman HH, Fried GM. A simplified simulator for the training and evaluation of laparoscopic skills. Surg Endosc 2000;14:149-153.

8. Gaba DM, DeAnda A. The response of anesthesia trainees to simulated critical incidents. Anesth Analg 1989;68:444-451.

9. Gaba DM. Anaesthesia simulators. Can J Anaesth 1995;42(10):952-953.

10. Gaba DM, Howard SK, Flanagan B, Smith BE, Fish KJ, Botney R. Assessment of clinical performance during simulated crises using both technical and behavioral ratings. Anesthesiology 1998;89:8-18.

11. Ruppert M, Reith MW, Widmann JH, et al. Checking for breathing: evaluation of the diagnostic capability of emergency medical services personnel, physicians, medical students, and medical laypersons. Ann Emerg Med 1999;34:720-729.

12. Dorfsman ML, Menegazzi JJ, Wadas RJ, Auble TE. Two-thumb vs. two-finger chest compression in an infant model of prolonged cardiopulmonary resuscitation. Acad Emerg Med 2000;7(10):1077-1082.

13. Cheung V, Critchley LA, Hazlett C, Wong EL, Oh TE. A survey of undergraduate teaching in anaesthesia. Anaesthesia 1999;54:4-12.

14. King PH, Blanks ST, Rummel DM, Patterson D. Simulator training in anesthesiology: an answer? Biomed Instrum Technol 1996;30:341-345.

15. DeAnda A, Gaba DM. Unplanned incidents during comprehensive anesthesia simulation. Anesth Analg 1990;71:77-82.

16. Schwid HA, O'Donnell D. Anesthesiologists' management of simulated critical incidents. Anesthesiology 1992;76:495-501.

17. Schwid HA, Rooke GA, Michalowski P, Ross BK. Screen-based anesthesia simulation with debriefing improves performance in a mannequin-based anesthesia simulator. Teach Learn Med 2001;13(2):92-96.

18. Chopra V, Gesink BJ, de Jong J, Bovill JG, Spierdijk J, Brand R. Does training on an anaesthesia simulator lead to improvement in performance? Br J Anaesth 1994;73:293-297.

19. Gaba DM, Howard,S.K., Fish,K.J., Smith,B.E., Sowb,Y.A. Simulation-based training in anesthesia crisis resource management (ACRM): a decade of experience. Simulation and Gaming 2001;32(2):175-193.

20. Sayre MR, Sakles JC, Mistler AF, Evans JL, Kramer AT, Pancioli AM. Field trial of endotracheal intubation by basic EMTs. Ann Emerg Med 1998;31:228-233.

21. New Technologies in anesthesia practice: anesthesia simulators [Audiocassette]. Kansas City, MO: Audio-Digest Anesthesiology; 1998.

22. Chait P. Future directions in interventional pediatric radiology. Pediatr Clin North Am 1997;44(3):763-782.

23. Sadler DJ, Parrish F, Coulthard A. Intravenous contrast media reactions: how do radiologists react? Clin Radiol 1994;49:879-882.

24. Medina LS, Racadio JM, Schwid HA. Computers in radiology. The sedation, analgesia, and contrast media computerized simulator: a new approach to train and evaluate radiologists' responses to critical incidents. Pediatr Radiol. 2000;30:299-305.

25. Sica GT, Barron DM, Blum R, Frenna TH, Raemer DB. Computerized realistic simulation: a teaching module for crisis management in radiology. AJR Am J Roentgenol 1999;172(2):301-304.

26 . Carrico CJ, Satava,R.M. Advanced simulation technologies for surgical education. Bull Am Coll Surg 1996;81:77.

27. Ota D, Loftin B, Saito T, Lea R, Keller J. Virtual reality in surgical education. Comput Biol Med 1995;25:127-137.

28. Chong CK, How TV, Black RA, Shortland AP, Harris PL. Development of a simulator for endovascular repair of abdominal aortic aneurysms. Ann Biomed Eng 1998;26:798-802.

29. Rudman DT, Stredney D, Sessanna D, et al. Functional endoscopic sinus surgery training simulator. Laryngoscope 1998;108(11 Pt 1):1643-1647.

30. Ecke U, Klimek L, Muller W, Ziegler R, Mann W. Virtual reality: preparation and execution of sinus surgery. Comput Aided Surg 1998;3:45-50.

31. Szekely G, Bajka M, Brechbuhler C, et al. Virtual reality based surgery simulation for endoscopic gynaecology. Stud Health Technol Inform 1999;62:351-357.

32. Poss R, Mabrey JD, Gillogly SD, et al. Development of a virtual reality arthroscopic knee simulator. J Bone Joint Surg Am 2000;82-A(10):1495-1499.

33. Ballaro A, Briggs T, Garcia-Montes F, MacDonald D, Emberton M, Mundy AR. A computer generated interactive transurethral prostatic resection simulator. J Urol 1999;162:1633-1635.

34. Maher JE, Kleinman GE, Lile W, Tolaymat L, Steele D, Bernard J. The construction and utility of an amniocentesis trainer. Am J Obstet Gynecol 1998;179:1225-1227.

35. Bettega G, Payan Y, Mollard B, Boyer A, Raphael B, Lavallee S. A simulator for maxillofacial surgery integrating 3D cephalometry and orthodontia. Comput Aided Surg 2000;5:156-165.

36. Derossis AM, Bothwell J, Sigman HH, Fried GM. The effect of practice on performance in a laparoscopic simulator. Surg Endosc 1998;12:1117-1120.

37. Fried GM, Derossis AM, Bothwell J, Sigman HH. Comparison of laparoscopic performance in vivo with performance measured in a laparoscopic simulator. Surg Endosc 1999;13:1077-1081; discussion 1082.

38. Scott DJ, Bergen PC, Rege RV, et al. Laparoscopic training on bench models: better and more cost effective than operating room experience? J Am Coll Surg 2000;191:272-283.

39. Wallwiener D, Rimbach S, Bastert G. The HysteroTrainer, a simulator for diagnostic and operative hysteroscopy. J Am Assoc Gynecol Laparosc 1994;2:61-63.

40. Wallwiener D, Rimbach S, Aydeniz B, Pollmann D, Bastert G. Operative Hysteroscopy: Results, Security Aspects, In Vitro Simulation Training (Hysterotrainer). J Am Assoc Gynecol Laparosc 1994;1(4, Part 2):S39.

41. Williams CB, Saunders BP, Bladen JS. Development of colonoscopy teaching simulation. Endoscopy 2000;32:901-905.

42. Aabakken L, Adamsen S, Kruse A. Performance of a colonoscopy simulator: experience from a hands-on endoscopy course. Endoscopy 2000;32:911-913.

43. Neumann M, Mayer G, Ell C, et al. The Erlangen Endo-Trainer: life-like simulation for diagnostic and interventional endoscopic retrograde cholangiography. Endoscopy 2000;32(11):906-910.

44. Tuggy ML. Virtual reality flexible sigmoidoscopy simulator training: impact on resident performance. J Am Board Fam Pract 1998;11:426-433.

45. Champagne MT, Harrell JS, Friedman BJ. Use of a heart sound simulator in teaching cardiac auscultation. Focus Crit Care 1989;16:448-456.

46. Ewy GA, Felner JM, Juul D, Mayer JW, Sajid AW, Waugh RA. Test of a cardiology patient simulator with students in fourth-year electives. J Med Educ 1987;62:738-743.

47. Swanson WM, Clark RE. A simple cardiovascular system simulator: design and performance. J Bioeng 1977;1:135-145.

48. Wang Y, Chui C, Lim H, Cai Y, Mak K. Real-time interactive simulator for percutaneous coronary revascularization procedures. Comput Aided Surg 1998;3:211-227.

49. Dawson SL, Cotin S, Meglan D, Shaffer DW, Ferrell MA. Designing a computer-based simulator for interventional cardiology training. Catheter Cardiovasc Interv 2000;51:522-527.

50. Gaba DM. Improving anesthesiologists' performance by simulating reality. Anesthesiology 1992;76:491-494.

51. Jollis JG, Romano PS. Volume-outcome relationship in acute myocardial infarction: the balloon and the needle. JAMA 2000;284:3169-3171.

52. Jollis JG, Peterson ED, Nelson CL, et al. Relationship between physician and hospital coronary angioplasty volume and outcome in elderly patients. Circulation 1997;95:2485-2491.

Return to Contents
Proceed to Next Chapter