Slide Presentation from the AHRQ 2009 Annual Conference
On September 16, 2009, Jean Loes made this presentation at the 2009 Annual Conference. Select to access the PowerPoint® presentation (2.39 MB) (Plugin Software Help).
Staff Perception Survey before and after EHR/CPOE Implementation
Marcia Ward, Douglas Wakefield, John O'Brien
- Among the most notable challenges to implementing clinical information systems are the varying levels of acceptance and use by healthcare providers and employees:
- Research has shown that experiences shape the degree to which users will accept the technology initially (Dixon, 1999; Herbert & Benbasat, 1994).
- Research has shown that users' attitudes regarding risks to service quality and disruptions in workflow hinder implementation (Hu, et al., 2002; Zheng, et al., 2005).
- All of the significant models of information technology use suggest that perceptions of the impact on work and outcomes are significant determinants of technology use and adoption (Kufafka, et al., 2003).
Background on Measures
- Measures have been created to explore attitudes toward technology including perceived usefulness and ease of use (Davis, 1989).
- Recently we developed a measure of healthcare worker perceptions of the effect of clinical information systems on workflow processes and outcomes (Wakefield et al., 2007).
- This Information Systems Expectations and Experiences (I-SEE) survey was administered before and after "Go-Live" of a comprehensive system change in several hospitals.
- Analysis of the survey factor structure identified scales that assessed respondents' perceptions related to communication changes, changes in selected work behaviors, perceptions of the implementation strategy, and the impact on quality of patient care (Wakefield et al., 2007).
- The purpose of this study is to examine changes in perceptions about quality before and after implementation of a comprehensive clinical information system.
- This study is the first to explore these changes in multiple small hospitals and across various employee categories.
- The I-SEE survey instrument was modified. Respondents were asked to indicate their current perceptions at each wave.
- The survey was administered at three times:
- Wave 1 - March 2007 - before any changes
- Wave 2 - March 2008 - during training and preparation
- Wave 3 - March 2009 - after implementation
- Implementation "Go-Live" occurred:
- July 2008 for 3 hospitals
- September 2008 for 4 hospitals
Survey Responses Number and (Response Rate)
|Ellsworth Municipal Hospital
|Franklin General County Hospital
|Hancock County Memorial Hospital
|Kossuth Regional Health Center
|Mercy Medical Center - New Hampton
|Mitchell County Regional Medical Center
|Palo Alto County Health Systems
|Total Returned Surveys
Survey Items with an Average Response of "Strongly Agree"
The table describes the highest rated (6-point Likert scale: strongly disagree to strongly agree) items in descending order.
||Overall Mean Response
|31. Patient care is consistently given according to the "9 Rights"
|25. I enjoy my job
|14. Overall patient care is safe in the areas I work
|1. Access to information to make good patient care decisions is available
|18. I get a great deal of professional satisfaction from my job
|29. Communications ensuring high quality and safe patient care routinely occur when patients are transferred to other facilities
Items that Changed across Waves
The line graph illustrates the items that indicated significant changes across the three waves of surveys.
Participants' response to question 15 (Staff are alerted to potential patient care errors before they occur) indicate a decline in favorable perception over the three waves.
Participants' response to question 22 (Patient related clinical data are available to decision makers in a timely manner) indicate no notable difference in perception between Wave 1 and 2, but an increase at Wave 3 in favorable perception.
Participants' response to question 3 (I can quickly access information that I need to share with patients and families) indicate an increase in favorable perception over the three waves.
Participants' response to question 26 (Patient care orders are consistently legible and clear) indicate a slight decline in perception between Wave 1 and 2, and a substantial increase at Wave 3.
Participants' response to question 8 (Too many verbal orders are made on my unit) indicate no notable difference in perception between Wave 1 and 2, but an increase at Wave 3 in favorable perception.
Number of Respondents in Each Staff Group
(physicians and midlevel providers)
(professional, technical, patient support)
(clerical, senior management, marketing)
Comparison across Staff Groups
- To explore whether the staff groups differed, we compared their responses across the three waves.
- The Other - NonClinical group was excluded because they had minimal if any contact with the clinical information system and were not directly involved in patient care.
- Significant interactions were found for three survey items shown on the next slides. The pattern from before to after implementation was:
- Other - Clinical respondents showed no change or increases in survey responses after implementation
- Registered nurses showed no change in survey responses after implementation
- Providers showed sizable decreases in survey responses after implementation.
Significant Differences among Staff Groups
The line graph illustrates the change in response to question 10 (I spend about the right amount of time recording diagnoses and symptoms) between three staff groups across the three waves. The providers show the most change in perception and are the only group to decrease after implementation. The Registered Nurses show little change while the Other-Clinical group shows an increase after implementation.
Significant Differences among Staff Groups
The line graph illustrates the change in response to question 11 (I spend about the right amount of time preparing discharge documents) between three staff groups across the three waves. The providers show the most change in perception and are the only group to decrease after implementations. Prior to implementation, the Registered Nurses show a slight increase while the Other-Clinical group shows a slight decrease.
Significant Differences among Staff Groups
The line graph illustrates the change in response to question 19 (The work processes I commonly use are efficient) between three staff groups across the three waves. The providers show the most change in perception and are the only group to decrease after implementation. The Registered Nurses show a slight decline and the Other-Clinical group shows a slight increase across waves.
Comparison between Physicians and Mid-level Providers
- To further explore whether subsets of the provider group differed, we compared their responses across the three waves.
- Physicians included 11 supervisory and 13 nonsupervisory physicians.
- Mid-level Providers included 22 nurse practitioners, physicians assistants, CRNAs, etc.
- Significant interactions were found for four survey items. As shown on the next slide, the pattern from before to after implementation was:
- Mid-level Providers showed increases in survey responses after implementation to two survey items
- Mid-level Providers showed no change in survey responses after implementation to two survey items
- Physicians showed increases in survey responses after implementation to two survey items
- Physicians showed decreases in survey responses after implementation to two survey items.
Significant Differences between Physicians and Mid-Level Providers
The line graphs illustrates the change in physicians' and mid-level providers' response across the three waves. Significant differences were found between the two groups for questions 3, 4, 14, and 22. In each case the physicians had a lower rating than the mid-level providers.
CPOE Rates after Implementation
The line graph illustrates computerized provider order entry (CPOE) use rate within each of the seven hospitals over a nine month period, from Aug. 2008 to June 2009. The average CPOE use rate is about 60% and if fairly consistent across the nine-month period.
Relationship between Survey Items at Wave 3 and CPOE Use Rates
- Correlations between survey items at Wave 3 and CPOE use rates indicate that staff at hospitals with higher CPOE use rates also tended to respond more in agreement to three survey items:
- "Patients are rarely asked the same questions by the staff" (r=.93)
- "Access to information to make good patient care decisions is available" (r=.79)
- "I get a great deal of professional satisfaction from my job" (r=.73)
- The strongest relationship indicated that staff at hospitals with higher CPOE use rates tended to strongly disagree with the survey item "Too many verbal orders are made on my unit" (r= -.96)
Relationship between Survey Items at Wave 2 and CPOE Use Rates
- Correlations between survey items at Wave 2 and CPOE use rates indicate that staff at hospitals with higher CPOE use rates tended to respond more in agreement to three survey items before Go-Live:
- "I spend about the right amount of time documenting patient care" (r=.69)
- "Patients are rarely asked the same questions by the staff" (r=.63)
- "Overall patient care is safe in the areas I work" (r=.59)
- Thus, higher agreement with these items may predict CPOE use rates after implementation.
Davis, F. D. (1989). Perceived usefulness, perceived ease of use, and use acceptance of information technology. MIS Quarterly, 13 (3), 318-340.
Dixon, D. R. (1999). The behavioral side of information technology. International Journal of Medical Informatics, 56, 117-123.
Halbesleben JRB, Wakefield DS, Ward MM, Brokel J, Crandall D. (2009). The relationship between super users' attitudes and employee experiences with clinical information systems. Medical Care Research & Review, 66: 82-96.
Hebert, M., & Benbasat, I. (1994). Adopting information technology in hospitals: the relationship between attitudes/expectations and behavior. Hosp Health Serv Adm, 39 (3), 369-383.
Hu, P. J., Chau, P. Y. K., & Sheng, O. R. L. (2002). Adoption of telemedicine technology by health care organizations: An exploratory study. Journal of Organizational Computing and Electronic Commerce, 12 (3), 197-221.
Kukafka, R., Johnson, S. B., Linfante, A., & Allegrante, J. P. (2003). Grounding a new information technology implementation framework in behavioral science: A systematic analysis of the literature on IT use. Journal of Biomedical Informatics, 36, 218-227.
Wakefield DS, Halbesleben JRB, Ward MM, Qiu Q, Brokel J, Crandall D. (2007) Development of a measure of clinical information systems expectations and experiences. Medical Care, 45: 884-890.
Zheng, K., Padman, R., Johnson, M. P., & Diamond, M. S. (2005). Understanding technology adoption in clinical care: clinician adoption behavior of a point-of-care reminder system. International Journal of Medical Informatics. 74(7-8), 535-543.
- This work was supported by funding from AHRQ - 5UC1HS016156 - "EHR Implementation for the Continuum of Care in Rural Iowa"
- Mercy Medical Center - North Iowa
- Mercy North Iowa Network
- The University of Iowa - Center for Health Policy and Research
- The University of Missouri - Center for Health Care Quality
Current as of December 2009
Staff Perception Survey before and after EHR/CPOE Implementation. Slide Presentation from the AHRQ 2009 Annual Conference (Text Version). December 2009.
Agency for Healthcare Research and Quality, Rockville, MD. http://www.ahrq.gov/about/annualconf09/Loes.htm