Evidence Report/Technology Assessment: Number 201
The Agency for Healthcare Research and Quality (AHRQ), through its Evidence-based Practice Centers (EPCs), sponsors the development of evidence reports and technology assessments to assist public- and private-sector organizations in their efforts to improve the quality of health care in the United States. The reports and assessments provide organizations with comprehensive, science-based information on common, costly medical conditions and new health care technologies. The EPCs systematically review the relevant scientific literature on topics assigned to them by AHRQ and conduct additional analyses when appropriate prior to developing their reports and assessments.
AHRQ expects that the EPC evidence reports and technology assessments will inform individual health plans, providers, and purchasers as well as the health care system as a whole by providing important information to help improve health care quality.
Medication management is a continuum that covers all aspects of prescription medications. Medication management includes prescribing and ordering, order communication (or order transmission) between prescribers and pharmacists, dispensing, administering, and monitoring, as well as reconciliation, adherence, and education.1 Medication management is complex and costly and enhances the health and well-being of more than half of the population in the developing world. Health information technology (health IT) holds great promise to improve the quality of health care and reduce potential and real errors in medication management while at the same time providing cost-effective care. The Agency for Healthcare Research and Quality (AHRQ) is committed to summarizing and providing the evidence base for health IT. It has produced evidence summaries on health IT related to costs and benefits;2 barriers and drivers of health IT for the elderly, chronically ill, and underserved;3 the impact of consumer informatics applications;4 and telemedicine.5 AHRQ also has contracted for evidence summaries on the use of health IT in decisionmaking,6 patient-centered care,7 and decision support for health care decisionmaking.8 The contracted reports will be available through http://www.healthit.ahrq.gov in mid-2011. Although these reports often mention medication management, the body of published evidence on all aspects of the medication management process and how it is affected by multiple health IT systems has not been consolidated. A single document is needed to summarize the evidence evaluating the effects of health IT on the medication management process across providers, settings, patients, and research methods.
The objectives of this report are to:
- Review the literature on the effects of health IT on medication management.
- Synthesize available evidence regarding the effectiveness and effects of health IT in all phases of medication management as well as reconciliation and education.
- Identify gaps in the literature.
- Make recommendations for future research.
For the purposes of this review, medication management includes the processes that encompass the five phases of the medication process (i.e., prescribing and ordering, order communication, dispensing, administering, and monitoring) across groups of health professionals, patients, and their informal caregivers, and two aspects of quality with respect to medication management across the five phases of medication management (medication reconciliation and education, both postprofessional education of training and patient education related to medication management). Medication management can also include procurement, storage, and reporting from the first assessment of patients to determine their need for drugs through to optimal care and monitoring after the drugs are prescribed. The organization of the information in this report is based on the Bell framework of the five phases across the continuum of medication management and reconciliation and education.1
To address the goals of this report, we further define medication management health IT (MMIT) applications as electronic systems that (1) collect, process, or exchange health information about patients; (2) are integrated with existing health IT systems such as electronic health records or electronic medical record (EMR) systems; and (3) provide advice or suggestions to either the health care provider or the patients and their families on issues or decisions related to medication management. We recognize that functional elements of the MMIT will vary across particular implementation approaches within a given phase of medication management. Many of the MMIT applications we found were designed to encompass more than one phase of medication management. The sophistication of the systems, degree of integration of the health IT into workflow systems, and the broad range of settings in which a particular health IT is implemented and used are also complex and varied. Many health professionals, support staff, patients, and patients' families were involved in medication management in the studies assessed.
The evidence assessing MMIT is large, diffuse, and published across many disciplines. People who can benefit from the knowledge in this report include health professionals, researchers, administrators, and other decisionmakers and those who develop and implement health IT applications. This report is timely because of the Federal emphasis on the use of health IT to improve health care while at the same time making health and wellness care more cost effective and safer. Seven questions structure this evidence report. Within reporting related to the questions, sections are based on phases of medication management. Reporting is done to address the multiple settings where medication management is important, the range of health care providers who deliver and support care using medications, and classes of medications, specific drugs, or a broad spectrum of medications.
Key Questions (KQs)
Within all phases of the medication management continuum, what evidence exists that health IT applications are effective in improving:
- Health care processes?
- Other intermediate outcomes (e.g., satisfaction with system, usability, knowledge, skills, and attitude)?
- Costs and economic outcomes?
- Clinical outcomes for patients?
- Population level outcomes?
- Composite outcomes?
- To what extent does the impact of health IT on improving health care processes, other outcomes, costs and economics, and clinical outcomes vary depending on the type of medication (controlled or noncontrolled substance) or the form of the medication (e.g., oral, injection, intravenous)?
KQ2: Gaps in Knowledge or Evidence
What knowledge or evidence deficits exist to support estimates of cost, benefit, impact, and net value with regard to health IT applications in all phases of medication management?
KQ3: Value Proposition for Implementers and Users
What critical information regarding the impact of health IT applications implemented to support the phases of medication management is needed to give clinicians, health care facility administrators, patients, and their families a clear understanding of the value proposition particular to them?
KQ4: System Characteristics
What evidence supports or refutes the impact of any of: open source, homegrown, proprietary, local configuration ability, system configuration ability, conformity with standards being Certification Commission for Healthcare Information Technology (CCHIT) certified, system architecture, or feature set on the decision to purchase, implement, or use health IT in medication management systems?
What factors influence sustainability of health IT applications that support a phase of the medication management continuum?
- What evidence exists to demonstrate that health care settings (ambulatory, long-term care, etc.) influence implementation, use, and effectiveness of such health IT applications?
- What is the impact (challenges, merits, costs, and benefits) of having electronic access to patient data on the quality and safety of care provided by health IT applications that support at least one phase of the continuum of medication management?
KQ6: Two-Way Prescription Electronic Data Interchange (EDI)
In a two-way electronic data interchange (EDI) between the prescribers and pharmacists:
- What evidence exists demonstrating the barriers and drivers of implementation of complete EDI that can support the prescription, transmittal and receipt, and perfection process of e-Prescriptions?
- How do barriers, facilitators, and economic incentives vary across pharmacists, physicians, and other relevant stakeholders with respect to adoption and use of complete EDI (e-Prescribing/ordering with e-Transmission)?
KQ7: Randomized Controlled Trials (RCTs) of Clinical Decision Support Systems (CDSS)
What evidence exists regarding the extent of integration of electronic clinical decision support in a health IT system for the prescribing, dispensing, and administering of medications, and to what extent does the use of clinical decision support systems impact the various outcomes (e.g., health care process, intermediate, cost and economics, and clinical) of interest?
We anticipated finding few RCTs across all phases of medication management and MMIT applications. Studies that employ other research methods can also provide valuable evidence for understanding MMIT applications. We therefore included studies employing a range of research methodologies. We restricted our analysis to hypothesis-driven studies with group comparisons and appropriate statistical analysis in addition to qualitative studies with explicit methods for KQ1: Effectiveness. The only methodological limit was for assessment of the effect of CDSSs on prescribing, for which sufficient RCTs were available to provide evidence for synthesis.
Through consultation with our internal team and AHRQ, we determined that the answers to KQ2: Gaps in Knowledge or Evidence and KQ3: Value Proposition for Implementers and Users would become evident from our review of the evidence in KQ1: Effectiveness. We supplemented these articles with other studies addressing values propositions by stakeholders. KQ4: System Characteristics addresses the impact of MMIT application features on the likelihood that the systems will be purchased, implemented, and used. The evidence for this question comes from studies of all designs that measure implementation, use, and purchasing decisions. KQ5: Sustainability addresses the factors influencing the sustainability of MMIT applications, specifically the impact of the setting and access to other electronic data within integrated systems on health care quality and safety. To identify articles that addressed this question, the team, in consultation with AHRQ, used the definition of sustainability by Humphreys et al.,9 which restricted our choice of articles to only a few. Their definition of sustainability was the ability of a health service to provide ongoing access to appropriate quality care in a cost- and health-effective manner. KQ6: Two-Way Prescription EDI relates to the barriers and facilitators to complete EDI between prescribers and pharmacies during the time between prescription writing and dispensing and how these vary across stakeholders. The best evidence available for KQ6 is found in articles studying EDI between prescribers and pharmacies that include original data (qualitative or quantitative). Because insufficient evidence was found on two-way EDI, we included one-way EDI as well. KQ7: RCTs of CDSS addresses the extent to which CDSS systems are integrated into health IT systems for medication management and the impact on outcomes as described in KQ1: Effectiveness. As a team we felt that adequate evidence was available to address this issue so that we could limit our scope to RCTs.
Given the broad range of questions and outcomes addressed, we searched peer-reviewed electronic databases by first using textwords relating to the various types of health IT applied to medication management (Appendix A of the full report). These searches were then combined with a search using subject headings related to the five medication management phases plus reconciliation and education as well as specific health IT application terms (e.g., CDSS). We combined these medication management terms with computer and technology terms. When possible, we excluded letters, editorials, commentaries, and animal studies. Because our interest was in all study designs, we did not limit based on methodology. We also put no limits on language or time to capture the global literature and early studies.
Databases searched included MEDLINE®, Embase, CINAHL (Cumulated Index to Nursing and Allied Health Literature), Cochrane Database of Systematic Reviews, International Pharmaceutical Abstracts, Compendex, Inspec (which includes IEEE Xplore), Library and Information Science Abstracts, E-Prints in Library and Information Science, PsycINFO, Sociological Abstracts, and Business Source Complete. We also looked for eligible studies by reviewing grey literature sites, performing hand searches of pertinent reviews, querying our experts, and by reviewing the AHRQ National Resource Center for Health IT Knowledge Library resources (available at: http://healthit.ahrq.gov/portal/server.pt/community/knowledge_library/653).
The search results were downloaded into Reference Manager® version 10 (ISI ResearchSoft) and uploaded into a customized systematic review management system (Health Information Research Unit, McMaster University).
Studies were eligible for inclusion if they used health IT in any aspect of the medication management process. We included articles on MMIT only if the system was integrated with at least one existing health IT system and if they processed patient-specific information and provided advice or suggestions. A critical inclusion requirement was the integration of information.
Personal digital assistants (PDAs), which integrated patient-specific information provided by either the clinicians or the patients, were analyzed to assist in medication management decisions (by request of AHRQ). This exception is made because PDAs and hand-held devices are considered an important, and perhaps unique, means of improving health care quality in relation to medications. The use of PDAs to manage medications is especially important for clinicians and patients who are in settings that do not have large, sophisticated, and integrated information systems. Other stand-alone devices with no integration of information with another health IT were excluded. Articles on all five phases of the management process plus medication reconciliation and postprofessional education related to MMIT were included. Once we tagged the articles for content, we assessed whether those that passed our inclusion criteria were pertinent to specific key questions. Many articles were analyzed in several phases of medication management and sections of the report.
Studies were classified as being observational, case-control, cohort, or RCTs. The quality of included studies was assessed using the same criteria employed by Jimison et al. in their AHRQ report.3 RCT scoring was based on the Delphi consensus work done by Verhagan and colleagues.10 This scale is referred to in this report as the Verhagen/AHRQ RCT quality scale. Observational studies with before�after, time series, surveys, or qualitative methods were not assessed for quality because few well-validated instruments exist. Bibliographies of systematic and narrative reviews were examined to identify studies, and select reviews were integrated into sections of the report.
Data were abstracted from relevant articles and tagged for applicability to the various key questions. Given the range of questions addressed, data abstraction was performed by a core group of staff and entered into online data abstraction forms. One reviewer did the abstraction, and a second, senior reviewer checked its accuracy. The authors of this report performed a final check on the abstracted data. The reviewers were not blinded to the identity of the article authors, institutions, or journal. Data abstraction was difficult in many instances because of the lack of accepted definitions and absence of important features of the study or MMIT application. For example, we identified problems with the differences between computerized provider order entry (CPOE) for ordering and e-Prescribing systems. Definitions for medication errors and related terms were often inconsistently used. To make data abstraction easier, we established working definitions, which can be found in Appendix F of the full report.
Meta-analysis was not performed on any data because of the heterogeneity of the studies in terms of interventions, populations, technologies used, and outcomes measured, as well as the presence of mostly descriptive and observational studies.
Throughout the project, the core team sought feedback from the internal advisors, our Task Officer from AHRQ, and the Technical Expert Panel.
Our literature search retrieved 40,582 articles. After duplicates were removed, 32,785 articles were screened at title and abstract stage. From a full-text screen of 4,578 articles, we identified 789 articles that were eligible for inclusion in this report. Of these articles, 361 met only our inclusion criteria for content and did not have group comparisons, hypothesis testing, or appropriate analysis. These are listed in the bibliography of the report. Across the seven key questions, we synthesized the information from 428 articles.
All outcomes. KQ1: Effectiveness contains 379 studies assessing changes in process, intermediate outcomes, clinical outcomes, and economic and cost outcomes. The majority of studies were observational, with a fair number of RCTs for prescribing and monitoring phases (Table A). Fifty-three qualitative studies are included in this total. Prescribing and monitoring were the most frequently studied phases of medication management (Table A), with hospital and ambulatory care settings well-represented to the near exclusion of long-term care, home, and community (Table B).
Though dealing with prescriptions and medications, pharmacists were poorly represented in studies, most focused on physicians (Table C). CDSS and CPOE systems were the most often studied MMIT technologies (Table D).
The results from this section suggest that care processes such as medication errors, time for tasks, workflow and knowledge, skills, and attitudes can be improved with the use of MMIT. The evidence is strongest specifically during the prescribing and monitoring phases. Few studies evaluated clinical outcomes associated with the use of MMIT. Those that did often did not show statistically significant improvements in clinical outcomes. Most of the studies with statistically significant differences in clinical outcomes found small differences. The small number of articles with data on clinical outcomes is probably due, at least in part, to the difficulty in evaluating and establishing a direct association between the use of MMIT and clinical outcomes. This difficulty arises because of the distant nature of the outcome compared with the application of the health IT. Other contributing factors could also be considered.
Much of the relatively new research is addressing the type of research needed to come to a realistic and useful assessment of MMIT: pilot and demonstration projects and quantitative studies. Limited evidence suggests that MMIT can likely be cost effective, although most of the economic data come from cost analyses, which were often incomplete and seldom from head-to-head cost-effectiveness, cost utility, or cost-benefit trials.
A substantial body of qualitative literature indicates support for the use of health IT in the various phases of medication management by a number of health care providers and patient groups. Survey studies of satisfaction and use reflect similar findings of acceptance and satisfaction, although most indicated room for improvement. Issues relating to changing care practices and workflow are frequently mentioned. The studies also provide useful summaries of unintended consequences of MMIT applications, which are discussed in detail in the full report.
Process changes. Most of the studies evaluating MMIT applications provided data on changes in process (225 of 378). Distribution in the number of studies across the five phases, plus reconciliation and education, was not equal. Prescribing was studied in 174 studies, order communication in 16 studies, dispensing in 9 studies, administering in 19 studies, and monitoring in 47 studies. Four studies evaluated reconciliation and one studied patient education. Studies often evaluated more than one phase.
Prescribing. The prescribing phase is well studied (174 studies), especially in hospital (61 percent of studies) and ambulatory care settings (39 percent). Long-term care centers (one study) and community and home settings (no studies) are not well studied. Physicians are by far the most studied group of health professionals. More studies are needed that evaluate nonphysician use of MMIT, specifically pharmacists, mental health professionals, nurses, and other nonphysician prescribers, as well as patients and their caregivers. Many of the studies of health care providers who were not physicians were purely descriptive of the people involved with them, and the systems themselves.
Based on the studies of process changes, CDSS and CPOE systems can play an important role in making prescribing and ordering more accurate, improving record keeping, and speeding up and improving communication. Both systems, either alone or, more often, integrated, are well studied (multiple studies with strong methods). Other MMIT applications lack evidence, especially those that involve nurses, pharmacists, and patients and their families.
MMIT in prescribing is associated with improvements in patient safety-related processes of the prescribing process, especially in hospital-based studies (87 percent, 52 of 60 studies), and somewhat less in ambulatory-based studies (68 percent, 28 of 41 studies). Errors related to prescribing and ordering were reduced in hospital-based studies (68 percent, 15 of 22 studies), but prescribing errors were not studied as often in ambulatory settings (two of two studies were positive). Reductions in time were related to the time taken to order or prescribe or the speed of the prescribing-to-administering processes. Most reductions in time were not seen as often in hospital-based studies (four of seven studies positive), but were positive more often in ambulatory settings (four of five studies). Adherence to treatment guidelines, reminders, and recommended practice was improved in hospital studies (83 percent, 19 of 23) and to a lesser but still significant extent in ambulatory studies (64 percent, nine of 14 studies). Workflow was not evaluated in these studies of changes in process, although issues of workflow are addressed in qualitative studies in other sections of this report.
Order communication. Order communication, like dispensing, is one of the two medication management phases with the least number of studies—only 16 were identified. Two-way EDI holds promise to increasing the effectiveness of perfecting the prescription/order interactions between clinician prescribers and pharmacists. Currently, evidence on one-way communication predominates. The changes in process were also varied (two studies of errors, two of prescribing changes, five on time considerations, and three on workflow). Most studies were done using quantitative observational methods and all showed positive results.
Dispensing. Nine studies (three RCTs) assessed process improvements in dispensing. All process changes that were evaluated were found to be positive: four on modifications of the drugs that the pharmacists dispensed, three on errors, two on workflow, and one on adherence to good practice. With these few studies and multiple outcomes, evidence is limited on the role of MMIT in improving dispensing. This supports the findings of a Canadian health technology assessment report on MMIT that evaluated hospital dispensing and administering medications in hospitals.11
Administering. Many articles dealing with administering medications were not included in this report because they were descriptive and did not include comparative data. Nineteen studies, 1 RCT, 1 cohort study, and 17 quantitative observational studies, were included. All studies were set in hospitals and included nurses. The MMIT systems were well integrated into multiple hospital IT systems. Error-reduction goals were common in the studies and almost always found to be improved (8 of 13 studies of errors). Errors were mixed, as some related to transcription and some to timing of administration, while some identified more serious errors. Four studies showed no improvement in errors while one study showed increases in errors, mostly related to timing of administration.12 Four of five studies showed reductions in time from ordering to administering medication. Two studies evaluated the allocation of nursing time: one showed change and one did not in the proportion of time spent on various nursing tasks, including direct patient care, with the introduction of integrated MMIT for medication administering.
Monitoring. In our analysis, 70 percent (33 of 47 studies) of the included studies were associated with a 50 percent improvement in half or more process measures. Of these studies, most targeted physicians exclusively (34 studies), were conducted in academic institutions (33 studies), were developed for use in the ambulatory care setting (28 studies), focused on the adult population (36 studies), and provided CDSS with alerts or reminders to support chronic disease management (12 studies).
Studies that involved laboratory-based medication monitoring were most likely (76 percent of the time) to be associated with a greater than 50 percent improvement in a process outcome(s) than sign- or symptom-based medication monitoring. The most successful types of studies focused on changing prescriber behavior, improving response time to generated alerts, and improving the diagnosis and management of chronic diseases.
Reconciliation. Two systematic reviews and four studies provided evidence for improved reconciliation of medications with health IT. Reconciliation is the matching of medication lists over time, from different health care systems or from different prescribers. The evidence on reconciliation of medication lists is sparse, especially for systems that are fully integrated and capable of providing electronic comparisons of historical and current medications for individual patients at hospital discharge or on transfer to other facilities. All four studies showed improvements in agreement among lists of medications and two extended the evaluation to show improved prescribing13 and reduced errors.14
Unintended consequences. Eighteen studies provided data on adverse effects or unintended consequences. Two qualitative studies identified classes or categories of unintended consequences of health IT, many of which apply to MMIT applications. Some unintended consequences are minor, and some are major. In addition, some are seen to be positive and helpful. Some consequences are serious. For example, a small but statistically significant increase in mortality was seen in a children's hospital that installed a CPOE system that did not match workflow needs.15 A similar study showed another children's hospital that did not see the same increase in mortality in admitted children after their careful planning and implementation of health IT.16 Several authors contend that all health IT has unintended consequences. Formal evaluations of health IT installations should seek these unintended consequences and report them in their publications related to the evaluation. The importance of unintended consequences of MMIT also depends on the severity of the event, the degree of invasiveness of the MMIT, and the extent to which the use of the MMIT system disrupts existing workflow and processes. Consideration of formal reporting of serious unintended consequences might benefit all involved in development and implementation of MMIT systems. The qualitative studies in this report supplied a richer understanding of the adverse effects of MMIT, and they can form a strong base for more qualitative and quantitative studies of unintended consequences.
Education. Education related to MMIT centers on three aspects: formal informatics training during professional education or after graduation, training to use the MMIT systems, and improved outcomes based on knowledge and skills because of the use of the MMIT systems for health care providers, patients, and their families. This report does not include preprofessional or professional education related to the use and understanding of MMIT systems or certification in informatics or eHealth, all important aspects of MMIT application development and integration. Although we sought articles assessing postprofessional education related to changes in process associated with MMIT systems, we did not identify any articles that met our criteria. Training in the use of systems was often mentioned in articles but was not evaluated. Only one article was related to the educational component of MMIT systems for patient and family use, and it was associated with improved clinical outcomes. More information on health care professional and patient education is included in the sections of this report dealing with intermediate outcomes.
Intermediate outcomes. Intermediate outcomes deal with use, usability, education, knowledge, skills, and attitudes. Most studies with intermediate main endpoints focused on measuring use, correlates of use, perceptions, and satisfaction in the prescribing phase (26 of 42 studies). As for changes in process, clinicians and prescribing were well-studied. Use, perceptions, and satisfaction were reported to be improved. Factors such as ease of use, perceived usefulness, and improved quality of care predominated. Satisfaction and attitudes varied depending on the role of the health care provider. Variation in needs and roles of health professionals with respect to use of health IT are real and should be considered when choosing or implementing any new IT system. Usability studies with comparison groups are sparse but can provide useful suggestions to improve systems. Usability studies are often difficult to generalize or transfer across settings, in part because MMIT effectiveness is linked strongly to the culture, institutional leadership, and other situation specific factors. Therefore, applicability of findings related to usability is problematic in MMIT applications.
Economic outcomes. Five of 31 articles dealing with costs conducted comprehensive economic evaluations (costs and consequences). Two evaluated a CPOE system and three evaluated CDSS. Most of the studies that included monetary data (22 of 31 studies) were partial economic evaluations in the form of cost analyses (assessing costs of alternatives without analysis of effectiveness or efficacy). Most of these partial economic analyses assessed costs of prescribed medications with the MMIT system compared with not having the MMIT system.
Several studies found that health IT interventions may offer cost advantages despite their increased acquisition costs. These studies showed that over time, a net benefit accrued based on cost reductions resulting from the MMIT (such as lower adverse drug events (ADEs), drug costs, and laboratory test usage). However, given the uncertainty that surrounds the cost and outcomes data, and limited study designs available in the literature, it is difficult to reach any definitive conclusion as to whether the additional costs and benefits represent value for money.
Clinical outcomes. A total of 76 studies sought to measure improvement in clinical outcomes or reduction in ADEs, of which 26 (34 percent) reported significant benefits of health IT. One reported harm—a small but clinically important increase in mortality when an inflexible CPOE was implemented in a children's hospital.15 Because of the seriousness of the implications of this study, many people reviewed this article and its methods.17 A later and similar study showed that with careful planning another children's hospital did not see the same increase in mortality in admitted children after the implementation of a health IT.16
An additional two studies implemented CDSSs to reduce costs and assessed whether reductions in drug use increased mortality15 and length of stay.18 Both studies lacked sufficient power to conduct a valid assessment.
Studies that used laboratory-, sign- and symptom-based monitoring approaches were mostly clinician based. If the MMIT monitoring was used to identify and intervene with patients with actual problems (e.g., excess blood pressure) or needed care (e.g., hemoglobin A1c monitoring), this appears to be more effective than CDSS approaches that identified theoretical problems (potential for ADEs), particularly if patients are also sent reminders and decision support recommendations.
Highly targeted interventions, which focused on specific problems that provide problem-related specific interventions, appear to be more effective than more diffusely focused systems such as CDSS and CPOE. Some of these highly targeted interventions involved CDSS tools for improving the effectiveness of anticoagulants (proportion of days with blood clotting parameters within the therapeutic range), improving the choice, route, and duration of antibiotics, and reducing ADEs related to antibiotic use, and most were successful.
Studies that have been successful in improving patient outcomes target high risk and vulnerable populations who have poor disease control, lack sufficient access to health care providers to manage their condition or subpopulations with sufficient economic resources to respond to the CDSS intervention. The effect of similar CPOE systems on mortality can vary substantially as a function of the extent to which implementation strategies disrupt or delay critical activities in the clinical setting and demand additional time for order entry from clinical staff. Critically ill patients (i.e., those who are most vulnerable) are most likely to be affected by dysfunctional technology and implementation strategies.
Qualitative studies. Qualitative studies seek to understand phenomena and answer questions of why and how as well as to gain insights into real life situations. They often study the more human or "soft" side of health and health care. The preceding sections concentrated on studies with quantitative outcomes. Fifty-three qualitative studies are included in this section. Patient safety was the main health aspect evaluated in qualitative studies. Before MMIT implementation most studies found that clinicians expected that MMIT would improve patient safety and once implemented most clinicians felt that MMIT had improved safety.
The qualitative studies focused on system design including workflow changes, challenges with the system interface, and new communication processes—all of which can generate new kinds of medical errors, which in some cases were detrimental to patient safety.
Early implementers associated MMIT with a lot of self-reported "hard work" by those who were expected to use the new systems. These people, most often health professionals, struggled, often independently, with limited guidance with respect to planning and implementation tactics during preparation for and implementation of the MMIT applications. During planning and early implementation, the users often experienced unanticipated effects. Frequently, the initial stage was disruptive and, consequently, clinicians found provision of care to be more challenging with the MMIT system than without. However, after the initial stage was over, the attitudes of the care providers changed, and the potential benefits of the system become clearer to most. Of special note is that the implementation of MMIT systems generated emotional responses in a broad range of health professionals, both positive and negative. For example, strong feelings were associated with reminders and alerts19 and CPOE.20
MMIT implementation did not just mean that a clinician needed to learn a new IT system, but the implementation also affected most of the other parts of the delivery of care processes, including how the interdisciplinary care team worked together.
KQ2: Knowledge and Evidence Gaps
We identified gaps in the report, some that we expected and some that we did not. We address the question of knowledge deficits across phases and outcomes, settings and participants, grouping similar gaps together.
Phases of medication management. Because of the preponderance of publications on the prescribing and monitoring phases, they are less in need of more study than the other phases of order communication, dispensing and administering, and medication reconciliation. In addition, the educational or training requirements for effective use of MMIT applications by health professionals need to be studied as well as education related to patients as new MMIT applications are developed for their use.
Research methods. MMIT applications are complex interventions and need to be studied in pragmatic (i.e., does it work in real settings?) evaluation projects and using complex interventions methods. The applications also should ideally be studied by teams of researchers with, or teams that seek consultation from, those who have experience in clinical practice, research methods, statistical analysis, and informatics training and experience. Qualitative studies are also vital to understand the complex nature of how systems are used and valued, especially across groups of health professionals who often have different needs and expectations.
Health care providers. Physicians are well studied. Nurses, midlevel practitioners (nurse practitioners, physician assistants, midwives), pharmacists, other prescribers such as dentists and mental health practitioners, and hospital administrators need studies directed at their needs, practice patterns, and health IT tools.
Patients. Many studies included data related to patients, usually in the measurement and reporting of process changes and other outcomes. Few studies, however, concentrated on how the MMIT systems directly affected patients and clinical outcomes important to them. Traditionally, MMIT systems were developed as clinician and administrator tools. Patient and family use of MMIT systems is becoming more important, and this gap in our understanding needs to be addressed.
Settings. Hospitals and ambulatory care settings are well studied. Gaps exist in our knowledge of the effectiveness of MMIT in long-term care facilities, the community, and homes. Long-term care facilities most need strong qualitative and quantitative studies because theyrely heavily on medication. Homes, schools, and other community settings will also become more important with shifting care to more self-reliance in relation to wellness care and chronic disease management.
Health IT. Much research has gone into evaluating CDSS and CPOE systems, either alone or integrated. For example, 77 of 88 RCTs evaluated some aspect of CDSSs. Other MMIT applications, especially those that are used by nonphysicians or outside the prescribing and monitoring phases, lack evidence. Examples with little evidence on effectiveness are bar coding for administering and dispensing, pharmacy information systems, electronic medication administration record systems, and fully integrated comprehensive information systems.
Process changes. Patient safety processes such as error reductions and improvement in prescribing have a strong evidence base. Issues related to workflow, communication changes, and unintended consequences are understudied. More study of laboratory-based monitoring of medications, especially in facilities that have highly integrated information systems, is important. More qualitative and controlled studies are needed as well as multicenter studies and those that use methods developed by groups focusing on health technology assessment (HTA). These HTA methods include integrated reports that bring together research syntheses, modeling of processes and full economic reports, and cost studies. Often these HTA reports do not, but can, involve additional collection of evidence.
Intermediate outcomes. More study is needed on the importance of usability testing in all stages of development and use. This must be done with all users and not just segments of those involved in using MMIT. Usability studies have not traditionally been generalizable or transferrable but more limited to a specific setting. AHRQ might consider a research program in how to make these usability studies more applicable to multiple institutions, training in usability methods, collection of usability tools and completed studies, and research into the need for standards of usability testing for new or modified systems. Usability studies must also include all users of systems. For example, systems that have been optimized only for physician users are usually systems that nurses and other health professionals have difficulty using. Workarounds have often been unofficially implemented by users instead of system modifications and improvements.
Clinical outcomes. Findings associated with improvement in clinical outcomes are still equivocal. These studies are difficult to do well, expensive, and time consuming, but they must be done. Multicentered trials planned by strong teams of experienced people from multiple backgrounds are vital.
Cost and economic outcomes. Although many studies exist that list costs and outcomes, few comprehensive and definitive studies of the economic value of MMIT applications exist. Both the potential for improvement and the costs of implementing and maintaining these systems are huge. Again, well-planned studies with broad input from many stakeholders are necessary for understanding the true worth of MMIT applications. HTA or other studies that integrate costs and consequences of MMIT systems would be ideal.
Qualitative. Qualitative studies have provided much valuable information about MMIT. Gaps in qualitative knowledge center on the lack of qualitative studies that address the effects of MMIT on health outcomes. In addition, very few qualitative studies examined the effects of MMIT from the perspective of the patient.
KQ3: Value Proposition for Implementers and Users
Value proposition is determined from a balance of financial, clinical, and organizational benefits. A clear assessment of each of these from the viewpoint of each stakeholder is needed to make a clear value judgment. For each stakeholder—and many are involved with MMIT implementation—the relative importance of these three elements is different. Values will also vary depending on the setting and the type of technology employed. Multiple stakeholders, some of whom may be distant from the MMIT, need to be considered in any value proposition study. Based on the evidence in KQ1: Effectiveness, knowledge about the three elements needed to make value judgments is slowly accumulating. We cite only 31 papers in this section, although some of our assessments come from sections of this report that have included more studies. Gains in productivity and process of care outcomes have been shown, but good evidence of improvement in patient outcomes with MMIT is weak or lacking. The body of economic literature is still sparse and lacks vigorous study. We found little theoretical work or actual studies that were done to determine what each stakeholder takes into account to reach value proposition judgments related to MMIT.
KQ4: System Characteristics
Few studies (n = 21) demonstrated evidence of the impact of the characteristics of MMIT applications on the likelihood to purchase, implement, and use such IT applications. No studies assessed open-source health IT applications, with only one study each on conformity with standards and CCHIT-certified systems. Twenty of the articles related to the prescribing and ordering phase. Almost all of the articles suggest that feature sets of health IT applications have been instrumental in reaching decisions to adopt MMIT applications. Certain features of systems improve the likelihood of purchase, implementation, and use of MMIT. The literature, however, is sparse and observational in nature. Most often authors described barriers and concerns toward implementation and acceptance rather than characteristics of MMIT that could facilitate implementation, purchase, and use of such systems. Authors seldom provided enough details about the technology to form conclusions about the value of feature sets and system characteristics. Head-to-head comparisons of systems differing in their features were not found.
Our literature review revealed three important findings: sustainability is frequently mentioned in the core biomedical informatics literature, it is poorly defined, and none of the articles included in this evidence report explicitly studied sustainability. These findings are not entirely surprising. A previous AHRQ-sponsored evidence report that assessed the costs and benefits of health IT in pediatrics found only one article that explicitly discussed sustainability.21
Future research would be beneficial for many if a study or group would develop an operational definition of sustainability that could be used to study its determinants. Moreover, it is likely that the Health Information Technology for Economic and Clinical Health (HITECH) Act of 2009 will lead to improvements and sustainability of health IT applications that specifically support the medication management continuum through meaningful use.
We have summarized a body of literature that uses surveys to detect patterns in the characteristics of people and organizations that are more likely to implement various technologies. These surveys are often the basis for further study into barriers and facilitators to increasing uptake and adoption.
Integration of MMIT with other systems was an inclusion criterion for our report (except for PDAs that analyzed patient-specific data). Some technologies were integrated with a greater number of components than others. Frequently, the descriptions of the systems were inadequate to fully determine how the systems were connected. Access to various other information sources, most notably laboratory reports, enhanced the performance and acceptance of the MMIT applications.
KQ6: Complete Two-Way Electronic Data Interchange
No reports documenting the use of complete two-way EDI systems were found. Evidence from the limited set of one-way, e-Prescribing studies was extrapolated to identify possible key facilitators and barriers to completely electronic, two-way, e-Prescribing systems. Possible facilitators include monetary or other incentives to providers, a permissive regulatory environment, and the existence of an established standard for prescription EDI. Barriers included the low rate of EMR adoption in the United States, regulatory and legal uncertainties, and inadequate consideration of the effects of e-Prescriptions on pharmacists and pharmacies and their processes. While answering this question, we found that the Bell model does not represent the two-way communication between pharmacists and prescribers—it shows only a one-way linear movement of information.
KQ7: Effectiveness of CDSS
Seventy-seven RCTs were designated as primarily studying CDSS related to medication management and integrated with other health IT. These studies involved 4,709 providers and 828,441 patients in total. All studies assisted with at least the prescribing or monitoring phases of medication management. Overall, we found a lack of RCTs addressing electronic decision support integrated with other types of health IT. Statistically significant process changes were often shown in these RCTs. Only a small minority of these focus on clinical outcomes, however. Studies with clinical outcomes are those that are most important to guide decisionmaking of patients' providers and policymakers about the usefulness and need for MMIT interventions. A very small number of studies reported improvement in clinical outcomes.