Skip Navigation U.S. Department of Health and Human Services www.hhs.gov/
Agency for Healthcare Research Quality www.ahrq.gov
AHRQ Home | Questions? | Contact Us | Site Map | What's New | Browse | Información en español | E-mail Updates
www.ahrq.gov/

You Are Here: AHRQ Home > Clinical Information > EPC Evidence Reports > Topics In Progress > Healthcare Decisionmaking

Healthcare Decisionmaking

Full Title: Enabling Health Care Decision Making through the Use of Health Information Technology (Health IT)

Evidence-based Practice Center Review Protocol

Expected Release Date: late 2010

Contents

Brief Background of Topic
Key Questions
Analytic Framework
Proposed Literature Search and Review
Criteria for Evaluating the Quality of Studies
Rating the Strength of the Body of Evidence
References
Definition of Terms
Summary of Protocol Amendments

Brief Background of Topic

Efforts to improve the quality and value of healthcare are increasingly emphasizing a critical role for the meaningful use of clinical decision support systems. The specific aim of clinical decision support systems (CDSS) is to provide patient-specific recommendations based on algorithms that allow comparison of patient information with a knowledge base.2-5 Examples of electronic CDSS include alerts and reminders, dashboards, computer-assisted diagnosis, order sets, and drug dosage calculations. In general, CDSS can enhance clinical effectiveness by improving the quality of care6 and patient outcomes by aiding health care providers in the decisionmaking process.7,8 However, in order for CDSS to improve the quality of health care, there needs to be evidence-based and practice-based information that provides evidentiary knowledge applicable to the clinical setting and the clinician and patient interaction. As a form of health information technology (health IT), CDSS can serve as an information tool to augment clinician decisionmaking with best practice guidelines and evidence at the point of care.

Within CDSS, there is a continuum of decision support aides that have the goal of obtaining knowledge to inform a decision at the point of care or for a specific care situation. This review will examine each type of decision support aide presented in Table 1.

Table 1 depicts:

  1. Three types of decision support aides.
  2. How context-specific queries are processed by the decision support aides to submit patient-specific information and retrieve patient-specific recommendations.

Table 1. Continuum of Decision Support

Types of Decision Support AidesClassic Clinical Decision SupportInformation Retrieval Tool Knowledge Resource
ExamplePreventative service reminderInfobuttonEpocrates
ProcessSubmit patient-specific informationAutomated (machine)Automated (machine)Manual (human)
Retrieve patient-specific informationAutomated (machine)Manual (human)Manual (human)

Classic clinical decision support (CDS) is defined as "any electronic system designed to aid directly in clinical decisionmaking, in which characteristics of individual patients are used to generate patient-specific assessments or recommendations that are then presented to clinicians for consideration".1 An example of a classic CDS is a preventative service reminder to remind the clinician of a specific action. For this type of decision support, the processes to submit patient-specific information and retrieve patient-specific recommendations are automated and performed by a machine.

Information retrieval tool is defined as an electronic tool designed to aid clinicians in the search and retrieval of context-specific knowledge from information sources based on patient-specific information from a clinical information system to facilitate decisionmaking at the point of care of for a specific care situation. An example of an information retrieval tool is an infobutton embedded in a clinical information system, such as an EHR, that when selected, provides context-specific links to various information sources. For this type of decision support, the process to submit patient-specific information is automated and performed by a machine and the process to retrieve patient-specific recommendations is manually performed by a human.

Knowledge resource is defined as an electronic resource comprised of distilled primary literature designed to facilitate decisionmaking at the point of care or for a specific care situation. Examples of knowledge resources include UpToDate, Epocrates, and MDConsult.

Return to Contents

Key Questions

Key Question 1. What evidence-based study designs can be used to determine the clinical effectiveness of clinical decision support systems (CDSS)?

Key Question 2. What contextual factors/features influence the implementation and use of electronic knowledge management and CDSS?

Key Question 3. What is the impact of introducing electronic knowledge management and CDSS?

  1. Changes in the organization of health care delivery.
  2. Changes in the workload and efficiency for the user.
  3. Changes in process and clinical outcomes.

Key Question 4. What generalizable knowledge can be integrated into electronic knowledge management and CDSS to improve health care quality?

  1. Knowledge from published evidence about electronic knowledge management and CDSS to improve health care quality based on different types of measures (health care process, relationship-centered, clinical, economic).
  2. How a clinicians' expertise/proficiency/informatics competency using the electronic knowledge management and CDSS effect patient outcomes (one type of measure)?

Return to Contents

Analytic Framework

Select for the analytic framework (47 KB).

Population(s)

  • Populations that will be examined in this review (Table 2) include the system user and health care organization. System user is defined as a clinician that interacts with the knowledge management or CDSS. Health care organization is defined as an organization that provides individuals access to health care services delivered by medical and allied health professionals.
  • The population group will apply to all of the Key Questions.

Table 2. A Summary of Populations Included in this Review

PopulationsPotential CategoriesDescription
System userNurse; Nurse Practitioner; Physician Assistant; Physician (Resident, Fellow, Attending); PharmacistClinician that interacts with the knowledge management or CDSS
Health care organizationHospital; Ambulatory clinic; Long-term care facilityOrganization that provides individuals access to health care services delivered by medical and allied health professionals

Interventions

  • The interventions evaluated will include electronic knowledge management and CDSS. Electronic knowledge management will be defined as any electronic system based on the distillation of primary literature used at the point-of-care to inform decisionmaking. Examples of electronic knowledge management include UpToDate, Epocrates, and infobuttons. CDSS will be defined as "Any electronic system designed to aid directly in clinical decisionmaking, in which characteristics of individual patients are used to generate patient-specific assessments or recommendations that are then presented to clinicians for consideration".1 Examples of electronic CDSS include alerts and reminders, dashboards, computer-assisted diagnosis, order sets, and drug dosage calculations.
  • The intervention will apply to all of the Key Questions.

Comparators

  • The comparator will include usual care and how usual care is defined in each study will be recorded.
  • Usual care will refer to the following scenarios:
    • CDS is compared with no electronic CDS
    • Basic CDS is compared with advanced CDS in CPOE
    • Basic CDS is compared with advanced CDS in a stand-alone system
  • The comparator will apply to all of the Key Questions.

Outcomes

The outcomes evaluated will differ by key question and are listed below:

KQ1
  • Study designs.
KQ2

Factors/features:

  • The Meaning of CDS.
  • Data as a Foundation for CDS.
  • Content Library Management.
  • New Roles for Special Essential People.
  • Translation for Collaboration.
  • User Computer Interaction.
  • Workflow.
  • Communication, Training, and Support.
  • Measurement and Metrics.
  • Governance.
  • Impact on the Culture of Safety and/or Quality.
  • Provider Compensation.
  • Technical Aspects.
KQ3

Outcomes and process measures:

  • Health care process outcomes (prevention, diagnosis, treatment, e.g. receiving appropriate treatment).
  • Relationship-centered outcomes (shared decisionmaking, clinician-patient communication, clinician-clinician communication, clinician workload, workflow, efficiency).
  • Clinical outcomes (length of stay, adverse drug event, morbidity, mortality, quality of life).
  • Economic outcomes (cost).
KQ4
  • Generalizable knowledge.

Timing

  • Currently we will not place any restrictions on the study duration for included studies. However, after a review of the literature, we may decide to restrict studies if such a restriction is needed based on the quality of the available studies.
  • The timing definition will apply to all of the Key Questions.

Setting

  • No restrictions. We are planning to include studies from all settings such as academic centers, community hospitals, federally-funded hospitals, and others.
  • The setting definition will apply to all of the Key Questions.

Return to Contents

Proposed Literature Search and Review

Sources

The comprehensive search will include electronic searching of peer reviewed literature databases and grey literature databases. Electronic peer reviewed literature databases that will be searched include Cumulative Index to Nursing and Allied Health Literature (CINAHL), The Cochrane Library, MEDLINE® accessed via PubMed, PsychInfo, and Web of Science®. Grey literature databases that will be searched include Health Services Technology, Administration, and Research (HealthSTAR), The Institute of Electrical and Electronics Conference Proceeding, Institute of Electrical and Electronics Engineers (IEEE) Conference Proceedings, and Proceedings of the American Society for Information Science and Technology (Wiley InterScience). Searches of those databases will be supplemented with hand-searching of reference lists contained in all included articles and in relevant review articles.

Search Strategies

Search strategies will be specific to each database to retrieve the articles most relevant to the Key Questions. However, the basic search strategy will use the National Library of Medicine's Medical Subject Headings (MeSH) key word nomenclature developed for MEDLINE®, limit searches to articles published in English, and hand-search retrieved articles and published reviews.

Abstract and Full-text Screening

Paired researchers from the Duke research team will independently review all abstracts and classify each as "include" or "exclude" according to project-specific criteria. An abstract will be included for further review if one of the paired reviewers recommends that it be included. We will test for inter-rater reliability for include/exclude decisions at the abstract stage.

At the full-text review stage, paired researchers will independently review the articles and indicate a decision to "include" or "exclude" the article for data abstraction. When the paired reviewers arrive at different decisions about whether to include or exclude an article, they will reconcile the difference through a third party arbitrator. The detailed exclusion criteria for full-text articles are listed below.

Full-text Screening Criteria

General screening criteria for Key Questions 1-4:

  1. Must be an implemented electronic CDSS.
  2. Evaluate the effectiveness of the CDSS.
  3. Describe at least one outcome or process measure.
  4. Present original data.
Key Question 1

Screening criteria for Key Question 1:

  1. Nothing additional to what is listed under General screening criteria.
Key Question 2

Screening criteria for Key Question 2:

  1. Describe factors/features that influenced the implementation and use of electronic knowledge management or CDSS:
    • The Meaning of CDS.
    • Data as a Foundation for CDS.
    • Content Library Management.
    • New Roles for Special Essential People.
    • Translation for Collaboration.
    • User Computer Interaction.
    • Workflow.
    • Communication, Training, and Support.
    • Measurement and Metrics.
    • Governance.
    • Impact on the Culture of Safety and/or Quality.
    • Provider Compensation.
    • Technical Aspects.
Key Question 3

Background clarifications:

  1. Changes in the organization of health care delivery: how CDSS enables care to be delivered differently; how CDSS can support tasks to be performed without a human or by someone with less training.
    1. Examples: Care managers; changes in workflow process (modifying intake process to allow nurses to perform a particular function); re-organizing teams (coordinate similar tasks, reduce duplication of tasks/orders).

Screening criteria for Key Question 3:

  1. Discuss changes in the organization of health care delivery, workload and efficiency for the user, or process and clinical outcomes.
Key Question 4

Background clarifications were:

  1. Generalizable knowledge will be defined as: "Published evidence of the effects of a health information technology (HIT) intervention on costs and benefits that other health care organizations can use to implement HIT and reasonably expect benefits similar to those reported in the original study. Therefore, generalizable knowledge from a study has two components: (1) the internal validity of the study and (2) the utility of the information to others considering implementing HIT".10
  2. Health care quality: can be assessed with different types of measures.11
    1. Health care process outcomes (prevention, diagnosis, treatment, e.g. receiving appropriate treatment).
    2. Relationship-centered outcomes (shared decision making, clinician-patient communication, clinician-clinician communication, clinician workload, workflow, efficiency).
    3. Clinical outcomes (length of stay, adverse drug event, morbidity, mortality, quality of life).
    4. Economic outcomes (cost).

Screening criteria for Key Question 4a:

  1. Describe how the electronic knowledge management or CDSS effected health care quality.
  2. RCT design.

Screening criteria for Key Question 4b:

  1. Sdf describe the clinician (user) expertise, proficiency, informatics competency using electronic knowledge management and CDSS.
  2. describe at least one patient (clinical) outcome.

Return to Contents

Criteria for Evaluating the Quality of Studies

The included studies will be assessed on the basis of the quality of their reporting of relevant data. We will evaluate the quality of individual studies using the approach described in the Methods Guide for Effectiveness and Comparative Effectiveness Reviews.

To assess quality, we will employ the strategy to:

  1. Classify the study design.
  2. Apply predefined criteria for quality and critical appraisal.
  3. Arrive at a summary judgment of the study's quality.

To evaluate methodological quality, we will apply criteria for each study type derived from core elements described in the Methods Guide for Effectiveness and Comparative Effectiveness Reviews, West et al. and Lohr.12,13 To indicate the summary judgment of the quality of the individual studies, we will use the summary ratings of good, fair, and poor.

To assess applicability, we will use the PICOTS format to identify specific issues that may limit the applicability of individual studies or a body of evidence as recommended in the Methods Guide for Effectiveness and Comparative Effectiveness Reviews.

Return to Contents

Rating the Strength of the Body of Evidence

The strength of evidence for each key question will be assessed using the approach described in the Methods Guide for Effectiveness and Comparative Effectiveness Reviews. The evidence will be evaluated using the four required domains: risk of bias, consistency, directness, and precision. Additionally, when appropriate, the studies will be evaluated for: coherence, dose-response association, residual confounding, strength of association (magnitude of effect), publication bias, and applicability. The strength of evidence will also be assigned an overall strength of evidence grade of high, moderate, low, or insufficient.

Return to Contents

References

1. Kawamoto K, Houlihan CA, Balas EA, Lobach DF. Improving clinical practice using clinical decision support systems: a systematic review of trials to identify features critical to success. BMJ 2005 Apr 2;330(7494):765.

2. Eccles M, McColl E, Steen N, Rousseau N, Grimshaw J, Parkin D, et al. Effect of computerised evidence based guidelines on management of asthma and angina in adults in primary care: cluster randomised controlled trial. BMJ 2002 Oct 26;325(7370):941.

3. Friedman C, Wyatt J. Evaluation methods in medical informatics. Springer-Verlag, editor 1997.

4. Garg AX, Adhikari NK, McDonald H, Rosas-Arellano MP, Devereaux PJ, Beyene J, et al. Effects of computerized clinical decision support systems on practitioner performance and patient outcomes: a systematic review. JAMA 2005 Mar 9;293(10):1223-38.

5. Grimshaw J, Freemantle N, Wallace S, Russell I, Hurwitz B, Watt I, et al. Developing and implementing clinical practice guidelines. Qual Health Care 1995 Mar;4(1):55-64.

6. Sim I, Gorman P, Greenes RA, Haynes RB, Kaplan B, Lehmann H, et al. Clinical decision support systems for the practice of evidence-based medicine. J Am Med Inform Assoc 2001 Nov-Dec;8(6):527-34.

7. Bates DW, Evans RS, Murff H, Stetson PD, Pizziferri L, Hripcsak G. Detecting adverse events using information technology. J Am Med Inform Assoc 2003 Mar-Apr;10(2):115-28.

8. Bates DW, Gawande AA. Improving safety with information technology. N Engl J Med 2003 Jun 19;348(25):2526-34.

9. Ash JS, Anderson NR, Tarczy-Hornoch P. People and organizational issues in research systems implementation. J Am Med Inform Assoc 2008 May-Jun;15(3):283-9.

10. Shekelle PG, Morton SC, Keeler EB. Costs and Benefits of Health Information Technology. Evidence Report/Technology Assessment No. 132. (Prepared by the Southern California Evidence-based Practice Center under Contract No. 290-02-0003.) AHRQ Publication No. 06-E006. Rockville, MD: Agency for Healthcare Research and Quality. April 2006.

11. Gibbons MC, Wilson RF, Samal L, Lehmann CU, Dickersin K, Lehmann HP, Aboumatar H, Finkelstein J, Shelton E, Sharma R, Bass EB. Impact of Consumer Health Informatics Applications. Evidence Report/Technology Assessment No. 188. (Prepared by Johns Hopkins University Evidence-based Practice Center under contract No. HHSA 290-2007-10061-I). AHRQ Publication No. 09(10)-E019. Rockville, MD. Agency for Healthcare Research and Quality. October 2009.

12. Lohr KN. Rating the strength of scientific evidence: relevance for quality improvement programs. Int J Qual Health Care 2004 Feb;16(1):9-18.

13. West S, King V, Carey TS, et al. Systems to Rate the Strength of Scientific Evidence. Evidence Report/Technology Assessment No. 47 (Prepared by the Research Triangle Institute-University of North Carolina Evidence-based Practice Center under Contract No. 290-97-0011). AHRQ Publication No. 02-E016. Rockville, MD: Agency for Healthcare Research and Quality. April 2002.

Return to Contents

Definition of Terms

AHRQ = Agency for Healthcare Research and Quality
CDSS = Clinical Decision Support System
CERs = Comparative Effectiveness Reviews
CINAHL = Cumulative Index to Nursing and Allied Health Literature (CINAHL)
EPC = Evidence-based Practice Center
HealthSTAR = Health Services Technology, Administration, and Research
IEEE = Institute of Electrical and Electronics Engineers
PICOTS = Population(s), Interventions, Comparators, Outcomes, Timing, Settings
RCTs = Randomized Controlled Trials
TEP = Technical Expert Panel
TOO = Task Order Officer

Return to Contents

Summary of Protocol Amendments

In the event of protocol amendments, the date of each amendment will be accompanied by a description of the change and the rationale.

Note: The following protocol elements are standard procedures for all protocols.

Review of Key Questions

For Comparative Effectiveness reviews the key questions were posted for public comment and finalized after review of the comments. For other systematic reviews, key questions submitted by partners are reviewed and refined as needed by the EPC and the Technical Expert Panel (TEP) to assure that the questions are specific and explicit about what information is being reviewed.

Technical Expert Panel (TEP)

A TEP panel is selected to provide broad expertise and perspectives specific to the topic under development. Divergent and conflicted opinions are common and perceived as health scientific discourse that results in a thoughtful, relevant systematic review. Therefore study questions, design and/or methodological approaches do not necessarily represent the views of individual technical and content experts. The TEP provides information to the EPC to identify literature search strategies, review the draft report and recommend approaches to specific issues as requested by the EPC. The TEP does not do analysis of any kind nor contribute to the writing of the report.

Peer Review (Standard Language)

Approximately five experts in the field will be asked to peer review the draft report and provide comments. The peer reviewer may represent stakeholder groups such as professional or advocacy organizations with knowledge of the topic. On some specific reports such as reports requested by the Office of Medical Applications of Research, National Institutes of Health there may be other rules that apply regarding participation in the peer review process. Peer review comments on the preliminary draft of the report are considered by the EPC in preparation of the final draft of the report. The synthesis of the scientific literature presented in the final report does not necessarily represent the views of individual reviewers. The dispositions of the peer review comments are documented and will, for Comparative Effectiveness Reviews (CERs) and Technical Briefs, be published three months after the publication of the Evidence Report.

It is our policy not to release the names of the peer reviewers or TEP panel members until the report is published so that they can maintain their objectivity during the review process. 

Return to Contents

Current as of March 2010

Internet Citation:

Enabling Health Care Decisionmaking through the Use of Health IT, Systematic Review Protocol. Agency for Healthcare Research and Quality, Rockville, MD. http://www.ahrq.gov/clinic/tp/knowmgttp.htm

 
AHRQAdvancing Excellence in Health Care