Conducting a Sound Systematic Review: Balancing Resources with Quality Control

AHRQ 2008 Annual Conference

Slide Presentation from the AHRQ 2008 Annual Conference

On September 10, 2008, Eric B. Bass, made this presentation at the 2008 Annual Conference. Select to access the PowerPoint® presentation (100 KB; Plugin Software Help).

Slide 1

Conducting a Sound Systematic Review: Balancing Resources with Quality Control

• Eric B. Bass, MD, MPH
  Johns Hopkins University
  Evidence-based Practice Center

Slide 2

Learning Objectives

• Identify steps in a systematic review that have greatest implications for resource needs.
• Explain how those steps are vulnerable to error/bias.
• Identify strategies for quality control when resources are limited.

Slide 3

Steps in the systematic review

1. Establish eligibility criteria for each question.
2. Search literature.
3. Review citations.
4. Extract data.
5. Evaluate study quality & applicability.
6. Summarize & synthesize evidence.

Slide 4

Focus the Question(s)

• Formulate in specific measurable terms.
• Clarify priorities.
• Be realistic about time & effort.
• Negotiate with sponsor.

Slide 5

Set Eligibility Criteria

• Study design
  • RCT only?
  • Comparison group?
  • Any observational study?
• Study population.
• Study setting.
• Size of study.
• Year of publication.
• Peer-reviewed?
• Language.

Slide 6

Search Literature

• Select highest yield sources.
  • MEDLINE®
  • EMBASE®
  • Cochrane database of reviews & trials.
• Consider other sources.
  • Use sampling to estimate incremental yield.
• Use hand searches for quality control.
  • Citations in eligible articles.
  • Table of contents of relevant journals.
  • Query experts.

Slide 7

Review Citations

• Titles:
  • Many citations obviously not relevant & can be excluded quickly.
• Abstracts:
  • Usually sufficient to determine eligibility of citations.
• Full text of articles:
  • For some studies, eligibility can only be determined by reading full text.

Slide 8

Review Citations: An Example

• Title review (n=10,475):
  • Title review: 6,863 excluded.
• Abstract review (n=3,612):
  • Abstract review: 3, 163 excluded.
• Full text review (n=449):
  • Full text review: 386 excluded.
• 63 included articles.

Slide 9

Review Citations: Quality Control

• Use independent dual reviewers.
• Assess samples early.
• Discuss discrepancies.

Slide 10

Extract Data

• Set priorities for data extraction.
  • Anticipate content of final evidence tables.
  • Resist temptation to extract everything.
• Choose data management method.
  • Systematic review software vs. Access vs. tables.
  • Consider complexity, consistency, completeness of data.
• Establish quality control.
  • Dual review vs. solo review with random checks.
  • Independent vs. sequential dual review.
  • Discuss discrepancies.

Slide 11

Evaluate Study Quality & Applicability

• Decide how evaluation will be used.
  • To determine eligibility of studies.
  • To give more or less weight to studies.
  • To improve future research.
• Focus on most important aspects of quality & applicability.
• Use established instrument(s).
  • Jadad criteria for Randomized Controlled Trials (RCTs).
• Establish quality control.
  • Independent dual review.
  • Decide how to reconcile discrepancies.

Slide 12

Summarize & Synthesize Evidence

• Assemble evidence tables.
• Reassess priorities.
• Exclude duplicative data.
• Prepare summary of evidence on each question.
• Assess strength of evidence.
• Use 2 or more reviewers.
• Decide how to reconcile differences.

Slide 13

Summary

• To conduct a sound systematic review with limited resources:
  • Pay attention to priorities when defining questions, eligibility criteria & elements of evidence tables.
  • Establish quality control measures for each step that could introduce error or bias.
    • Reviewing citations.
    • Extracting data.
    • Evaluating quality, applicability & overall strength of evidence.