A Tool for the Classification of Study Designs in Systematic Reviews o

Slide Presentation from the AHRQ 2009 Annual Conferenc

Slide presentation from the AHRQ 2009 conference.

On September 15, 2009, Meera Viswanathan made this presentation at the 2009 Annual Conference. Select to access the PowerPoint® presentation (432 KB) (Plugin Software Help).

Slide 1

A Tool for the Classification of Study Designs in Systematic Reviews of Interventions and Exposures

Meera Viswanathan, PhD
for the University of Alberta EPC
 

 

Slide 2

Steering Committee

  • Ken Bond, UAEPC
  • Donna Dryden, UAEPC
  • Lisa Hartling, UAEPC
  • Krystal Harvey, UAEPC
  • P. Lina Santaguida, McMaster EPC
  • Karen Siegel, AHRQ
  • Meera Viswanathan, RTI-UNC EPC

 

Slide 3

Background

  • EPC reports, particularly comparative effectiveness reviews, are increasingly including evidence from nonrandomized and observational designs
  • In systematic reviews, study design classification is essential for study selection, risk of bias assessment, approach to data analysis (e.g., pooling), interpretation of results, grading body of evidence
  • Assignment of study designs is often given inadequate attention

 

Slide 4

Objectives

  • Identify tools for classification of studies by design
  • Select a classification tool for evaluation
  • Develop guidelines for application of the tool
  • Test the tool for accuracy and inter-rater reliability

 

Slide 5

Objective 1: Identification of tools

31 organizations/individuals contacted
11 organizations/individuals responded
23 classification tools received
10 tools selected for closer evaluation
1 tool selected for modification and testing
 

 

Slide 6

Objective 2: Tool selection

  • Steering Committee ranked tools based on:
    • Ease of use
    • Unique classification for each study design
    • Unambiguous nomenclature and decision rules/definitions
    • Comprehensiveness
    • Potentially allows for identification of threats to validity and provides a guide to strength of inference
    • Developed by a well-established organization

 

Slide 7

Objective 3: Tool development

  • Three top-ranked tools:
    • Cochrane Non-Randomised Studies Methods Group
    • American Dietetic Association
    • RTI-UNC
  • Incorporated positive elements of other tools
  • Developed glossary

 

Slide 8

Objective 4: Testing round 1

Overall agreement (30 studies, 6 testers)κ=0.26 (fair)
Graduate level training complete (3 testers)κ=0.38 (fair)
Graduate level training in progress (3 testers)κ=0.17 (slight)
Item agreement: 
6/6 testers agreed0
5/6 testers agreed7 (23%)
4/6 testers agreed5 (17%)
3/6 testers agreed9 (30%)
2/6 testers agreed8 (27%)
No agreement1 (3%)

 

Slide 9

Objective 4: Testing round 1

  • No clear patterns in disagreements
  • Disagreements occurred at all decision points
  • Tool vs. studies
  • Variations in application of the tool

 

Slide 10

Objective 4: Reference standard

Overall agreement (30 studies, 3 raters)κ=0.33 (fair)
  
Item agreement: 
3/3 raters agreed7 (23%)
2/3 raters agreed14 (47%)
No agreement9 (30%)

 

Slide 11

Objective 4: Testing round 2

Overall agreement (15 studies, 6 testers)κ=0.45 (moderate)
Graduate level training complete (3 testers)κ=0.45 (moderate)
Graduate level training in progress (3 testers)κ=0.39 (fair)
Item agreement: 
6/6 testers agreed3 (20%)
5/6 testers agreed2 (13%)
4/6 testers agreed6 (40%)
3/6 testers agreed2 (13%)
2/6 testers agreed2 (13%)
No agreement0

 

Slide 12

Discussion

  • Moderate reliability, low agreement with reference standard
  • Studies vs. tool as source of disagreement
    • Tool not comprehensive, e.g., quasi-experimental designs
    • Studies challenging, e.g., sample of difficult studies, poor study reporting
  • To optimize agreement and reliability:
    • Training in research methods
    • Training in use of tool
    • Pilot testing
    • Decision rules

 

Slide 13

Next Steps

  • Test within a real systematic review
  • Further testing for specific study designs
  • Further evaluation of differences in reliability by education, training, and experience

 

Slide 14

Acknowledgments

  • Ahmed Abou-Setta
  • Liza Bialy
  • Michele Hamm
  • Nicola Hooton
  • David Jones
  • Andrea Milne
  • Kelly Russell
  • Jennifer Seida
  • Kai Wong
  • Ben Vandermeer (statistical analysis)

 

Slide 15

Questions?

University of Alberta EPC.Edmonton, Alberta, Canada

Current as of December 2009
Internet Citation: A Tool for the Classification of Study Designs in Systematic Reviews o: Slide Presentation from the AHRQ 2009 Annual Conferenc. December 2009. Agency for Healthcare Research and Quality, Rockville, MD. http://www.ahrq.gov/news/events/conference/2009/viswanathan/index.html