Selecting Evidence for Comparative Effectiveness Reviews: When to use

Slide Presentation from the AHRQ 2009 Annual Conferenc

On September 16, 2009, Meera Viswanathan made this presentation at the 2009 Annual Conference. Select to access the PowerPoint® presentation (1 MB) (Plugin Software Help).

Slide 1

Selecting Evidence for Comparative Effectiveness Reviews: When to use Observational Studies

Dan Jonas, MD, MPH
Meera Viswanathan, PhD
Karen Crotty, PhD, MPH

RTI-UNC Evidence-based Practice Center

Slide 2

Sources

• AHRQ Methods Guide, Chapters 4 and 8, http://www.effectivehealthcare.ahrq.gov/repFiles/2007_10DraftMethodsGuide.pdf
• Draft manuscript, Norris et al., Observational Studies in Systematic Reviews of Comparative Effectiveness.
• Chou R, Aronson N, Atkins D, et al. Assessing harms when comparing medical interventions: AHRQ and the Effective Health Care Program. J Clin Epidemiol 2008 Sep 25.

Slide 3

Overview

• Why should reviewers consider including observational studies (OS) in comparative effectiveness reviews (CERs)?
• When should OS be included in CERs?
• What are the differences in considering inclusion of OS for benefits as opposed to OS of harms?

Slide 4

Current Perspective

• CERs should consider including observational studies
  • *this should be the default strategy*
• Reviewers should explicitly state the rationale for including or excluding OS

Slide 5

Comparative Effectiveness Reviews (CERs)

• Systematic reviews that compare the relative benefits and harms among a range of available treatments or interventions for a given condition

Slide 6

CER Process Overview

• Refine key questions
• Develop analytic frameworks

• Indentify eligibility
• Search for relecant studies
• Select evidence for inclusion

• Estract evidence from studies
• Construct evidence tables

• Assess quality of studies
• Assess applicability of studies
• Apply qualitative methods
• Apply qualitative methods (meta-anyayses
• Rate the strength of a body of evidence

Slide 7

Hierarchy of Evidence

• Systematic Reviews
• RCTs
• Controlled Clinical Trials and Observational Studies
• Uncontrolled Observational Studies
• Case reports and case series
• Expert Opinions

Slide 8

Danger of Over-reliance on RCTs

• May be unnecessary, inappropriate, inadequate, or impractical
• May be too short in duration
• May report intermediate outcomes rather than main health outcomes of interest
• Often not available for vulnerable populations
• Generally report efficacy rather than effectiveness
• AHRQ Evidence-based Practice Centers include wide variety of study designs (not only RCTs)

Slide 9

Observational Studies (OS)

• Definition: Studies where the investigators did not assign the exposure/intervention
  • I.e. non-experimental studies
  • Controlled clinical trials are quasi-experimental studies, not OS
• We present considerations for including OS to assess benefits and to assess harms separately

Slide 10

OS to Assess Benefits

• Often insufficient evidence from trials to answer all KQs in CERs (think PICOTS)
  • Population: may not be available for sub-populations and vulnerable populations
  • Interventions: may not be able to assign high-risk interventions randomly
  • Outcomes: may report intermediate outcomes rather than main health outcomes of interest
  • Timing: may be too short in duration
  • Setting: may not represent typical practice

Slide 11

Group Exercise

• What should reviewers consider when deciding whether or not to include observational studies in CERs?

Slide 12

OS to Assess Benefits

• Reviewers should consider 2 questions:
• 1. Are there gaps in trial evidence for the review questions under consideration?
• 2. Will observational studies provide valid and useful information to address key questions?

Slide 13

Flowchart of OS to Assess Benefits

1.Systematic review question (including PICOTS)
2. Always consider: Controlled Trials
3. Are there gaps in trial evidence?
4. Yes - Consider OS
4. No - Confine review to Controlled Trials

Will OS provide valid and useful information?
5. Refocus the review question on gaps
6. Assess whether OS address the review question
7. Assess the suitability of OS: Natural history of the disease or exposure Potential biases

Slide 14

Group Exercise: Include OS?

• CER of PCI vs. CABG for coronary disease identified 23 RCTs. Experts (TEP) raised concerns that the studies enrolled patients with a relatively narrow spectrum of disease relative to those having the procedures in current practice
• Review of antioxidant supplementation to prevent heart disease found numerous large clinical trials, including over 20,000 elevated-risk subjects in the Heart Protection Study. No beneficial effects were seen in CV outcomes, including mortality. Findings were consistent across trials with varying populations, sizes, etc.

Slide 15

Group Exercise: include OS?

• CER of PCI vs. CABG----Need to look for OS
  • OS from 10 large cardiovascular registries were identified
  • These confirmed that the use of the procedures in the community included patients with wider variation in disease
  • For patients similar to those enrolled in trials, mortality results in the registries were similar to trials (no difference between interventions)
  • Relative benefits of the procedures varied markedly with extent of disease, raising caution about extending trial conclusions to patients with greater or lesser disease than those in trial populations
• Review of antioxidant supplementation to prevent heart disease----Trial data are sufficient

Slide 16

Gaps in Trial Evidence: PICOTS

• Trial data may be insufficient for a number of reasons
  • PICOTS
  • Populations included (missing certain groups)
  • Interventions included
  • Outcomes reported (only intermediate)
  • Duration
  • All trials may be efficacy studies

Slide 17

Are Trial Data Sufficient?
PICOTS and Beyond

• Risk of bias (internal validity)
  • Degree to which the findings may be attributed to factors other than the intervention under review
• Consistency
  • Extent to which effect size and direction vary within and across studies
  • Inconsistency may be due to heterogeneity across PICOTS
• Directness
  • Degree to which outcomes that are important to users of the CER (patients, clinicians, or policymakers) are encompassed by trial data
  • Health outcomes generally most important

Slide 18

Are Trial Data Sufficient? PICOTS and Beyond

• Precision
  • Includes sample size, number of studies, and heterogeneity within or across studies
• Reporting bias
  • Extent to which trial authors appear to have reported all outcomes examined
• Applicability
  • Extent to which the trial data are likely to be applicable to populations, interventions, and settings of interest to the user
  • The review questions should reflect the PICOTS characteristics of interest

Slide 19

When to Identify Gaps in Trial Evidence

• Identification of gaps in trial evidence available to answer review questions can occur at a number of points in the review
  • When first scoping the review
  • Consultation with Technical Expert Panel
  • Initial review of titles and abstracts
  • After detailed review of trial data

Slide 20

CER Process Overview

• Refine key questions
• Develop analytic frameworks

• Indentify eligibility
• Search for relecant studies
• Select evidence for inclusion

• Estract evidence from studies
• Construct evidence tables

• Assess quality of studies
• Assess applicability of studies
• Apply qualitative methods
• Apply qualitative methods (meta-anyayses
• Rate the strength of a body of evidence

Slide 21

Gaps in Trial Evidence

• Operationally, may perform initial searches broadly, to identify both OS and trials, or may do searches sequentially and search for OS after reviewing trials in detail to identify gaps in evidence

Slide 22

2. Will observational studies provide valid and useful information to address key questions?

Reviewers should:

• Refocus the study question on gaps in trial evidence
  • Specify the PICOTS characteristics for gaps in trial evidence
• Assess whether available OS may address the review questions (applicable to PICOTS?)
• Assess suitability of OS to answer the review questions

Slide 23

Valid and Useful Information

• Assess suitability of OS to answer the review questions
• After gaps have been identified in trial literature and that OS potentially fill those gaps
  • Consider the clinical context and natural history of the condition under study
  • Assess how potential biases may influence the results of OS

Slide 24

Clinical context

• Fluctuating or intermittent conditions are more difficult to assess with OS
  • Especially if there is no comparison group
• OS may be more useful for conditions with steady progression or decline

Slide 25

Group Exercise

• Here are two very different conditions:
  • Acute low back pain
  • Amyotrophic lateral sclerosis (ALS)
• How might the differences in these conditions impact whether OS would provide useful information?

Slide 26

Group Exercise

• Main considerations here are the natural history of the condition under study
• People with acute low back pain often recover spontaneously
  • A cohort study of treatments for acute low back pain can't establish, with any degree of certainty, whether the treatments affected patient outcomes
• ALS has a course of steady decline
  • An uncontrolled cohort study of treatments for ALS may well be able to demonstrate meaningful effects

Slide 27

Potential biases

• Selection bias (and confounding by indication)
• Performance bias
• Detection bias
• Attrition bias

Slide 28

Group Exercise

• Suppose you're conducting a CER of medications for rheumatoid arthritis (RA)
• You find several retrospective analyses of administrative databases comparing outcomes of RA patients taking etanercept vs. methotrexate
• Suppose that etanercept is restricted in many of the health systems to patients with more severe RA who have failed other treatments
• Should you include these OS?
• What considerations will influence your decision?

Slide 29

Group Exercise

• Confounding by indication
  • A type of selection bias
  • When different diagnoses, severity of illness, or comorbid conditions are important reasons for physicians to assign different treatments
  • Common problem in pharmacoepidemiology studies comparing beneficial effects of interventions
• Generally would not include this information due to a high risk of bias (poor internal validity), unless studies had a good way to adjust for severity of disease

Slide 30

Harms

• Assessing harms can be difficult
  • Trials often focus on benefits, with little effort to balance assessment of benefits and harms
  • OS are almost always necessary to assess harms adequately
• There are tradeoffs between increasing comprehensiveness of reviewing all possible harms data and decreasing quality (increasing risk of bias) for harms data

Slide 31

Trials to Assess Harms

• Randomized controlled trials = gold standard for evaluating efficacy
• But, relying solely on RCTs to evaluate harms in CERs is problematic
  • Most lack prespecified hypotheses for harms as they are designed to evaluate benefits
  • Assessment of harms is often a secondary consideration
  • Quality and quantity of reporting of harms is frequently inadequate
  • Few have sufficient sample sizes or duration to adequately assess uncommon or long-term harms

Slide 32

Trials to Assess Harms

• Most RCTs are "efficacy" trials
  • They assess benefits and harms in ideal, homogenous populations and settings
  • Patients who are more susceptible to harms are often under-represented
• Few RCTs directly compare alternative treatment strategies
• Publication bias and selective outcome reporting bias
• RCTs may not be available

Slide 33

Trials to Assess Harms

• Nevertheless, head-to-head RCTs provide the most direct evidence on comparative harms
• In addition, placebo-controlled RCTs can provide important information
• In general, CERs should routinely include both head-to-head and placebo-controlled trials for assessment of harms
  • In lieu of placebo-controlled RCTs, CERs may incorporate findings of well-conducted systematic reviews if they evaluated the specific harms of interest

Slide 34

Unpublished Supplemental Trials Data

• Consider including results of completed or terminated unpublished RCTs and unpublished results from published trials
  • FDA website, www.ClinicalTrials.gov , etc.
  • Must contemplate ability to fully assess risk of bias
• When significant # of published trials fails to report an important AE, CER authors should report this gap in the evidence and consider efforts to obtain unpublished data

Slide 35

OS to Assess Harms

• OS are almost always necessary to assess harms adequately
• Exception is when there are sufficient data from RCTs to reliably estimate harms
• May provide best or only data for assessing harms in minority or vulnerable populations who are under-represented in trials
• Types of OS included in a CER will vary; different types of OS might be included or rendered irrelevant by availability of data from stronger study types

Slide 36

Hypothesis Testing vs. Hypothesis Generating

• Important consideration in determining which OS to include
  • Case reports are hypothesis generating
  • Cohort and case-control studies are well suited for testing hypotheses of whether one intervention is associated with a greater risk for an adverse event than another and for quantifying the risk*

Slide 37

Hierarchy of Evidence

• Systematic Reviews
• RCTs
• Controlled Clinical Trials and Observational Studies
• Uncontrolled Observational Studies
• Case reports and case series
• Expert Opinions

Slide 38

OS to Assess Harms

• Cohort and case-control studies
  • CERs should routinely search for and include, except when RCT data are sufficient and valid
• OS based on patient registries
• OS based on analyses of large databases
• Case reports and post-marketing surveillance
  • New medications
• Other OS

Slide 39

OS to Assess Harms

• Criteria to select OS for inclusion
  • There are often many more OS than trials; evaluating a large number of OS can be impractical when conducting a CER
  • Several criteria commonly uses in CERs to screen OS for inclusion (empirical data lacking)
  • • Minimum duration of follow-up
    • Minimum sample size
    • Defined threshold for risk of bias
    • Study design (cohort and case-control)
    • Specific population of interest

Slide 40

Key Take-home Points

• Often insufficient evidence from trials to answer all Key Questions in CERs
• CERs should consider including OS *default strategy*
• Should explicitly state the rationale for including or excluding OS
• For OS to assess benefits, reviewers should consider 2 questions:
  • Are there gaps in trial evidence for the review questions under consideration?
  • Will observational studies provide valid and useful information to address key questions?
• For harms, should routinely search for and include cohort and case-control studies