Searching Grey Literature: Is the Effort Worth the Yield? (Text Version)

Slide presentation from the AHRQ 2011 conference.

On September 20, 2011, Marian McDonagh made this presentation at the 2011 Annual Conference. Select to access the PowerPoint® presentation (520 KB). Plugin Software Help.


Slide 1

 Searching Grey Literature: Is the Effort Worth the Yield?

Searching Grey Literature: Is the Effort Worth the Yield?
AHRQ Annual Meeting Track C, Session 105
Methods for Synthesizing Evidence

Marian McDonagh, PharmD
Evidence-based Practice Center

Oregon Health & Science University

Slide 2

 The Problem:

The Problem:
 

  • Searching for and including grey literature adds substantial workload to a systematic review.
  • The benefit is unclear in terms of value for time spent.
  • Information from grey literature may be limited and risk of bias cannot be determined.

Slide 3

 Session Purpose and Goals

Session Purpose and Goals

  • This session will explore:
    • The methodological argument in favor of searching for grey literature.
    • Different approaches to searching for grey literature with highest yield.
    • The usefulness of including grey literature based on real-world examples.

Slide 4

Discussion Leaders  

Discussion Leaders

  • Dr. David Moher:
    • Ottawa Hospital Research Institute.
    • University of Ottawa.
    • Ottawa Evidence-based Practice Center.
  • Rose Relevo, MLIS, MS:
    • Oregon Evidence-based Practice Center.
    • Oregon Health & Science University (OHSU).
  • Marian McDonagh, PharmD:
    • Oregon Evidence-based Practice Center.
    • Oregon Health & Science University.

Slide 5

 Usefulness of Grey Literature

Usefulness of Grey Literature

  • What have we discussed so far today:
    • Evidence that failure to include grey literature can lead to biased result (Dr. Moher).
      • Methods to identify grey literature for comparative effectiveness reviews (Ms. Relevo).
  • Interactive Session: What's it like in practice?
    • Examples from the Drug Effectiveness Review Project use of Food and Drug Administration (FDA) materials as grey literature.

Slide 6

 Drug Effectiveness Review Project (DERP)

Drug Effectiveness Review Project (DERP)

  • DERP is a collaboration of primarily state Medicaid agencies that have joined together to commission systematic reviews of the comparative effectiveness and harms of drugs in many widely used drug classes, and to apply the findings to inform public policy in local settings.
  • Project began in 2003:
    • OHSU Center for Evidence-based Policy coordinates.
    • EPCs at Oregon and University of North Carolina produce systematic reviews for the project.

Slide 7

 DERP Organizations

DERP Organizations

DERP I (15)
2003-2006
DERP II (17)
2006-2009
DERP III (12)
2009—2012
Alaska
Arkansas
CADTH*
California HCF
Idaho
Kansas
Michigan
Minnesota
Missouri
Montana
North Carolina
Oregon
Washington
Wisconsin
Wyoming
Alaska
Arkansas
California
California HCF
CADTH*
Idaho
Kansas
Michigan
Minnesota
Missouri
Montana
New York
North Carolina
Oregon
Washington
Wisconsin
Wyoming
Arkansas
CADTH*
Colorado
Idaho
Maryland
Missouri
Montana
New York
Oregon
Washington
Wisconsin
Wyoming

*Canadian Agency for Drugs and Technologies in Health

Slide 8

114 Reports Completed to date

114 Reports Completed to date

  • Proton Pump Inhibitors.
  • Long-acting Opioids.
  • Statins.
  • Non-steroidal Anti-Inflammatory Drugs (NSAIDs).
  • Estrogens.
  • Triptans.
  • Skeletal Muscle Relaxants.
  • Oral Hypoglycemics.
  • Drugs for Overactive Bladder.
  • ACE Inhibitors.
  • Beta Blockers.
  • Calcium Channel Blockers.
  • Angiotensin II Receptor Antagonists.
  • 2nd Generation Antidepressants.
  • Drugs for Constipation.
  • Direct Renin Inhibitors/ACEI/AIIRA.
  • Fibromyalgia, Drugs to treat.
  • Drugs for Neuropathic Pain.
  • Antiepileptic Drugs in Bipolar Disorder/Pain.
  • 2nd Generation Antihistamines.
  • Atypical Antipsychotics.
  • Inhaled Corticosteroids.
  • ADHD and ADD, Drugs to treat.
  • Alzheimer's, Drugs to treat.
  • Antiplatelet Drugs.
  • Thiazolidinediones (glitazones).
  • Drugs for Hepatitis C.
  • Newer Drugs for Insomnia.
  • Targeted Immune Modulators.
  • Beta Agonists.
  • Newer Anti-emetics.
  • Drugs for Multiple Sclerosis.
  • Combination Products.
  • Controller Drugs for Asthma.
  • Newer Drugs for Diabetes.
  • Topical Calcineurin Inhibitors.

Slide 9

DERP Methods in Using FDA Documents

DERP Methods in Using FDA Documents

  • Systematic review methodology for DERP includes searching FDA Web site:
    • FDA Medical and Statistical reviews of NDAs:
      • Documents available on FDA Web site > 1997.
      • Approved drugs, newly approved for updates.
      • Include studies if enough information to rate quality.
  • What we are looking for:
    • Unique unpublished studies.
    • Unpublished data related to published studies.

Slide 10

Workload?

Workload?

  • Number of documents reviewed varies:
    • Range 0 to 9 FDA documents per report (mean 1.5).
  • Document Size and Organization Issues:
    • Several hundred pages each.
    • Organization varies.
    • Historically pdfs of scanned originals.
  • Matching published and unpublished studies:
    • Review after electronic searching and selection.
  • Dual review: eligibility, quality, abstraction.

Slide 11

DERP Studying Experience with FDA Documents

DERP Studying Experience with FDA Documents

  • We are analyzing DERP reports for:
    • Frequency of information found in FDA documents being included in a report.
    • Value of information from FDA documents included.
    • Value is being assessed by assigning an impact category to each instance, e.g., fills a gap in evidence.
  • 175 FDA documents eligible for review in 114 reports over 9 years.
    • 48 instances of evidence from FDA documents included (27%).

Slide 12

Example 1: Unique unpublished study

Example 1: Unique unpublished study

  • Atypical Antipsychotic Drugs (2004):
    • Head to head comparisons of primary relevance to review.
    • No eligible published studies of aripiprazole (Abilify®) identified through usual searches and request for manufacturer information.
    • Abilify® FDA document found:
      • > 200 hundred pages, Examined 5 trials.
      • Identified one trial head to head trial (N = 255).

Slide 13

Your Thoughts?

Your thoughts?

If you came across this evidence what would do with it?

  1. Nothing. Don't use grey literature.
  2. Include in report qualitatively only.
  3. Include in report qualitatively, but influences publication bias assessment.
  4. Fully include, incorporate into meta-analysis, adjust strength of evidence.

Slide 14

Your Thoughts?

Your Thoughts?

If you had included this information, how do you think it would have changed the SOE?

  1. Yes.
  2. No.
  3. Unsure.

Slide 15

Your Thoughts?

Your Thoughts?

Do you believe the added information was valuable to the review's conclusions?

  1. Yes.
  2. No.
  3. Unsure.

Slide 16

Your Thoughts?

Your thoughts?

Do you think that the time and resources spent to obtain this information was worthwhile?

  1. Yes.
  2. No.
  3. Unsure.

Slide 17

Discussion

Discussion

  • Panelists Thoughts?
  • What we did in DERP:
    • Included study in report; changed conclusions only in a minor way:
      • Secondary outcome measure used (cognitive function).
      • Aripiprazole superior to olanzapine on some, but not all measures.
  • Further discussion?

Slide 18

Example 2: Unpublished data

Example 2: Unpublished data

  • Drugs for Insomnia, Update 2 (2008).
  • Zolpidem ER (Ambien CR®) new:
    • 3 published placebo-controlled trials identified.
    • 5 placebo-controlled trials in FDA documents:
      • Matched 2 of the published trials.
  • FDA docs: Primary outcome at 8 hours:
    • Ambien superior to placebo up to 6 hours, not different at hours 7 and 8.
    • Publication: reports only 6-hour results.

Slide 19

Your Thoughts?

Your thoughts?

If you came across this evidence what would do with it?

  1. Nothing. Don't use grey literature.
  2. Include in report qualitatively only.
  3. Include in report qualitatively, but influences publication bias assessment.
  4. Fully include, incorporate into meta-analysis, adjust strength of evidence.

Slide 20

Your Thoughts?

Your Thoughts?

If you had included this information, how do you think it would have changed the SOE?

  1. Yes.
  2. No.
  3. Unsure.

Slide 21

Your Thoughts?

Your Thoughts?

Do you believe the added information was valuable to the review's conclusions?

  1. Yes.
  2. No.
  3. Unsure.

Slide 22

Your Thoughts?

Your thoughts?

Do you think that the time and resources spent to obtain this information was worthwhile?

  1. Yes.
  2. No.
  3. Unsure.

Slide 23

Discussion

Discussion

  • Panelists Thoughts?
  • What we did in DERP:
    • Included results from FDA docs—changed conclusions of report on Ambien CR®.
  • Further discussion?

Slide 24

Example 3: Unique unpublished study

Example 3: Unique unpublished study

  • Proton Pump Inhibitors, Update 2 (2004).
  • Esomeprazole (Nexium®) vs omeprazole:
    • FDA documents: 4 trials submitted prior to the 2001 approval of Nexium®.
    • 2 published studies identified:
      • Both found esomeprazole superior to omeprazole.
    • 2 unpublished studies (in FDA docs) found no statistically significant difference between the drugs

Slide 25

Your Thoughts?  

Your thoughts?

If you came across this evidence what would do with it?

  1. Nothing. Don't use grey literature.
  2. Include in report qualitatively only.
  3. Include in report qualitatively, but influences publication bias assessment.
  4. Fully include, incorporate into meta-analysis, adjust strength of evidence.

Slide 26

 Your Thoughts?

Your Thoughts?

If you had included this information, how do you think it would have changed the SOE?

  1. Yes.
  2. No.
  3. Unsure.

Slide 27

 Your Thoughts?

Your Thoughts?

Do you believe the added information was valuable to the review's conclusions?

  1. Yes.
  2. No.
  3. Unsure.

Slide 28

 Your Thoughts?

Your thoughts?

Do you think that the time and resources spent to obtain this information was worthwhile?

  1. Yes.
  2. No.
  3. Unsure.

Slide 29

 Discussion

Discussion

  • Panelists Thoughts?
  • What we did in DERP:
    • Included study in report only qualitatively.
    • Inadequate information in FDA docs to evaluate quality.
    • Subsequent update: obtained data from manufacturer and incorporated into meta-analysis.
  • Further discussion?

Slide 30

 Wrap up

Wrap up

  • The Problem Revisited:
    • Searching for and including grey literature adds substantial workload to a systematic review.
    • The benefit is unclear in terms of value for time spent.
    • Information from grey literature may be limited and risk of bias cannot be determined.
Current as of March 2012
Internet Citation: Searching Grey Literature: Is the Effort Worth the Yield? (Text Version). March 2012. Agency for Healthcare Research and Quality, Rockville, MD. http://www.ahrq.gov/news/events/conference/2011/mcdonagh/index.html