Searching Grey Literature: Is the Effort Worth the Yield? (Text Versio

Slide Presentation from the AHRQ 2011 Annual Conference

Searching Grey Literature: Is the Effort Worth the Yield?

Slide Presentation from the AHRQ 2011 Annual Conference

On September 20, 2011, Marian McDonagh made this presentation at the 2011 Annual Conference. Select to access the PowerPoint® presentation (520 KB). Plugin Software Help.

Slide 1

Searching Grey Literature: Is the Effort Worth the Yield?
AHRQ Annual Meeting Track C, Session 105
Methods for Synthesizing Evidence

Marian McDonagh, PharmD
Evidence-based Practice Center

Slide 2

The Problem:

• Searching for and including grey literature adds substantial workload to a systematic review.
• The benefit is unclear in terms of value for time spent.
• Information from grey literature may be limited and risk of bias cannot be determined.

Slide 3

Session Purpose and Goals

• This session will explore:
  • The methodological argument in favor of searching for grey literature.
  • Different approaches to searching for grey literature with highest yield.
  • The usefulness of including grey literature based on real-world examples.

Slide 4

Discussion Leaders

• Dr. David Moher:
  • Ottawa Hospital Research Institute.
  • University of Ottawa.
  • Ottawa Evidence-based Practice Center.
• Rose Relevo, MLIS, MS:
  • Oregon Evidence-based Practice Center.
  • Oregon Health & Science University (OHSU).
• Marian McDonagh, PharmD:
  • Oregon Evidence-based Practice Center.
  • Oregon Health & Science University.

Slide 5

Usefulness of Grey Literature

• What have we discussed so far today:
  • Evidence that failure to include grey literature can lead to biased result (Dr. Moher).
    • Methods to identify grey literature for comparative effectiveness reviews (Ms. Relevo).
• Interactive Session: What's it like in practice?
  • Examples from the Drug Effectiveness Review Project use of Food and Drug Administration (FDA) materials as grey literature.

Slide 6

Drug Effectiveness Review Project (DERP)

• DERP is a collaboration of primarily state Medicaid agencies that have joined together to commission systematic reviews of the comparative effectiveness and harms of drugs in many widely used drug classes, and to apply the findings to inform public policy in local settings.
• Project began in 2003:
  • OHSU Center for Evidence-based Policy coordinates.
  • EPCs at Oregon and University of North Carolina produce systematic reviews for the project.

Slide 7

DERP Organizations

*Canadian Agency for Drugs and Technologies in Health

Slide 8

114 Reports Completed to date

• Proton Pump Inhibitors.
• Long-acting Opioids.
• Statins.
• Non-steroidal Anti-Inflammatory Drugs (NSAIDs).
• Estrogens.
• Triptans.
• Skeletal Muscle Relaxants.
• Oral Hypoglycemics.
• Drugs for Overactive Bladder.
• ACE Inhibitors.
• Beta Blockers.
• Calcium Channel Blockers.
• Angiotensin II Receptor Antagonists.
• 2nd Generation Antidepressants.
• Drugs for Constipation.
• Direct Renin Inhibitors/ACEI/AIIRA.
• Fibromyalgia, Drugs to treat.
• Drugs for Neuropathic Pain.
• Antiepileptic Drugs in Bipolar Disorder/Pain.
• 2nd Generation Antihistamines.
• Atypical Antipsychotics.
• Inhaled Corticosteroids.
• ADHD and ADD, Drugs to treat.
• Alzheimer's, Drugs to treat.
• Antiplatelet Drugs.
• Thiazolidinediones (glitazones).
• Drugs for Hepatitis C.
• Newer Drugs for Insomnia.
• Targeted Immune Modulators.
• Beta Agonists.
• Newer Anti-emetics.
• Drugs for Multiple Sclerosis.
• Combination Products.
• Controller Drugs for Asthma.
• Newer Drugs for Diabetes.
• Topical Calcineurin Inhibitors.

Slide 9

DERP Methods in Using FDA Documents

• Systematic review methodology for DERP includes searching FDA Web site:
  • FDA Medical and Statistical reviews of NDAs:
    • Documents available on FDA Web site > 1997.
    • Approved drugs, newly approved for updates.
    • Include studies if enough information to rate quality.
• What we are looking for:
  • Unique unpublished studies.
  • Unpublished data related to published studies.

Slide 10

Workload?

• Number of documents reviewed varies:
  • Range 0 to 9 FDA documents per report (mean 1.5).
• Document Size and Organization Issues:
  • Several hundred pages each.
  • Organization varies.
  • Historically pdfs of scanned originals.
• Matching published and unpublished studies:
  • Review after electronic searching and selection.
• Dual review: eligibility, quality, abstraction.

Slide 11

DERP Studying Experience with FDA Documents

• We are analyzing DERP reports for:
  • Frequency of information found in FDA documents being included in a report.
  • Value of information from FDA documents included.
  • Value is being assessed by assigning an impact category to each instance, e.g., fills a gap in evidence.
• 175 FDA documents eligible for review in 114 reports over 9 years.
  • 48 instances of evidence from FDA documents included (27%).

Slide 12

Example 1: Unique unpublished study

• Atypical Antipsychotic Drugs (2004):
  • Head to head comparisons of primary relevance to review.
  • No eligible published studies of aripiprazole (Abilify®) identified through usual searches and request for manufacturer information.
  • Abilify® FDA document found:
    • > 200 hundred pages, Examined 5 trials.
    • Identified one trial head to head trial (N = 255).

Slide 13

Your thoughts?

• If you came across this evidence what would do with it?
• Nothing. Don't use grey literature.
• Include in report qualitatively only.
• Include in report qualitatively, but influences publication bias assessment.
• Fully include, incorporate into meta-analysis, adjust strength of evidence.

Slide 14

Your Thoughts?

If you had included this information, how do you think it would have changed the SOE?

1. Yes.
2. No.
3. Unsure.

Slide 15

Your Thoughts?

Do you believe the added information was valuable to the review's conclusions?

1. Yes.
2. No.
3. Unsure.

Slide 16

Your thoughts?

Do you think that the time and resources spent to obtain this information was worthwhile?

1. Yes.
2. No.
3. Unsure.

Slide 17

Discussion

• Panelists Thoughts?
• What we did in DERP:
  • Included study in report; changed conclusions only in a minor way:
    • Secondary outcome measure used (cognitive function).
    • Aripiprazole superior to olanzapine on some, but not all measures.
• Further discussion?

Slide 18

Example 2: Unpublished data

• Drugs for Insomnia, Update 2 (2008).
• Zolpidem ER (Ambien CR®) new:
  • 3 published placebo-controlled trials identified.
  • 5 placebo-controlled trials in FDA documents:
    • Matched 2 of the published trials.
• FDA docs: Primary outcome at 8 hours:
  • Ambien superior to placebo up to 6 hours, not different at hours 7 and 8.
  • Publication: reports only 6-hour results.

Slide 19

Your thoughts?

If you came across this evidence what would do with it?

1. Nothing. Don't use grey literature.
2. Include in report qualitatively only.
3. Include in report qualitatively, but influences publication bias assessment.
4. Fully include, incorporate into meta-analysis, adjust strength of evidence.

Slide 20

Your Thoughts?

If you had included this information, how do you think it would have changed the SOE?

1. Yes.
2. No.
3. Unsure.

Slide 21

Your Thoughts?

Do you believe the added information was valuable to the review's conclusions?

1. Yes.
2. No.
3. Unsure.

Slide 22

Your thoughts?

Do you think that the time and resources spent to obtain this information was worthwhile?

1. Yes.
2. No.
3. Unsure.

Slide 23

Discussion

• Panelists Thoughts?
• What we did in DERP:
  • Included results from FDA docs—changed conclusions of report on Ambien CR®.
• Further discussion?

Slide 24

Example 3: Unique unpublished study

• Proton Pump Inhibitors, Update 2 (2004).
• Esomeprazole (Nexium®) vs omeprazole:
  • FDA documents: 4 trials submitted prior to the 2001 approval of Nexium®.
  • 2 published studies identified:
    • Both found esomeprazole superior to omeprazole.
  • 2 unpublished studies (in FDA docs) found no statistically significant difference between the drugs

Slide 25

Your thoughts?

If you came across this evidence what would do with it?

1. Nothing. Don't use grey literature.
2. Include in report qualitatively only.
3. Include in report qualitatively, but influences publication bias assessment.
4. Fully include, incorporate into meta-analysis, adjust strength of evidence.

Slide 26

Your Thoughts?

If you had included this information, how do you think it would have changed the SOE?

1. Yes.
2. No.
3. Unsure.

Slide 27

Your Thoughts?

Do you believe the added information was valuable to the review's conclusions?

1. Yes.
2. No.
3. Unsure.

Slide 28

Your thoughts?

Do you think that the time and resources spent to obtain this information was worthwhile?

1. Yes.
2. No.
3. Unsure.

Slide 29

Discussion

• Panelists Thoughts?
• What we did in DERP:
  • Included study in report only qualitatively.
  • Inadequate information in FDA docs to evaluate quality.
  • Subsequent update: obtained data from manufacturer and incorporated into meta-analysis.
• Further discussion?

Slide 30

Wrap up

• The Problem Revisited:
  • Searching for and including grey literature adds substantial workload to a systematic review.
  • The benefit is unclear in terms of value for time spent.
  • Information from grey literature may be limited and risk of bias cannot be determined.

Current as of December 2011

Internet Citation:

Searching Grey Literature: Is the Effort Worth the Yield? Slide Presentation from the AHRQ 2011 Annual Conference (Text Version). December 2011. Agency for Healthcare Research and Quality, Rockville, MD. http://www.ahrq.gov/about/annualconf11/mcdonagh_moher_relevo/mcdonagh.htm