A Professional Association Perspective on Systematic Reviews and Their Utility for Guideline Development

On September 19, 2011, Joel Yager made this presentation at the 2011 Annual Conference. Select to access the PowerPoint® presentation (225 KB). Plugin Software Help.

Slide 1

A Professional Association Perspective on Systematic Reviews and their Utility for Guideline Development

Joel Yager, M.D., Laura J. Fochtmann, M.D., M.S., Robert Kunkle, M.A., Robert M. Plovnick, M.D., M.S.

Executive Committee for Practice Guidelines
American Psychiatric Association

Slide 2

Disclosures

• The authors are with the Practice Guidelines Project of the American Psychiatric Association.
• Dr. Yager is Professor of Psychiatry, University of Colorado School of Medicine.
• Dr. Fochtmann is Professor of Psychiatry, School of Medicine, SUNY Stony Brook.
• Mr. Kunkle is Director, Practice Guidelines, APA.
• Dr. Plovnick is Director, Quality Care, APA.
• None of the authors has any financial disclosures pertinent to this presentation.

Slide 3

Available APA Guidelines

Slide 4

Major Themes

• Positive aspects of rigorous systematic reviews.
• Limitations, Challenges and Dangers of guidelines based on limited high quality evidence.
• Opportunities for AHRQ to help professional associations to develop trustworthy guidelines.

Slide 5

Positive Aspects of Rigorous Systematic Reviews

• Where robust evidence bases exit, systematic reviews are worth the effort and cost.
• Such reviews will help develop trustworthy guidelines in the following ways:

Slide 6

Rigorous Systematic Reviews Can Help

• Equilibrate ratings regarding strength of evidence and, in turn, strength of recommendations, across guidelines.
• Differentiate strong from weak recommendations.
• Determine treatment priorities.
• Reduce vague, underspecified recommendations.
• Reduce biases resulting from informal group dynamics, differential power status of "experts", and potential competing interests.
• Increase transparency regarding evidence-base to guideline users.
• Meet IOM standards.

Slide 7

But, let's face it...
relying primarily on systematic evidence-based reviews poses......

• Limitations.
• Challenges.
• Dangers.

Slide 8

Limitations of Relying Primarily on Systematic Evidence-Based Reviews for Guideline Development

• For many important clinical questions there's not much high quality evidence out there:
  • Example from eating disorders.

Slide 9

A Sidebar on Evidence-Based Medicine: Strength of Recommendations in Eating Disorders Practice Guideline

APA Guideline Recommendation

• Level [I]: 118.
• Level [II]: 57.
• Level [III]: 21.

British NICE Recommendation

• Grade A: 3.
• Grade B: 13.
• Grade C: 85.

Slide 10

National Institute for Clinical Excellence (NICE) Recommendations

1.1.1.1 Assessment of people with eating disorders should be comprehensive and include physical, psychological and social needs, and a comprehensive assessment of risk to self.

Slide 11

Parachute use to prevent death and major trauma related to gravitational challenge: systematic review of randomized controlled trials

(BMJ 327: 1459 Published 18 December 2003)

Image: A photograph of parachuters is shown below the text.

Slide 12

Limitations

• The more precise the PICO-TS question with regard to patient population characteristics, comparative treatments, and meaningful outcomes, the less high quality evidence is available:
  • Think co-occurring medical and psychiatric comorbidities, patient preferences, prior longitudinal illness course, and treatment experience.

Slide 13

Evidence for PICO-TS questions?

• For patients with paranoid schizophrenia who have developed tardive dyskinesia on first generation antipsychotic medications and who suffer from obesity and type II diabetes mellitus, which of the following treatments (clozapine vs. aripiprazole vs. olanzapine vs. perphenazine) will result in the most desirable outcomes with respect to psychiatric symptoms, psychiatric impairment, medical morbidity and mortality?
• For currently depressed patients with bipolar disorder type I, with co-occurring alcohol and marijuana dependence, which of the following combinations of medications and psychosocial interventions will result in the most desirable outcomes with respect to psychiatric symptoms, psychiatric impairment, medical morbidity and mortality?

Slide 14

Limitations

• When you factor in patient/family preferences and access to/costs of care, don't expect to find much.
• There will always be lots of wiggle room in framing assumptions and interpreting results.
• Patients and practitioners seek clinical guidance for complex questions where randomized controlled trials are unlikely to ever be conducted. Are we supposed to ignore those needs?

Slide 15

Limitations

• Common clinical questions may vary depending upon the level of generalist/specialist/subspecialist using the guidelines.
• AHRQ funded EBM reviews may not address situations where lack of access limits clinicians' abilities to offer best practices.

Slide 16

Limitations (continued)

• AHRQ reports may focus on particular treatments (e.g. second generation antipsychotics) rather than on comparisons needed for choosing a treatment approach best suited for an individual patient.

Slide 17

Challenges of Relying Primarily on Systematic Evidence-Based Reviews for Guideline Development

• AHRQ evidence reports may cover only a fraction of a professional societies' guideline development efforts. Do we have "trustworthy" guidelines and "good enough" guidelines?
• How are professional associations expected to pay for guideline development processes where AHRQ is not funding an EBC effort and/or where high quality evidence is not available?

Slide 18

Challenges:
Determining When the Juice is Worth the Squeeze.

• What degree of pre-screening should be conducted and what type of evidence should be required before an EBC or PA heads off on detailed review?
• For example, should a minimum of two high quality RCTs that specifically address a PICO-TS question be required before embarking on a detailed search?
• How closely to the precise PICO questions should these studies be expected to adhere?

Slide 19

Dangers of Relying Primarily on Systematic Evidence-Based Reviews for Guideline Development

• Systematic reviews may ignore or minimize important systematic biases, e.g. clinical trials design biases.
• The IOM Systematic Review recommendations are admittedly riddled with strong recommendations for which there is no evidence base! A cynic might think that COI was at play.

Slide 20

Dangers of Relying Primarily on Systematic Evidence-Based Reviews for Guideline Development

• "Without an assessment of hard, irrefutable measures of clinical decision-making that include individual preferences for treatment, decisions about the appropriateness of clinical treatments and variations of care cannot be made."

Livingston EH, McNutt RA. The Hazards of Evidence-Based Medicine: Assessing Variations in Care. JAMA 2011;306(7):762-763. doi: 10.1001/jama.2011.1181.

Slide 21

Opportunities for AHRQ to Help Professional Associations Developing Practice Guidelines

• Establish and disseminate criteria for trustworthy pre-screening searches to determine potential value of full-scale systematic reviews.
• Fund grants to professional associations and other guideline developing groups for conducting pre-screening searches.

Slide 22

Opportunities for AHRQ to Help Professional Associations Developing Practice Guidelines (continued)

• When extensive systematic reviews are conducted:
  • Organize and distribute evidence tables to enable guideline developers to use them to address clinically different questions than those for which they were initially created.
  • Require EBC evidence tables to link to the PubMed abstracts for all related RCTs.
  • Require EBC evidence tables to contain clinically relevant calculations, effect sizes, NNT, NNH, as well as proportion of individuals experiencing particular adverse effects and global outcomes.

Slide 23

Opportunities for AHRQ to help Professional Associations Developing Practice Guidelines

• More broadly:
  • Work with other agencies to require investigators to post data in a manner suitable for incorporation into formal medical decision analyses.
  • Work with other agencies to require investigators to post individual subject or group descriptive data (means, standard deviations) split according to key factors such as age, sex, race/ethnicity even when the numbers are too small to analyze otherwise.

Slide 24

Opportunities for AHRQ to help Professional Associations and Other Groups Developing Practice Guidelines

• For circumstances when only low-quality (or no) data is available:
  • Help develop acceptable standardized methods for acquiring expert consensus.
  • Help develop policies and procedures authorizing the development of trustworthy guidelines based on available evidence plus expert consensus.

Slide 25

Thanks for Listening

joel.yager@ucdenver.edu

Current as of December 2011

Internet Citation:

A Professional Association Perspective on Systematic Reviews and Their Utility for Guideline Development. Slide Presentation from the AHRQ 2011 Annual Conference (Text Version). December 2011. Agency for Healthcare Research and Quality, Rockville, MD. http://www.ahrq.gov/about/annualconf11/crystal_tyutyulkova_yager/yager.htm