Rating the Evidence: Using GRADE to Develop Clinical Practice Guidelines (Text Version)

Slide presentation from the AHRQ 2009 conference.

On September 14, 2009, Yngve Falck-Ytter and Holger Schüenemann made this presentation at the 2009 Annual Conference. Select to access the PowerPoint® presentation (4.6 MB). Plugin Software Help.


Slide 1

Slide  1. Rating the Evidence: Using GRADE to Develop Clinical Practice Guidelines.

Rating the Evidence: Using GRADE to Develop Clinical Practice Guidelines

AHRQ Annual Meeting 2009:
"Research to Reform: Achieving Health System Change"

September 14, 2009

Yngve Falck-Ytter, M.D.
Case Western Reserve University, Cleveland, Ohio

Holger Schüenemann, M.D., Ph.D.
Chair, Department of Clinical Epidemiology & Biostatistics
Michael Gent Chair in Healthcare Research
McMaster University, Hamilton, Canada

 

Slide 2

Slide  2. Disclosure

Disclosure

In the past 5 years, Dr. Falck-Ytter received no personal payments for services from industry. His research group received research grants from Three Rivers, Valeant and Roche that were deposited into non-profit research accounts. He is a member of the GRADE working group which has received funding from various governmental entities in the US and Europe. Some of the GRADE work he has done is supported in part by grant # 1 R13 HS016880-01 from the Agency for Healthcare Research and Quality (AHRQ).

 

Slide 3

Slide  3. Content

Content

  • Part 1
    • Introduction
  • Part 2
    • Why revisiting guideline methodology?
  • Part 3
    • The GRADE approach
      • Quality of evidence
  • Part 4
    • The GRADE approach
      • Strength of recommendations

 

Slide 4

Slide  4. Q to audience

Q to audience

  • Involved in giving recommendations?
    • Using any form of grading system?
  • Familiarity with GRADE:
    • Heard about GRADE before this conference?
    • Read a GRADE article published by the GRADE working group?
    • Attended a GRADE presentation?
    • Attended a hands-on GRADE workshop?

 

Slide 5

Slide  5. Reassessment of clinical practice guidelines

Reassessment of clinical practice guidelines

  • Editorial by Shaneyfelt and Centor (JAMA 2009)
    • "Too many current guidelines have become marketing and opinion-based pieces."
    • "AHA CPG: 48% of recommendations are based on level C = expert opinion."
    • ".clinicians do not use CPG [.] greater concern [.] some CPG are turned into performance measures."
    • "Time has come for CPG development to again be centralized, e.g., AHQR."

 

Slide 6

Slide  6. Evidence-based clinical decisions

Evidence-based clinical decisions

Diagram of three interlocking circles representing:

  • Clinical state and circumstances
  • Patient values and preferences
  • Research evidence

The word "Expertise" is superimposed over the circles and "Equal for all" is below it.

Haynes et al. 2002

 

Slide 7

Slide  7. Before GRADE

Before GRADE

Level of evidenceSource of evidenceGrades of recommend.
ISR, RCTsA
IICohort studiesB
IIICase-control studies
IVCase seriesC
VExpert opinionD

Arrows point from cell "I" to cells "A" and "B."

Oxford Centre of Evidence Based Medicine; http://www.cebm.net

 

Slide 8

Slide  8. Where GRADE fits in

Where GRADE fits in

Prioritize problems, establish panel
Systematic review
Searches, selection of studies, data collection and analysis
 

 

 
Assess the relative importance of outcomes
Prepare evidence profile: Quality of evidence for each outcome and summary of findings
Assess overall quality of evidence
Decide direction and strength of recommendation
}GRADE

 

 
Draft guideline
Consult with stakeholders and / or external peer reviewer
Disseminate guideline
Implement the guideline and evaluate
 

 

Slide 9

Slide  9. GRADE uptake

GRADE uptake

A collage of many different logos.

 

Slide 10

Slide  10. GRADE - Why revisiting guideline methodology?

GRADE—
Why revisiting guideline methodology?
 

 

Slide 11

Slide  11. Disclosure

Disclosure

Dr. Schüenemann receives no personal payments for service from the pharmaceutical industry. The research group he belongs to received research grants from the industry that are deposited into research accounts.
Institutions or organizations that he is affiliated with likely receive funding from for-profit sponsors that are supporting infrastructure and research that may serve his work.
He is documents editor for the American Thoracic Society and co-chair of the GRADE Working Group.

 

Slide 12

Slide  12. Content

Content

  • Why grading
  • Confidence in information and recommendations
  • Intro to:
    • Quality of evidence
    • Strength of recommendations

 

Slide 13

Slide  13. Please discuss the difference between consensus statements and guidelines?

Please discuss the difference between consensus statements and guidelines?

Be prepared to discuss your answer

 

Slide 14

Slide  14. There are no RCTs!

There are no RCTs!

  • Do you think that users of recommendations would like to be informed about the basis (explanation) for a recommendation or coverage decision if they were asked (by their patients)?
  • I suspect the answer is "yes"
  • If we need to provide the basis for recommendations, we need to say whether the evidence is good or not so good; in other words perhaps "no RCTs"

 

Slide 15

Slide  15. Hierarchy of evidence

Hierarchy of evidence

  • STUDY DESIGN
    • Randomized Controlled Trials
    • Cohort Studies and Case Control Studies
    • Case Reports and Case Series, Non-systematic observations

A diagram of a pyramid with "BIAS" written at the top and "Expert Opinion" at the bottom.

 

Slide 16

Slide  16. Confidence in evidence

Confidence in evidence

  • There always is evidence
    • "When there is a question there is evidence"
  • Better research >> greater confidence in the evidence and decisions

 

Slide 17

Slide  17. Who can explain the following?

Who can explain the following?

  • Concealment of randomization
  • Bias, confounding and effect modification
  • Blinding (who is blinded in a double blinded trial?)
  • Intention to treat analysis and its correct application
  • Why trials stopped early for benefit overestimate treatment effects?
  • P-values and confidence intervals

 

Slide 18

Slide  18. Hierarchy of evidence

Hierarchy of evidence

  • STUDY DESIGN
    • Randomized Controlled Trials
    • Cohort Studies and Case Control Studies
    • Case Reports and Case Series, Non-systematic observations
    • Expert Opinion

A diagram of a pyramid with "BIAS" written at the top and "Expert Opinion" at the bottom.

 

Slide 19

Slide  19. Reasons for grading evidence?

Reasons for grading evidence?

  • Appraisal of evidence has become complex and daunting
  • People draw conclusions about the
    • Quality of evidence and strength of recommendations
  • Systematic and explicit approaches can help
    • Protect against errors, resolve disagreements
    • Communicate information and fulfil needs
  • Change practitioner behavior
  • However, wide variation in approaches

GRADE working group. BMJ. 2004 & 2008

 

Slide 20

Slide  20. Which grading system?

Which grading system?

Recommendation for use of oral anticoagulation in patients with atrial fibrillation and rheumatic mitral valve disease
 

EvidenceRecommendationOrganization
BClass IAHA
A1ACCP
IVCSIGN

 

Slide 21

Slide  21. What to do?

What to do?

Graphic image of a doctor and a photo of a large grouping of traffic signal lights.

 

Slide 22

Slide  22. Recommendations vs statements!

Recommendations vs statements!

Other options are available but not recommended for routine use as initial or first-line controllers in Step 2. Sustained-release theophylline has only weak anti-inflammatory and controller efficacy 126-130 (Evidence B) and is commonly associated with side effects that range from trivial to intolerable 131-132. Cromones (nedocromil sodium and sodium cromoglycate) have comparatively low efficacy, though a favorable safety profile 133-136 (Evidence A).

 

Slide 23

Slide  23. Limitations of older systems and approaches

Limitations of older systems & approaches

  • Confuse quality of evidence with strength of recommendations

 

Slide 24

Slide  24. Levels of evidence

Levels of evidence

Level of evidenceType of evidence
1++High-quality meta-analyses, systematic reviews of RCTs, or RCTs with a very low risk of bias
1+Well-conducted meta-analyses, systematic reviews of RCTs, or RCTs with a low risk of bias
1-Meta-analyses, systematic reviews of RCTs, or RCTs with a high risk of bias
2++High-quality systematic reviews of case-control or cohort studies

high-quality case-control or cohort studies with a very low risk of confounding, bias, or chance and a high probability that the relationship is causal
2+Well-conducted case-control or cohort studies with a low risk of confounding, bias, or change and a moderate probability that the relationship is causal
2-Case-control or cohort studies with a high risk of confounding bias, or chance and a significant risk that the relationshiop is not causal
3Non-analytic studies (for example, case reports, case series)
4Expert opinion

 

Slide 25

Slide  25. Recommendations

Recommendations

GradeEvidence
A
  • At least one meta-analysis, systematic review, or RCT rated as 1++, and directly applicable to the target population, or
  • A systematic review of RCTs or a body of evidence consisting principally of studies rated as 1+, directly applicable to the target population, and demonstrating overall consistency of results
  • Evidence drawn from a NICE technology appraisal
B
  • A body of evidence including studies rated as 2++, directly applicable to the target population, and demonstrating overall consistency of results, or
  • Extrapolated evidence from studies rated as 1++ or 1+
C
  • A body of evidence including studies rated as 2+, directly applicable to the target population and demonstrating overall consistency of results, or
  • Extrapolated evidence from studies rated as 2++
D
  • Evidence level 3 or 4, or
  • Extrapolated evidence from studies rated as 2+, or
  • Formal consensus
D (GPP)
  • A good practice point (GPP) is a recommendation for best practice based on the experience of the Guideline Development Group

 

Slide 26

Slide  26. Limitations of older systems and approaches

Limitations of older systems & approaches

  • Confuse quality of evidence with strength of recommendations
  • Lack well-articulated conceptual framework
  • Criteria not comprehensive or transparent
  • Focus on single outcomes

 

Slide 27

Slide  27. GRADE Quality of Evidence

GRADE Quality of Evidence

  • In the context of a systematic review
    • The quality of evidence reflects the extent to which we are confident that an estimate of effect is correct.
  • In the context of making recommendations
    • The quality of evidence reflects the extent to which our confidence in an estimate of the effect is adequate to support a particular recommendation.

 

Slide 28

Slide  28. What makes you confident in health care decisions?

What makes you confident in health care decisions

 

Slide 29

Slide  29. Confident in the evidence?

Confident in the evidence?

A meta-analysis of observational studies showed that bicycle helmets reduce the risk of head injuries in cyclists.
OR: 0.31, 95%CI: 0.26 to 0.37

A meta-analysis of observational studies showed that warfarin prophylaxis reduces the risk of thromboembolism in patients with cardiac valve replacement.
RR: 0.17, 95%CI: 0.13 to 0.24

 

Slide 30

Slide  30. This slide is blank.

No titles

 

Slide 31

Slide  31. GRADE: Quality of evidence

GRADE: Quality of evidence

  • The extent to which our confidence
    in an estimate of the treatment effect
    is adequate to support a particular recommendation.
  • GRADE defines 4 categories of quality:
    • High
    • Moderate
    • Low
    • Very low

 

Slide 32

Slide  32. Quality of evidence across studies

Quality of evidence across studies

Diagram of several studies showing quality outcomes of high, moderate and low.

 

Slide 33

Slide  33. Determinants of quality

Determinants of quality

  • RCTs start high
  • Observational studies start low

 

Slide 34

Slide  34. What is the study design?

What is the study design?

1: Rev Esp Enferm Dig. 1998 Nov;90(11):788-93

Surgical treatment of the acute cholecystitis in the laparoscopic age. A comparative study: laparoscopic against laparatomy.

[Article in English, Spanish]

Carbajo Caballero MA, Martin del Olmo JC, Blanco Alvarez JI, Cuesta de la Llave C. Atienza Sanchez R, Inglada Daliana L, Vaquero Puerta C.

Department of Surgery, Medina del Campo Hospital, Valladolid, Spain.

OBJECTIVE: The aim of this study was to assess the complications and results of the laparoscopic opposite to open treatment of the acute choecystitis. METHODS: A retrospective randomized study with two groups of 30 patients each one. The parameters tested were age, sex, risk factors, surgical time, hospital stay, cholecystitis tyep, and early or late complications. RESULTS: In the two groups there were no significant differences in age, sex, risk factors, type of cholecystitis and surgical time. The average of hospital stay was significantly longer for open cholecystectomy (9.5) than for laparoscopic technique (2.30) (p < 0.001). The complication rate was higher (7.30%) in open cholecystectomy. CONCLUSIONS: The laparoscopic cholecystectomy should be the standard procedure for the treatment of the acute cholecystitis.

The words "METHODS: A retrospective randomized study" are highlighted.

 

Slide 35

Slide  35. Determinants of quality

Determinants of quality

What lowers quality of evidence? 5 factors:

  • Methodological limitations
  • Inconsistency of results
  • Indirectness of evidence
  • Imprecision of results
  • Publication bias

 

Slide 36

Slide  36. Methodological limitations

Methodological limitations

Assessment of detailed design and execution (risk of bias)

  • For RCTs:
    • Lack of allocation concealment
    • No true intention to treat principle
    • Inadequate blinding
    • Loss to follow-up
    • Early stopping for benefit

 

Slide 37

Slide  37. Allocation concealment

Allocation concealment

250 RCTs out of 33 meta-analyses
Allocation concealment:
 
Effect
(Ratio of OR)
adequate1.00 (Ref.)
unclear0.67 [0.60—0.75]
not adequate0.59 [0.48—0.73]

 

Slide 38

Slide  38. 5 vs 4 chemo-Rx cycles for AML

5 vs 4 chemo-Rx cycles for AML

Hazard ratio plot of mortality in the five versus four courses randomization in the MRC AML12 trials.

 

Slide 39

Slide  39. Studies stopped early becasue of benefit

Studies stopped early becasue of benefit

Hazard ratio plot of mortality in the five versus four courses randomization in the MRC AML12 trials.

 

Slide 40

Slide  40. What about scoring tools?

What about scoring tools?

Example: Jadad score 
Was the study described as randomized?1
Adequate description of randomization?1
Double blind?1
Method of double blinding described?1
Description of withdrawals and dropouts?1

Max 5 points for quality

 

Slide 41

Slide  41. Cochrane Risk of bias graph in RevMan 5

Cochrane Risk of bias graph in RevMan 5

Example of a "Risk of bias graph"

 

Slide 42

Slide  42. Inconsistency of results

Inconsistency of results

  • Look for explanation for inconsistency
    • Patients, intervention, comparator, outcome, methods
  • Judgment
    • Variation in size of effect
    • Overlap in confidence intervals
    • Statistical significance of heterogeneity
    • I2

 

Slide 43

Slide  43. Heterogeneity

Heterogeneity

Chart showing neurological or vascular complications or death within 30 days of endovascular treatment (stent, balloon angioplasty) vs. surgical carotid endarterectomy (CEA)
 

 

Slide 44

Slide  44. Indirectness of evidence

Indirectness of evidence

  • Indirect comparisons
    • Interested in head-to-head comparison
    • Drug A versus drug B
    • Tenofovir versus entecavir in hepatitis B treatment
  • Differences in
    • Patients (early cirrhosis vs end-stage cirrhosis)
    • Interventions (CRC screening: flex. sig. vs colonoscopy)
    • Comparator (e.g., differences in dose)
    • Outcomes (non-steroidal safety: ulcer on endoscopy vs symptomatic ulcer complications)

 

Slide 45

Slide  45. Imprecision of results

Imprecision of results

  • Small sample size
    • Small number of events
    • Wide confidence intervals
    • Uncertainty about magnitude of effect

 

Slide 46

Slide  46. Imprecision

 

Imprecision

Chart showing any stroke (or death) within 30 days of endovascular treatment (stent, balloon angioplasty) vs. surgical carotid endarterectomy (CEA)
 

 

Slide 47

Slide  47. Publication bias

Publication bias

  • Reporting of studies
    • Publication bias
      • Number of small studies

 

Slide 48

Slide  48. All phase II and III licensing trial for antidepressant drugs between 1987 and 2004 (74 trials - 23 were not published)

All phase II and III licensing trial for antidepressant drugs between 1987 and 2004 (74 trials—23 were not published)

Two charts showing Journal and FDA estimates.

 

Slide 49

Slide  49. Quality assessment criteria

Quality assessment criteria

Quality of evidenceStudy design
HighRandomized trial
Moderate 
LowObservational study
Very low 
Lower if.
Study limitations (design and execution)
Inconsistency
Indirectness
Imprecision
Publication bias
Higher if.

What can raise the quality of evidence?

 

Slide 50

Slide  50. Image of an X-Ray

Photo of an x-ray.

 

Slide 51

Slide  51. Quality assessment criteria

Quality assessment criteria

Quality of evidenceStudy design
HighRandomized trial
Moderate 
LowObservational study
Very low 
Lower if.
Study limitations
Inconsistency
Indirectness
Imprecision
Publication bias
Higher if.
Large effect (e.g., RR 0.5) Very large effect (e.g., RR 0.2)
Evidence of dose-response gradient
All plausible confounding would reduce a demonstrated effect

 

Slide 52

Slide  52. Conceptualizing quality

Conceptualizing quality

LevelDescriptionQuality
Rating
HighFurther research is very unlikely to change our
confidence in the estimate of effect
+ + + +
ModerateFurther research is likely to have an important impact on
our confidence in the estimate of effect and may change
the estimate
+ + +
LowFurther research is very likely to have an important
impact on our confidence in the estimate of effect and is
likely to change the estimate
+ +
Very lowAny estimate of effect is very uncertain+

 

Slide 53

Slide  53. Process flow

Process flow

Graphic showing process flow including:

  • Clinical question
  • Rate importance
  • Select outcomes
  • Quality rating outcomes across studies

 

Slide 54

Slide  54. GRADE evidence profile

GRADE evidence profile

Example of a Quality Assessment.

 

Slide 55

Slide  55. GRADE - From evidence to decisions

GRADE—From evidence to decisions

 

Slide 56

Slide  56. Strength of recommendations

Strength of recommendations

  • Desirable effects
    • Health benefits
    • Less burden
    • Savings
  • Undesirable effects
    • Harms
    • More burden
    • Costs

 

Slide 57

Slide  57. Developing recommendations

Developing recommendations

Figure describing the balance between important benefits and downsides related to a recommendation. The process begins by evaluating whether desirable effects outweigh undersirable effects or vice versa. Moving on to making a recommendation requires a decision: if the balance is clear, a strong recommendation for or against an action follows. If the balance is not clear, a weak recommendation for or against an action follows. Widely differing values can also lead to a less clear balance of benefits versus downsides.

 

Slide 58

Slide  58. Strength of recommendation

Strength of recommendation

  • "The strength of a recommendation reflects the extent to which we can, across the range of patients for whom the recommendations are intended, be confident that desirable effects of a management strategy outweigh undesirable effects."
  • Strong or weak/conditional

 

Slide 59

Slide  59. Quality of evidence and strength of recommendation

Quality of evidence & strength of recommendation

  • GRADE separates quality of evidence from strength of recommendation
  • Linked but no automatism
  • Other factors beyond the quality of evidence influence our confidence that adherence to a recommendation causes more benefit than harm

 

Slide 60

Slide  60. What makes Guidelines Evidence-Based in 2009?

What makes Guidelines Evidence-Based in 2009?

Standardized Reporting of Clinical Practice Guidelines: A Proposal from the Conference on Guideline Standardization
Checklist for reporting: 18 items

  1. Recommendations and rationale—state the recommended action precisely. Indicate the quality of evidence and the recommendation strength.

Ann Intern Med. 2003

 

Slide 61

Slide  61. What makes Guidelines Evidence-Based in 2009?

What makes Guidelines Evidence-Based in 2009?

Standardized Reporting of Clinical Practice Guidelines: A Proposal from the Conference on Guideline Standardization
Checklist for reporting: 18 items

  1. Patient preferences—describe the role of patient preferences when a recommendation involves a substantial element of personal choice or values.

Ann Intern Med. 2003

 

Slide 62

Slide  62. A COPD guideline - do you want your review used like this?

A COPD guideline—do you want your review used like this?

7.6. Mucolytic/antioxidant therapy

  • These include drugs such as:
    • Ambroxol
    • Erdosteine
    • Carbocysteine
    • Iodinated glycerol

The regular use of these drugs has been evaluated in a number of studies with little evidence of any effect on lung function.

Data from a Cochrane review of the studies supports a role for these drugs in reducing the number of exacerbations of chronic bronchitis [33].

There is better evidence that N-acetylcysteine, a drug with mucolytic and anti-oxidant actions, can reduce the number of exacerbations of COPD and this is currently under study in a large prospective trial [34].

 

Slide 63

Slide  63. Another COPD guideline

Another COPD guideline

Mucolytic (mucokinetic, mucoregulator) agents (ambroxol, erdosteine, carbocysteine, iodinated glycerol). The regular use of mucolytics in COPD has been evaluated in a number of long-term studies with controversial results. Although a few patients with viscous sputum may benefit from mucolytics, the overall benefits seem to be very small, and the widespread use of these agents cannot be recommended at present.

Antioxidan agents. Antioxidants, in particular N-acetylcysteine, have been reported in small studies to reduce the frequency of exacerbations, leading to speculation that these medications could have a role in the treatment of patients with recurrent exacerbations. However, a large randomized controlled trial found no effect of N-acetylcysteine on the frequency of exacerbations, except in patients not treated with inhaled glucocorticosteroids.

 

Slide 64

Slide  64. And another COPD guideline

And another COPD guideline

1.2.14 Mucolytic therapy
1.2.14.1 Mucolytic drug therapy should be considered in patients with a chronic cough productive of sputum. B
1.2.14.2 Mucolytic therapy should be continued if there is symptomatic improvement (for example, reduction in frequency of cough and sputum production). D

1.2.15 Anti-oxidant therapy
1.2.15.1 Treatment with alpha-tocopherol and beta-carotene supplements, alone or in combination, is not recommended. A

 

Slide 65

Slide  65. What to do?

What to do?

Graphic image of a doctor and a photo of a large grouping of traffic signal lights.

 

Slide 66

Slide  66. Current state of recommendations

Current state of recommendations

Photo of the International Journal of Medical Informatics titled "The Yale Guideline Recommendation Corpus: A representative sample of the knowledge content of guidelines"

 

Slide 67

Slide  67. Current state of recommendations

Current state of recommendations

  • Reviewed 7527 recommendations
    • 1275 randomly selected
  • Inconsistency across/within
  • 31.6% did not recommendations clearly
    • Most of them not written as executable actions
  • 52.7% did not indicated strength

 

Slide 68

Slide  68. Yale Guideline Corpus

Yale Guideline Corpus

  1. Identify the critical recommendations in guideline text using semantic indicators
  2. Use consistent semantic and formatting indicators throughout the publication
  3. Group recommendations together in a summary section
  4. Do not use assertions of fact as recommendations.
  5. Clearly and consistently assign evidence quality and recommendation strength in proximity
    • Distinguish between the distinct concepts of quality of evidence and strength of recommendation.

 

Slide 69

Slide  69. Challenges in wording recommendations

Challenges in wording recommendations

  • Need to express (two) levels
  • Need to express direction
  • Differences across languages
  • Need codes (letters, symbols, numbers)

 

Slide 70

Slide  70. Letters, numbers, symbols and words: how to communicate grades of evidence and recommendations

Letters, numbers, symbols and words: how to communicate grades of evidence and recommendations

Photo of a clipping titled "Letters, numbers, symbols and words: how to communicate grades of evidence and recommendations" with highlighted text:

We did not find any studies comparing different systems of communicating grades in health care. A number of studies have compared alternative ways of presenting information about risk, but none addressed the use of codes and grades.

 

Slide 71

Slide  71. Categories of recommendations  

Categories of recommendations

  • Although the degree of confidence is a continuum, we suggest using two categories: strong and weak/conditional.
  • Strong recommendation: the panel is confident that the desirable effects of adherence to a recommendation outweigh the undesirable effects.
  • Weak recommendation: the panel concludes that the desirable effects of adherence to a recommendation probably outweigh the undesirable effects, but is not confident.

 

Slide 72

Slide  72. Implications of a strong recommendation

Implications of a strong recommendation

  • Patients: Most people in your situation would want the recommended course of action and only a small proportion would not
  • Clinicians: Most patients should receive the recommended course of action
  • Policy makers: The recommendation can be adapted as a policy in most situations

 

Slide 73

Slide  73. Implications of a weak/conditional recommendation

Implications of a weak/conditional recommendation

  • Patients: The majority of people in your situation would want the recommended course of action, but many would not
  • Clinicians: Be prepared to help patients to make a decision that is consistent with their own values
  • Policy makers: There is a need for substantial debate and involvement of stakeholders

 

Slide 74

Slide  74. Case scenario

Case scenario

A 13 year old girl who lives in rural Indonesia presented with flu symptoms and developed severe respiratory distress over the course of the last 2 days. She required intubation. The history reveals that she shares her living quarters with her parents and her three siblings. At night the family's chicken stock shares this room too and several chicken had died unexpectedly a few days before the girl fell sick.

Interventions: antivirals, such as neuraminidase inhibitors oseltamivir and zanamivir

 

Slide 75

Slide  75. Relevant healthcare question?

Relevant healthcare question?

  • Clinical question:
    • Population: Avian Flu/influenza A (H5N1) patients
    • Intervention: Oseltamivir (or Zanamivir)
    • Comparison: No pharmacological intervention
    • Outcomes: Mortality, hospitalizations, resource use, adverse outcomes, antimicrobial resistance

WHO Avian Influenza GL. Schunemann et al., The Lancet ID, 2007

 

Slide 76

Slide  76. How would you make decisions?

How would you make decisions?

 

Slide 77

Slide  77. Judgements about the strength of a recommendation

Judgments about the strength of a recommendation

  • No precise threshold for going from a strong to a weak recommendation
  • The presence of important concerns about one or more of these factors make a weak recommendation more likely.
  • Panels should consider all of these factors and make the reasons for their judgements explicit.
  • Recommendations should specify the perspective that is taken (e.g. individual patient, health system) and which outcomes were considered (including which, if any costs).

 

Slide 78

Slide  78. Evidence Profile

Evidence Profile

Oseltamivir for treatment of H5N1 infection:

Example of a Quality Assessment.

 

Slide 79

Slide  79. Oseltamivir for Girl with Avian Flu

Oseltamivir for Girl with Avian Flu

  • Summary of findings:
    • No clinical trial of oseltamivir for treatment of H5N1 patients.
    • 4 systematic reviews and health technology assessments (HTA) reporting on 5 studies of oseltamivir in seasonal influenza.
      • Hospitalization: OR 0.22 (0.02—2.16)
      • Pneumonia: OR 0.15 (0.03 —0.69)
    • 3 published case series.
    • Many in vitro and animal studies.
    • No alternative that is more promising at present.
    • Cost: ~ $45 per treatment course

 

Slide 80

Slide  80. What are the factors that determine your decisions?

What are the factors that determine your decisions?

 

Slide 81

Slide  81. GRADE: Factors influencing decisions and recommendations

GRADE: Factors influencing decisions and recommendations

  • Quality of Evidence
  • Balance of desirable and undesirable consequences
  • Values and preferences
  • Cost

 

Slide 82

Slide  82. Determinants of the strength of recommendation

Determinants of the strength of recommendation

Factors that can strengthen a recommendationComment
Quality of the evidenceThe higher the quality of evidence, the more likely is a strong recommendation.
Balance between desirable and undesirable effectsThe larger the difference between the desirable and undesirable consequences, the more likely a strong recommendation warranted. The smaller the net benefit and the lower certainty for that benefit, the more likely weak recommendation warranted.
Values and preferencesThe greater the variability in values and preferences, or uncertainty in values and preferences, the more likely weak recommendation warranted.
Costs (resource allocation)The higher the costs of an intervention—that is, the more resources consumed—the less likely is a strong recommendation warranted.

 

Slide 83

Slide  83. Determinants of the strength of recommendation

Determinants of the strength of recommendation

Factors that can weaken the strength of a recommendation. Example:DecisionExplanation
Lower quality evidenceYes/No 
Uncertainty about the balance of benefits versus harms and burdensYes/No 
Uncertainty or differences in valuesYes/No 
Uncertainty about whether the net benefits are worth the costsYes/No 

Table. Decisions about the strength of a recommendation
Frequent "yes" answers will increase the likelihood of a weak recommendation

 

Slide 84

Slide  84. Oseltamivir - Avian Influenza

Oseltamivir—Avian Influenza

Factors that can weaken the strength of a recommendation. Example: treatment of H5N1 patients with oseltamivirDecisionExplanation
Lower quality evidenceYesThe quality of evidence is very low
Uncertainty about the balance of benefits versus harms and burdensYesThe benefits are uncertain because several important or critical outcomes where not measured. However, the potential benefit is very large despite potentially small relative risk reductions.
Uncertainty or differences in valuesNoAll patients and care providers would accept treatment for H5N1 disease
Uncertainty about whether the net benefits are worth the costsNoFor treatment of sporadic patients the price is not high ($45)

Frequent "yes" answers will increase the likelihood of a weak recommendation

 

Slide 85

Slide  85. Example: Oseltamivir for Avian Flu

Example: Oseltamivir for Avian Flu

Recommendation: In patients with confirmed or strongly suspected infection with avian influenza A (H5N1) virus, clinicians should administer oseltamivir treatment as soon as possible (????? recommendation, very low quality evidence).

Schunemann et al. The Lancet ID, 2007

 

Slide 86

Slide  86. Are values important? Should resources be considered?

Are values important? Should resources be considered?

 

Slide 87

 Slide  87. Example: Oseltamivir for Avian Flu

Example: Oseltamivir for Avian Flu

Recommendation: In patients with confirmed or strongly suspected infection with avian influenza A (H5N1) virus, clinicians should administer oseltamivir treatment as soon as possible (strong recommendation, very low quality evidence).

Values and Preferences
Remarks: This recommendation places a high value on the prevention of death in an illness with a high case fatality. It places relatively low values on adverse reactions, the development of resistance and costs of treatment.

Schunemann et al. The Lancet ID, 2007

 

Slide 88

Slide  88. Other explanations

Other explanations

Remarks: Despite the lack of controlled treatment data for H5N1, this is a strong recommendation, in part, because there is a lack of known effective alternative pharmacological interventions at this time.

The panel voted on whether this recommendation should be strong or weak and there was one abstention and one dissenting vote.

 

Slide 89

Slide  89. Health Care Question (PICO) Systematic reviews

Health Care Question (PICO) Systematic reviews

 

Slide 90

Slide  90. Process flow diagrams

Process flow diagrams

Diagrams showing the process flows of the Systematic review and the Guideling development.

 

Slide 91

Slide  91. This slide is blank.

No titles

 

Slide 92

Slide  92. Ideal vs. practical ad hoc GRADE approaches

Ideal vs. practical ad hoc GRADE approaches

StageElementsAdvantageComment
IdealSystematic review
GRADE eTables
Qual. of evidence
Strength of rec.
Follows int. standards
Methodol. most rigorous
Easily maintainable
Fully transparent process
Access to methodologist
Access to evidence centers
Initially more resource
intensive, long-term savings
Inter-mediaryAd hoc review
GRADE eTables
Qual. of evidence
Strength of rec.
Still retaining major
advantages of the
"ideal approach"
Risk of bias may be higher
Access methodologist rec.
Only minimal addl. cost
InitiationAd hoc review
GRADE eTables
Qual. of evidence
Strength of rec.
Option to fully "upgrade" to
an "ideal approach"
Foundation of a methodo-
logically sound system
Risk of bias may be higher
Access methodologist prn
No additional cost

 

Slide 93

Slide  93. Evidence-based guidelines?

Evidence-based guidelines?

  • What is evidence?

 

Slide 94

Slide  94. What is evidence?

What is evidence?

  • a: An outward sign, an indication
  • b: Something that furnishes proof
Current as of December 2009
Internet Citation: Rating the Evidence: Using GRADE to Develop Clinical Practice Guidelines (Text Version). December 2009. Agency for Healthcare Research and Quality, Rockville, MD. http://www.ahrq.gov/news/events/conference/2009/falck-ytter-schunemann/index.html