Value of Information Calculations to Inform and Prioritize Clinical Research Investments

On September 29, 2010, David Meltzer made this presentation at the 2010 Annual Conference. Select to access the PowerPoint® presentation (422 KB). Free PowerPoint® Viewer (Plugin Software Help).

Slide 1

Value of Information Calculations to Inform and Prioritize Clinical Research Investments

David Meltzer MD, PhD
The University of Chicago

Slide 2

Overview

• Cost-effectiveness analysis has long been used to assess the value of medical treatments and the information that comes from diagnostic tests.
  • Cost-effectiveness measured in Cost/QALY
  • Net health benefits = gain in QALYs—opportunity cost of spending in QALYs
  • Net monetary benefit = $ value of improved health—costs
• Newer value of information techniques has extended these tools to assess the value of medical research.

Slide 3

Research as Value of Information: Analogy to Diagnostic Testing

Image: A chart displays the following decision formula:

Test

• S
  • U(T|S)
    • pU(N|S)+(1-p)U(N|H)}
• H
  • U(N|H)

Don't Test

• S (Images of photographs and a reel of film)
  • Max{pU(T|S)+(1-p)U(T|H),
• H
  • pU(T|S)+(1-p)U(N|H)

Notes: The value of research can be understood as analogous to the value of diagnostic testing. With ideal information one always makes the best choice given the true state of the world, which provides the greatest possible expected value of outcomes. Without information, a decision has to be made that is best on average, and this will produce a lower expected value.

Slide 4

Value of Information Approach to Value of Research

• Without information:
  • Make best compromise choice not knowing true state of the world (e.g. don't know if intervention is good, bad).
    • With probability p: get V(Compromise|G)
    • With probability 1-p: get V(Compromise|B)
• With information:
  • Make best decision knowing true state:
    • With probability p: get V(Best choice|G)
    • With probability 1-p: get V(Best choice|B)
• Value of information:
  • = E(outcome) with information—E(outcome) w/o information
• = {p*V(Best choice|G) + (1-p)*V(Best choice|B)}—
• {p*V(Compromise|G) + (1-p)*V(Compromise|B)}
• = Value of Research

Slide 5

Practical Applications of Value of Information (VOI)

• VOI requires modeling population value of information
  VOI = Σ_t Β^t x D(t) x I(t) x N_t x IVOI
  where

  Β ^t is time preference discount factor.
  D(t) is depreciation of knowledge over time
  I(t) is extent of implementation
  N _t is number of eligible individuals in each cohort
  IVOI is individual VOI

• VOI based on decision models:
  • IVOI modeled with decision model
  • UK (NICE): Alzheimer's Disease Tx, wisdom teeth removal
• Minimal modeling approaches to VOI:
  • IVOI comes (nearly) directly from clinical trial
  • US (NIH): CATIE Trial of atypical antipsychotics
• Bound with more limited data (burden of illness)

Slide 6

"Bayesian Value of information analysis: An application to a policy model of Alzheimer's disease."

Image: A Markov model of disease progression. At the top are two ovals captioned "Mild (com/NH)" and "Moderate (com/NH)." Arrows point back and forth between these two ovals; arrows also point down from each oval to a third oval beneath them, captioned "Dead." An arrow also points down from "Moderate (com/NH)" to a fourth oval at the bottom of the model, captioned "Severe (com/NH)." An arrow points up from "Severe (com/NH)" to "Dead."

Notes: The authors of this study constructed a Markov model for the progression of Alzheimer Disease from mild to moderate to severe and to death.

Intl J of Technology Assessment in Health Care 17:1, 2001.

Slide 7

Uncertainty in Incremental Net Benefits

Image: A line graph depicts the prior distribution of incremental net benefit of donepezil. The line for "INB at 24 weeks" rises to a sharp peak from .000 to .059 at -250, then decreases as sharply. The line for "INB at 210 weeks" rises from .000 to .015 at -250, then decreases slowly back to .000 over 1,500, 3.250, and 5,000.

Notes: It is quite certain that the INB of donepezil is negative at 24 weeks. However, there is a great deal of uncertainty about whether the INB at 210 weeks is positive or negative and the magnitude of those benefits.

Intl J of Technology Assessment in Health Care 17:1, 2001.

Slide 8

Cost-Effectiveness Acceptability Curve

Image: A line graph depicts the acceptability curve of donepezil, charting the probability that donepezil is cost-effective (p[INB>0]) against the monetary valuation of health outcome (1/g). The line starts with the probability that donepezil is cost-effective at 0.2, and rises to 0.45 at $25,000. The line rises further to nearly 0.7 at $50,000 (a baseline is set here, noted as p[INB>0 - 0.68]). The line continues to rise thereafter: to 0.75 at $75,000, to 0.8 at $100,000, to 0.9 at $125,000 and $150,000.

Notes: The probability that donepezil is cost-effective varies from 20% up to 90% as the WTP for a QALY increases from $0 to $150,000.

Intl J of Technology Assessment in Health Care 17:1, 2001.

Slide 9

Value of Research by Value of Health

Image: A line graph depicts the expected value of perfect information (EVPI) for treatment choice (U.S. population), charting EVPI against the monetary valuation of health outcome (1/g). The line for "Week 210" begins at $600,000,000 EVPI and rises to nearly $800,000,000 EVPI at $15,000, then drops steadily, back to $600,000,000 EVPI at $20,000, $400,000,000 EVPI at $40,000, $250,000,000 EVPI at $60,000, $150,000,000 EVPI at $80,000, and $100,000,000 EVPI at $100,000. The line for "Week 54" begins barely above $0 EVPI and rises to $50,000,000 EVPI at $20,000, then slightly above $100,000,000 EVPI at $40,000 before it reaches its peak at $200,000,000 EVPI at $50,000; thereafter, the line drops steadily to slightly above $100,000,000 EVPI at $60,000, $50,000,000 EVPI at $80,000, and barely above $0 EVPI at $100,000. The line for "Week 24" remains at $0 EVPI across the graph.

Notes: The value of research is largest at 210 weeks and almost zero at 24 weeks. Value of research varies by WTP for a QALY.

Slide 10

Value of Research by Time Horizon

Image: A line graph depicts the EVPI for an individual patient, charting EVPI against a time horizon (weeks). The line begins at $0 EVPI at about 20 weeks, and rises just above $200 EVPI at 45 weeks, then drops slightly below $200 EVPI at 75 weeks before rising steadily, back to just above $200 EVPI at 105 weeks, $250 EVPI at 125 weeks, just above $300 EVPI at 160 weeks and just below $400 EVPI at 210 weeks.

Notes: The value of information is zero up to about 20 weeks treatment duration and then rises almost continuously up to 210 weeks.

Slide 11

Contributors to Value of Research

Image: A bar graph depicts EVPI for model inputs (210 weeks). Total: $340,000,000 EVPI; Efficacy duration: $275,000,000 EVPI; RRR>24: $100,000,000 EVPI; RRR<24: just under $100,000,000 EVPI; Transitions: $50,000,000 EVPI; Dropout rate: $40,000,000 EVPI; Direct costs: $40,000,000 EVPI; Utilities: $40,000,000 EVPI; Indirect costs: $40,000,000 EVPI.

Notes: Efficacy duration is the major component of the value of additional information in this study.

Slide 12

Practical Applications of Value of Information

• VOI requires modeling population value of information
  VOI = Σ_t Β^t x D(t) x I(t) x N_t x IVOI
  where

  Β ^t is time preference discount factor.
  D(t) is depreciation of knowledge over time
  I(t) is extent of implementation
  N _t is number of eligible individuals in each cohort
  IVOI is individual VOI

• VOI based on decision models
  • IVOI modeled with decision model
  • UK (NICE): Alzheimer's Disease Tx, wisdom teeth removal
• Minimal modeling approaches to VOI
  • IVOI comes (nearly) directly from clinical trial
  • US (NIH): CATIE Trial of atypical antipsychotics
• Bound with more limited data (burden of illness)

Slide 13

Expected Value of Research on the Comparative Cost-Effectiveness of Antipsychotics Drugs

David Meltzer MD PhD

Department of Medicine, Department of Economics,
Harris School of Public Policy, &
University of Chicago CERT
Chicago IL

(Joint work with Anirban Basu PhD, University of Chicago & Dr. Herbert Y. Meltzer, Vanderbilt University)
 

Slide 14

The Clinical Antipsychotic Trials in Intervention Effectiveness (CATIE)

• $42.6 million, NIMH-funded randomized trial of Atypical Antipsychotic Drugs (A-APDs) and a Neuroleptic (Perhphenazine) in patients with established schizophrenia
• Major findings:
  • Discontinuation rates similar with A-APDs and Perphenazine
  • Perphenazine cost-effective first-line treatment
• Impact:
  • Frequently discussed in coverage decisions
  • Some have argued results should be considered definitive

Slide 15

The Clinical Antipsychotic Trials in Intervention Effectiveness (CATIE)

• Limitations:
  • Continuation was major endpoint
  • Limited precision in estimates of effectiveness, costs
  • Small differences in effectiveness/costs across many persons could be of great value
• Important to know value of potential future research
  • Help prioritize individual research opportunities
  • Facilitate rational investment decisions

Slide 16

CATIE Cost-Effectiveness Results

Only statistically significant difference:
QALY_Perphenazine > QALY_Risperidone (p-val < 0.001).

(Ref: Rosenheck et al, 2006; Private Communications with Dr. Rosenheck).

Slide 17

Aims

Primary Aim: To determine the expected value of more precise determination of effects of AAPDs and Perphenazine on costs and QALYs.

Secondary Aim: To determine the optimal sample size for a future trial of the effects of AAPDs and Perphenazine on costs and QALYs

Slide 18

Methods Used for Value of Research

Expected value calculated based on the welfare of the prevalent cohort over their lifetimes and the welfare of next 20 incident cohorts over their lifetimes

3% discounting was used.

Slide 19

Simulated Distribution of Mean QALYS
(Based on uncertainty around CATIE results)

Image: Graph compares the density vs E(QALY)/per patient per year for four antipsychotic drugs: Olanzapine: 0.723 (0.0063); Quetiapine: 0.721 (0.0065); Risperidone: 0.706 (0.0066); Perphenazine: 0.722 (0.0064).

Notes: The distribution of QALY weights in CATIE  is centered around a mean of 0.72 for all treatments studies except risperidone, which has a mean of 0.70. The standard errors of these means is small (about 0.0065)

Slide 20

Simulated Distribution of Mean Costs
(Based on uncertainty around CATIE results)

Image: Graph compares the density vs E(QALY)/per patient per year for four antipsychotic drugs: Olanzapine: $1606 (1421); Quetiapine: $1685 (14855); Risperidone: $1621 (1439); Perphenazine: $810 (723).

Notes: The mean costs in CATIE for perphenazine is $810 compared to about $1600 for the three atypical antipsychotics studied. The distribution of CATIE costs is heavily skewed to the right.

Slide 21

Realizations of Value of Research Over Time

Image: Line graph shows the value of future research to prevalent and incident cohorts at $50k/QALY for incidents from 2007 through 2036. All lines start at $12 billion in 2007 and drop steadily together to about $7 billion in 2017, $4 billion in 2027, $2 billion in 2037, $1 billion in 2047 and reach 0 between 2067 and 2087.

Total Value to Each Incident Cohort: $6.6 billion
Total Value to Prevalent & Next 20 Incident Cohorts: $342 billion

Notes: The total value to each incident cohort is about $6.6 billion and the total value to the prevalent and next 20 incident cohorts is $342 billion.

Slide 22

Value of Research and Acceptability Profile

Image: Line graph shows the value of research (VofR) in billions by marginal WTP for a QALY for Max VofR and Pr(CATIE decision is incorrect). The Max VofR line begins at $350 billion at 10,000 marginal WTP and drops slightly to just above $340 billion at 30,000, $340 billion at 50,000, just below $340 billion at 70,000, 90,000, and 11,000, ~$380 billion at 130,000, and ~$375 billion at 150,000. The Pr line begins just below $360 billion at 10,000 and drops slightly to just above $350 billion at 30,000, just above $340 billion at 50,000, ~$340 billion at 70,000 and 90,000, below $340 billion at 11,000, ~$375 billion at 130,000, and 150,000.

Notes: The maximal value of research is about $340 million and the probability that perphenazine is cost-effective is about 53 percent across WTP for a QALY values from $10,000 to $150,000.

Slide 23

Optimal Sample Size for A Future Trial
 

• Traditional (Deterministic) Power Calculations (at $50K/QALY):
  • Largest effect size in NMB between an atypical & perphenazine based on CATIE results: $15,680 (sd=$315,000) vs. $26,296 (sd=$140,000).
  • Sample size required for alpha = 0.05 & power = 0.80: 8,300 for each arm.
  • Power associated with n of CATIE = 400/arm & alpha = 0.05: 10%.

Slide 24

Net Expected Value of Sample Information
(at $50K, $100K and $150K/QALY)

Image: Line graph shows EVSI in billions by sample size for each arm (Cost of research $3 million + (sample size *4)*($5000/month)*18 months). The $50K/QALY line begins at ~310 EVSI at 5,000 sample size, rises to 330 by 15,000 and remains roughly level, dropping to 350 by 50,000. The $100K/QALY line begins at ~305 EVSI at 5,000 sample size, rises to ~324 by 15,000 and remains roughly level, dropping to ~322 by 50,000. The $150K/QALY line begins at 300 EVSI at 5,000 sample size, rises to ~320 by 15,000 and remains roughly level, dropping to ~318 by 50,000.

Optimal sample size for each arm = 22,500.

Notes: This graph of the relationship between sample size and value of research shows that the highest value of research net of costs occurs at a sample size of about 22,500 subjects in each arm.

Slide 25

CATIE VOI Conclusions

• The value to more precisely establishing the cost-effectiveness of typical/atypical antipsychotics is enormous.
• The results of CATIE should not be viewed as definitive.
• Further studies of the comparative cost-effectiveness of typical/atypical antipsychotics with adequate sample size to answer such questions have high expected value.
• Optimal sample sizes may be exceptionally large, raising interesting questions as to how clinical trials on such a scale might be executed.
  • Large scale social experiments may provide an interesting model for such studies.

Slide 26

Conclusions

• Value of information (VOI) analysis can be used to develop prospective estimates of the value of research.
• VOI can be used to prioritize areas of clinical research or choose among study designs:
  • Decision models or minimal modeling approaches
  • Additional minimal model applications in progress
  • Erlotinib (Tarceva) (+gemcitabine) in advanced pancreatic CA
  • Azithromycin vs. Augmentin in acute sinusitis
• Incorporating VOI into research prioritization is a work in progress:
  • Use by funding agencies?
  • Use by investigators?

Slide 27

Acknowledgments

• Collaborators: Anirban Basu Ph.D., Jeanette Chung Ph.D., Ties Hoomans Ph.D., Herbert Meltzer M.D.
• Funding: AHRQ BCBS Evidence-Based Practice Center, AHRQ Hospital Medicine and Economics Center for Education and Research in Therapeutics, Best Practice Inc, National Institute of Aging, National Institute of Mental Health