Transparency in the Use of Propensity Score Methods (Text Version)

Slide Presentation from the AHRQ 2008 Annual Conference

On September 9, 2008, John D. Seeger, made this presentation at the 2008 Annual Conference. Select to access the PowerPoint® presentation (610 KB; Plugin Software Help).

Slide 1

Transparency in the Use of Propensity Score Methods

John D. Seeger, PharmD, DrPH
Chief Scientist, i3 Drug Safety
Adjunct Assistant Professor, Harvard School of Public Health
September 9, 2008

With thanks to: Alec Walker, Tobias Kurth, Jeanne Loughlin, Mona Eng, and Alex Cole

Slide 2

Propensity Score Analysis—When?

• Strategies to control for confounding in non-experimental pharmacoepidemiology:
  • Measured confounders**
    • Design**
      • Restriction
      • Matching
    • Analysis**
      • Standardization
      • Stratification
      • Multivariate regression
  • Unmeasured confounders*
    • Unmeasured but measurable in a validation study:
      • Two-stage sampling
      • External adjustment
    • Unmeasurable:
      • Design
        • Crossover designs
        • Active comparison group (restriction)
      • Analysis:
        • Instrumental variables
        • Sensitivity analysis
• Note: *Thanks to S. Schneeweiss
  **Amenable to Propensity Techniques

Slide 3

Motivation

• Assume matching when comparing 2 treatments:
  • For every drug user with given characteristics
  • Find a comparator with identical characteristics
• Example: Male, age 45, smoker, with hypertension (HTN)...
  • Matching fails:
    • Age (10 categories) x
    • Sex (2 categories) x
    • Prior diagnoses (5 @ 2 categories each)
    • Prior drug therapy (5 @ 2 categories each)
    • Preceding cost of care (5 categories)
  • Leads to 102,400 potential matching groups.

Slide 4

Propensity Score Collapses Exposure Predictors

The slide shows a line graph which presents the "Hypothetical Distribution of Propensity Scores." The vertical axis, number of people, goes from 0 to 1400 and the horizontal axis, propensity score, goes from 0 to 1. The results show "Initiators" beginning at 0, reaching a maximum of 400 people at 0.65, and ending at less than 100 people at 1. The results show "Non-Initiators" beginning at 300, reaching a maximum of 1199 people at 0.35, and ending at 0.

• Single value:
  • Probability: subject will receive therapy vs comparator
  • Removes confounding by components of the score
    • Patient characteristics that favor one therapy over another
• Permits:
  • Restriction
  • Matching
  • Stratification
  • Modeling
  • Weighting

Slide 5

Should Propensity Scores Always be Used?

The slide shows a line graph which presents "Logistic Regression."

• Note: Figure 1. Median percentage of bias with the logistic regression, by strength of the exposure and number of events per confounder. In the logistic regression, the bias declines as the number of events per confounder increases. Values greater than zero indicate an overestimation of the effect of the exposure on the outcome. Negative values indicate an underestimation of the effect of the exposure on the outcome.
• Note: Cepeda S, et al. Am J Epidemiol 2003;158:280-7.
• More than 8 events per covariate leads to unbiased estimates
• So propensity score favored when:
• Many more persons exposed to drug of interest than study outcomes.
  • Common exposure
  • Rare outcome
• Allows for richer model (more predictors) of exposure than outcome.
  • Alternative hypotheses

Slide 6

Estimate Propensity Score

• Predict treatment from baseline covariates within database
• Inclusion of predictors:
  • A priori (what characteristics are used to prescribe?)
  • Empiric (what differentiates initiators?)
  • Generic (what patterns of healthcare predict initiation?)
• Coefficients of propensity score:
  • Interpretable and Informative

Slide 7

Propensity Score Restriction

The slide shows a bell curve graph presenting the results for "Exposed subjects" and "Unexposed subjects."

• Note: *In this example subjects with low propensity scores are never exposed while subjects with high propensity scores are always exposed.
• Note: Sturmer T, et al. J Clin Epidemiol 2006;59:437-47.

Slide 8

Propensity Score Restriction

• Potential for serious adverse events from error (name confusion):
  • Amaryl (glimepiride an oral hypoglycemic)
  • Reminyl (galantamine for Alzheimer's disease)
• 36,816 people with AD diagnosis (14,626 Reminyl dispensings):
  • 236 Amaryl recipients
  • 24 Amaryl recipients in the lowest decile of the propensity score
  • 13 with a single dispensing of Amaryl or no diabetes diagnoses
  • 2 with no diabetes-related claims across entire claim history
  • Medical record review suggested no error
• Propensity score restriction may be used as a screening method to identify unusual patterns of healthcare for closer scrutiny:
  • Possible medication dispensing errors
  • Others
• Confirmation requires additional data, which could be obtained through medical record review.

Slide 9

Propensity Score Distribution and Strata

The line graph presents the propensity scores on the x axis and percent on the y axis. There are two lines representing "Zopiclone" and "Temazepam."

• C-statistic equals 0.739.

Slide 10

Effect of Temazepam Relative to Zopiclone

The table presents statistical data of "Relative Risk" and "Upper and Lower 95th Percentile." The data shown is for Quintiles 1-5 and totals adjusted for propensity score and continuous propensity score.

• Transparent analysis:
  • Within-stratum balance
  • Stratum-specific effect estimates as well as pooled estimate
  • Explicit evaluation of potential for effect measure modification

Slide 11

Matching on the Propensity Score

• Matching can be performed by:
  • Standard automated case-control matching programs where the matching range is specified
  • Nearest available match based on the propensity score
  • Greedy matching techniques (http://www2.sas.com.proceedings/sugi26/p214-26.pdf)

Slide 12

Distribution of Propensity Score

The line graph presents propensity score on the x axis and number of persons on the y axis. The categories are "Statin Initiators" and "Non-Initiators." The results show a consistent trend for both categories, with a low propensity score for higher numbers of persons and exponentially higher propensity score as the number of persons decrease.

Slide 13

Propensity Score Distribution (After Matching)

The line graph presents propensity score on the x axis and number of persons on the y axis. The categories are "Statin Initiators" and "Non-Initiators." The data for both categories shows a sharp increase in number of persons in the lower propensity scores, between 0 and 0.05. Beyond propensity scores of 0.05, the number of persons decreases as the propensity scores increase, approaching 0 beyond a propensity score of 0.7.

Slide 14

Characteristics Before Matching

The table presents the results for "Initiators: N=4144," "Non-Initiators: N=4144," and "P-value" for various "Variables" such as Lipid-Related Labs, Different Prescription Drugs, Low-density lipoprotein (LDL) Level, Angina, Smoking, Hypertension, etc.

Slide 15

Balance Achieved by Matching

The table presents the results for "Initiators: N=2901," "Non-Initiators: N=2901," and "P-value" for various "Variables" such as Lipid-Related Labs, Different Prescription Drugs, LDL Level, Angina, Smoking, Hypertension, etc.

• Note: Matched at 0.01 Propensity Score

Slide 16

Analysis by 2X2 Table

The slide presents a "Table of mi by statin."

Slide 17

MI Outcome (After Matching)

The line graph presents months of follow up on the x axis and cumulative incidence on the y axis. The two data sets are "Statin Initiators" and "Statin Non-Initiators." Both data lines begin at approximately, 0, 0. Both show a somewhat linear increase in cumulative incidence as months of follow-up increase. The trend is stronger in the "Statin Non-Initiators" data.

• Note: HR=0.69 (0.52-0.93)
• Note: 31% (7%-48%) Risk Reduction

Slide 18

Regression Adjustment with Propensity Scores

• Regression adjustment:
  • Note: An image of the equation is portrayed.
  • All study participants are used
  • Still a two-step approach (exposure and outcome)
  • More power compared to including all covariates into the model, since degrees of freedom are gained
  • However, assumes the underlying association between the score and the outcome is modeled appropriately

Slide 19

Weighting

The slide presents three separate line graphs showing the distribution of propensity score on the x axis relative to the number of people on the y axis. The lower two charts are entitled "IPTW" and "SMR" and both show nearly identical distributions of data for "Initiators" and "Non-Initiators". The "IPTW" chart shows a bell curve distribution of with a peak in number of people, approximately 1300, at a propensity score of approximately 0.45. The "SMR" chart shows a bell curve distribution of with a peak in number of people, approximately 400, at a propensity score of approximately 0.65.

Slide 20

Baseline Characteristics

Table 1. Variables included in the propensity scores, LABA ans ICS cohorts

The table presents statistical data of "LABA cohort" and "ICS cohort." The data is shown for various age categories, genders, geographic regions of health plan, Asthma-related drug dispensings (0-2 days), Asthma-related drug dispensings (3-365), and Asthma-related physician visits.

Slide 21

Cohort Results

Incidence rates (per 1000) and hazard ratios

The table presents statistical data of "Events," "Person Years," "Incidence Rate," and "Hazard Ratio" for the following categories:

• All-cause Mortality
• Asthma-related emergency room visits
• Asthma-related hospitalizations
• Intubations

Slide 22

Are Divergent Results Possible?

The table presents statistical data for "No.," "OR*," and "95% Cl*" for the following categories:

• Crude model
• Multivariable model
• Matched on propensity score
• Regression adjusted with propensity score
• Propensity score, continuous
• Multivariable
• Propensity score, deciles
• Multivariable
• Weighted models
• IPTW*
• SMR* weighted
• Note: Kurth T, et al. Am J Epidemiol 2006;163:262-70.

Slide 23

What About Unmeasured Confounding?

• Obesity, Smoking, Exercise

Pharmacoepidemiology and Drug Safety (2008)
Published online in Wiley InterScience (www.interscience.wiley.com)
DOI: 10.1002/pds.1554
Original Report

Supplementary data collection with case-cohort analysis to address potential confounding in a cohort study of thromboembolism in oral contraceptive initiators matched on claims-based propensity scores.

P. Mona Eng ScD^1*, John D. Seeger PharmD, DrPH^1,2, Jeanne Loughlin MS^1*, C. Robin Clifford MS^1*, Sherry Mentor BA^1* and Alexander M. Walker MD, DrPH^1,2

 ¹Ingenix, i3 Drug Safety, Waltham, MA, USA

 ²Harvard School of Public Health, Boston, MA, USA

Slide 24

Accounting for Variables had Little Effect

Table 3. Relative risks of thromboembolism comparing ethinyl estradiol/drospirenone (EE/DRSP) and other oral contraceptive (OC) initiators from case-cohort and propensity score matched cohort analyses, Ingenix Research Data Mart 2001-2004, United States.

Slide 25

Conclusion

• Propensity score can be useful for addressing confounding (by indication).
• Allows for rich model of exposure to be developed.
• Advantageous when number of people with a study outcome is small relative to number of exposed persons and number of potential confounders is large.
  • Drug effects (particularly adverse ones)
• Consider transparency:
  • When selecting propensity score
  • When building propensity score
  • When using propensity score

Slide 26

Thank You

John.Seeger@i3DrugSafety.com