Advanced Methods in Delivery System Research - Planning, Executing, Analyzing, and Reporting Research on Delivery System Improvement

Webinar #3: Logic Models (Slide Presentation)

This slide presentation was presented at a webinar on June 4, 2013.

Presenter: Dana Petersen, PhD
Discussant: Todd Gilmer, PhD
Moderator: Michael I. Harrison, PhD

Sponsored by AHRQ's Delivery System Initiative in partnership with the AHRQ PCMH program.

Slide 1

Advanced Methods in Delivery System Research –Planning, Executing, Analyzing, and Reporting Research on Delivery System Improvement 

Webinar #3: Logic Models

Text description is shown below.

Presenter: Dana Petersen, PhD
Discussant: Todd Gilmer, PhD
Moderator: Michael I. Harrison, PhD

Sponsored by AHRQ's Delivery System Initiative in partnership with the AHRQ PCMH program.

June 4, 2013

Slide 2

Speaker Introductions

Text description is shown below.

Dana Petersen is a Health Researcher at Mathematica Policy Research. Dana is the lead author of the brief on Logic Models in AHRQ’s PCMH research methods series. Her presentation today is based on that methods brief.

Todd Gilmer, Ph.D., is a Professor of Health Economics and Chief of the Division of Health Policy in the Department of Family and Preventive Medicine at the University of California, San Diego. His presentation today draws on his AHRQ-funded research on Full Service Partnerships for housing and treating people who are homeless mentally ill.

Images of Dr. Petersen and Dr. Gilmer are also shown on this slide.

Slide 3

The Logic Model: A Foundation to Implement, Study, and Refine Patient-Centered Medical Home (PCMH) Models

Text description is shown below.

June 4, 2013
AHRQ Advanced Methods Webinar

Dana Petersen -- Erin Fries Taylor -- Deborah Peikes

Mathematica Policy Research

Slide 4

Acknowledgements

Text description is shown below.

  • David Meyers and Jan Genevro (AHRQ) for commissioning the methods brief from which this presentation is derived.
  • Source – Petersen D, Taylor EF, Peikes D. Logic Models: The Foundation to Implement, Study, and Refine Patient-Centered Medical Home Models. Rockville, MD: Agency for Healthcare Research and Quality, February 2013. AHRQ Publication No. 13-0029-EF.
  • Available for download from pcmh.ahrq.gov.

Slide 5

Outline of Presentation

Text description is shown below.

  1. Definition and Elements
  2. Purposes and Users
  3. How to Build a Logic Model
  4. Applicability to PCMH (Medical Home) Models
  5. Advantages and Limitations

Slide 6

Definition: What is a Logic Model?

Text description is shown below.

  • A graphical depiction that shows:
    • Necessary resources and inputs.
    • Intervention activities and processes.
    • Intermediate and ultimate outcomes of the intervention and the pathways through which it produces those outcomes.
    • Interrelationships among intervention components.
    • Contextual factors that may influence the intervention’s ability to produce results.
  • Shows the links in the chain of reasoning about “what causes what”.
  • Can vary in complexity and take many forms.

Slide 7

Elements of a Basic Logic Model

Text description is shown below.

Image of a flow chart illustrating the elements of a basic logic model. Adapted from McCawley, PF. The logic model for program planning and evaluation. University of Idaho Extension. Available at: http://www.uiweb.uidaho.edu/extension/LogicModel.pdf.

Slide 8

Purposes and Users of Logic Models

Text description is shown below.

 PlannersImplementersEvaluators
Facilitate understanding of multi-faceted interventions
Clarify goals and conceptual gaps
Track progress and changing needs 
Guide choice and timing of process and outcome measures  
Provide a framework for interpreting results  

Slide 9

Building a Logic Model

Text description is shown below.

  • There is no single right way:
    • Depends on purpose and preference.
  • Discuss these questions:
    • What problem are you trying to solve?
    • What outcomes would represent success?
    • What intervention is required to achieve these outcomes?
    • What inputs and resources are needed to deliver the intervention?
    • What contextual factors are likely to affect the process?
  • Graphically illustrate answers and relationships.

Slide 10

Applicability to Medical Home Studies

Text description is shown below.

  • Medical home models are multifaceted interventions:
    • Consist of multiple and interrelated components.
    • Affect change on multiple levels (patients, providers, practices, health systems).
    • Include synergies and iterative feedback loops based on a continuous quality improvement framework.
    • Are tested in complex health systems contexts.
  • Using logic models helps to overcome evaluation challenges raised by these complexities.

Slide 11

Example: Overarching Medical Home Logic Model

Text description is shown below.

Image of a flowchart showing the medical home logic model.

Slide 12

Example: Logic Model Related to Accessible Services

Text description is shown below.

Image of a flowchart showing a logic model related to accessible services.

Slide 13

Advantages of Logic Models

Text description is shown below.

  • Maintain a focus on intervention process and context.
  • Balance brevity and complexity in data collection and analysis.
  • Highlight systems change and interactions.
  • Adapt to interventions of any scale.

Slide 14

Limitations of Logic Models

Text description is shown below.

  • Might oversimplify reality.
  • Involve balancing depth and detail.
  • Require mid-course review.
  • Could inhibit creative thinking.
  • Must be combined with other tools.

Slide 15

More Information

Text description is shown below.

Slide 16

Using Logic Models to Understand Variation in Implementation of Full Service Partnerships

Text description is shown below.

June 4, 2013
AHRQ Advanced Methods Webinar

Todd Gilmer, PhD
Department of Family and Preventive Medicine
University of California, San Diego

AHRQ R01 HS01986-1

Slide 17

Overview

Text description is shown below.

  • Background on the Mental Health Services Act and our study of Full Service Partnerships.
  • Using a logic model to understand variation in implementation of FSPs.
  • Using a logic model to explore how implementation factors related to fidelity relate to fidelity domains.
  • Using a logic model to describe how one implementation factor is related to outcomes.

Slide 18

Mental Health Services Act

Text description is shown below.

  • In 2004, California voters approved proposition 63, which was signed into law as the MHSA.
    • Policy history included AB3777 (1988), AB34 (1999), and AB2034 (2000) which funded integrated models 1% tax on incomes > $ million to fund public mental health services in specific areas:
  • 1% tax on incomes > $ million to fund public mental health services in specific areas:
    • Community Services and Supports:
      • Recovery oriented programs targeting the underserved: homeless, Latinos, Asians, older adults, transitional age youth.
      • 28% of CSS funding to Full Service Partnerships.
    • Prevention and Early Intervention (e.g. stigma, suicide).
    • Innovations (integrated mental and physical health).

Slide 19

Full Service Partnerships

Text description is shown below.

  • FSPs provide supported housing and team based services with a focus on rehabilitation and recovery.
  • FSPs are client centered and recovery oriented programs that do ‘whatever it takes’ to improve residential stability and mental health outcomes.
  • FSPs were implemented with substantial stakeholder input, and were adapted to local environments, resulting in a wide diversity in approaches to both housing and services.

Slide 20

Housing First

Text description is shown below.

  • Developed in New York City by Pathways to Housing.
  • Traditional housing model required treatment adherence and sobriety before placement.
  • Housing first model emphasized immediate housing in scatter site apartments with tenancy rights.
  • Adherence to the Housing First model can be measured using a fidelity scale:
    • Choice/affordability, scatter site housing, separation of housing and treatment, service philosophy, service array, team structure.

Slide 21

Why Use a Fidelity Scale to Study FSPs?

Text description is shown below.

  • FSPs provide a natural experiment to study various approaches to housing and services.
  • Housing First model provides a gold standard.
  • Fidelity to Housing First provides a method of mapping FSP practices.
  • Opportunity to identify both best practices among FSPs and the important elements of Housing First.

Slide 22

Figure 2. Modified Gelberg-Andersen Behavioral Model for Clients of Full Service Partnerships

Text description is shown below.

Image of a flowchart showing predisposing variables such as age, gender, and homelessness leading to enabling variables, such as insurance and housing leading to need variables, such as serious mental illness. The Full Services Partnership addresses needs and risks. Engagement and retention in treatment improves retention in housing and improves outcomes.

Source: Derived from Gelberg, et al., 2002.

Slide 23

Mixed Methods Study

Text description is shown below.

  • Quantitative data:
    • Administrative data (N=8,553, 60% schizophrenia).
      • Provides information on housing, service utilization and costs.
      • Difference-in-difference analysis.
      • Propensity score matched control group.
    • Fidelity to Housing First obtained through a survey of 94 FSP practices:
      • Survey based on the HF Fidelity Scale.
      • Respondents were FSP teams + clients.
      • Allows us to link practices to outcomes.
  • Qualitative data:
    • Fidelity to Housing First obtained through 20 site visits.

Slide 24

FSP Survey Psychometrics Reveal a Two Factor Model of Fidelity

Text description is shown below.

  • Housing / service philosophy:
    • Housing process and structure.
    • Separation of housing and services.
    • Service philosophy.
  • Service array / team structure:
    • Service array.
    • Team structure.

Slide 25

Qualitative Analysis Reveal Implementation Factors

Text description is shown below.

  • Individuals: Program director knowledge and beliefs about the intervention.
  • Inner setting: Program culture, compatibility, communication, and readiness for implementation.
  • Outer setting: Clients’ needs and resources (i.e. Target pop.), External policy and incentives.

Slide 26

Image of a Flowchart

Text description is shown below.

Image of a flowchart.  It shows two domains: implementation and fidelity. The Implementation Domain has three elements: Individuals, Inner Setting and Outer Setting; all leading to Fidelity. Fidelity leads to the Fidelity Domains: Housing/Service Philosophy and Service Array Treatment.

Slide 27

Gnomes Plan for Profit

Text description is shown below.

Screen shot from the South Park cartoon series. It shows a poster with three phases on it.  Phase 1 says Collect underpants underneath. Phase 2 says "?" underneath; and Phase 3 says Profit underneath. From South Park, S2, Ep17.

Slide 28

Image of a Flowchart

Text description is shown below.

Image of a flow chart. It shows the direct, indirect, and total effect of fidelity to HF. There are three ovals with text in them. The top oval says Target Population with an arrow to Client Outcomes as a direct effect. The left oval says Fidelity to HF with an arrow to Target Population as an indirect effect, and an arrow to the right oval Client Outcomes as a total effect.

Slide 29

Changes in Days Homeless Over One Year by Quintiles of Fidelity, Adjusted and Unadjusted

Text description is shown below.

Image of a bar chart showing homelessness decreasing over one year.

Slide 30

Difference in Difference Estimates of Costs by Quintile of Propensity Score

Text description is shown below.

Image of two bar charts. One shows inpatient costs staying fairly steady over the first four quintiles, then dropping dramatically the last quintile. Outpatient costs are also shown as fairly steady with the last quintile going up slightly.

Slide 31

Conclusions

Text description is shown below.

  • Logic models are a useful tool in helping us to understand how variation in implementation is related to outcomes.
  • Logic models can help identify gaps in our understanding mid-research.
  • A logic model in this study helps to illustrates how program fidelity relates to the illness severity of the population enrolled, and how that related to outcomes.

Slide 32

Thank You for Attending!

Text description is shown below.

Webinar #4: Formative Evaluation

Presenter: Jeffrey Smith, PhD
Discussant: Cheryl McDonnell, PhD
Moderator: Michael  I. Harrison, PhD

Sponsored by AHRQ’s Delivery System Initiative in partnership with the AHRQ PCMH program

July 15, 2013 1 p.m.

Register for this and other events in our series on advanced methods in delivery system research, here: http://bit.ly/EconometricaAHRQ

Slide 33

For More Information...

Text description is shown below.

For more information about the AHRQ PCMH Research Methods briefs, please visit:
 http://www.pcmh.ahrq.gov/portal/server.pt/community/pcmh__home/1483/pcmh_evidence___evaluation_v2

Page last reviewed July 2013
Internet Citation: Advanced Methods in Delivery System Research - Planning, Executing, Analyzing, and Reporting Research on Delivery System Improvement: Webinar #3: Logic Models (Slide Presentation). July 2013. Agency for Healthcare Research and Quality, Rockville, MD. http://www.ahrq.gov/professionals/prevention-chronic-care/improve/coordination/webinar03/logicmodelsl.html