Methods and Metrics Issues in Delivery Systems Research

Slide Presentation by Jeff Alexander

On February 16, Jeff Alexander made this slide presentation at the AHRQ Expert Meeting on the Challenge and Promise of Delivery System Research.

Slide 1

Text Description is below the image.

Methods and Metrics Issues in Delivery System Research

Jeff Alexander
The University of Michigan

The Challenge and Promise of Delivery System Research: A Meeting of AHRQ Grantees, Experts, and Stakeholders
Doubletree Dulles — Sterling, Virginia
February 16, 2011

Slide 2

Text Description is below the image.

Challenges

  • Framing the issues around a common theme.
  • Striking a balance between depth and breadth.
  • Pushing the envelope without being unrealistic.
  • Examples from proposals.

Slide 3

Text Description is below the image.

What do We Want to Know about Delivery System Effectiveness?

  • Can it work?
  • Will it work?
    • When will it work (for which patients and settings)?
    • What is necessary for it to work?
  • Is it worth it?
    • Value
  • How can we put it into practice?
    • Implementation research

Slide 4

Text Description is below the image.

Delivery System Change

  • Manner in which a given DSC is implemented and practiced is influenced by range of factors surrounding the specific change.
  • These factors constitute the social system or social context of change.
  • These factors may explain why, when, and how a DSC works (or not).

Slide 5

Text Description is below the image.

Limits of Experimentalism

  • "Experimentalists have pursued too single-mindedly the question of whether a [social] program works at the expense of knowing why it works."
    • Pawson and Tilley (1997)
  • "....although the OXO model seeks generalizable knowledge, in that pursuit it relies on — it depends on — removing most of the local details about "how" something works and about the "what" of contexts."
    • Berwick (2008)

Slide 6

Text Description is below the image.

Five Methods and Metrics Issues

  • Modeling intervention context.
  • Assessing intervention fidelity and sustainability.
  • Incorporating time in delivery system models.
  • Measuring readiness for change.
  • Assessing complex, multi-component interventions.

Slide 7

Text Description is below the image.

Modeling Intervention Context

  • Interventions can display different strengths, causal directions, and base rates depending on the ecological conditions under which the processes or programs are observed.
  • Measure context as an analytic variable in our models– not as study setting descriptions.
  • Analytic techniques for assessing contextual effects are robust–- concepts and measures of context are not.

Slide 8

Text Description is below the image.

Assessing Intervention Fidelity and Sustainability

  • Measurement of treatment fidelity provides a means for determining whether key program components were delivered as specified by the program logic model/theory.
  • Sustainability– are intervention components active long enough to produce the desired effect on individuals?
  • Fidelity and sustainability follow from interrelationships among a range of internal and external factors that constitute the social system that surrounds the intervention– not just individual attitudes.

Slide 9

Text Description is below the image.

Time

  • Time as an important analytic concept in its own right, not simply as an element of the research design.
  • Individual growth trajectories.
  • Organizational/system level developmental trajectories.

Slide 10

Text Description is below the image.

Readiness for Change

  • Not all delivery systems/organizations are good candidates for interventions and change.
  • Readiness for change a precursor to the successful implementation of complex changes in health care settings.
  • Readiness measured as collective motivation and collective capability.

Slide 11

Text Description is below the image.

Complex, Multi-Component Interventions

  • How the combined effects of multiple intervention elements affect outcomes.
  • How, and the extent to which, individual elements of the intervention contribute to these collective efforts.
  • Mixed methods designs are difficult to implement in a manner that creates synergistic benefits from the use of different forms of data collection and analysis.

Slide 12

Text Description is below the image.

General Recommendations

  • Don't reject traditional designs– complement them.
  • Consider methods and metrics challenges as a package of related issues.
  • Increase the synergistic value of mixed methods research.
  • Time as an analytic variable.
  • Robust theoretical frameworks to guide the application of analytic methods and measures.

Slide 13

Text Description is below the image.

From DSC Model to Practice

A graphic depiction of the delivery service change (DSC) model. It shows how the science of DSC and market demand can increase the number of systems, providers, and individuals who provide and receive evidence-based delivery system change to improve patient well being.

Slide 14

Text Description is below the image.

Thank you

Page last reviewed May 2011
Internet Citation: Methods and Metrics Issues in Delivery Systems Research: Slide Presentation by Jeff Alexander. May 2011. Agency for Healthcare Research and Quality, Rockville, MD. http://www.ahrq.gov/professionals/systems/system/delivery-system-initiative/alexander/alexander.html