Building the Science of Health Care Quality Improvement Intervention (Text Version)

On September 28, 2010, Denise Dougherty made this presentation at the 2010 Annual Conference. Select to access the PowerPoint® presentation (676 KB).

Slide 1

Building the Science of Health Care Quality Improvement Intervention

Denise Dougherty, Ph.D.

Senior Advisor, Child Health and Quality Improvement

AHRQ Annual Conference 2010

Coordinator and Moderator

Implementation, Change, and Improving Health Care Quality and Safety: Lessons Learned From AHRQ’s Implementation Science Awards
September 28, 2010

Slide 2

Overview (15 minutes)

• Introductory Comments: Why and how does AHRQ Focus on Implementation Science?
• An emerging framework
• Implementation Science grantees work
  • Rita Mangione-Smith
  • Carrie Byington
• Interactive Discussion

Slide 3

The "3T's" Road Map to Transforming U.S. Health Care

Source: JAMA, May 21, 2008: D. Dougherty and P.H. Conway, pp. 2319-2321. The "3T's Roadmap to Transform U.S. Health Care: The 'How' of High-Quality Care."
 

Slide 4

Relevant Provisions in New Laws: American Recovery and Revitalization Act (ARRA)

• "Meaningful Use" of Health IT =
• • Beyond getting the electrons in place
  • Using health IT to improve quality and safety of health care
• Funding for Comparative Effectiveness Research (CER) beyond purely clinical interventions
• • System redesign
  • Enhanced registries for QI and CER
  • Accelerating Implementation of Comparative Effectiveness Findings on Clinical and Delivery System Interventions by Leveraging AHRQ Networks
  • Other (http://www.ahrq.gov/fund/granarch.html#RFA)

Slide 5

New Laws: Patient Protection and Affordable Care Act

• Demonstration Projects for Quality Improvement
• • "demonstration"—312 mentions
  • "pilot" -80 mentions#
• Creation of the Center for Medicare and Medicaid Innovation (CMI)
• National Strategy for Quality Improvement
• More (see CRS report)

# http://e-caremanagement.com/pilots-demonstrations-innovation-in-the-ppaca-healthcare-reform-legislation/
http://www.aamc.org/reform/summary/ph.pdf

Slide 6

CHIPRA—CMS Quality Demonstration Grants to States

• Aims:
  • A) Experiment with, and evaluate the use of new measures for quality of Medicaid/CHIP children's health care;
  • B) Promote the use of HIT for the delivery of care for children covered by Medicaid/CHIP;
  • C) Evaluate provider-based models which improve the delivery of Medicaid/CHIP children's health care services; or
  • D) Demonstrate the impact of the model Electronic Health Record format for children (developed and disseminated under section 401(f)) on improving pediatric health, and pediatric health care quality, as well as reducing health care costs.
  • E) Broad systems approaches/medical home
• 10 awards made Feb. 2010 to individual States and consortia of States
• National Evaluation (in planning stage -AHRQ has lead)

http://www.cms.gov/CHIPRA/15_StateDemo.asp

Slide 7

AHRQ Funding to Test and Disseminate Strategies to Improve Quality and Patient Safety

• ACTION II
• Program Announcements
• Evaluation of Spread of the Keystone projects (health care associated infections)
• Assessing the Evidence for Context-Sensitive Effectiveness and Safety of Patient Safety Practices: Developing Criteria (forthcoming report)
• Co-sponsorship of the NIH Science of D & I conference (2011)
• Innovations Clearinghouse
• Knowledge Transfer Projects
• Value Exchanges
• PAR 08-136

Slide 8

What are We Trying to Learn from all This Work?

• Answer: Not Only the What, but the How and the Why of Healthcare Quality Improvement

Slide 9

Current State of the Science: QII and Evaluation Designs-A Personal View

• Problem identification (vaguely defined)
• Theory of action to solve the problem (often omitted, vague or in-name-only)
• Interventions
  • Vaguely described;
  • Not replicable;
  • Conceptual confusion between "intervention" and "implementation"
• Focus on internal validity and related designs
  • Context of intervention/ implementation processes:
    • Not considered or
    • Considered post hoc and descriptive/idiosyncratic
    • Effects of context/variation in context not considered in assessing results and variation in results
    • "qualitative" research does not mean standards of the qualitative research field
    • Lack of validated measures of contextual variables (leadership, culture, teamwork, resources)
  • For publication: design driven by clinical hierarchy of evidence standards (RCTs at patient level)
  • If not for publication:
    • Threats to internal validity rarely considered; post only studies or simple pre-post without comparisons; implications for knowledge base not widely recognized.
• Few comparison studies (one QI intervention to another; multiple settings)

Slide 10

Specific Example: Context- Multiple potential influences on QII Results

• External factors—e.g.:
• • Regulatory requirement
  • Payments or penalties
  • Local sentinel event
• Structural/organizational characteristics (organization site)
• Culture, Teamwork, Leadership
• Implementation Processes and Tools
  • Staff education and training
  • Audit and feedback

Source: Shekelle, Pronovost, and Wachter, Contract Report to AHRQ, Contract #HHSA-290-2009-10001C, forthcoming.

Slide 11

Specific Example: Quantitative Approaches to Context Heterogeneity —Progress

• Premise: Context often moderates intervention effectiveness
• This moderation effect can be represented statistically through the "intervention x context" interaction:
• • Yi = b0 + b1  Ti + b2  Ci + b12  Ti  Ci + ei,
  • • Where i denotes the unit of analysis (usually the various sites in the study, but can also be dyads of sites in matched comparisons), Yi denotes the outcome measure, Ti denotes the intervention status (Ti=1 for intervention, Ti=0 for control), Ci denotes the contextual factor, Ti  Ci denotes the "intervention  context" interaction, ei denotes random error, b0 denotes the intercept for the model, b1 denotes the main effect for the intervention, b2 denotes the main effect for the contextual factor, and b12 denotes the moderation effect for the contextual factor, i.e., the influence of the contextual factor on intervention effectiveness.
• Looks like progress, assuming we can quantify contextual variables

Source: Shekelle, Pronovost, and Wachter, Contract Report to AHRQ, Contract #HHSA-290-2009-10001C, Chapter 12, Special contribution from Naihua Duan, Columbia University, New York, New York forthcoming

Slide 12

Meta-Science Issues

• IRBs
• Study sections
• Promotion and Tenure
• Little collaborative research (understanding effects of variations in context)
• Conflict between research and evaluation
• Limited knowledge of evaluation "how to"?

Slide 13

Research in Implementation and Change While Improving Quality—The Answer?

• PAR -08-136
  • A relatively small attempt to specifically try to understand the how and why in a rigorous way
    • $300K/year
  • No dedicated pot of funds at AHRQ—highly competitive
• This session:
  • Two examples—in process—methods and interim results, not definitive findings
  • Interactive discussion—
    • What would you add?
    • What else do you need to know?