Evaluation of the Effectiveness of AHRQ's Grant Supported Research on Healthcare Costs, Productivity, Organization, and Market Forces: Overview of Project Findings

On September 28, 2010, Marsha Gold made this presentation at the 2010 Annual Conference. Select to access the PowerPoint® presentation (1.1 MB). Free PowerPoint® Viewer (Plugin Software Help).

Slide 1

Evaluation of the Effectiveness of AHRQ's Grant Supported Research on Healthcare Costs, Productivity, Organization, and Market Forces:
Overview of Project Findings

by

Marsha Gold, Sc.D., Project Director
Timothy Lake, Ph.D., Deputy Director
Kate Stewart, Ph.D., Researcher
Tara Krissik, M.P.P., Researcher

December 2008

This project was funded by the Agency for Healthcare Research and Quality.

Note: On the bottom of every slide is the logo for Mathematica Policy Research, Inc.

Slide 2

Rationale for Project

• While AHRQ's statutory mission explicitly includes support for grant funded research on healthcare costs, productivity, organization, and market forces, the agency does not systematically track the work it funds in this area and what has been accomplished through it.
• Purpose of this project is to address this gap and identify how AHRQ may improve the visibility and impact of this research.

Slide 3

Project Details

• MPR evaluation began in October 2006; the two phased project was completed in December 2008.
• Work was guided by an Expert Panel: Sharon Arnold (AcademyHealth), John Christianson (University of Minnesota), Paul Ginsburg (Center for Studying Health Systems Change), Robert Helms (AEI), and Gail Wilensky (Project Hope).
• Michael Hagan was the AHRQ project officer.

Slide 4

Key Questions of Interest

1. Since the late 1990s, what grant research has AHRQ funded that relates to healthcare costs, productivity, organization, and market forces?
2. How are the research findings disseminated to public and private decision makers and what factors contribute to their use?
3. What is AHRQ's role in supporting research in this area and how does it compare with that of others, such as NIH and private funders?
4. What actions, if any, could enhance AHRQ's efforts to track, disseminate and encourage use of these research findings?

Slide 5

Components of the Study—I

• Descriptive Analysis: Reviewed AHRQ administrative data on all grants, and conducted indepth analysis of a stratified random sample of nine grants.
• Conceptual Framework: Using relevant social science literature, identified pathways for translation and key variables that are likely to influence use of research.
• Case Studies: In depth description and analysis of 7 grants that illustrate diverse ways in which study findings reach target audiences and influence policy debates; based on PI and user interviews.

Slide 6

Components of the Study—II

• Comparative Funder Analysis: Comparison of AHRQ's infrastructure and funding for grants in this area to that of NIH (government) and HCFO (private sector).
• Federal Research Translator/User Interviews: Identify how and when this type of research gets used, and awareness of AHRQ research. (Included CBO, GAO, MedPAC, FTC, DOJ, and selected others).
• PI Survey: Web based survey in July-August 2008 to all 149 grantees (70% response). Detail on research focus and key findings, dissemination modes and preferences, uses made of results, and PIs interaction with policymakers. Also obtained feedback on AHRQ's role and performance and how AHRQ compares to other funders.

Slide 7

Components of the Study—III

Slide 8

Key Findings: What has AHRQ Funded in this Area between 1998-2006?

Slide 9

Number of New AHRQ Grants for Research on Healthcare Costs, Productivity, Organization, and Market Forces Funded, 1998-2006

Image: A bar chart shows the following data:

• 1998: 1
• 1999: 2
• 2000: 31
• 2001: 41
• 2002: 34
• 2003: 17
• 2004: 5
• 2005: 9
• 2006: 10

Source: MPR analysis of AHRQ Administrative Data.

Slide 10

AHRQ Funding for Research on Healthcare Costs, Productivity, Organization, and Market Forces, 1998-2006

Image: A bar chart shows the following data:

• 1998: $393,559
• 1999: $887,243
• 2000: $9,822,272
• 2001: $15,864,342
• 2002: $19,292,051
• 2003: $14,177,886
• 2004: $10,752,562
• 2005: $5,103,536
• 2006: $5,096,191

Source: MPR analysis of AHRQ Administrative data.

Note: Includes funding for new grants and continuing grants that were funded in 1998 or later. The combined spending over the period was $81.4 million, of which $12.6 million was for three PO1 grants. 47 RO3 grants under $100,00 accounted for about $4 million in spending.

Slide 11

Characteristics of Funded Studies

• Of the 149 studies:
  • 97 studies had organizations as the unit of analysis. Studies of hospitals and health plans were particularly common.
  • 37 studies examined consumer behavior.
  • 15 studies examined markets or purchaser behavior.
• 75 percent of the 149 were national in scope.
• Multiple outcomes studied including quality (53%), use (44%), cost (38%) and access (19%), among others.

Slide 12

Nature of Research

PI survey indicates that findings commonly examine how specific outcomes of care are influenced by:

• Economic factors (e.g., provider payment, insurance coverage)
• Organizational characteristics (e.g., nurse leadership, volume)
• Systems and markets (e.g., HMO penetration, capacity constraints)
• Patient characteristics and preferences (e.g., percent minority, DNR orders)

Findings address questions of substantial policy interest today about influences on the performance of health care system.

Slide 13

Key Findings: How are Findings Disseminated and What Factors Contribute to Use

Slide 14

Dissemination May Occur Through Diverse Pathways—I

• We identified 10 pathways that differ in means used to communicate findings and the role researchers, intermediaries and users play in the process. Pathways include:
  • Traditional: "Big Bang," Gradual Accumulation and Diffusion, Gradual Communication and Formal Syntheses
  • Effective Use of Intermediaries: Researcher as Messenger/Expert, Formal Intermediary Broker, Press Publicizes
  • User Guided: User Directed Syntheses, User Partner Grant Review, User Commissioned Studies, Researcher as User

Slide 15

Dissemination May Occur Through Diverse Pathways—II

• Some pathways involve researchers directly engaging with users and others involve intermediaries that help synthesize, apply or communicate the findings.
• Diverse pathways will be effective in different circumstances; effectively reaching diverse audiences for the work enhances the use of well targeted high quality research.

Slide 16

PI Survey Shows Preference for Publication as Dissemination Strategy but Use of Multiple Modes

Source: MPR Survey of AHRQ-Funded Principal Investigators.

Note: Responses based on 97 responding PIs (70% of grantees surveyed).

Slide 17

Support from Host Institution for Dissemination Limited

Source: MPR Survey of AHRQ-Funded Principal Investigators.

Notes: N = 91. These analyses exclude 6 respondents who did not respond to any questions about dissemination support.

Slide 18

Mean Percentage of Time PIs Report Interacting with Policymakers

Image: A bar chart shows the following data:

• Actual: 10%
• Desire: 15%

Source: MPR Survey of AHRQ-Funded Principal Investigators.

Slide 19

The Impact of Research on Policy—I

• Research is one of several influences on decision-making. Importance of research likely to vary with the topic, user, environment and other factors.
• Some research may have an immediate impact but most accumulates and gets applied to relevant topics as issues arise.
• Accumulation of research contributes to a "research reservoir"—use more likely if findings readily available to potential users and applicability of findings clear.

Slide 20

The Impact of Research on Policy—II

• Accumulation of findings across multiple studies adds weight to findings.
• Some forms of communication are likely to be stronger in reaching users than others.
• Researchers can enhance usefulness by thinking carefully about the key findings and identify the main "message" of the study ("elevator test").

Slide 21

PI Perception of Outcomes Relevant to Their Grants

Image: Bar chart shows percentages of outcomes investigators say are relevant to their goals:

• Add to general knowledge/wisdom: 100%
• Inform other research: 100%
• Contribute to policy debate: 92%
• Target areas for future research: 87%
• New research methods: 73%
• Help organizations improve: 69%
• New models: care/policy design: 67%

Source: MPR Survey of AHRQ-Funded Principal Investigators.

Note: N = 85.

Slide 22

Lessons from Case Studies on Ways to Enhance Use

• Develop relationships with potential users and involve them early.
• Be aware of how results may be relevant to different policy decisions and the timing of those decisions.
• Identify where research "fits" in stream of research.
• Develop expertise—and reputation for it.

Slide 23

What Federal Research Translators Told Us

• They make extensive use of this type of research.
• They are familiar with ongoing work and consult the literature as time allows.
• They value timeliness but also place a high value on quality and objectivity.
• Regulatory entities find general studies of markets useful in providing context for case specific analysis.
• They see critical gaps in the existing body of research in this area.

Slide 24

Key Findings: What Is AHRQ's Role in Context of NIH, HCFO, and Other Funders?

Slide 25

General Findings Across the Project

• Researchers view AHRQ as a major source of funding for this research but have turned elsewhere as AHRQ funds have become less available. Low funding levels for this kind of research at AHRQ was the most common problem cited by PIs.
• PIs view AHRQ's oversight over grants more positively than the agency's work on dissemination and communication.
• While NIH processes mirror those of AHRQ, HCFO has more emphasis on dissemination, and researchers score HCFO higher than other funders on technical assistance
• Federal translators perceive AHRQ as more a primary resource for databases and clinical research than for this type of research.

Slide 26

Share of Respondents Who Consider Various Funding Organizations to be Major, Moderate, Minor, or Not a Source of Funding for Research on Health Care Costs, Productivity, Organization, and Market Forces (percentages unless otherwise noted)

Source: MPR Survey of AHRQ-Funded Principal Investigators.

Note: N = 96.

Slide 27

Ratings of AHRQ Technical Support

Source: MPR Survey of AHRQ-Funded Principal Investigators.

Note: Mean scores calculated based on respondent ratings of AHRQ technical support, where 1 = Excellent; 2 = Very Good; 3 = Good; 4 = Fair; 5 = Poor

Slide 28

Comparison of Technical Assistance Provided by AHRQ, NIH, HCFO, and Others

Source: MPR Survey of AHRQ-Funded Principal Investigators.

Note: Mean scores calculated based on respondent ratings of funders' technical support, where 1 = Excellent; 2 = Very Good; 3 = Good; 4 = Fair; 5 = Poor.

Slide 29

Key Findings: Actions That Could Enhance Use of Research Findings

Slide 30

Conclusions From the Project

• AHRQ currently is not highly visible as a source of research on health care costs, productivity, organization and market forces despite a legislative mandate for it and $81 million spent on grants for this kind of research from 1998 through 2006.
• Our findings show that the research AHRQ has funded addresses critical aspects of health system performance that are important to decision-makers.
• Research AHRQ has funded has had an impact but mainly project by project at the initiative of the investigator.
• AHRQ can better track its work and leverage its investments to make its research more visible, make findings more accessible, and enhance their use.

Slide 31

Recommendations for AHRQ—I

• Use our study as basis for ongoing tracking of research.
• Create a visible place on the Agency Web site to communicate AHRQ's efforts in this area.
• Create internal capacity for monitoring grant progress to identify emerging outcomes that may be of interest; reward staff for support in this area.
• Communicate with PIs why AHRQ would value ongoing reports of dissemination and use after grants end and develop systems to support such monitoring.

Slide 32

Recommendations for AHRQ—II

• Use our project to develop an annotated bibliography with abstracts of publications from AHRQ funded research in this area.
• Work with others on targeted syntheses of findings on topics which AHRQ has invested.
• Help others contact experts on given research topics through online contact information on the Web site or other means.
• Offer researchers a "webinar" on ways of thinking about, developing, and communicating "messages" from the study.

Slide 33

Recommendations for AHRQ (and Others)—III

• Elicit more feedback from users on their needs to identify areas that can be supported by indepth, high quality research.
• Work with others to better articulate what this type of research is and its relevance to understanding health system performance and how to enhance outcomes.