Evaluation of the Effectiveness of AHRQ's Grant Supported Research on Healthcare Costs, Productivity, Organization, and Market Forces: Overview of Project Findings

Slide presentation from the AHRQ 2010 conference.

On September 28, 2010, Marsha Gold made this presentation at the 2010 Annual Conference. Select to access the PowerPoint® presentation (1.1 MB). Free PowerPoint® Viewer (Plugin Software Help).


Slide 1

 Evaluation of the Effectiveness of AHRQ's Grant Supported Research on Healthcare Costs, Productivity, Organization, and Market Forces: Overview of Project Findings

Evaluation of the Effectiveness of AHRQ's Grant Supported Research on Healthcare Costs, Productivity, Organization, and Market Forces:
Overview of Project Findings

by

Marsha Gold, Sc.D., Project Director
Timothy Lake, Ph.D., Deputy Director
Kate Stewart, Ph.D., Researcher
Tara Krissik, M.P.P., Researcher

December 2008

This project was funded by the Agency for Healthcare Research and Quality.

Note: On the bottom of every slide is the logo for Mathematica Policy Research, Inc.

Slide 2

 Rationale for Project

Rationale for Project

  • While AHRQ's statutory mission explicitly includes support for grant funded research on healthcare costs, productivity, organization, and market forces, the agency does not systematically track the work it funds in this area and what has been accomplished through it.
  • Purpose of this project is to address this gap and identify how AHRQ may improve the visibility and impact of this research.

Slide 3

 Project Details

Project Details

  • MPR evaluation began in October 2006; the two phased project was completed in December 2008.
  • Work was guided by an Expert Panel: Sharon Arnold (AcademyHealth), John Christianson (University of Minnesota), Paul Ginsburg (Center for Studying Health Systems Change), Robert Helms (AEI), and Gail Wilensky (Project Hope).
  • Michael Hagan was the AHRQ project officer.

Slide 4

 Key Questions of Interest

Key Questions of Interest

  1. Since the late 1990s, what grant research has AHRQ funded that relates to healthcare costs, productivity, organization, and market forces?
  2. How are the research findings disseminated to public and private decision makers and what factors contribute to their use?
  3. What is AHRQ's role in supporting research in this area and how does it compare with that of others, such as NIH and private funders?
  4. What actions, if any, could enhance AHRQ's efforts to track, disseminate and encourage use of these research findings?

Slide 5

 Components of the Study-I

Components of the Study—I

  • Descriptive Analysis: Reviewed AHRQ administrative data on all grants, and conducted indepth analysis of a stratified random sample of nine grants.
  • Conceptual Framework: Using relevant social science literature, identified pathways for translation and key variables that are likely to influence use of research.
  • Case Studies: In depth description and analysis of 7 grants that illustrate diverse ways in which study findings reach target audiences and influence policy debates; based on PI and user interviews.

Slide 6

 Components of the Study-II

Components of the Study—II

  • Comparative Funder Analysis: Comparison of AHRQ's infrastructure and funding for grants in this area to that of NIH (government) and HCFO (private sector).
  • Federal Research Translator/User Interviews: Identify how and when this type of research gets used, and awareness of AHRQ research. (Included CBO, GAO, MedPAC, FTC, DOJ, and selected others).
  • PI Survey: Web based survey in July-August 2008 to all 149 grantees (70% response). Detail on research focus and key findings, dissemination modes and preferences, uses made of results, and PIs interaction with policymakers. Also obtained feedback on AHRQ's role and performance and how AHRQ compares to other funders.

Slide 7

 Components of the Study-III

Components of the Study—III

Key Research QuestionsDescriptive Analysis of GrantsFramework on Research UseCase Studies of UseComparative Funder AnalysisFederal Research Translator InterviewsPrincipal Investigator Survey
1. What has AHRQ funded in these areas and how is it used?X X  X
2. How are the findings disseminated and what contributes to their use? XX XX
3. What is AHRQ's role in this research area and how does its performance compare with other funders?   X X
4. What actions, if any, could enhance AHRQ's efforts to track, disseminate, and encourage use of these research findings?XXXXXX

Slide 8

Key Findings: What has AHRQ Funded in this Area between 1998-2006?  

Key Findings: What has AHRQ Funded in this Area between 1998-2006?

Slide 9

Number of New AHRQ Grants for Research on Healthcare Costs, Productivity, Organization, and Market Forces Funded, 1998-2006  

Number of New AHRQ Grants for Research on Healthcare Costs, Productivity, Organization, and Market Forces Funded, 1998-2006

Image: A bar chart shows the following data:

  • 1998: 1
  • 1999: 2
  • 2000: 31
  • 2001: 41
  • 2002: 34
  • 2003: 17
  • 2004: 5
  • 2005: 9
  • 2006: 10

Source: MPR analysis of AHRQ Administrative Data.

Slide 10

 AHRQ Funding for Research on Healthcare Costs, Productivity, Organization, and Market Forces, 1998-2006

AHRQ Funding for Research on Healthcare Costs, Productivity, Organization, and Market Forces, 1998-2006

Image: A bar chart shows the following data:

  • 1998: $393,559
  • 1999: $887,243
  • 2000: $9,822,272
  • 2001: $15,864,342
  • 2002: $19,292,051
  • 2003: $14,177,886
  • 2004: $10,752,562
  • 2005: $5,103,536
  • 2006: $5,096,191

Source: MPR analysis of AHRQ Administrative data.

Note: Includes funding for new grants and continuing grants that were funded in 1998 or later. The combined spending over the period was $81.4 million, of which $12.6 million was for three PO1 grants. 47 RO3 grants under $100,00 accounted for about $4 million in spending.

Slide 11

 Characteristics of Funded Studies

Characteristics of Funded Studies

  • Of the 149 studies:
    • 97 studies had organizations as the unit of analysis. Studies of hospitals and health plans were particularly common.
    • 37 studies examined consumer behavior.
    • 15 studies examined markets or purchaser behavior.
  • 75 percent of the 149 were national in scope.
  • Multiple outcomes studied including quality (53%), use (44%), cost (38%) and access (19%), among others.

Slide 12

 Nature of Research

Nature of Research

PI survey indicates that findings commonly examine how specific outcomes of care are influenced by:

  • Economic factors (e.g., provider payment, insurance coverage)
  • Organizational characteristics (e.g., nurse leadership, volume)
  • Systems and markets (e.g., HMO penetration, capacity constraints)
  • Patient characteristics and preferences (e.g., percent minority, DNR orders)

Findings address questions of substantial policy interest today about influences on the performance of health care system.

Slide 13

 Key Findings: How are Findings Disseminated and What Factors Contribute to Use

Key Findings: How are Findings Disseminated and What Factors Contribute to Use

Slide 14

Dissemination May Occur Through Diverse Pathways-I  

Dissemination May Occur Through Diverse Pathways—I

  • We identified 10 pathways that differ in means used to communicate findings and the role researchers, intermediaries and users play in the process. Pathways include:
    • Traditional: "Big Bang," Gradual Accumulation and Diffusion, Gradual Communication and Formal Syntheses
    • Effective Use of Intermediaries: Researcher as Messenger/Expert, Formal Intermediary Broker, Press Publicizes
    • User Guided: User Directed Syntheses, User Partner Grant Review, User Commissioned Studies, Researcher as User

Slide 15

Dissemination May Occur Through Diverse Pathways-II  

Dissemination May Occur Through Diverse Pathways—II

  • Some pathways involve researchers directly engaging with users and others involve intermediaries that help synthesize, apply or communicate the findings.
  • Diverse pathways will be effective in different circumstances; effectively reaching diverse audiences for the work enhances the use of well targeted high quality research.

Slide 16

 PI Survey Shows Preference for Publication as Dissemination Strategy but Use of Multiple Modes

PI Survey Shows Preference for Publication as Dissemination Strategy but Use of Multiple Modes

Dissemination StrategyPercentage of Respondents Reporting
Major FocusMinor Focus
Publications (any)
Journal article(s)
Research report/working paper
User-focused research brief/issue paper
Chapter
Book
97
91
29
12
6
3
2
5
27
19
18
0
Conference Presentations (any)
Paper or poster at research conference
Presentation at policy-/user-focused meeting
70
57
34
28
28
32
Briefings (any)
Policymaker briefings
Managerial briefings
Interest group briefings
27
18
11
11
26
19
17
18
Other Forms of Dissemination (any)
Mass media
Federal or state testimony
Expert witness
Other
9
5
2
0
3
27
23
8
3
4

Source: MPR Survey of AHRQ-Funded Principal Investigators.

Note: Responses based on 97 responding PIs (70% of grantees surveyed).

Slide 17

 Support from Host Institution for Dissemination Limited

Support from Host Institution for Dissemination Limited

Type of AssistanceAmount of Assistance (Reported %)
SubstantialModerateLimitedNone/Did Not
Use Resource
Press office for interaction with media9%28%24%40%
Newsletters reporting on key findings from research10%17%20%54%
Established working paper series3%11%7%79%
Established series of research/issue briefs6%7%14%74%
Funds available to develop user-oriented materials 0%2%4%93%
Information or training on how to understand and interact with potential users of research1%3%13%83%

Source: MPR Survey of AHRQ-Funded Principal Investigators.

Notes: N = 91. These analyses exclude 6 respondents who did not respond to any questions about dissemination support.

Slide 18

 Mean Percentage of Time PIs Report Interacting with Policymakers

Mean Percentage of Time PIs Report Interacting with Policymakers

Image: A bar chart shows the following data:

  • Actual: 10%
  • Desire: 15%

Source: MPR Survey of AHRQ-Funded Principal Investigators.

Slide 19

The Impact of Research on Policy-I  

The Impact of Research on Policy—I

  • Research is one of several influences on decision-making. Importance of research likely to vary with the topic, user, environment and other factors.
  • Some research may have an immediate impact but most accumulates and gets applied to relevant topics as issues arise.
  • Accumulation of research contributes to a "research reservoir"—use more likely if findings readily available to potential users and applicability of findings clear.

Slide 20

 The Impact of Research on Policy-II

The Impact of Research on Policy—II

  • Accumulation of findings across multiple studies adds weight to findings.
  • Some forms of communication are likely to be stronger in reaching users than others.
  • Researchers can enhance usefulness by thinking carefully about the key findings and identify the main "message" of the study ("elevator test").

Slide 21

 PI Perception of Outcomes Relevant to Their Grants

PI Perception of Outcomes Relevant to Their Grants

Image: Bar chart shows percentages of outcomes investigators say are relevant to their goals:

  • Add to general knowledge/wisdom: 100%
  • Inform other research: 100%
  • Contribute to policy debate: 92%
  • Target areas for future research: 87%
  • New research methods: 73%
  • Help organizations improve: 69%
  • New models: care/policy design: 67%

Source: MPR Survey of AHRQ-Funded Principal Investigators.

Note: N = 85.

Slide 22

Lessons from Case Studies on Ways to Enhance Use  

Lessons from Case Studies on Ways to Enhance Use

  • Develop relationships with potential users and involve them early.
  • Be aware of how results may be relevant to different policy decisions and the timing of those decisions.
  • Identify where research "fits" in stream of research.
  • Develop expertise—and reputation for it.

Slide 23

What Federal Research Translators Told Us  

What Federal Research Translators Told Us

  • They make extensive use of this type of research.
  • They are familiar with ongoing work and consult the literature as time allows.
  • They value timeliness but also place a high value on quality and objectivity.
  • Regulatory entities find general studies of markets useful in providing context for case specific analysis.
  • They see critical gaps in the existing body of research in this area.

Slide 24

 Key Findings: What Is AHRQ's Role in Context of NIH, HCFO, and Other Funders?

Key Findings: What Is AHRQ's Role in Context of NIH, HCFO, and Other Funders?

Slide 25

 General Findings Across the Project

General Findings Across the Project

  • Researchers view AHRQ as a major source of funding for this research but have turned elsewhere as AHRQ funds have become less available. Low funding levels for this kind of research at AHRQ was the most common problem cited by PIs.
  • PIs view AHRQ's oversight over grants more positively than the agency's work on dissemination and communication.
  • While NIH processes mirror those of AHRQ, HCFO has more emphasis on dissemination, and researchers score HCFO higher than other funders on technical assistance
  • Federal translators perceive AHRQ as more a primary resource for databases and clinical research than for this type of research.

Slide 26

 Share of Respondents Who Consider Various Funding Organizations to be Major, Moderate, Minor, or Not a Source of Funding for Research on Health Care Costs, Productivity, Organization, and Market Forces (percentages unless otherwise noted)

Share of Respondents Who Consider Various Funding Organizations to be Major, Moderate, Minor, or Not a Source of Funding for Research on Health Care Costs, Productivity, Organization, and Market Forces (percentages unless otherwise noted)

OrganizationsMajor Source of FundingModerate FundingLittle FundingNo FundingNo Opinion/ Refused
Agency for Healthcare Research and Quality (AHRQ)40%302730
National Institutes of Health (NIH)40%302433
Robert Wood Johnson Foundation's Healthcare Financing and Organization (HCFO) program39%341917
Centers for Medicare & Medicaid Services (CMS)7%1362613
Robert Wood Johnson Foundation, and other programs10%432549
Other federal or state government4%1513564
Other private foundations4%2416353

Source: MPR Survey of AHRQ-Funded Principal Investigators.

Note: N = 96.

Slide 27

Ratings of AHRQ Technical Support  

Ratings of AHRQ Technical Support

All RespondentsRatings Among Those Who Consider Task Part of AHRQ's Job
Consider Part of
AHRQ's Job
(Number)
Mean ScorePercent Who
Say Excellent
or Very Good
Pre-award guidance822.168%
Grant award862.065%
Grant management802.359%
Research methods402.553%
Dissemination and communication of findings732.740%
Linkages with others interested in the topic of your research622.944%

Source: MPR Survey of AHRQ-Funded Principal Investigators.

Note: Mean scores calculated based on respondent ratings of AHRQ technical support, where 1 = Excellent; 2 = Very Good; 3 = Good; 4 = Fair; 5 = Poor

Slide 28

 Comparison of Technical Assistance Provided by AHRQ, NIH, HCFO, and Others

Comparison of Technical Assistance Provided by AHRQ, NIH, HCFO, and Others

 Number Reported Funding Source and Provided RatingMean Score (SD)
Robert Wood Johnson Foundation's Healthcare Financing and Organization (HCFO) program351.8 (0.9)
Agency for Healthcare Research and Quality (AHRQ)882.3 (1.1)
Robert Wood Johnson Foundation, other programs452.3 (0.9)
Other private foundations282.4 (0.8)
National Institutes of Health (NIH)602.5 (1.1)
Other federal or state government232.6 (1.2)
Centers for Medicare & Medicaid Services (CMS)163.4 (1.0)

Source: MPR Survey of AHRQ-Funded Principal Investigators.

Note: Mean scores calculated based on respondent ratings of funders' technical support, where 1 = Excellent; 2 = Very Good; 3 = Good; 4 = Fair; 5 = Poor.

Slide 29

Key Findings: Actions That Could Enhance Use of Research Findings  

Key Findings: Actions That Could Enhance Use of Research Findings

Slide 30

 Conclusions From the Project

Conclusions From the Project

  • AHRQ currently is not highly visible as a source of research on health care costs, productivity, organization and market forces despite a legislative mandate for it and $81 million spent on grants for this kind of research from 1998 through 2006.
  • Our findings show that the research AHRQ has funded addresses critical aspects of health system performance that are important to decision-makers.
  • Research AHRQ has funded has had an impact but mainly project by project at the initiative of the investigator.
  • AHRQ can better track its work and leverage its investments to make its research more visible, make findings more accessible, and enhance their use.

Slide 31

 Recommendations for AHRQ-I

Recommendations for AHRQ—I

  • Use our study as basis for ongoing tracking of research.
  • Create a visible place on the Agency Web site to communicate AHRQ's efforts in this area.
  • Create internal capacity for monitoring grant progress to identify emerging outcomes that may be of interest; reward staff for support in this area.
  • Communicate with PIs why AHRQ would value ongoing reports of dissemination and use after grants end and develop systems to support such monitoring.

Slide 32

 Recommendations for AHRQ-II

Recommendations for AHRQ—II

  • Use our project to develop an annotated bibliography with abstracts of publications from AHRQ funded research in this area.
  • Work with others on targeted syntheses of findings on topics which AHRQ has invested.
  • Help others contact experts on given research topics through online contact information on the Web site or other means.
  • Offer researchers a "webinar" on ways of thinking about, developing, and communicating "messages" from the study.

Slide 33

 Recommendations for AHRQ-III

Recommendations for AHRQ (and Others)—III

  • Elicit more feedback from users on their needs to identify areas that can be supported by indepth, high quality research.
  • Work with others to better articulate what this type of research is and its relevance to understanding health system performance and how to enhance outcomes.
Current as of December 2010
Internet Citation: Evaluation of the Effectiveness of AHRQ's Grant Supported Research on Healthcare Costs, Productivity, Organization, and Market Forces: Overview of Project Findings . December 2010. Agency for Healthcare Research and Quality, Rockville, MD. http://www.ahrq.gov/news/events/conference/2010/gold/index.html