Stakeholder Engagement in the Evidence-based Center (EPC) Practice Program

Select to access the PowerPoint® presentation (1.7 MB).

Slide 1

Engaging Stakeholders, Improving Research: Stakeholder Engagement in the Developing Evidence to Inform Decisions about Effectiveness (DEcIDE) Program

Jeanne-Marie Guise, MD, MPH
Professor
Oregon Health & Science University

Image: A cable car and a mountain in the background are shown.

Slide 2

Objectives

• Describe ways in which the EPC program has engaged stakeholders to improve relevance and applicability of its products.
• Present methods used for stakeholder engagement.
• Identify challenges in stakeholder engagement and strategies employed by the EPC program.

Slide 3

EPCs & SRC

Evidence-based Practice Centers (EPCs)

• 11 centers throughout the United States and Canada.
• Systematically and critically appraise existing research and synthesize knowledge.
• Produce comparative effectiveness and effectiveness reviews.

Scientific Resource Center (SRC)

• Responsible for scientific methodological work of reviews and other research projects for EPCs and DEcIDES.

Slide 4

EPCs & SRC

• Blue Cross and Blue Shield Association, Technology Evaluation Center, Chicago, IL.
• Brown University, Providence Rhode Island.
• ECRI Institute, Plymouth Meeting, PA.
• Johns Hopkins University, Baltimore, MD.
• Kaiser Foundation Research Institute, Oakland, CA.
• Oregon Health & Science University, Portland, OR.
• RTI International—University of North Carolina at Chapel Hill, Chapel Hill, NC.
• RAND Corporation, Santa Monica, CA.
• University of Alberta, Edmonton, Edmonton, Alberta.
• University of Minnesota EPC, Minneapolis, MN.
• Vanderbilt University Medical Center, Nashville, TN.
• Scientific Resource Center (SRC) for EPCs and DEcIDES—Portland VA Research Foundation, Portland OR.

Slide 5

Opportunities for Stakeholder Input

Image: At the center of the image is a green oval labeled "Public." Arrows point out from this oval to other ovals in a circle around it. These ovals are labeled:

• Disseminate Reports.
• Nominate Review Topics.
• Key Informants.
• Key Research Questions.
• Technical Expert Panel.
• Peer and Public Review.
• Feedback on Summary Guides.
• Future Research.

Slide 6

EHC Report on Engaging Stakeholders for Development and Prioritization of Future Research Needs

• O'Haire C, McPheeters M, Nakamoto E, LaBrant L, Most C, Lee K, Graham E, Cottrell E, Guise J-M. Engaging Stakeholders To Identify and Prioritize Future Research Needs. Rockville (MD): Agency for Healthcare Research and Quality (US); 2011 Jun. (Methods Future Research Needs Reports, No. 4.) Available from: http://www.ncbi.nlm.nih.gov/books/NBK62565.

Slide 7

Image: A chart displays the following process:

Phase I: Literature Summary (Oregon EPC) →

• Literature Search →
• Literature Review →
• Literature Synthesis →

Phase II: Key Informant Interviews (Oregon EPC) →

• Recruitment and Interview  Guide Development →
• Key Informant Interviews →
• Analysis of Interviews →

Phase III: EPC interviews (Vanderbilt EPC) →

• EPC Protocol Review →
• Invitations and EPC Discussion Guide →
• EPC Discussions →
• Analysis of Discussions →

All three sets of phases lead to Cumulative Synthesis.

Slide 8

Research Needs Development

Stakeholder Identification & Recruitment

• Identify pertinent stakeholder groups (intentional based upon stakeholder groupings).
• Ensure a balance of stakeholder perspectives (i.e., consumer, clinician, researcher, research funder, insurer/payer, manufacturer).
• The results of EPC FRN pilot projects suggest that engaging 6-10 stakeholders is appropriate.
• OMB restricts the number of participants in surveys to no more than 9 non-federal employees.

Slide 9

Stakeholders

Slide 10

Methods of Stakeholder Engagement

Engagement

• Most common: Focus Groups, Symposia/Conferences.

Prioritization

• Most common: Priority Questionnaire, Consensus.

Engagement & Prioritization Simultaneously

• Priority Survey.
• Delphi.

Slide 11

Slide 12

EHC Program Research Reviews

Topic Generation:

• Nominate topics via the Web.

Topic Development:

• Clarify nomination intent.
• Frame clinical context of nomination.

Topic Refinement:

• Provide "real world" input.
• Provide input on clinical logic.
• Public comment via Web.

Review:

• Provide methodological & clinical input.
• Provide input via peer review.
• Provide public comment via Web.

Research Needs Development:

• Participate in discussions to describe research gaps & prioritize research needs.

Report Translation & Dissemination:

• Participate in focus groups.
• Participate in individual interviews.
• Provide expert review of summary guides.

Image: A large arrow points from left to right across the top of these listed items. It is captioned along its length with "Topics," "Evidence Review," "Research," and "D&I."

Slide 13

Topic Generation Approaches

Nominal Group Process

• In-person.
• Idea generating.
• Intended to promote group participation

Our experience:

In-person brainstorming with flip chart used to document topics

Each state had 8 stickers to vote for priority topics:

#1 (2 red dots)
#2 (2 green dots)
#3 (2 blue dots), and
#4 (2 yellow dots)

Weighting system applied to priorities:

#1—4 points for each vote
#2—3 points
#3—2 points
#4—1 point

Images: Three photographs show the stickers with topic ideas, first as individual responses, then grouped for clarification and consolidation of ideas, then ranked.

Slide 14

Topic Generation Approaches

Medicaid Medical Directors

Analytic Process:

Weighting system:

#1—4 points
#2—3 points
#3—2 points
#4—1 point

Slide 15

Topic Generation Approaches

Prevention in Women's Health

Modified Process:

• Local Key Informants.
• Three web meetings:
  • May 11 – Framework & Initial Topics
    (Extranet & email topic generation)
  • June 8 – Topic Generation
    (Evidence Scan)
  • 13 v Final Prioritization.
• Prioritization:
  • Survey—survey monkey.
  • Web whiteboard during 3rd meeting.

Image: The logo of Prevention in Women's Health is shown.

Slide 16

Scoping CER

Stakeholders Engaged in:

• Development of CER key questions.
• Refinement of topic.
• Review of Draft.

Methods used:

• Group or individual calls.
• Web-based forums with survey type format to provide responses.

Slide 17

Preparation of Stakeholders Is Critical:

• Assume that all stakeholders need orientation to the CER process.
• Very important for them to know where they fit in and what to expect from their feedback.
• Familiarity with research methods likely to be very variable.
• Conflicts of interest are to be expected with stakeholder groups, but must be disclosed.

Slide 18

Stakeholder Preparatory Materials

Topic Generation:

• Nominate topics via the Web.

Topic Development:

• Clarify nomination intent.
• Frame clinical context of nomination.

Topic Refinement:

• Provide "real world" input from provider, patient, and policy perspectives.
• Provide input on clinical logic.
• Public comment via Web.

Review:

• Provide methodological & clinical input as a TEP member.
• Provide input via peer review.
• Provide public comment via Web.

Research Needs Development:

• Participate in discussions to describe research gaps & prioritize research needs.

Report Translation & Dissemination:

• Participate in focus groups.
• Participate in individual interviews.
• Provide expert review of summary guides.

Slide 19

Stakeholder Preparatory Materials

Slide 20

Future Research Needs Development

Purpose:

• Comparative Effectiveness Reviews (CERs) critically appraise and synthesize bodies of literature and identify areas where research is lacking or insufficient in either quantity or quality.
• There are limitations in funds available to support research.
• To fully actualize their potential, CERs should be utilized to inform and guide future research.
• Future Research Needs (FRN) documents are designed to facilitate this process by engaging stakeholders to prioritize future research topics.

Slide 21

Future Research Needs Development

• Step 1: Research gaps from CER interviews.
• Step 2: Identify & recruit stakeholders.
• Step 3: Orient stakeholders to project.
• Step 4: Solicit input on gaps, generate topics.
• Step 5: Engage stakeholders to prioritize FRN (at least 2 rounds).
• Step 6: Categorize by PICOs.
• Step 7: Search for ongoing Research & Reviews.
• Step 8: Write up report.
• Step 9: Elicit stakeholder feedback.
• Step 10: Final report.

Slide 22

Research Needs Development

Combined Methods:

• Semi-structured interviews to identify topics.
• Modified Delphi process for prioritization.

Semi-structured interviews.

• Telephone.
• Conference calls.
• Webinars.

Delphi

• #1 Survey:
  • Topics organized by PICOs.
  • Likert Scale—6-point scale of no, low, highest priority.
• #2 Survey:
  • Likert.
  • Ranking.
  • Indicate top 3 in order.

Slide 23

Research Needs Development

Slide 24

Research Needs Development

Image: A double-headed arrow captioned "Research Continuum" extends above the following items:

Topic Generation  →

• Clinicians.
• Consumers.
• Funders.
• Insurers.
• Researchers.
• Policymakers.

Research Prioritization →

• Clinicians.
• Consumers.
• Funders.
• Insurers.
• Manufacturers.
• Researchers.
• Payers.
• Policymakers.

Study Design →

• Clinicians.
• Manufacturers.
• Researchers.
• Policymakers.

Implementation →

• Clinicians.
• Consumers.
• Researchers.
• Policymakers.

Analysis/Interpretation →

• Clinicians.
• Researchers.

Dissemination →

• Clinicians.
• Consumers.
• Funders.
• Insurers.
• Manufacturers.
• Researchers.
• Payers.
• Policymakers.

A second double-headed arrow extends below these lists, with the text "EPC Stakeholder Engagement Expanded Scope of Future Research Needs Beyond CER."

Slide 25

Summary

• Several options for methods to use when engaging stakeholders.
• Method depends on several factors including:
  • Intent (identifying topics, prioritization).
  • Resources.
  • Number and variety of stakeholders.
  • What is already known in topic.
  • Experience.
• Important to outline approach and intent a priori.
• Relationships with stakeholders are important:
  • Contact potential stakeholders through multiple venues.
  • Consistent communication and follow-up in the form of emails or phone calls is important to ensuring a balanced selection of stakeholders and a high response rate.
• Preparation of stakeholders is critical to define expectations and allow engagement.