Panning for Research Questions: Using Systematic Reviews and Stakeholder Input to Identify Needed Research in Comparative Effectiveness
AHRQ's 2012 Annual Conference Slide Presentation
Select to access the PowerPoint® presentation (3.1 MB).
Slide 1
Panning for Research Questions: Using Systematic Reviews and Stakeholder Input to Identify Needed
Tim Carey, MD
RTI-UNC Evidence-based Practice Center (EPC)
Elisabeth Kato, MD, MRP
Effective Health Care Program, AHRQ
Jeanne-Marie Guise, MD, MPH
Oregon Evidence-based Practice Center (OHSU)
Image: The logos for UNC's Cecil G. Sheps Center for Health Services Research, AHRQ's EPC program, and OHSU are shown.
Slide 2
Overview
- The Future Research Needs Program.
- Working with Stakeholders.
- Prioritization Strategies.
- Exercise.
- Discussion and Future Directions.
Slide 3
The Effective Health Care Program
The 2003 Medicare Prescription Drug, Improvement, and Modernization Act authorized the Agency for Healthcare Research and Quality (AHRQ) to improve the quality, effectiveness, and efficiency of health care delivered through Medicare, Medicaid, and State Children's Health Insurance programs.
The Effective Healthcare Program carries out this goals through:
- Evidence Synthesis.
- Evidence Generation.
- Dissemination and Translation.
Slide 4
How Do We Decide Which Comparative Effectiveness Questions to Research?
Traditional research paradigm: unplanned, researcher driven, inconsistent public input.
- Decentralized generation of questions.
- Peer review by researchers.
- Closed-door funder decisions.
New CER/PCOR [comparative effectiveness and patient-centered outcomes research] paradigm: inclusive, rapid, and targeted process.
- Systematic identification of gaps.
- Involvement of a range of stakeholders, including patients.
- Open process.
Slide 5
AHRQ's Position with Stakeholders
"The Agency for Healthcare Research and Quality (AHRQ) firmly believes that involving all stakeholders in the research enterprise from the beginning improves the end product and facilitates the diffusion and implementation of the findings by getting early buy-in from users. Involving stakeholders also helps to ensure that the research reflects the various needs of all diverse users."
Operating Plan for American Recovery and Reinvestment Act Funds for Comparative Effectiveness Research as the Agency for Healthcare Research and Quality (2009)
Slide 6
PCORI's Mission Statement
"PCORI helps people make informed healthcare decisions, and improves healthcare delivery and outcomes, by producing and promoting high-integrity, evidenced-based information, that comes from research guided by patients, caregivers, and the broader health care community."
http://www.pcori.org/about/mission-and-vision/
Slide 7
- Where do gaps come from?
- Who should be involved?
- What is the best way to prioritize needs?
- How much time is required?
- How much money does it take?
Carey et al. (2012)
Slide 8
Systematic Reviews = Evidence Gaps
Image: The word EVIDENCE is shown in black block letters on a white background. Below it, EVIDENCE appears as white block letters on a black background.
Evidence Gaps = Future Research Needs?
Slide 9
Not Necessarily.....
- The review team may not identify all gaps.
- Not every evidence gap is equally important.
- Different goals.
"More, bigger, longer, better randomized controlled trials are needed".
Slide 10
AHRQ Effective Healthcare Program Future Research Needs Projects
- 2010: AHRQ piloted 8 Future Research Needs (FRNs) Projects to extract research gaps from a systematic review, transforming them into prioritized research questions aided by diverse stakeholders.
Goal: alert researchers and funders to important evidence gaps affecting healthcare decisions.
- 2011: pilot program reviewed, resulting in a common 7 step framework for AHRQ FRN projects.
- 2012: 29 "second generation" FRNs have been or are being completed, 8 FRN methods papers published to date.
- Systematic review is published with researcher-determined research gaps.
- Orientation of stakeholders to CER question, FRN process, and prioritization criteria.
- Elaboration and consolidation of research gaps through iterative process with stakeholders.
- Priority ranking of the research gaps.
- Transformation of research gaps into needs.
- Refinement and re-ranking of priorities by stakeholders.
- Addition of study design considerations.
Slide 11
Working With Stakeholders
How do you ensure meaningful and representative engagement?
Image: A cartoon shows a man making a presentation at a podium in front of a crowd; another man at the back of the crowd shouts, "You can engage us better down here!"
Slide 12
What Is Meaningful Engagement?
PCORI:
- "Extending well beyond the simple notion of securing input from a variety of stakeholders."
- "Projects must include stakeholders as partners with significant involvement in all appropriate stages of the research project."
- "True partnership of shared accountability."
Slide 13
Who Do You Need at the Table?
Image: King Arthur and his knights are shown standing around the Round Table.
Slide 14
Stakeholder Groups Defined
| Stakeholder Group | Description |
|---|---|
| Consumer (public, patient, and caregiver) | An individual or advocacy group representing individuals who have a health condition, use health care services and/or who are members of the community. |
| Clinician (Health professional) | Health care or public health provider (e.g., academic, rural/frontier, and community) and medical and/or public health organizations. |
| Social Service Organizations | An individual or organization that advances human welfare or social work (e.g., community or condition service providers, school-based programs, justice system). |
| Policymaker | An individual or organization who is involved in health care policy (e.g., local, state, provincial and Federal legislators and staff).
|
| Researcher | An individual who conducts and/or facilitates research activities in:
|
| Research Funder | A public or private organization that funds research (e.g., National Institute of Health, Susan G. Komen Foundation, and American Cancer Society). |
| Insurer/Payer | An organization or agency that pays for health-related goods and services (e.g., Blue Cross Blue Shield, Medicaid, and Medicare) or a business group that pays for health insurance (e.g., employers and government). |
| Manufacturer | A business group that produces health-related items (e.g., pharmaceuticals and medical devices). |
Slide 15
Stakeholder Involvement Across Research Phases
Image: A double-headed arrow captioned "Research Continuum" extends above the following items:
Topic Generation →
- Clinicians.
- Consumers.
- Funders.
- Insurers.
- Researchers.
- Policymakers.
Research Prioritization →
- Clinicians.
- Consumers.
- Funders.
- Insurers.
- Manufacturers.
- Researchers.
- Payers.
- Policymakers.
Study Design →
- Clinicians.
- Manufacturers.
- Researchers.
- Policymakers.
Implementation →
- Clinicians.
- Consumers.
- Researchers.
- Policymakers.
Analysis/Interpretation →
- Clinicians.
- Researchers.
Dissemination →
- Clinicians.
- Consumers.
- Funders.
- Insurers.
- Manufacturers.
- Researchers.
- Payers.
- Policymakers.
A second double-headed arrow extends below these lists, with the text "EPC Stakeholder Engagement Expanded Scope of Future Research Needs Beyond CER."
Slide 16
Things to Consider in Selecting Stakeholders
Make sure your expectations and those of stakeholders are aligned:
- Stakeholders can play multiple roles.
- Categories are broad, need to consider this carefully.
- Do you want patients, general public, or patient advocates?
- When working with organizations make sure you understand their interests.
- Individual vs. broader interests.
- Preparedness to understand topic & research.
Image: Two construction teams are shown, each standing on one side of a bridge they have been working on; the two sides do not align and will not be able to connect.
Slide 17
How Do You Ensure Meaningful Engagement?
Image: A group of blue figures stand in a circle around the pieces of a partially finished jigsaw puzzle.
Slide 18
Preparation of Stakeholders
- Assume that all stakeholders need orientation to the systematic review and evidence grading processes.
- Familiarity with research methods likely to be very variable.
- Conflicts of interest are to be expected with stakeholder groups, but must be disclosed.
- Patient/consumer/caregivers may need significant orientation and training regarding research the research process.
Image: A cartoon shows two painters working on a large sign that reads "ALWAYS BE PREPARED." One painter says to the other, "I thought you were bringing the paint."
Slide 19
Provide Background Materials in Lay Terms
Brief Description of Project:
Purpose
Process
Timeline
Background
Expectations
Final Product
Potential Formats:
Hard Copy
Web Conference
In-person
Consider offering personal sessions.
Slide 20
Example: Purpose
- Goals: To establish an agenda to guide the Agency for Healthcare Research and Quality's (AHRQ's) comparative effectiveness review and research activities regarding prevention in women's health.
- Identify topics stakeholders struggle with in clinical decisionmaking and health policy, where answers will improve the quality, effectiveness, and efficiency of United States health care.
Slide 21
Example: The Process
| Where We Are in the Comparative Effectiveness Review Process | Technical Expert Panel | Evidence-based Practice Center |
|---|---|---|
In an earlier phase of the Comparative Effectiveness Review (CER) process, Key Informants were involved in refinement of the Key Questions and development of the Analytic Framework. Some of the Key Informants are also members of the Technical Expert Panel. Subsequently, the Key Questions and Analytic Framework were posted on the Effective Health Care Program Website for public comment. The research team at the Oregon Evidence-based Practice Center reviewed the public comments and considered any modifications required to the Key Questions. | The Technical Expert Panel (TEP) is constituted to inform the scientific content of the research review and provide guidance to the EPCs throughout the evidence review process. The panel is typically composed of 5-8 members and includes clinicians, methods experts, researchers, relevant professional society and health care representatives and other content experts. TEP may assist the EPCs by providing input to:
| The role of the Evidence-based Practice Center (EPC) is to synthesize a report of the existing body of evidence through objective and systematic review. The Comparative Effectiveness Review (CER) report is meant to provide understandable and actionable information for patients, clinicians, policymakers and other decisionmakers—not to suggest a set of guidelines for practice. Key Principles:
Reviews commissioned under the Effective Healthcare Program (EHC) are posted publically at different stages of the review process, including at the stage of proposed Key Questions and the draft report stage. |
Slide 22
Example: Timeline
- Three meetings:
- May 11 – Framework.
- June 8 – Topic Generation.
- July 13 – Final Prioritization.
- Extranet and email topic generation and participant discussion.
Slide 23
Example: Background
Comparative Effectiveness Reviews
Definition: Comparative Effectiveness Reviews (CERs) systematically and critically appraise existing research comparing the benefits and harms of alternative methods to prevent, diagnose, treat and monitor a clinical condition, or to improve the delivery of care.
Purpose: To assist consumers, clinicians, purchasers and policymakers to make informed decisions that will improve health care at both the individual and population levels.
Sox, HC & Greenfield, S (2009).
Slide 24
Example: Expectations
Topic Generation:
- Nominate topics via the Web. [This text is circled in red, with a red text box noting "You are here."]
Topic Development:
- Clarify nomination intent.
- Frame clinical context of nomination.
Topic Refinement:
- Provide "real world" input from provider, patient, and policy perspectives.
- Provide input on clinical logic.
- Public comment via Web.
Review:
- Provide methodological & clinical input as a TEP member.
- Provide input via peer review.
- Provide public comment via Web.
Research Needs Development:
- Participate in discussions to describe research gaps & prioritize research needs.
Report Translation & Dissemination:
- Participate in focus groups.
- Participate in individual interviews.
- Provide expert review of summary guides.
Technical Expert (TEP) Panel
The TEP is constituted to inform the scientific content of the research review and provide guidance to the EPCs throughout the evidence review process. Technical Expert Panels may assist the EPCs by providing input to:
- Focus the literature search.
- Identify inclusion/exclusion criteria.
- Answer specific technical or clinical questions.
- Identify specific subgroups for which there may be heterogeneity of treatment effect.
*Note: The role of the EPC is to synthesize a report of the existing body of evidence. The final EPC report will be published by AHRQ and may be used by other agencies to inform policy or determine clinical guidelines.
Slide 25
Recommendations
- Key Informants: Optional.
- Stakeholder Number: 2-3 stakeholders from each SH category (One stakeholder is not representative; Different stakeholders of the same "type" have different perspectives)—Office of Management and Budget (OMB).
- Stakeholder Constituency: Depends on research phase:
- Most times clinicians, researchers, funders, consumer advocacy representatives.
- Methods of Engagement: Mixed:
- In person (telephone or face to face with or without Web)—orientation clarification and expectations delineation; also can be used to generate topics clarify questions.
- Materials—summarizing overview for stakeholders.
Slide 26
Current Dilemmas
- Different stakeholders need different levels of training.
- What extent can stakeholders be reused from project to project?
- Preparation and experience may change the stakeholder over time to be more like us.
Image: A white figure stands at the center for a circle of red arrows pointing outward in all directions. Above the figure's head is a question mark.
Slide 27
Prioritization
- When to do it.
- How to do it.
- What method is best?
- How can we do it better?
Slide 28
AHRQ Effective Healthcare Program Future Research Needs Projects
- 2010: AHRQ piloted 8 Future Research Needs (FRNs) Projects to extract research gaps from a systematic review, transforming them into prioritized research questions aided by diverse stakeholders.
Goal: alert researchers and funders to important evidence gaps affecting healthcare decisions.
- 2011: pilot program reviewed, resulting in a common 7 step framework for AHRQ FRN projects.
- 2012: 29 "second generation" FRNs have been or are being completed, 8 FRN methods papers published to date.
- Systematic review is published with researcher-determined research gaps.
- Orientation of stakeholders to CER question, FRN process, and prioritization criteria.
- Elaboration and consolidation of research gaps through iterative process with stakeholders.
- Priority ranking of the research gaps.
- Transformation of research gaps into needs.
- Refinement and re-ranking of priorities by stakeholders.
- Addition of study design considerations.
Slide 29
Priority Ranking
- Reviews may generate many gaps, need for prioritization.
- Some organizations use broad internet data gathering:
- Will the participants understand all of the issues?
- Multiple methods currently used:
- Ranking 1-xx.
- Likert scale 1-7.
- Multi-voting, multiple (but limited) votes per choice.
- Pair-wise comparisons.
- Delphi methods.
- Nominal group process.
- Consensus conference.
Slide 30
Transformation of Research Gaps into Needs
- Gaps are generally in the form of a declarative sentence.
- Needs are questions similar to research questions in a grant proposal.
- Most organizations use PICOTS framework:
- Population, Intervention, Comparator, Outcome, Timeframe, Setting.
- Methods questions may be important, but may not be a fit for PICOTS.
Slide 31
Refinements and Re-Ranking
- When number of gaps is large, multiple rounds of prioritization may be needed.
- Many groups use 2-3 rounds.
- Gaps may be combined or split depending on the level of granularity that is useful.
- Is consensus needed?
Slide 32
Images: Screenshots show pages from the Stars Selection Web site concerning prioritization of research needs for ADHD.
Slide 33
Example: AHRQ Future research needs on ADHD
Key Questions from Comparative Effectiveness Review
| KQ1 | Among children less than 6 years of age with Attention Deficit Hyperactivity Disorder or Disruptive Behavior Disorder, what are the effectiveness and adverse event outcomes following treatment? |
| KQ2 | Among people 6 years of age or older with Attention Deficit Hyperactivity Disorder, what are the effectiveness and adverse event outcomes following 12 months or more of any combination of follow-up or treatment, including, but not limited to, 12 months or more of continuous treatment? |
| KQ3 | How do A) underlying prevalence of Attention Deficit Hyperactivity Disorder, and B) rates of diagnosis (clinical identification) and treatment for Attention Deficit Hyperactivity Disorder vary by geography, time period, provider type, and sociodemographic characteristics? |
20 research gaps from the review mapped to the key questions, presented to a group of 12 stakeholders, including funders, advocates, clinicians, regulators, researchers, and policymakers.
After stakeholder input, 29 research gaps. 8 gaps emerged as the top future research needs after two rounds of prioritization using an online prioritization tool.
The next two slides show the presentation of one gap from identification to study design.
Slide 34
Identify Research Gap
For children less than 6 years of age with disruptive behavior disorder or ADHD, limited data are available about the efficacy and effectiveness of psychosocial treatment programs (e.g., parent training and summer behavior treatment programs), alone or in combination with pharmacological interventions, compared with other psychosocial treatment programs, alone or in combination with pharmacological interventions. (KQ 1).
After One Round of Prioritization Apply PICOTS and Develop Research Question:
| P | I | C | O | T/S |
|---|---|---|---|---|
| Age <6 years Diagnosed with ADHD or at risk for ADHD or diagnosed with disruptive behavior disorder (including ODD and CD by DSM) | Psychosocial interventions alone (including parent training and school-based interventions) | Pharmacological treatments, alone or in combination with psychosocial treatments | Outcomes for children and parents* | 6 Months/ 1 Year Private clinic, community clinic |
Research Question: For children less than 6 years of age with disruptive behavior disorder or ADHD, what is the comparative efficacy and effectiveness of specific psychosocial treatments alone compared with pharmacological treatments alone or in combination with psychosocial treatments for patient outcomes?
Slide 35
After Second Round of Prioritization Develop Study Design Considerations:
- Randomized controlled trials
Randomized trials could be designed to test various components in a 2x2 matrix of psychosocial treatment variants (parent training, school-based intervention, combination, or pharmacological).- Advantages of study design for producing a valid result
Allows isolation of causal inferences related to the intervention being tested. Multiple-armed trials would allow testing of several hypotheses regarding relative efficacy of singular or combination treatment components. - Ability to recruit/availability of data
Common condition in this age group with uncertainty regarding treatment choice; all arms receive some treatment. - Resource use, size, and duration
Large sample size (N = 840; n = 210 per treatment arm) needed. Key outcomes such as school achievement will require follow-up of several years. - Ethical, legal, and social issues
Vulnerable population, careful informed consent will need to occur.
- Advantages of study design for producing a valid result
Slide 36
State of the Science
- Multiple groups are currently conducting work in this area.
- Tendency to emphasize cross-cutting issues across diverse stakeholder groups.
- Sufficient common aspects to serve as a consensus:
- Criteria for gaps identification.
- Broad aspects of stakeholder panel composition.
- Need to train stakeholders in PCOR.
- Explicit prioritization method—but multiple methods currently used, no empiric guidance as to optimal method.
- Decisions regarding study design considerations.
- Organizations can use existing methods now while refining the approaches
Slide 37
Timelines
- Work can expand to fill a substantial amount of time.
- Rapid turnaround important for:
- Working from the current systematic review.
- Dissemination.
- 4-7 month time frame:
- Step 1: Identification/Recruitment (1-2 months).
- Step 2: Engagement & Prioritization (2-4 months).
- Step 3: Analysis/Drafting Report (1-2 months).
- After publication: Dissemination—targeted vs broad, role of Web vs peer review (may not be the EPC conducting this step)
Slide 38
Recommendations
- Evaluate different stakeholder panel sizes and compositions in prioritization.
- Evaluate the reliability of stakeholder prioritization through replication studies.
- Test different methods of prioritization to assess for transparency, reproducibility and efficiency.
- Clarify role of gap identification and prioritization with other methods such as value of information modeling.
- Collaborate across PCOR programs in improving methods, developing consistent terminology.
Slide 39
Current Dilemmas
- What prioritization works best?
- General versus specific research topics or questions.
- Do you propose study designs?
- How many rounds are required?
Image: A white figure stands at the center for a circle of red arrows pointing outward in all directions. Above the figure's head is a question mark.
Slide 40
Exercise
Future research needs in Attention Deficit Hyperactivity Disorder (ADHD)
Slide 41
Key Questions from Comparative Effectiveness Review
| KQ1 | Among children less than 6 years of age with Attention Deficit Hyperactivity Disorder or Disruptive Behavior Disorder, what are the effectiveness and adverse event outcomes following treatment? |
| KQ2 | Among people 6 years of age or older with Attention Deficit Hyperactivity Disorder, what are the effectiveness and adverse event outcomes following 12 months or more of any combination of follow-up or treatment, including, but not limited to, 12 months or more of continuous treatment? |
| KQ3 | How do A) underlying prevalence of Attention Deficit Hyperactivity Disorder, and B) rates of diagnosis (clinical identification) and treatment for Attention Deficit Hyperactivity Disorder vary by geography, time period, provider type, and sociodemographic characteristics? |
Slide 42
Analytic Framework
Image: The analytic framework for the ADHD Comparative Effectiveness Review is shown.
Slide 43
Evidence Gaps
- How does the use of different data capturing methods (such as child self-rating scales, parent scales, semistructured interviews) affect apparent treatment efficacy in people ≥6 yrs with ADHD.
- What are the prescriptive tx or optimal circumstances for adding specific psychosocial and/or pharmacological tx components based on the needs of people ≥6 yrs with ADHD.
- What are the prescriptive tx or optimal circumstances for adding specific psychosocial and/or pharmacological tx components based on the needs of preschoolers with disruptive behavior disorder or ADHD.
- What is the amount of variation in case identification and prevalence across geographic areas, age groups, settings and cultures.
- What is the comparative effectiveness of different preschool interventions on long term outcomes.
Slide 44
Nominal Group Exercise
1) Individual responses.
2) Clarification and consolidation.
3) Ranking responses.
Images: Three photographs show paper stickers with topic ideas written on them, first as individual responses, then grouped for clarification and consolidation of ideas, then ranked.
Slide 45
Summing It All Up
- Systematic reviews provide a feasible basis for inclusive, rapid, and targeted identification of future research needs.
- Meaningful stakeholder involvement requires careful selection and preparation.
- There are many ways to involve stakeholders in prioritization but the optimal approach is not yet clear.
Slide 46
Future Directions for Future Research Needs
- How should research needs identified through topic specific systematic reviews fit into broader topic identification efforts?
- Will the gaps and prioritization resonate with funders and policymakers?
- What are the best ways to communicate with the public and with funders?
- Who should be involved in refining this area?
Slide 47
Questions? Comments?
Slide 48
References
- Carey T, Yon A, Beadles C, Wines R. Prioritizing Future Research through Examination of Research Gaps in Systematic Reviews. Prepared for the Patient-centered Outcomes Research Institute. Washington DC: March 15, 2012.
- Chang SM, Carey T, Kato EU, Guise JM, Sanders GD. Ann Intern Med 2012 Jul 31. doi
- Kane RL, Guise JM, Hartman K, Rothenberg B, Trikalinos T, Wilt T. Presentation of Future Research Needs. Methods Future Research Needs Report No.9. AHRQ Publication No. 12-EHC053-EF. Rockville, MD: Agency for Healthcare Research and Quality. April 2012. http://www.effectivehealthcare.ahrq.gov/reports/final.cfm.
- Sox, HC & Greenfield, S. Comparative effectiveness research: a report from the Institute of Medicine. Annals of Internal Medicine 2009; 151(3):203-205.
