AHRQ Challenge on Integrating Healthcare System Data With Systematic Review Findings

The Agency for Healthcare Research and Quality (AHRQ), U.S. Department of Health and Human Services (HHS), is announcing a challenge competition to explore the resources and infrastructure needed to integrate real-world evidence from healthcare systems into systematic review findings.

• About the Challenge
• How to Enter the Challenge
• Review Process

About the Challenge

This healthcare systems data can augment study findings synthesized in systematic reviews in several ways, including by filling evidence gaps identified in the systematic review to strengthen the available evidence and by examining the applicability of systematic review findings to real-world populations, including population subgroups not examined in published studies. Ultimately, this will help AHRQ understand if and how sources of data and information outside of traditional systematic reviews, particularly from healthcare systems, could be used alongside systematic reviews to improve clinical decision making, healthcare delivery, and patient outcomes.

Challenge Goal

The AHRQ Evidence-based Practice Center (EPC) Program is interested in learning how analysis of real-world data collected by healthcare systems can be used in conjunction with findings from an AHRQ systematic review to inform healthcare decision-making in the context of a specific local setting. 

This challenge aims to explore and determine the feasibility, resources, and infrastructure needed to incorporate unpublished healthcare system data into systematic review findings. Ideally, these data will enable the healthcare system to decide which practices to incorporate locally and how to overcome barriers to implement the evidence to improve clinical practice, healthcare system operations, and, potentially, health outcomes.

Background

AHRQ’s Evidence-based Practice Center (EPC) Program produces systematic reviews which synthesize information from the peer-reviewed literature and provide the state of the science on available healthcare technologies and healthcare delivery strategies. Creating these reviews is stakeholder-driven, methodologically rigorous, and transparent. Reviews are used to inform healthcare decisions such as clinical practice guidelines and national coverage determinations by Medicare. AHRQ also supports healthcare systems in their efforts to improve the quality of care and optimize patient outcomes, and systematic reviews are scoped to address issues of priority to healthcare systems. Yet, due to limitations in the literature base, EPC systematic reviews may be inconclusive or only represent a narrow patient population, making it difficult to generalize or implement the findings within heterogeneous healthcare systems.  

Systematic reviews may also lack contextual details that can inform successful implementation. Improving healthcare delivery (and thus patient outcomes) often entails addressing issues beyond the benefits or harms of an intervention, traditionally the objective of a systematic review. Traditional reviews may not explain gaps in uptake or use of a clinical service and questions about how best to implement a given clinical service (e.g., details around implementation of a service or intervention).

A recent EPC methods report articulates specific scenarios with examples where healthcare system data may most effectively complement systematic reviews (i.e., to improve the strength, applicability, and implementation of evidence). For example, investigators at the Mayo clinic found that published evidence on outcomes following total pancreatectomy was sparse, so they supplemented a meta-analysis of published studies with their unpublished healthcare system data, which more than doubled the sample size and improved the strength of evidence available. 

In another instance, secondary analyses of Veterans Affairs (VA) data confirmed the applicability to the VA of findings from a published systematic review.

The recent EPC methods report also outlines important limitations and considerations when using unpublished healthcare system data alongside systematic reviews, such as relevant limitations in information and data quality. However, the report  did not address the necessary resources, skills, partnerships, and processes required to utilize healthcare system data alongside systematic reviews to strengthen the actionability of systematic reviews.

This Challenge, therefore, invites applicants to conduct an analysis of healthcare system data to supplement an existing AHRQ EPC Program systematic review. This will help AHRQ understand if and how sources of data and information outside of traditional systematic reviews, particularly from healthcare systems, could be used alongside systematic reviews to improve decision making, healthcare delivery, and patient outcomes.

Challenge Design

• All evidence reports (systematic reviews, rapid reviews, and technical briefs) published on the AHRQ website since January 2018 may be considered for this Challenge.
• Healthcare systems and other provider groups interested in implementing evidence at their sites may apply. 
• An organization may choose to address no more than two systematic reviews, submitting a separate proposal for each review.
• Teams should have expertise in the clinical topic, evidence-based practice, data analysis, and quality improvement.

This Challenge consists of two phases:

Phase 1: Proposal

Elicit written proposals on a topic for which an AHRQ evidence report (systematic review, rapid review, or technical brief) has been published since January 2018 and that is relevant to a dilemma in the applicant’s healthcare system. Each proposal is to be written in the form of a narrative that:

1. explicitly states the rationale (e.g., addressing evidence gaps to strengthen available evidence, examining the applicability of findings to real-world patients) to complement conclusions of an EPC report with an analysis of health system data, including a discussion of possible limitations of the analysis.
2. develops and describes an analytic plan for the use of healthcare system data (EHR data from an individual healthcare system or networks of healthcare systems [for example, PCORnet, Epic's Cosmos, etc.]).
3. provides an approach for decision-making based on the data analysis and evidence report results and an evaluation of the decision-making process and results.
4. describes potential challenges, barriers, and strategies to complete the analysis successfully.
5. lists team members, their roles, areas of expertise, and hours on the project.

Each proposal will be considered and evaluated on its own merit.

A total of 5 proposals will be selected as winners for Phase 1.

Phase 2: Analyses

Healthcare systems selected as winners in Phase 1 will be invited to provide a written narrative that:

1. includes a complete analysis of internal real-world data and appraisal of the analysis for risk of bias using a proper tool such as the ROBINS I.
2. specifies how the findings from unpublished data support, refute, and otherwise complement findings from the published evidence examined in the systematic review. If the unpublished data conflict with the AHRQ review’s conclusions, discuss possible reasons for the discrepancy (e.g., challenges with internal validity of healthcare system data analysis related to study design and methods used, or challenges with  external validity concerning population subgroups [gender, race/ethnicity, multimorbidity] examined in the healthcare system data).
3. describes how the unpublished data informed decision making (e.g., adapt, adopt, abandon, prioritize).
4. reports on solutions to any barriers encountered to using healthcare system data alongside a published evidence review, including barriers to access to healthcare system data, interoperability of data sources, and data analysis.
5. briefly describes potential approaches to implement or deimplement the evidence, including the use of clinical advisories, clinical pathways, clinical decision support, or any other method. The plan should describe anticipated risks and barriers and strategies for successful implementation. Decisions made against uptake should be justified.

Timeline and Prize Amounts

AHRQ is hosting this challenge as a two-phase competition. All costs associated with developing and submitting proposals and conducting real-world data analysis of real-world data will be the responsibility of the Challenge participant. Cash prizes will be awarded only after the projects are evaluated and determined acceptable at the end of Phase 2.

Timeline

• September 26, 2022—Challenge launch.
• January 23, 2023, by 7:00 p.m. ET—Submissions for Phase 1 (written proposals) are due. AHRQ will complete the review of the proposals within 6-8 weeks of closing the announcement.
• March 30, 2023—AHRQ will announce the Phase 1 winners. Phase 2 of the Challenge will commence once the Phase 1 winners are announced and notified by March 31, 2023. The AHRQ team will schedule a live, virtual technical assistance webinar with all winners of Phase 1 to discuss scope of content, accessibility/compliance with Section 508, and address questions that the winners may have.
• March 31, 2023—Phase 2 participants will have 120 calendar days from notification to create and submit their analyses as described in their proposal(s). The deadline to submit the analysis is July 31, 2023, by 7:00 p.m. ET.
• Fall 2023—The final winners of Phase 2 of the competition will be announced in October 2023.

Prize Amounts

The top five entries in Phase 1 will be invited to participate in Phase 2. Upon completion of Phase 2, each of the top five entries will each receive cash awards of $50,000. Additionally, the Phase 2 first-place winner will be awarded an additional $150,000, and the second-place winner will be awarded an additional $100,000. The winner and runner up from Phase 2 will be posted on the AHRQ website.

Participants in Phase 2 may be disqualified if their submitted analyses deviate from their winning proposals.