Methodology

Methods Guidance
   Methods Guide for Effectiveness and Comparative Effectiveness Reviews
   Methods Guide for Medical Test Reviews
Methods Research Reports
   Health Systems Partnership Project Reports
   Methods Research on Future Research Needs
   Nutritional Research Series
White Papers

Methods Guidance

Methods Guide for Effectiveness and Comparative Effectiveness Reviews

Download the full Methods Guide for Effectiveness and Comparative Effectiveness Reviews (PDF, 5.58 MB)

• Overview and Table of Contents
• Foreword: Comparing Medical Interventions: AHRQ and the Effective Health Care Program (PDF, 329 KB)
• Principles in Developing and Applying Guidance for Comparing Medical Interventions (PDF, 113 KB).
• Identifying, Selecting, and Refining Topics for Comparative Effectiveness Systematic Reviews (PDF, 589 KB)
• Developing and Selecting Topic Nominations for Systematic Reviews (PDF, 743 KB)
• The Refinement of Topics for Systematic Reviews: Lessons and Recommendations From the Effective Health Care Program (PDF, 743 KB)
• Finding Evidence for Comparing Medical Interventions (PDF, 357 KB)
• Finding Grey Literature Evidence and Assessing for Outcome and Analysis Reporting Biases When Comparing Medical Interventions: AHRQ and the Effective Health Care Program (PDF, 364 KB)
• Avoiding Bias in Selecting Studies (PDF, 201 KB)
• Selecting Observational Studies for Comparing Medical Interventions (PDF, 297 KB)
• Assessing the Risk of Bias in Systematic Reviews of Health Care Interventions (PDF, 413 KB)
• Assessing the Applicability of Studies When Comparing Medical Interventions (PDF, 340 KB)
• Prioritization and Selection of Harms for Inclusion in Systematic Reviews (PDF, 285 KB)
• Assessing Harms When Comparing Medical Interventions (PDF, 524 KB)
• Quantitative Synthesis—An Update (PDF, 703 KB)
• Grading the Strength of a Body of Evidence When Assessing Health Care Interventions for the Effective Health Care Program of the Agency for Healthcare Research and Quality: An Update (PDF, 334 KB)
• Using Existing Systematic Reviews to Replace De Novo Processes in CERs (PDF, 77 KB)
• Integrating Bodies of Evidence: Existing Systematic Reviews and Primary Studies (PDF, 226 KB)
• Updating Comparative Effectiveness Reviews: Current Efforts in AHRQ's Effective Health Care Program (PDF, (213 KB)
• Guidance for the Conduct and Reporting of Modeling and Simulation Studies in the Context of Health Technology Assessment (PDF, 411 KB)

Top of Page

Methods Guide for Medical Test Reviews

Download the complete Methods Guide for Medical Test Reviews (PDF, 2.4 MB)

• Overview and Table of Contents
• Chapter 1: Introduction to the Methods Guide for Medical Test Reviews
• Chapter 2: Developing the Topic and Structuring Systematic Reviews of Medical Tests: Utility of PICOTS, Analytic Frameworks, Decision Trees, and Other Frameworks
• Chapter 3: Choosing the Important Outcomes for a Systematic Review of a Medical Test
• Chapter 4: Effective Search Strategies for Systematic Reviews of Medical Tests
• Chapter 5: Assessing Risk of Bias as a Domain of Quality in Medical Test Studies
• Chapter 6: Assessing Applicability of Medical Test Studies in Systematic Reviews
• Chapter 7: Grading a Body of Evidence on Diagnostic Tests
• Chapter 8: Meta-Analysis of Test Performance When There Is a Gold Standard 
• Chapter 9: Options for Summarizing Medical Test Performance in the Absence of a "Gold Standard"
• Chapter 10: Deciding Whether To Complement a Systematic Review of Medical Tests With Decision Modeling 
• Chapter 11: Challenges in and Principles for Conducting Systematic Reviews of Genetic Tests Used as Predictive Indicators
• Chapter 12: Systematic Review of Prognostic Tests
• Appendix: Test Performance Metrics

Top of Page

Methods Research Reports

Developing Consistent and Useful Quality Improvement Study Data Extraction for Health Systems (September 2020)

Integrating Health System Data With Systematic Reviews: A Framework for When and How Unpublished Health System Data Can Be Used With Systematic Reviews (April 2020)

Pilot To Promote Entry of Structured Data Into the Systematic Review Data Repository (October 2019)

Improving Health Systems’ Access to High-Quality Evidence (August 2019)

Supplemental Project To Assess the Transparency of Reporting Requirements for Studies Evaluating the Effectiveness of Treatment Options for Symptoms of Diabetic Peripheral Neuropathy (July 2017)

Supplemental Project To Assess the Transparency of Reporting for Trials Evaluating Treatment for Infertility (June 2017)

Supplemental Project To Assess the Transparency of Reporting Requirements: Omega-3 Fatty Acids and Cardiovascular Disease (June 2017)

Supplemental Project To Assess the Transparency of Reporting Requirements: Tympanostomy Tubes in Children With Otitis Media (June 2017)

Modeling and Simulation in the Context of Health Technology Assessment: Review of Existing Guidance, Future Research Needs, and Validity Assessment (January 2017)

Supplemental Project To Assess the Transparency of Reporting for Strategies To Improve Mental Health Care for Children and Adolescents (January 2017)

The Empirical Evidence of Bias in Trials Measuring Treatment Differences (September 2014)

Use of Evidence-based Practice Center Comparative Effectiveness Reviews by Clinical Point-of-Care Tools (April 2014)

Developing and Assessing Contextual Frameworks for Research on the Implementation of Complex System Interventions (March 2014)

Simulation-Based Comparison of Methods for Meta-Analysis of Proportions and Rates (November 2013)

Towards a Framework for Communicating Confidence in Methodological Recommendations for Systematic Reviews and Meta-Analyses (September 2013)

Assessing Risk of Bias and Confounding in Observational Studies of Interventions or Exposures: Further Development of the RTI Item Bank (August 2013)

Surveillance and Identification of Signals for Updating Systematic Reviews: Implementation and Early Experience (June 2013)

Identifying and Managing Nonfinancial Conflicts of Interest for Systematic Reviews (May 2013)

Progression-Free Survival: What Does It Mean for Psychological Well-Being or Quality of Life? (April 2013)

Meta-Analytic Statistical Inferences for Continuous Measure Outcomes as a Function of Effect Size Metric and Other Assumptions (April 2013)

Case Study Comparing Bayesian and Frequentist Approaches for Multiple Treatment Comparisons (March 2013)

Empirical Simulation-Based Comparison of Univariate and Multivariate Meta-Analysis for Binary Outcomes (March 2013)

Harmonization of Cognitive Measures in Individual Participant Data and Aggregate Data Meta-Analysis (March 2013)

Decision and Simulation Modeling in Systematic Reviews (February 2013)

Framework for Determining Research Gaps During Systematic Review: Evaluation (February 2013)

Findings of Bayesian Mixed Treatment Comparison Meta-Analyses: Comparison and Exploration Using Real-World Trial Data and Simulation (January 2013)

Evaluating Practices and Developing Tools for Comparative Effectiveness Reviews of Diagnostic Test Accuracy: Methods for the Joint Meta-Analysis of Multiple Tests (January 2013)

A Bayesian Missing Data Framework for Multiple Continuous Outcome Mixed Treatment Comparisons (January 2013)

Assessing the Accuracy of Google Translate To Allow Data Extraction From Trials Published in Non-English Languages (January 2013)

An Empirical Assessment of Bivariate Methods for Meta-Analysis of Test Accuracy (November 2012)

Empirical Assessment of Within-Arm Correlation Imputation in Trials of Continuous Outcomes (November 2012)

Methods for Benefit and Harm Assessment in Systematic Reviews (November 2012)

Quality of Reporting in Systematic Reviews of Implantable Medical Devices (November 2012)

Assessing the Impact of Economic Evidence on Policymakers in Health Care—A Systematic Review (October 2012)

Usefulness of Economic Evaluation Data in Systematic Reviews of Evidence (October 2012)

Best Practices for Conducting Economic Evaluations in Health Care: A Systematic Review of Quality Assessment Tools (October 2012)

A Pilot Study Using Machine Learning and Domain Knowledge To Facilitate Comparative Effectiveness Review Updating (September 2012)

Machine Learning Methods in Systematic Reviews: Identifying Quality Improvement Intervention Evaluations (September 2012)

Use of Mixed Treatment Comparisons in Systematic Reviews (August 2012)

Systematizing the Use of Value of Information Analysis in Prioritizing Systematic Reviews (August 2012)

Peer Review of Search Strategies (June 2012)

Assessing Equivalence and Noninferiority (June 2012)

Reliability Testing of the AHRQ EPC Approach To Grading the Strength of Evidence in Comparative Effectiveness Reviews (May 2012)

Accuracy of Data Extraction of Non–English-Language Trials With Google Translate (April 2012)

Validity and Inter-rater Reliability Testing of Quality Assessment Instruments (March 2012)

Comprehensive Overview of Methods and Reporting of Meta-Analyses of Test Accuracy (March 2012)

OpenMetaAnalyst: Powerful Open-Source Software for Meta-Analysis (February 2012)

Detection of Associations Between Trial Quality and Effect Sizes (January 2012)

A Process for Robust and Transparent Rating of Study Quality: Phase 1 (November 2011)

Correcting for Publication Bias in the Presence of Covariates (September 2011)

Development of the RTI Item Bank on Risk of Bias and Precision of Observational Studies (September 2011)

Feasibility Study of a Wiki Collaboration Platform for Systematic Reviews (September 2011)

Bayesian Approaches to Indirect Comparisons (September 2011)

A Framework for "Best Evidence" Approaches in Systematic Reviews (June 2011)

Empirical Evidence of Associations Between Trial Quality and Effect Size (June 2011)

Addressing Challenges in Genetic Test Evaluation: Evaluation Frameworks and Assessment of Analytic Validity (June 2011)

Identifying Signals for Updating Systematic Reviews: A Comparison of Two Methods (June 2011)

Development of Quality Criteria To Evaluate Nontherapeutic Studies of Incidence, Prevalence, or Risk Factors of Chronic Diseases: Pilot Study of New Checklists (February 2011)

Developing and Testing a Tool for the Classification of Study Designs in Systematic Reviews of Interventions and Exposures  (January 2011)

Comparative Effectiveness Review Methods: Clinical Heterogeneity (September 2010)

Updating Systematic Reviews (September 2007)

Consumer-Oriented Strategies for Improving Health Benefit Design: An Overview (Archived) (July 2007)

A Critical Analysis of Care Coordination Strategies for Children With Special Health Care Needs (Archived) (June 2007)

Empirical Evaluation of the Association Between Methodological Shortcomings and Estimates of Adverse Events (October 2006)

Criteria for Distinguishing Effectiveness From Efficacy Trials in Systematic Reviews (April 2006)

Developing a Methodology for Establishing a Statement of Work for a Policy-Relevant Technical Analysis (Archived) (January 2006)

Meta-Regression Approaches: What, Why, When, and How? (March 2004)

Determinants of Increases in Medicare Expenditures for Physicians' Services (Archived) (October 2003)

Current Validity of AHRQ Clinical Practice Guidelines (Archived) (September 2002)

Health Systems Partnership Project Reports

• Translation of a C. difficile Treatment Clinical Pathway Into Machine-Readable Clinical Decision Support Artifacts Prototyped for Electronic Health Record Integration
• Assessing the Accuracy of Machine-Assisted Abstract Screening With DistillerAI: A User Study
• Performance and Usability of Machine Learning for Screening in Systematic Reviews: A Comparative Evaluation of Three Tools
• Facilitating the Implementation of EPC Reports in Learning Health Systems Engaged in Quality Initiatives: an EPC Pilot Project on ADHD
• Web Interactive Presentation of EPC Reports: A Foray Into Interactive Reports
• Improving Access to and Usability of Systematic Review Data for Health Systems Guidelines Development
• Use of a Clinical Pathway To Facilitate the Translation and Utilization of AHRQ EPC Report Findings
• Development and Usability Testing of EPC Evidence Review Dissemination Summaries for Health Systems Decisionmakers
• Linking Evidence Reviews to Organizational Guideline Planning: A Pilot Test of an Interactive, Web-Based Presentation and Discussion of Evidence 
• Disseminating Findings From EPC Reports: Pilot Project of Three Products
• EPC Pilot Project: A Dual Approach To Facilitate Health Systems Uptake of Evidence Synthesis Reports. Anxiety in Children
• Assessing the Impact of Indexing Performance Measure Codes on the Perceived Value of EPC Reports to Health-Systems
• Development of a Primary Care Guide for Implementing Evidence-based Screening and Counseling for Unhealthy Alcohol Use With Epic-based Electronic Health Record Tools

Methods Research on Future Research Needs

• Overview
• Prioritization Criteria Methodology for Future Research Needs Proposals Within the Effective Health Care Program (January 2013)
• Presentation of Future Research Needs (April 2012)
• Framework for Considering Study Designs for Future Research Needs (March 2012)
• Frameworks for Determining Research Gaps During Systematic Reviews (August 2011)
• Assessing the Impact of AHRQ Evidence-based Practice Center (EPC) Reports on Future Research (June 2011)
• Minimal Modeling Approaches to Value of Information Analysis for Health Research (June 2011)
• Evaluating the Potential Use of Modeling and Value-of-Information Analysis for Future Research Prioritization Within the Evidence-based Practice Center Program (June 2011)
• Engaging Stakeholders To Identify and Prioritize Future Research Needs (June 2011)
• Defining an Optimal Format for Presenting Research Needs (June 2011)
• Finding Evidence on Ongoing Studies (December 2010)

Nutritional Research Series: Advancing the Role of Systematic Reviews in Nutrition Research and Applications

• Vol. 6: Concordance Between the Findings of Epidemiological Studies and Randomized Trials in Nutrition (May 2013)
• Vol. 5: Comparison of Translational Patterns in Two Nutrient–Disease Associations (February 2012)
• Vol. 4: Effects of Eicosapentanoic and Docosahexanoic Acid on Mortality (February 2012)
• Vol. 3: Reporting of Systematic Reviews of Micronutrients and Health (March 2009)
• Vol. 2: Issues & Challenges in Conducting Systematic Reviews to Support Development of Nutrient Reference Values (March 2009)
• Vol. 1: Application of Systematic Review Methodology to the Field of Nutrition (January 2009)

Top of Page

White Papers

The Evidence Base for Telehealth: Reassurance in the Face of Rapid Expansion During the COVID-19 Pandemic (May 2020)

Standardized Library of Depression Outcome Measures (May 2020)

Standardized Library of Asthma Outcome Measures (April 2019)

A Framework for Conceptualizing Evidence Needs of Health Systems (December 2017)

Understanding Health-Systems' Use of and Need for Evidence To Inform Decisionmaking (October 2017)

Discerning the Perception and Impact of Patients Involved in Evidence-based Practice Center Key Informant Interviews (September 2017)

EPC Methods: AHRQ End-User Perspectives of Rapid Reviews (April 2016)

EPC Methods: An Exploration of the Use of Text Mining Software in Systematic Reviews (April 2016)

Change Score or Followup Score? An Empirical Evaluation of the Impact of Choice of Mean Difference Estimates (April 2015)

Comparison of Effects as Evidence Evolves From Single Trials to High-Quality Bodies of Evidence (March 2015)

EPC Methods: An Exploration of Methods and Context for the Production of Rapid Reviews (January 2015)

Integration of Existing Systematic Reviews (June 2014)

Impact of Contacting Study Authors on Systematic Review Conclusions: Diagnostic Tests for Hepatic Fibrosis (April 2014)

Observational Evidence and Strength of Evidence Domains: Case Examples (April 2014)

Systematic Reviews of Complex Multicomponent Health Care Interventions (March 2014)

Assessment of a Method To Detect Signals for Updating Systematic Reviews (March 2014)

Defining the Benefits of Stakeholder Engagement in Systematic Reviews (March 2014)

Payer Perspectives on Improving Usability of Effective Health Care Products: Bridging the Gap Between Information Needs and Evidence Translation (February 2014)

Observational Studies: Empirical Evidence of Their Contributions to Comparative Effectiveness Reviews (June 2013)

A Primer for Systematic Reviewers on the Measurement of Functional Status and Health-Related Quality of Life in Older Adults (September 2013)

Global Health Evidence Evaluation Framework  (June 2013)

Mechanistic Evidence in Evidence-Based Medicine: A Conceptual Framework (June 2013)

Reporting the Findings of Updated Systematic Reviews of Comparative Effectiveness: How Do Users Want To View New Information? (May 2013)

EPC Response to IOM Standards for Systematic Reviews (April 2013)

Role of Single Group Studies in Agency for Healthcare Research and Quality Comparative Effectiveness Reviews (January 2013)

Proposed Governance and Data Management Policy for the Systematic Review Data Repository (September 2012)

Systematic Review Data Repository (July 2012)

A Framework to Facilitate the Use of Systematic Reviews and Meta-Analyses in the Design of Primary Research Studies (January 2012)

Developmental Disabilities Issues Exploration Forum: Cerebral Palsy (November 2011)

Stakeholder Involvement in Improving Comparative Effectiveness Reviews: AHRQ and the Effective Health Care Program (September 2011)

Identifying Priorities for Patient-Centered Outcomes Research for Serious Mental Illness (September 2011)

Using the Principles of Randomized Controlled Trial Design To Guide Test Evaluation (November 2009)

Additional Patient Outcomes and Alternative Pathways in Evaluations of Testing (November 2009)

Medical Tests—White Paper Series (November 2009)

Proposals for a Phased Evaluation of Medical Tests (November 2009)

Top of Page