Development of a Tool to Evaluate the Quality of Non-randomized Studies of Interventions or Exposures

On September 15, 2009, Nancy D Berkman, PhD & Meera Viswanathan, PhD made this presentation at the 2009 Annual Conference. Select to access the PowerPoint® presentation (334 KB) (Plugin Software Help).

Slide 1

Development of a Tool to Evaluate the Quality of Non-randomized Studies of Interventions or Exposures

Presented by
Nancy D Berkman, PhD & Meera Viswanathan, PhD
Presented at.AHRQ 2009 Annual Conference
Bethesda, Maryland.September 15, 2009

Slide 2

Acknowledgements

• Project funding provided by
  • Phase 1:
  • • Grant from RTI Independent Research and Development (IR&D) funds
  • Phase 2:
  • • Contract from Agency for Healthcare Research and Quality (AHRQ), U.S. Department of Health and Human Services, through the Evidence-based Practice Centers program (EPC)

Slide 3

Context for the Project

• Increasing demand to include non-randomized studies in systematic literature reviews and comparative effectiveness reviews to capture
  • The effects of interventions or exposures on a more broadly defined population than can be observed through RCTs
  • Topics where RCTs would be logistically or ethically inappropriate
  • Longer term outcomes and harms (side effects)
• The trade-off for wider applicability of findings among observational studies, compared with RCTs, is a potentially wider range of sources of bias, including in selection, performance, detection of effects, and attrition.

Slide 4

Background: Rating the Quality of Non-randomized Studies

The quality (internal validity) of each study included in a review needs to be evaluated:

• Well-established criteria and instruments exist for evaluating the quality of RCTs, but not non-randomized (observational) studies
• PIs conducting systematic reviews generally lack access to validated and adaptable instruments for evaluating the quality of observational studies.
• Each new review often develops its own quality rating tool, "reinvents the wheel", leading to "inconsistent standards" within and across reviews.

Slide 5

Project Goals

To create a practical and validated tool for evaluating the quality of non-randomized studies of interventions or exposures that is:

• Reflects a comprehensive theoretical framework: captures all relevant domains
• Broad applicability: can be used "off the shelf" by different PIs
• Modifiable: can be adapted to different topic areas
• Easy to use and understand: can be used by reviewers with varying levels of expertise or experience
• Validated: users can be confident of their evaluation of study quality
• Advances the methodology in the field
• Disseminated widely

Slide 6

Methods: Phase 1

Item development

• Reviewed the literature on the evaluation of the quality of observational studies
• Collected quality review items used in early tools to evaluate non-RCTs through
  • Published literature
  • 90 AHRQ-sponsored EPC reviews
• Categorized all potential items into the 12 quality domains identified in Evaluating non-randomized intervention studies (Deeks et al., 2003)

Slide 7

Methods: Phase 1 (continued)

Item Bank development

• Selected the best items for measuring each of the included domains
• Modified selected items where necessary to ensure that critical domains were included and to improve readability
• Developed a pre-specified set of responses
• Developed explanatory text to be used by PIs and abstractors to individualize as well as standardize interpretation

Slide 8

Methods: Phase 2

• Technical Expert Panel input
  • Conceptual framework to ensure that we included all relevant domains
  • Face validity
• Cognitive interviews with potential users
  • Readability
  • Conceptualization
• Validation
  • Content/face validity
  • Inter-rater reliability testing

Slide 9

Conceptual Underpinnings of the Instrument

Evaluation of quality can rely on either a description of methods or an assessment of validity and precision

• Methods description approach
  • Follows the reporting structure of many manuscripts
  • Relies less on judgment than on reporting
• Validity and precision approach
  • What we really care about
  • More challenging to evaluate
  • Greater reliance on judgment

Slide 10

Domains for quality evaluation approaches

Methods description approach

• Background/context
• Sample definition and selection
• Intervention/exposure
• Creation of treatment groups
• Follow-up
• Specification of outcomes
• Analysis: comparability of groups
• Analysis: outcomes
• Interpretation

• Selection bias
• Performance bias
• Information bias
• Detection bias
• Attrition bias
• Reporting bias
• Precision

Slide 11

Tool Results

• Comprehensive: bank of 39 questions
• Modifiable: includes relevant items appropriate for all non-randomized study types
• Easy to use: instructions for PIs and abstractors to assist in appropriate interpretation of questions. Example:

What is the level of detail in describing the intervention or exposure? [PI: specify which details need to be stated, e.g., intensity, duration, frequency, route, setting, and timing of intervention/exposure. For case-control studies, consider if the condition, timing, frequency, and setting of symptoms is provided in the case definition]

Slide 12

Next Steps

• Finalize inter-rater reliability results
• Publish findings and disseminate the tool
• Proposed Phase III:
  • Design specific validation including inter-rater reliability testing by study type
  • Reduce the number of questions needed to address specific domains
  • Develop a web-based platform for generating design and topic-specific instruments from the item bank.