Appendix 2. Study Quality Rating Criteria

Diagnostic Accuracy Studies

Criteria:

• Screening test relevant, available for primary care, adequately described.
• Study uses a credible reference standard, performed regardless of test results.
• Reference standard interpreted independently of screening test.
• Handles indeterminate results in a reasonable manner.
• Spectrum of patients included in study.
• Adequate sample size.
• Administration of reliable screening test.

Definition of ratings based on above criteria:

Good: Evaluates relevant, available screening test; uses a credible reference standard; interprets reference standard independently of screening test; reliability of test assessed; has few or handles indeterminate results in a reasonable manner; includes large number (> 100) broad-spectrum patients with and without disease.

Fair: Evaluates relevant available screening test; uses reasonable although not the best standard; interprets reference standard independent of screening test; moderate sample size (50-100 patients) and a "medium" spectrum of patients.

Poor: Has important limitations such as: uses inappropriate reference standard; screening test improperly administered; biased ascertainment of reference standard; small sample size of narrow selected spectrum of patients.

Randomized Controlled Trials (RCTs) and Cohort Studies

Criteria:

• Initial assembly of comparable groups: RCTs—adequate randomization, including concealment and whether potential confounders were distributed equally among groups; cohort studies—consideration of potential confounders with either restriction or measurement for adjustment in the analysis; consideration of inception cohorts.
• Maintenance of comparable groups (includes attrition, crossovers, adherence, contamination).
• Important differential loss to followup or overall high loss to followup.
• Measurements: equal, reliable, and valid (includes masking of outcome assessment).
• Clear definition of interventions.
• Important outcomes considered.
• Analysis: adjustment for potential confounders for cohort studies, or intention-to-treat analysis for RCTs.

Definition of ratings based on above criteria:

Good: Meets all criteria: comparable groups are assembled initially and maintained throughout the study (followup ≥ 80%); reliable and valid measurement instruments are used and applied equally to the groups; interventions are spelled out clearly; important outcomes are considered; and appropriate attention to confounders in analysis.

Fair: Studies will be graded "fair" if any or all of the following problems occur, without the important limitations noted in the "poor" category below: generally comparable groups are assembled initially, but some question remains whether some (although not major) differences occurred in followup; measurement instruments are acceptable (although not the best) and generally applied equally; some but not all important outcomes are considered; and some but not all potential confounders are accounted for.

Poor: Studies will be graded "poor" if any of the following major limitations exist: groups assembled initially are not close to being comparable or are not maintained throughout the study; unreliable or invalid measurement instruments are used or not applied at all equally among groups (including not masking outcome assessment); and key confounders are given little or no attention.

Case Control Studies

Criteria:

• Accurate ascertainment of cases.
• Nonbiased selection of cases/controls with exclusion criteria applied equally to both.
• Adequate response rate.
• Diagnostic testing procedures applied equally to each group.
• Measurement of exposure accurate and applied equally to each group.
• Appropriate attention to potential confounding variable.

Definition of ratings based on above criteria:

Good: Appropriate ascertainment of cases and nonbiased selection of case and control participants; exclusion criteria applied equally to cases and controls; response rate equal to or greater than 80%; diagnostic procedures and measurements accurate and applied equally to cases and controls; and appropriate attention to confounding variables.

Fair: Recent, relevant, without major apparent selection or diagnostic work-up bias, but with response rate less than 80% or attention to some but not all important confounding variables.

Poor: Major selection or diagnostic work-up biases, response rates less than 50%, or inattention to confounding variables.

Return to Document