Appendix A: Measure Evaluation Framework

Selecting Quality and Resource Use Measures: A Decision Guide for Community Quality Collaboratives

Appendix A presents a matrix of measure evaluation criteria* used by five major national organizations. The matrix is borrowed from AHRQ's Guidance for Using the AHRQ Quality Indicators for Public Reporting or Payment - Appendix B: Public Reporting Evaluation Framework. Please note that terminology may vary by framework.
Public Reporting Evaluation Framework
Evaluation criteria and requirementsNational Quality ForumAHRQ Quality IndicatorsNational Healthcare Quality ReportThe Joint CommissionNational Committee for Quality Assurance
1. ImportanceImportant—Face validity
—Foster real quality improvement
Importance Relevance
  • Assesses an important leverage point for improving quality; significant to target audiences; impact on health
  • Opportunity for improvement, considerable variation in quality of care exists
  • Aspect of quality is under provider or health system control
  • Should not create incentives or rewards to improve without truly improving quality of care
  • Leverage point for improving quality
  • Considerable variation in quality of care exists
  • Performance in the area is suboptimal
  • Aspect of quality is under provider or health system control1
  • Measure an important aspect of quality that is subject to provider or health system control
  • Should not create incentives or rewards to improve without truly improving quality of care
  • Impact on health
  • Meaningfulness
  • Susceptibility to being influenced by health care
  • Targets improvement in the health of populations
  • Under provider control
  • Strategic importance
  • Health importance
  • Meaningfulness to decision makers
  • Variance among systems
  • Potential for improvement
  • Controllability
  • Financial importance

* Criteria are taken verbatim from the various sources and have not been edited.

1 This criterion is in the NQF framework at the scope/priority level and not at the individual measure evaluation level.

Evaluation criteria and requirementsNational Quality ForumAHRQ Quality IndicatorsNational Healthcare Quality ReportThe Joint CommissionNational Committee for Quality Assurance
2. Scientific acceptabilityScientifically acceptablePrecision
—Minimum bias
—Construct validity
Scientific soundness Scientific soundness
  • Relationship to quality is based on scientific evidence
  • Well defined and precisely specified
  • Valid, measures the intended aspect of quality; accurately represents the concept being evaluated; data sources are comparable
  • Adequate proportion of total variation is explained by provider performance and amount of variation in measurement is small after provider performance and patient characteristics are taken into account
  • Reliable, producing the same results a high proportion of time in the same population
  • Precise, adequately discriminating between real differences in provider performance and reasonable sample size exists to detect actual differences; captures all possible cases and bias related to case exclusion or limited data are minimal.
  • Risk adjustment is adequate to address confounding bias
  • Well defined and precisely specified
  • Reliable, producing the same results a high proportion of time in the same population
  • Valid, accurately representing the concept being evaluated
  • Precise, adequately discriminating between real differences in provider performance
  • Adaptable to patient preferences and variety of settings
  • Adequate and specified risk adjustment strategy exists
  • Evidence is available linking process measures to outcomes
  • Have relatively large variation among providers that is not due to random variation or patient characteristics
  • Should not be affected by systematic differences in patient case-mix
  • When systematic differences exist, an adequate risk adjustment system is available based on HCUP discharge data
  • Supported by evidence of a relationship to quality
  • Related to other indicators intended to measure the same or related aspects of quality
  • Explicitness of the evidence base
  • Reliability
  • Validity
  • Precisely defined and specified
  • Reliable
  • Valid
  • Risk-adjusted or stratified
  • Clinical evidence linking processes, outcomes, interventions
  • Reproducibility
  • Validity (face, construct, content)
  • Accuracy
  • Case-mix risk adjustment methods
  • Comparability of data sources
 
Evaluation criteria and requirementsNational Quality ForumAHRQ Quality IndicatorsNational Healthcare Quality ReportThe Joint CommissionNational Committee for Quality Assurance
3. UsabilityUsableApplication   
  • Effective (understandable and clear) presentation and dissemination strategies exist
  • Statistical testing can be applied to communicate when differences in performance levels are greater than would be expected by chance
  • Has been used effectively in the past and/or has high potential for working well with other indicators currently in use
  • Compelling content for stakeholder decision making
  • Measure can be used by stakeholders for decision making
  • Performance differences are statistically meaningful
  • Performance differences are practically and clinically meaningful
  • Risk stratification, risk adjustment, and other forms of recommended analyses can be applied appropriately
  • Effective presentation and dissemination strategies exist
  • Information produced can be used by at least one health care stakeholder audience to make a decision or take action
  • Information about specific conditions under which the measure is appropriate to use has been given
  • Methods to aggregate the measure with related measures are defined if determined to be more under-standable and useful
  • Have been used effectively in the past
  • Have high potential for working well with other indicators currently in use
 
  • Can be interpreted and useful in the accreditation process
 
 
4. FeasibilityFeasibility Feasibility Feasibility
  • Consistent construction and assessment of the measure
  • Feasible to calculate; benefits exceed financial and administrative burden of implementation
  • Confidentiality concerns are addressed
  • Audit strategy can be implemented, quality of data are known
  • Data collection tied to care delivery when feasible
  • Timing and frequency of measure collection are specified
  • Benefit evaluated against financial and administrative burden of implementation
  • Confidentiality concerns are addressed
  • Audit strategy is designed and can be implemented
 
  • Availability of required data across the system
  • Cost or burden of measurement
  • Existence of prototypes
  • Capacity of data and measure to support subgroup analyses
  • Data collection effort is assessed
  • Precise definition (under scientific soundness in other frameworks)
  • Reasonable cost
  • Logistical feasibility
  • Confidentiality
  • Auditability

Source: Remus D, Fraser I. Guidance for using the AHRQ Quality Indicators for hospital-level public reporting or payment. Rockville, MD: Agency for Healthcare Research and Quality; 2004. AHRQ Pub. No. 04-0086-EF. Available at: http://qualityindicators.ahrq.gov/downloads/technical/qi_guidance_appendix_B.pdf.

Page last reviewed May 2010
Internet Citation: Appendix A: Measure Evaluation Framework: Selecting Quality and Resource Use Measures: A Decision Guide for Community Quality Collaboratives. May 2010. Agency for Healthcare Research and Quality, Rockville, MD. http://www.ahrq.gov/professionals/quality-patient-safety/quality-resources/tools/perfmeasguide/perfmeasappa.html