Methods and Measures

To achieve the evaluation aims of EvidenceNOW, the cooperatives and national evaluation team collected information on the harmonized evaluation measures, including the key outcome measures: ABCS and practice capacity. In addition, the evaluations collected sizeable information on practices’ characteristics, infrastructure, and context.  Given some of the unique aims of each evaluation, cooperatives also collected information on additional measures. These measures are briefly described below.

Practice characteristics and infrastructure

The EvidenceNOW evaluators collected information on the organizational characteristics of practices and their infrastructure. These factors were largely captured in the practice survey, with some information collected in the practice member survey.  The following are the main practice characteristic and infrastructure components that were captured:

• Staff composition (e.g., number of clinicians, staff/FTE, panel size/clinician).
• Practice type.
• Ownership.
• Panel characteristics (e.g., race, ethnicity, age, sex, insurance).
• Recent changes (e.g., changed electronic health record) or disruptions.
• Participation in ACO (e.g., registry use).
• Ability to produce quality reports.
• Patient needs and resources.
• Clinician attitudes towards new guidelines.
• Respondent characteristics.

Oklahoma’s practice characteristics survey provides examples of these items (go to Practice Survey section for more examples).  Several articles from EvidenceNOW grantees report on the relationship between practice characteristics and ABCS outcomes (Shelley et al., 2018) and quality improvement strategy use (Balasubramanian et al., 2018).  Cuellar et al (2018) examined differences in practice ownership on psychological safety, burnout, and select aspects of practice capacity.  Oklahoma collected data from multiple sources and reported on the rates of “major disruptive events” over a 2-year period (Mold et al., 2018)

Context

In addition to organization and infrastructure, the surveys included questions about contextual factors, including participation in an accountable care organization (ACO), patient-centered medical home (PCMH) recognition, receiving incentives for meeting quality standards, and meaningful use. Additional work was done by the EvidenceNOW evaluators, including ESCALATES, to capture in a structured manner the context, such as cooperative structure and external factors, potentially affecting cooperatives’ projects.

Practice Capacity

For the EvidenceNOW initiative, improving practice capacity was a primary aim and key outcome of interest. Practice capacity could also be used as a mediator in evaluations.  Practice capacity was measured using two instruments:

1. Change Process Capability Questionnaire (CPCQ) instrument. The CPCQ was completed by each practice. It has been used in other studies in primary care (for example, Scholle et al., 2013, Solberg et al., 2013, Rubenstein et al., 2014) 
   1. Go to scoring guidance and codebook put together by ESCALATES.
   2. Go to EvidenceNOW articles on CPCQ (Balasubramanian et al., 2018).
2. Adaptive Reserve (AR) instrument.  This instrument was administered to practice staff as part of the practice member survey. The Adaptive Reserve instrument has been used in other studies (for example, McAllister et al., 2013, Tu et al., 2015)
   1. Go to scoring guidance put together by ESCALATES.
   2. Go to EvidenceNOW articles that report on use of the AR (Cuellar et al., 2018, Henderson et al., 2018, Shelley et al., 2018)

ABCS outcomes

The Agency for Healthcare Research and Quality’s (AHRQ) EvidenceNOW initiative is supporting over 1,500 primary care practices to build capacity to use evidence to improve the delivery of cardiovascular care. The clinical quality measures that EvidenceNOW practices used to monitor the delivery of cardiovascular care included Aspirin, Blood Pressure, Cholesterol, and Smoking Cessation (i.e., ABCS clinical quality measures overview and handout. Cooperatives and practices used EHRs to extract ABCS clinical quality measures data to answer the initiative’s research questions. Each regional cooperative employed different approaches to extract EHR data from practices. The national evaluation team (ESCALATES), in collaboration with Minnesota Community Measurement, developed measure narratives and flow charts that cooperatives used to harmonize research data collected across the seven EvidenceNOW regions:

• Aspirin: Measures narrative and flow chart.
• Blood Pressure: Measures narrative and flow chart.
• Cholesterol: Measures narrative and flow chart.
• Smoking Cessation: Measures narrative and flow chart.

Regional EvidenceNOW cooperatives also supported practices  to improve how providers input data into their EHRs, generate reports, and engage with their data. Cooperatives offered five core services to support practices in their efforts to use clinical quality measures for QI and research.

How did EvidenceNOW cooperatives engage practices to share data for research purposes?

The practice-based model used a “teach-to-fish” approach, aiming to increase capacity for practices to generate ABCS clinical quality measures. This approach aimed to activate practice-based learning and increase team buy-in for QI and research. The model accommodated practices that lacked the opportunity, infrastructure, or willingness to link to a centralized electronic reporting platform (i.e., data warehouse, hub, health information exchange).

Practices faced challenges in generating data for research and QI, such as capturing the desired measurement period, generating trustworthy or credible EHR reports, modifying measure specifications when guidelines changed, and producing clinical quality reports at the practice, clinical team, clinician, and patient levels. Cooperatives developed resources for practice facilitators to support practice-based models of generating reports using EHRs. Two examples of these resources are:

• Southwest Cooperative: instructions for practices on submitting measures, and overview of data reporting.
• Northwest Cooperative: Database user guide.

Because of the challenges in generating trustworthy and credible reports from EHRs, some cooperatives used chart abstraction to collect the ABCS clinical quality data from a subset of their practices. Resources that cooperatives developed to support chart abstraction at the practice level included:

General Resources

• Northwest Cooperative: Chart abstraction tracking sheet.
• Midwest Cooperative: Chart review submission form.

ABCS Chart Abstraction Templates

• Aspirin: Instructions and template.
• Blood Pressure: Instructions, protocol, and template.
• Cholesterol: Template.
• Smoking: Instructions and template.

Cooperatives also used centralized infrastructure (i.e., data warehouse, hub, health information exchange) to generate ABCS clinical quality measure data from research. One of the aims of the centralized approach was to facilitate patient-centric, rather than practice-centric, clinical quality measurement by expanding the scope of patient data available to include other community providers and health care settings. Cooperatives using these approaches faced similar challenges to the practice-based model, depending on how EHR vendors made data available.

• popHealth® Implementation.

Each approach has strengths and weaknesses, including the engagement of practices with diverse interests for research and QI, variation across EHR vendors, and scalability given the multitude of EHR vendors within each Cooperative (cite). EvidenceNOW leveraged internal technical and policy expertise and engaged vendors, professional associations, and other primary care initiatives to collectively address these challenges.

Additional Measures

The cooperatives’ local evaluations had slightly different aims and unique interventions, and several of the cooperatives used measures in addition to the core set.  For example, individual cooperatives assessed readiness for implementation using the Key Driver Implementation Scale (KDIS) used an adapted assessment of medical home components, assessed the quality of the practice facilitation team, and collected information from patients directly.