Supplements to Evaluation Highlight No. 2

Notes: Massachusetts is collecting the MHI in intervention practices and the MHI-RSF in comparison practices. Idaho and Utah are collecting components of the MHI and the Clinical Microsystem Assessment Tool. Florida and Maine will be providing MHI and MHI-RSF data as they become available.

3. Qualitative Data Collection Methods

The national evaluation team gathered information on the medical home assessment tools chosen by each State and the reasons for their choices through a series of conversations with State officials and other stakeholders during the first year of the demonstration. During the planning phase, presentations were made to the States by the NCQA, developers of the MHI, and the national evaluation team about the different medical home assessment tools and their potential pros and cons for States and evaluation purposes.

In addition to these early discussions, the national evaluation team conducted site visits to demonstration States from March to August 2012, when States were in the early phases of project implementation, approximately 2 years into their 5-year CHIPRA quality demonstration projects. The site visits consisted of semi-structured interviews conducted with purposively selected key informants. Respondents included demonstration staff, practices participating in the demonstration, representatives from consumer advocacy organizations and professional associations, and other key stakeholders. Interviews were conducted in person, during site visits to demonstration States. The few respondents who were unavailable for interviews on site were interviewed by phone on dates preceding or following that State’s site visit. One interviewer and one note-taker participated in each interview. Interviews were audio-recorded with respondents’ expressed consent. These interviews provided additional insights into how the States and practices were using the medical home assessment tools during the implementation period and additional information on their perceived strengths and weaknesses.

4. Psychometric Assessment of the Medical Home Index-Revised Short Form

Creating the Medical Home Index-Revised Short Form

The MHI-RSF was developed as a low-burden medical home assessment tool for the national evaluation of the CHIPRA Quality Demonstration Grant Program. The tool modifies the Center for Medical Home Improvement’s (CMHI) MHI-Short Version (MHI-SV) by adding four additional items from the full Medical Home Index (MHI). W. Carl Cooley and Jeanne McCallister of CMHI provided input into the modification of the MHI tool to create the MHI-RSF.

The 25 item MHI has been shown to be a reliable and valid tool to assess medical homeness at the practice level and its six domains align with the American Academy of Pediatrics and the Maternal and Child Health Bureau definition of a medical home.[4] CMHI used statistical techniques (factor analysis) on the full MHI to develop the 10 item MHI-SV as an interval measure for practices undergoing medical home transformation.[5]

The MHI-RSF includes all 10 items from CMHI’s MHI-SV and adds four items from the full MHI to capture additional components of medical homeness (Table 2). Two of the added items fall under the medical home domain of Data Management, which was not included in the MHI-SV but has become more important in recent years as technology has advanced. The final two new items, Communication/Access and Family Involvement, were prioritized by CMHI as important additions.

Table 2. Domains and Topics on the MHI and MHI-RSF

Notes: CSHCN= children with special health care needs

* indicates that the topic was not on the original MHI-SV

Like the full MHI, each item on the MHI-RSF is scored 1 to 8, with 1 representing the most basic care and 8 representing the most comprehensive care. Scores are totaled and then standardized to a scale of 0-100 for ease of interpretation. In addition, overall and domain specific means (range: 1-8) are calculated.

The resulting MHI-RSF strikes a balance between comprehensively representing the domains of medical homeness as defined by the American Academy of Pediatrics (AAP) and the Maternal Child Health Bureau (MCHB), capturing all six domains in the full MHI, and being low burden for intervention and comparison practices participating in the demonstration.

Psychometric Analysis of the MHI-RSF

The MHI-RSF is an adaptation of two previously validated medical home assessment tools. To further assess the scientific properties of the MHI-RSF, we conducted descriptive and psychometric analyses on baseline data from 104 pediatric and other child-servicing practices participating as intervention or comparison practices in the CHIPRA Quality Demonstration Grant Program in six States. The evaluation team:

1. Compared scores from the MHI-RSF and full MHI among the same practices.
2. Calculated rank-order correlations between the MHI-RSF scores and MHI scores among the same practices.
3. Calculated the internal reliability of the MHI-RSF.
4. Analyzed performance on the MHI-RSF by several practice characteristics thought to be associated with medical homeness, providing evidence of the tool’s validity.

MHI-RSF versus MHI Scores

At baseline, 33 practices shared data on the full MHI and 71 practices on the MHI-RSF; counts include both intervention and comparison practices. Among the practices that provided data for the full MHI, we compared the scores from the 25 MHI items to the subset of 14 items included in the MHI-RSF. Table 3 shows descriptive statistics for the overall standardized total score (standardized to 100 points for both the MHI and MHI-RSF), the overall mean score (range: 1-8), and six domain mean scores (range: 1-8) for the 25 MHI items versus the subset of 14 MHI-RSF items. Figure 1 depicts the mean scores graphically. 

Table 3. Comparison Between Scores on the 25-Item MHI and the Subset of 14 Items in the MHI-RSF, Among 33 Practices that Completed the MHI

† standardized to a 100-point scale

Figure 1. Comparison Between Scores on the 25-Item MHI and the Subset of 14 Items in the MHI-RSF, Among 33 Practices that Completed the MHI

To assess whether the 33 practices rank similarly in medical homeness scores when measured by the MHI-RSF and MHI, we calculated Spearman rank-order correlation coefficients (Table 4). These correlation coefficients can range from 0 to 1. Correlations closer to 1 indicate that practices rank similarly when measured on the two tools. That is, practices that rank highest on the MHI also rank highest on the MHI-RSF. For the overall standardized total, overall mean, and domain-specific means, we observed high correlation coefficients ranging from 0.81 to 1.00. All coefficients were statistically significant. This analysis provides evidence that the MHI-RSF and MHI tools rank practices similarly on medical homeness.  

Table 4. Spearman Rank-Order Correlations Between MHI-RSF Scores and MHI Scores (n= 104 practices)

*p<0.05

† The items in the data management domain are the same for the MHI-RSF and the MHI, resulting in a perfect correlation.

Reliability

We assessed the internal reliability of the MHI-RSF among 104 intervention and comparison practices that provided baseline data on either the MHI-RSF or MHI.[6] We calculated the raw score Cronbach’s alpha, a coefficient ranging from 0 to 1 that is a function of the degree of correlation among items in the tool. Higher scores indicate greater correlation among items. A Cronbach’s alpha above 0.7 is considered a marker of a good scale.[7] The Cronbach’s alpha for the MHI-RSF was 0.89 (Table 5). All 14 items showed moderate to strong item-to-total correlations, ranging from r= 0.47 to 0.72 (not shown) indicating that they are measuring different dimensions of the same construct. Inter-item correlations ranged from r= 0.14 to 0.73 and were statistically significant for nearly all (82 of 91) of the item pairings (not shown). Deletion of any one item did not increase the Cronbach’s alpha above 0.89, suggesting that no items are adding error and no items should be cut (not shown).

Table 5. Internal Reliability of the MHI-RSF (n= 104 practices)

Additionally, among the 33 practices that completed the full MHI, we compared the internal reliability of the 25 MHI items (Cronbach’s alpha= 0.93) to the subset of 14 MHI-RSF items (Cronbach’s alpha= 0.88). Because the Cronbach’s alpha is in part determined by the number of items in a scale, we expect that the internal reliability will decrease simply by reducing the number of items. The Spearman Brown Prophecy Formula can predict the Cronbach’s alpha that would be expected based on subtracting a specified number of items. The Spearman Brown Prophecy Formula predicted that by cutting the 25-item MHI to the 14-item MHI-RSF, we would expect the internal reliability to drop from 0.93 to 0.89 in this subset of practices. We observed an MHI-RSF internal reliability (Cronbach’s alpha= 0.88) very close to the prediction, suggesting that the difference in the reliability of the MHI-RSF and MHI is related only to the decrease in items.

In the full sample of 104 practices, we also calculated the internal reliability of the domain scores on the MHI-RSF (Table 5). Two of the MHI-RSF domains (Community Outreach and Quality Improvement) have only one item, so internal reliability is not applicable. The Cronbach’s alphas of three domains (Chronic Condition Management, Care Coordination, and Data Management) were 0.7 or greater, indicating acceptable reliability; however, the internal reliability of the Organizational Capacity domain was low (Cronbach’s alpha= 0.61). The three items in the Organizational Capacity domain had fair to moderate inter-item correlations (range: r= 0.24 to 0.45; all p<0.05), indicating the low internal reliability may be driven by the small number of items in the domain rather than by poor inter-item correlation.

We did not test inter-rater reliability of the MHI-RSF. See Cooley et al (2003) for information on inter-rater reliability of the full MHI.3

Validity

While we did not design a validity test prior to the baseline analysis of the MHI-RSF, we were able to examine indicators of validity by calculating the MHI-RSF Overall Standardized Total Score stratified by practice characteristics that are expected to be related to medical homeness. Information on these additional practice characteristics were collected on the MHI-RSF. Among all 104 practices, the following characteristics are statistically associated with having higher MHI-RSF overall total scores: being involved in other medical home or quality improvement initiatives; having a care coordinator present in the practice; being knowledgeable about and regularly applying the concepts of the AAP medical home definition; and being knowledgeable about and regularly applying the concepts of the MCHB elements of family-centered care (Table 6). This analysis provides evidence of known-group validity of the MHI-RSF, indicating that the MHI-RSF is measuring the concept it was designed to assess.

Table 6. MHI-RSF Standardized Total Scores by Practice Characteristics Likely to be Associated with Medical Homeness

Notes: *p<0.05; not all practices answered all questions on the survey, so the total number of observations across questions may vary.

AAP = American Academy of Pediatrics; MCHB = Maternal and Child Health Bureau

5. State-by-State Analysis of the MHI-RSF 

In addition to the descriptive analysis presented in Evaluation Highlight No. 2, we explored the MHI-RSF scores by State (Table 7). The methods for selecting demonstration practices and collecting data varied across States. Each State sought to include practices that varied along key dimensions, such as size, ownership, and geographic location. Despite these efforts, the practices selected in a particular State may not reflect the mix of practices in the State as a whole. Moreover, some States have a very small number of demonstration practices. Together, these limitations indicate that the results presented below should not be interpreted as representative of a State as a whole, and comparisons across States should be interpreted with caution.

The mean standardized total score among intervention practices ranged from 51.5 in Alaska to 61.8 in North Carolina. Practice scores varied widely within each State, as indicated by the minimum and maximum total scores. While States do vary in baseline scores, the results suggest that intervention practices in all States have considerable opportunities to benefit from the PCMH interventions implemented through the CHIPRA demonstration grants.

Table 7. Distribution of MHI-RSF Standardized Total Scores Across Intervention Practices in Six States 

Note: Overall standardized total scores are standardized to a scale of 1-100. SD is standard deviation.