As noted in Section 4, we intended to conduct impact analyses of Category C projects. Of the 17 States participating in Category C, 12 focused on PCMH models, three were focused on finding new strategies for funding CMEs or developing new ones (Maryland, Georgia, Wyoming), and two were focused on SBHCs (Colorado, New Mexico). We wanted to ensure that we had the potential to combine data across the States to maximize the chances of having sufficient analytic power to estimate impact. Hence, we excluded the five States not working to implement PCMH models because those models and the expected effects were sufficiently different from the PCMH approach that combining them would not have been conceptually plausible.
CMS did not require or encourage States to develop comparison groups for any of their CHIPRA projects, although some were planning to do so. By the time the national evaluation had developed a foundation of knowledge about the State projects (in December 2010, about 10 months after the start of the States’ demonstrations), the States were completing their planning process and, in some cases, were well underway with implementation—too late for major design changes that might have supported a comparison group and a more rigorous evaluation. However, the NET was able to work with some of the PCMH States to develop a comparison group when they otherwise would not have done so.
Of the 12 PCMH States, we excluded from our planned analysis the States that did not agree to recruit comparison practices (Alaska, Florida, Idaho) or were unable to identify practices that were not participating in other PCMH initiatives in the State (Oregon, Vermont). Of the seven States that agreed to recruit comparison practices, Utah and West Virginia were unable to provide the Medicaid administrative data necessary for the analysis despite several months of negotiation.
We received data from the five remaining States (Illinois, Maine, Massachusetts, North Carolina, and South Carolina), including Medicaid enrollment and claims data and PCMH survey data,a from more than 140 intervention and comparison practices. There were severe data quality issues in the baseline data for three of the States (Maine, Massachusetts, and North Carolina) that required considerable time and resources to resolve via multiple rounds of data submission and discussions with the States. Unable to overcome data limitations and faced with budget constraints, we ultimately excluded Maine and Massachusetts from the analysis.
In 2013, we analyzed baseline data (2009–2010) from the three remaining States (Illinois, North Carolina, South Carolina), presenting findings at the 2013 and 2015 AcademyHealth annual research conferences and publishing a manuscript in the journal Academic Pediatrics.b
Site visits conducted by the NET in 2014 revealed that North Carolina and South Carolina shifted the focus of their projects from general PCMH improvement efforts as originally conceived to more specific QI projects that each practice conducted on targeted topics (for example, increasing dental visit rates, lowering obesity rates, or improving rates of documented developmental screenings). We decided not to conduct an impact evaluation on these projects because (1) no uniform set of outcomes applied to all practices; (2) the number of practices focusing on a given outcome was too small to expect an impact analysis to detect changes; and (3) we would be unable to assess some of the key outcomes (for example, BMI screening) through analysis of claims data, and alternative data collection methods, such as chart reviews, were not feasible.
Illinois was the only remaining State whose Category C project was designed to help practices enhance their PCMH features. Illinois provided PCMH survey data and Medicaid administrative data for calendar years 2009–2013 to support our impact evaluation. However, the State cannot yet provide data for the full post-intervention period (which includes calendar year 2014) because of the recent transition in CMS’ Medicaid data systems. For 2009–2013 data, Illinois sent us Medicaid Statistical Information System (MSIS) data approved by CMS. The State is transitioning to the new data system (the Transformed Medicaid Statistical Information System, or T-MSIS) for all claims occurring January 2014 and later. Illinois halted MSIS production beyond that date and will not provide 2014 claims until CMS has transitioned to and validated the T-MSIS claims, the timeline of which has been delayed and remains unknown. Hence, we have no way of accessing the post-intervention data we need to conduct an impact analysis.
a Maine, Massachusetts, North Carolina, and South Carolina used the Medical Home Index or the Medical Home Index-Revised Short Form developed for the evaluation. Illinois used the National Committee for Quality Improvement’s Patient-Centered Medical Home practice self-assessment.
b Christensen AL, Zickafoose JS, Natzke B, et al. Associations between practice-reported medical homeness and health care utilization among publicly insured children. Acad Pediatr 2015 May-June;15(3):267-74. Available at http://www.academicpedsjnl.net/article/S1876-2859(14)00429-X/abstract. Accessed October 27, 2015.
