More About the National Evaluation

This page includes additional information about the national evaluation’s objectives, research questions, and data sources.

Contents 

 Overview

In August 2010, the Agency for Healthcare Research and Quality (AHRQ), in partnership with the Centers for Medicare & Medicaid Services (CMS), awarded a contract to Mathematica Policy Research and its partners, the Urban Institute and AcademyHealth, to evaluate this demonstration grant program—the largest evaluation of child health care quality currently underway in the United States. Together, the grant program and its evaluation are among the Nation's most important efforts to improve the quality of health care for children enrolled in Medicaid and the Children's Health Insurance Program (CHIP).

The national evaluation team is charged with conducting a rigorous evaluation of the CHIPRA Quality Demonstration Grant Program to determine the impact of grantee activities on the quality of children's health care. (Contact the National Evaluation Team.)

Top of Page

 Evaluation Objectives

As described elsewhere, 18 States are implementing 52 projects in 5 broad categories (go to Table 1 below). The national evaluation team will be gathering many different kinds of qualitative and quantitative data about these projects and about the collaboration among grantees and their partners. We will analyze this information to conduct a multi-level evaluation that aims to:

  • Assess the implementation of single projects independently of all others, focusing on whether the project's goals and objectives were achieved.
  • Combine information across projects within a single grant category to identify effective strategies and successful outcomes.
  • Examine how specific States improved the quality of children's health care by implementing multiple projects and describe how the activities in one grant category supported or enhanced projects in other categories.
  • Determine the extent to which collaborations among States contributed to the success of the demonstration activities.
  • Assess the overall benefits of the demonstration program by comparing selected outcomes of the participating States with those of non-participating States.
  • Examine the contributions of demonstration activities to improvements in quality of care with respect to four special interest areas: oral health, obesity, behavioral health, and Early and Periodic Screening, Diagnostic, and Treatment (EPSDT) programs.
  • Provide insights into the successes and limitations of the program to inform future Federal demonstration efforts.

Top of Page

 Data Sources

Sources of quantitative data include administrative and claims data, original survey data from providers, and data from the State or the grantee-specific evaluation teams. Qualitative data sources include program documents and reports, key informant interviews with program staff and stakeholders, and information gleaned from focus groups with families.

Top of Page 

Demonstration Projects by Grant Category

 Table 1. CHIPRA Quality Demonstration Projects by Grant Category

  Using Quality Measures to Improve Care Quality Promoting Use of Health IT to Enhance Quality of Care Evaluating a Provider- Based Model Testing a Model EHR Format for Children Testing State-Specified Approaches to Improving Quality
Alaska    
Colorado*      
Florida*  
Georgia      
Idaho    
Illinois  
Maine*    
Maryland*        √
Massachusetts*    
New Mexico      
North Carolina*    
Oregon*    
Pennsylvania*    
South Carolina*    
Utah*    
Vermont    
West Virginia    
Wyoming      √
Total Projects in Category 10 12 17 2 11

Source: Final operational plans, evaluation addenda, and evaluation planning calls with grantees.
*Grantees.
Note: Health IT= health information technology; EHR= electronic health record.

Top of Page

 Evaluation Questions

AHRQ and CMS have identified 20 broad research questions and well over 200 detailed ones that the national evaluation might address in evaluating the demonstration projects. Examples of the broad questions include the following:

  • How did stakeholders use the initial set of core quality measures for children, and what was their impact on the delivery system?
  • What health information technology (IT) or health IT enhancements were effective in improving quality of care or reducing costs?
  • How were the provider-based models implemented by the demonstration States, and did they change children's health care quality?

Findings from the national evaluation will be organized, published, and disseminated in ways that address the needs of stakeholders, including Congress, AHRQ, CMS, States, the provider community, and family organizations. The national evaluation team will disseminate results of analyses with evaluation highlights, implementation guides, journal manuscripts, and presentations. For published reports, visit the What We Learned page.

Top of Page

Evaluation Design Plan

This final design plan describes the national evaluation team's current plans for evaluating the CHIPRA Quality Demonstration Grant Program. It is a "living document" that will evolve as the States' programmatic and evaluation activities are shaped by actual implementation experiences. At this point, grantees and partner States vary widely with respect to implementation schedules for specific activities. This document describes the evaluation design based on information available from the States as of February, 2014.

 Learn More About the CHIPRA Quality Demonstration

  • Read more about what the national evaluation has learned  through our various publications.
  • Learn more about the demonstration projects from reports and resources produced by the demonstration States.
  • To learn about each of the grant categories, please use the left navigational bar, organized by 5 areas of focus.
  • Learn more about the projects in each demonstration State.
  • Contact: CHIPRADemoEval@ahrq.hhs.gov.

Top of Page

Please note: This Web site uses the term "national evaluation" to distinguish this evaluation of the entire demonstration program from evaluations commissioned or undertaken by grantees. The word "national" should not be interpreted to mean that findings are representative of the United States as a whole.

Page last reviewed June 2014
Internet Citation: More About the National Evaluation. June 2014. Agency for Healthcare Research and Quality, Rockville, MD. http://www.ahrq.gov/policymakers/chipra/demoeval/evaluate-more/index.html