Design and Evaluation of Three Administration on Aging (AoA) Programs: Chronic Disease Self-Management Program Evaluation Design—Final Evaluation Design Report
The goal of this project is to develop a research design that estimates the effect of Chronic Disease Self-Management Programs (CDSMP), as administered in Administration on Aging (AoA)-funded settings, on outcomes of participants among AoA's target population, i.e., individuals aged 60 or older. The 24 states which were original CDSMP grantees in 2006 and 2007 are eligible for the evaluation. Public health care expenditure in the Medicare and Medicaid programs is an outcome of particular interest to AoA, because there has been little research examining this outcome for CDSMP participants. AoA seeks the most rigorous evaluation methodology and is therefore interested in the feasibility of a randomized controlled trial (RCT) evaluation of CDSMPs. AoA is also interested in a rigorous alternative design if there are feasibility challenges with the RCT design. We propose Propensity Score Matching (PSM) design with matched controls from Medicare administrative records. An overview of the proposed design elements is provided below. After the evaluation design is finalized, AoA will procure the services of an independent evaluation contractor to carry out a national evaluation of the CDSMP.
Experimental Option (RCT): After reviewing a large number of successfully implemented randomized controlled trials, we recommend this approach for the CDSMP evaluation. After applicants give informed consent, they will be randomly assigned to either a treatment or a control group, with equal probability. This method is the gold standard in program evaluation since it is the only method that can produce a comparison group that is similar to the treatment group on observed and unobserved characteristics thanks to the randomization. Due to ethical concerns, the control group will be put on a 6-month wait list to receive the program rather than being denied.
At the time of application to the CDSMP, sites will collect from applicants paper-and-pencil baseline data, and a 6-month follow-up phone survey will be conducted by the evaluation contractor. In parallel, Medicare administrative data spanning the 12 months prior to and the 12 months after the CDSMP workshop will be obtained from CMS. This procedure allows estimation of impacts on self-reported and Medicare outcomes at 6 months with the highest rigor. A 12-month follow-up phone survey (again, conducted by the evaluation contractor) is also worth considering because a longer period may be necessary to observe the impacts of CDSMP. However, the 12-month impacts cannot be estimated with as much rigor as the 6-month impacts, because the control group will be allowed to take the CDSMP workshops after 6 months. 12-month impact estimates will be biased downwards. Given the lack of a clean control group, we also recommend a pre-post test to estimate the 12-month impacts. This approach has been adopted frequently in the CDSMP evaluation literature.
In conclusion, the proposed RCT design is the most rigorous design given the ethical limitation of waitlisting the control group for six months only rather a year or two years.
Quasi-Experimental Option (PSM): We are proposing PSM as the next best option after RCT. When RCT is not feasible, PSM is an ideal alternative in situations where there is a rich set of characteristics available for a large population pool from which matches to the treatment group can be selected. Treatment group is the group of people who take the CDSMP workshops. Availability of rich administrative records for Medicare beneficiaries residing in service areas of the CDSMP host sites makes it feasible to implement a rigorous PSM. This method basically provides a mechanism to select a comparison group (administrative control group) by simultaneously matching on a multitude of characteristics of the treatment group including recent Medicare utilization and expenditures, health status, and demographics. Unlike the RCT, however, PSM cannot guarantee matching on unobserved characteristics. The PSM results are valid to the extent that these unobserved characteristics are not important in determining outcomes of interest or that they are captured by the extensive set of observed characteristics.
Similar to the RCT option, Medicare administrative data spanning the 12 months prior to and the 12 months after the CSMP workshop will be obtained for the treatment and comparison groups. An advantage over the RCT option, this option allows rigorous estimation of CDSMP impact on Medicare utilization and expenditures not only at 6 months, but also at 12 months since the comparison group will be contamination-free beyond 6 months. However, unlike the RCT option, the baseline, 6-month, and 12-month follow up surveys will only be administered to the treatment group as it will be very difficult to reach out to an adequate number of administrative controls. Due to lack of survey data for the comparison group, impact on self-reported 6-month outcomes cannot be estimated as rigorously as in the RCT option. Instead, a pre-post test is going to be utilized.
Outcomes and Survey Instruments
Self-reported outcomes include health status, health behavior, self-efficacy, quality of life, and cognitive symptom management. For comparability of results with earlier studies, we recommend that these self-reported outcomes be measured with the survey instruments developed by Stanford University. Claim-based outcomes include Medicare utilization and expenditures (number of hospitalizations, length of hospital stay, number of emergency room visits), and Medicare expenditures in various settings (inpatient, outpatient, physician, home health, skilled nursing facility, durable medical equipment, prescription drug/Part D). Even though CDSMP has the potential to improve outcomes for Medicaid beneficiaries (i.e., the dual-eligible population), it does not seem feasible to include Medicaid outcomes in the study for a number of reasons, including (1) a long lag time in receiving data and (2) inconsistencies across states that make the data difficult and costly to analyze. Although we had discussions with the sites and the technical assistance contractor, we were unable to determine the degree to which AoA grantees are specifically targeting dual-eligible beneficiaries. Furthermore, there were no data available to our team to assess whether dual eligibles constitute a reasonably large group among CDSMP participants. Nevertheless, we recommend keeping the dual eligibles in the study and evaluating the impact of CDSMP on their self-reported and Medicare outcomes if there happens to be an adequate number of them participating in the study. Furthermore, assuming an adequately large number of dual eligibles, it may also be possible to assess the impact of CDSMP on some components of Medicaid outcomes such co-pays paid by Medicaid for dual eligibles.
Selection of Sites and Sample Size
Due to the decentralized way in which the CDSMP program is implemented, the evaluation activity will be coordinated at the host site level (rather than the state level). March 2011 data from the CDSMP technical assistance contractor indicate that approximately 30,000 people attended the workshop in the prior 12 months. We estimate that evaluation sample frame is 14,500 of these in about 500 host sites. The frame consists of individuals 60 years of age or older, taking the generic CDSMP workshop in the 24 states that are the original grantees. A key finding is that most of the host sites are very small (average is 30 attendees per site) and a few large sites account for many of the workshop participants.
Based on the statistical power calculations we conducted, we recommend a baseline sample of 3,000. In order to recruit this many study participants we estimate that at least 20 of the largest sites will need to be recruited for the study along with a long intake period of 12 months. In addition, based on our analysis of the technical assistance contractor data and calls with sites, we strongly recommend that AoA provides additional funds to the evaluation sites in order for them to sustain and double the number of participants. The biggest challenge for the RCT option is being able to recruit the target sample size. To ensure national representation, up to 30 other sites may be probabilistically selected for a total of up to 50 sites. However, given the potential challenges of recruiting enough participants, it is advisable to include as many large sites as possible.
There are many uncertainties around the circumstances influencing the ability to recruit adequate sample size. It is necessary that the evaluator reassess these circumstances. Since the PSM option significantly reduces the recruitment burden, it may end up being the feasible design option based on this future assessment.
Our power calculations also indicate that an unrealistically large sample size (50,000) would be required to detect statistically significant CDSMP effects on Medicare expenditures. This is due to very high variability in these expenditures. The estimated impact will be the best estimate available, but it will be imprecise.
Evaluation Grants from AoA
Given the absence of a centralized, robust, and streamlined registration system, and because local staff are overstretched, we strongly recommend that AoA provides funds to help the selected sites develop a uniform registration system. To alleviate concerns about loss of clients to randomization, we also strongly recommend that AoA provides additional funds to help selected sites with marketing, advertising, and recruiting with the intent of doubling the customers.
Evaluation Timeline and Cost
Both the RCT and PSM evaluations are expected to take four years and approximately $2.6 million to complete with 50 evaluation sites. For 20 evaluation sites, estimated costs for RCT and PSM are $2.2 million and $2.1 million, respectively. Details are provided in Appendix A.
Combination Option (First RCT, then PSM, if necessary)
Considering the desire to conduct an RCT and the uncertainties around its feasibility, an optimal evaluation strategy would be to start with the RCT implementation and to switch to PSM, if necessary. At the midpoint of the 12-month intake period, the evaluator would assess the number of study recruits and will make a determination on the likelihood of reaching the target sample size by the end of the intake period. If the assessment is negative, the evaluator would switch to the PSM design. This switch should not cause any significant delays or significant additional evaluation costs if planned carefully before the evaluation is implemented.
Page originally created May 2011