Hispanic Diabetes Disparities Learning Network in Community Health Centers
Chapter 4. Evaluation Design
Table of Contents
The evaluation design of this initiative was to answer the following questions:
- What were the results at the clinic level in increasing goal-setting performance?
- Summary of Action Plan data.
- Assessment scores of clinic progress during the initiative.
- What were lessons learned at the clinic level in implementing this patient self-management intervention for each component of the Chronic Care Model? For the Hispanic population?
- What is the potential for spread within clinics at the end of this initiative? Likelihood of changes being institutionalized; further data collection being conducted to demonstrate the impact of goal setting on the control of the disease; and continuation and spread within the clinic?
- What are the plans and likelihood of regional/national spread of what has been learned?
- What were the key lessons learned relative to conducting this learning network initiative for community health centers?
1. Clinic-Level Results for Intervention
1A. Increasing Goal-Setting Rates
Self-management of a chronic condition relies primarily upon the ability of patients to change those behaviors that can improve the control of the condition. Self-management is more than patient education and includes communication between the patient and provider that promotes interactive goal setting and problem solving. Through the goal-setting process the patient can often develop the self-confidence needed to achieve the selected goal.
Improving the goal-setting rate in patients with diabetes was the key measure of this intervention. Table 1 summarizes characteristics of the six clinics, differences in their implementation strategies, and their progress with the intervention.
Table 1. Clinic characteristics, goal-setting rates, and HbA1c levels
|Clinic||FQHC||Chronic Disease Registry||Intervention Target Population||Team Members||Primary Clinician||Goal-Setting Rate||HbA1c Level||Additional Information|
|District of Columbia Metropolitan Area|
|Clinic A||No||Yes||All patients in clinic with diabetes||Medical director, nurse practitioner (NP), and diabetes educator (DEd)||NP, who performed interactive goal setting, but often referred to DEd 2 days each week for additional counseling and goal setting.||Post-intervention goal-setting rate was 39%.||
Clinics A and B had combined baseline HbA1c level of 9%
Post-intervention HbA1c level for Clinic A was 8.6%.
|DEd, who was a physician in her Central American country, conducted classes at Clinic A for persons with diabetes.|
|Clinic B||No||Yes||All patients in clinic with diabetes||Same three individuals as for Clinic A||
Most interactive goal setting performed by DEd who spent 3 days in Clinic B and held classes at night or on weekend for persons with diabetes.
|Increased from 9% to 41%.||
Combined baseline HbA1c level for Clinics A and B was 9%.
Post-intervention HbA1c level for Clinic B was 8%.
|Clinic B completed most Action Plans, with almost all involving DEd.|
|Clinic C||No||Yes||All patients in diabetes school during intervention period||Clinic director/administrator and DEd (clinic physician present for first Learning Session only)||No clinician was involved in goal setting and no one-on-one interactive goal setting occurred.||Not available (Clinic has since implemented electronic medical record [EMR]; plans to evaluate impact of goal setting in diabetes school.)||Not available||Clinic used Action Plan for goal setting only in diabetes school. Trainer was not able to provide one-on-one interactive goal setting. Instead, patients used form to select goals while in class. Copies of Action Plans were not placed in medical records to allow followup by clinicians.|
|Clinic D||Yes||Yes||Diabetic patients of one physician assistant (PA)||PA and quality coordinator||PA, whose performance on many indicators exceeded that of other clinicians in clinic. PA achieved patient continuity and goal-setting rate of 76%. PA worked closely with medical assistant for intervention.||No significant change in rate (75%)||No change in 8.1% level||Other clinicians participated in on-site patient self-management training to facilitate spread. Willing to try interactive goal setting after training because they learned that process did not have to be time consuming.|
|Clinic E||Yes||Yes||All patients in clinic with diabetes||Quality improvement coordinator and medical director||Mid-level practitioners and physicians||Increased from 25.9% to 49%.||Decreased from 8.2% to 8.0%.||
Did not implement Action Plan because similar tool was unsuccessful. Instead, redesigned processes to facilitate interactive goal setting by clinic team with clinicians providing one-on-one support. Also, on-site training was provided to clinicians on patient self-management and goal setting.
Clinic began implementation of EMR during latter part of intervention.
|Clinic F||No||No||All patients in clinic with diabetes||Clinic director only participant||Volunteer physicians (clinic heavily dependent on volunteers)||Not available||Not available||Unable to implement Action Plan or any goal-setting strategy during intervention period.|
Reviewing the data for Clinics D and E from July 2005 through January 2007 provided additional information. As shown for Clinic E in Table 2, the number of patients with diabetes in the registry has remained constant, but the goal-setting rate was higher a year before the intervention started, and after the intervention it is still almost 10 percentage points lower than in July 2005. However, the HbA1c has steadily declined from 8.3 to 8.0, raising concerns that the goal-setting rate had little to do with the outcome results or control of diabetes for this clinic. This same finding has been noted for other clinics with a registry, and one potential explanation offered by clinics not participating in this intervention is that more goal setting occurs than is documented in the registry. They doubt that the goal-setting rate truly falls from 60 percent or higher to 25 percent or lower in a year, but instead believe that clinics document goal setting better when they are participating in an intervention focusing on patient self-management.
Clinic D presents a different picture. For the same time period, 72 patients with diabetes have been added to the registry for a 52 percent increase. The percentage of these patients with a goal has gradually declined about 9 percentage points, which is understandable given the increase in patients. However, the HbA1c remained the same over the 19 months.
Table 2. Additional HbA1c and goal-setting data for Clinics D and E
|Date||DM Patients in Registry||HbA1c||No. with Goals||% With Goals|
Overall, clinics with strong team participation in the learning network were more successful. Having a strong clinic champion improved the likelihood for success. Furthermore, having a medical director as the clinic champion may be the best scenario for intervention support. Following are additional findings from the intervention:
- Clinics with a diabetes educator (even part time) completed the greatest number of Action Plans for patients with diabetes during the intervention.
- Clinics with volunteer clinicians perceive greater difficulty implementing any quality improvement strategies with the volunteer staff.
- Clinics without registries are at a disadvantage in collecting data needed to assess progress with quality improvement interventions.
- Clinics may use the same chronic disease registry but vary in the amount of data they enter into the system and use routinely for performance improvement.
- A clinic can redesign its clinic processes and increase the goal-setting rate similar to a clinic using a tool such as the Action Plan. Clinic team acceptance of a tool is important for success. Implementing a paper tool in any intervention may meet more resistance as clinics move to electronic medical records (EMRs).
- A high-performing clinician with a goal-setting rate already in the highest quartile may not benefit significantly from a tool such as the Action Plan.
- An improvement in the process and outcome measures of an intervention may not provide evidence that the intervention was effective in achieving the outcome. Ongoing monitoring and analysis of clinic data in the context of the environment are necessary to know when the intervention is the primary reason for the outcome results. Patient goal setting is a common measure for community clinics, but few clinics have designed a methodology to determine the effectiveness of their goal setting in changing patient behaviors.
1B. Analysis of Action Plan and Activation Tool Data
Action Plan. During the intervention, 211 Action Plans were completed, with more than 96 percent by Hispanic patients. The three most frequently chosen goals are shown in Table 3.
Table 3. Most frequently identified Action Plan goals
|Type of Goal||Number of Action Plans
With Goal Identified
|% of Action Plans|
Differences in patient goals by clinic. The diabetes educator in Clinic B with 91 Action Plans had 51 patients or 56 percent choosing diet as their goal. Other clinics varied from 22 percent to 42 percent for diet. Additional findings included:
- Clinic D with physician assistant performing interactive goal setting had the highest percentage (27 percent) choosing reduction of stress as a goal.
- Clinic C did not provide one-on-one interactive goal setting and had patients in diabetes school complete the Action Plan without assistance. Table 4 highlights the difference in rates for goals chosen in Clinic C compared with those for all clinics.
Table 4. Rates for Action Plan goals chosen by all clinics versus Clinic C
|Type of Goal||% of Action Plans|
|All Clinics||Clinic C|
Clinicians in Clinic C referred newly diagnosed patients with diabetes and those having the most difficulty with control to the diabetes school classes. Without linking the Action Plans with the medical records, it was not possible to determine if this group was significantly different from the patients targeted in the other clinics. The higher rate for medication goal setting for this group raises the concern that patients may be more likely to acknowledge lack of medication adherence when completing the Action Plan alone than when completing with a clinic team member.
Comfort levels chosen by patients indicated confidence in achieving their selected goal. If a form had a goal specified, but no comfort rating was provided, this form was dropped from calculations on comfort level. Every clinic had a few Action Plans without a comfort level noted.
However, an outlier was one clinic with 40 percent of Action Plans having no comfort-level rating.
- For the four clinics implementing the Action Plan, the percentage of forms with a "10" rating ranged from 44 percent to 70 percent with the latter in Clinic C without a one-on-one goal-setting process. The lowest was in the clinic where comfort level was not rated in 40 percent of the Action Plans.
- No Action Plan had a rating less than 5; the second most frequent rating was 5, with the third and fourth being ratings of 7 or 8.
Activation Assessment Tool. All clinics agreed from the beginning of the intervention that although this tool had been validated in Spanish, it was above the literacy level of their clinic population. However, a greater concern was that the concepts contained within the survey were not familiar to their Hispanic population. One clinician explained that Hispanics tend to be emotional and rate their feelings and well-being at the extreme poles of any rating scale and rarely select the middle ratings.
The two Spanish Catholic clinics continued to use the Activation Tool for 3 months to give the instrument a trial.
At Clinic A, 26 Activation Assessment Tools were completed as follows: 8 by the diabetes educator, 13 by the nurse practitioner, and 5 by the medical director. Table 5 provides the percentage of total responses (4 x 26 = 104) for each rating on the 10-point scale under the four areas of patient confidence.
Table 5. Clinic A: Percentage of total responses for each scale rating
At Clinic B, 74 forms were completed primarily with the assistance of the diabetes educator. Table 6 provides the percentage of total responses (4 x 74 = 296) for each rating on the 10-point scale under the four areas of patient confidence.
Table 6. Clinic B: Percentage of total responses for each scale rating
In addition, of the total responses for Clinic B:
- 51.4 percent answered 10 under all four areas of patient confidence.
- 64.8 percent rated all four areas of patient confidence with the same number.
- 81.2 percent rated all four areas of patient confidence between 7 and 10.
For Clinics A and B, Latinos were more likely to choose ratings demonstrating a high confidence level in managing their own care and understanding their treatment.
One drawback in having six clinics with such varying characteristics and intervention designs is that the findings may be interesting for consideration and further study, but may not be applicable to other clinics. For example:
- The patients in a diabetes school that completed the Action Plan themselves without one-on-one interactive goal setting were more likely to choose medication for their goal than patients setting a goal with a team member. They were also more likely to rate their comfort with reaching their goal at a "10" (70 percent).
- Having a diabetes educator in the clinic for clinicians to refer patients for goal setting might result in more patients having a goal.
- The discipline of the team member performing interactive goal setting might influence the type of goal selected by the patient.
- Hispanic patients' responses for the Action Plan on a scale from 0 to 10 relating their comfort or the likelihood of reaching their goal were predominately from 7 to 10. A concern might be that the patients were expressing confidence to please the clinic team.
2. Lessons Learned in All Components of Chronic Care Model
The Chronic Care Model is a framework for improving the health care of individuals with chronic conditions and includes six components:
- Delivery System Design.
- Decision Support.
- Clinical Information System.
- Patient Self-Management.
- Health System Organization.
- Community Resources and Policies.
Though this intervention focused on self-management, Tables 7 through 12 summarize lessons earned in each of the components of the Chronic Care Model.
Table 7. Lessons learned in patient self-management
|Select tools proven successful in implementing intervention to improve goal-setting rates.|
|1. Use Action Plan to facilitate patient self-management and goal setting.||
|2. Use Activation Assessment Tool to assess patient's comfort level in making behavioral changes. Knowing this level helps provider understand where to begin in goal-setting process and in education.||
|3. Make clinic team aware of potential benefit of intervention and assure they have skills for patient self-management.||
|4. Identify clinic team members involved in intervention and define their roles and responsibilities to maximize resources and patient outcomes.||
|5. Followup for progress on goals should not wait until next scheduled visit but occur within 2 weeks.||
|6. Invite family to participate in patient care.||
|7. Create comfortable, caring atmosphere for patients in clinic and classes.||
|8. Patient and all members of provider team must accept responsibility for care and outcomes.||
|9. Debunk myth in Latino population that insulin can worsen symptoms and complications of diabetes.||
Table 8. Lessons learned in decision support
|Determine any changes that could support better decision making in clinic.|
|1. Develop standing orders for medical assistants to use in ordering blood tests, procedures, and referrals for patients with diabetes.||
|2. Develop sheet for front of patient record with diabetes clinical guidelines stating what tests or procedures should be performed at every visit and periodicity of others.||
|3. Use screening tools for behavioral health problems, especially depression, to facilitate detection in all patients with chronic diseases. Screening for substance abuse has been added in one clinic.||
Page originally created September 2012