Effective improvement work is data driven. Information on the practice and its performance on key measures is used to:
- Create buy-in for improvement work.
- Identify areas in need of improvement and strengths that can be leveraged to support improvement work.
- Compare the practice’s performance to that of others (benchmarking).
- Prioritize improvement efforts and activities.
- Set improvement goals.
- Track progress toward improvement goals.
- Monitor maintenance of improvements once achieved.
Identifying What to Assess
What you assess should be determined by the scope and goals of the facilitation intervention. You will need to work closely with your practices to prioritize areas for assessment as you begin to work with them. Table 6.1 contains some important metrics you will need to talk to your practices about assessing.
Identifying Assessment Tools
A variety of tools exist to assess a practice. They fall into four categories:
- Surveys and rating scales (patient, provider, staff, whole practice).
- Chart/medical record audits.
- Direct observation and interviews.
- Document review.
Assessments can look at practice processes. For example, the Clinical Microsystems Assessment is a comprehensive assessment package for assessing multiple domains of a practice from clinical systems and performance to patient satisfaction and experience to financial issues.
Specialized tools exist for assessing particular aspects of practice performance and functioning. For example, the Assessing Chronic Illness Care (ACIC) tool evaluates the degree to which a practice’s processes and methods are consistent with the different elements of the Care Model. (Go to Module 16 on Care Model.) Similarly, the Medical Home Index evaluates the degree to which a practice reflects aspects of the patient-centered medical home.
Surveys let you gather information in a systematic fashion. You may want to survey staff satisfaction with the work environment, skills staff members have, and ways staff members spend their time. You can identify sources of stress and underutilized staff and collect ideas for improvement.
Surveys of patient satisfaction or experiences with care are another important source of information. Patient surveys can be broad (e.g., Clinician/Group CAHPS®) or specific to certain types of transformations (e.g., Patient Assessment of Chronic Illness Care, CAHPS® Patient-Centered Medical Home Survey). Qualitative methods, such as focus groups, can also be useful in gauging patient opinions.
Chart or electronic health record audits can be used to examine how the practice does on specific metrics for clinical performance and patient outcomes. Different groups have defined sets of quality metrics available that you can use to guide your assessments in these areas. Your practices will most likely be familiar with them and may already be tracking their performance on some of these metrics. A few examples are listed below:
- National Committee for Quality Assurance Healthcare Effectiveness Data and Information Set (HEDIS) indicators (www.ncqa.org).
- Measures endorsed by the National Quality Forum (http://www.qualityforum.org/).
- Measures approved by the AQA Alliance (www.aqaalliance.org/).
Direct observation will be one of the most powerful assessment tools available to you as a facilitator and something that can make you very valuable to your practices. Direct observation can be used to gather information about patient experience at the practice. For example, you might use a “secret shopper” approach to better understand what a patient experiences in a practice if this is an area of focus for a practice. You may spend time in the waiting room or observe the interaction of a care team to better understand and assess how they work together. You may use observation to assess factors affecting staff workflow or satisfaction, or to evaluate the implementation of new policies or procedures by staff in the practice environment. Observations of specific elements of the practice can be captured using field notes or checklists and then provided to the quality improvement (QI) team to use in designing improvements.
Document review is another important tool for assessing a practice. Examining documents and archival information produced as part of clinical care or various aspects of care can provide valuable insight into what is working and not working with a practice’s systems. You can use document review and observation as tools to conduct “fall-out” assessments, where a forensic analysis of system “failures” is conducted to identify the reasons for the failure and make suggestions for improvement. For example, in a practice interested in improving its lab reporting process, you can use observation and document review to track patients who failed to receive their lab results within the specified time period and identify failures in the system. The practice can use these data to correct and improve the process and reduce failures in the future.
Many different tools are available to examine different domains of practice functioning, from clinical care processes to administrative systems and financial stability. You will need to work with each practice to pick the tools and approaches that best fit the goals for the facilitation intervention and those of the practice and are most likely to yield information that can be acted on to make improvements.
Choosing Assessment Tools
Your facilitation program may have a set of basic assessment tools and measures that are routinely used to assess practices. You may also choose to augment those with additional assessments individual practices you are supporting would like you to conduct. Describe the options to the practice, but be careful or you’ll wind up drowning in assessment data.
The goal is to select a set of measures that will yield information that is “actionable” for the practice, but not to overwhelm them with data. Many variables might be interesting to assess but are not essential information. Part of your job will be to help the practice focus the assessment on those items that are directly relevant to the improvement goals at hand.
Identifying Assets as Well as Challenges
It is important to approach the assessment from an assets-based rather than deficit-based perspective. More than likely you are working with a practice because it is having difficulty implementing desired changes on its own. Thus, the tendency can be to focus only or mainly on the practice’s problems and weaknesses. This can result in a very negative dynamic in which the practice facilitator feels as though he or she must “rescue” the practice. This approach is debilitating to the practice and inhibits the sustained improvement and increased practice capacity that are goals of facilitation.
To avoid this trap, try developing an “asset” map of the practice that includes a list of the skills and talents of staff and clinicians as well as the resources the practice may already have that are relevant to practice improvement. John McKnight’s book, Building Communities From the Inside Out: A Path Toward Identifying and Mobilizing a Community’s Assets, can assist you in shifting paradigms from one that is deficit driven to one that is asset based.
Leveraging Existing Data Resources
Practices, especially those in the safety net, already collect a considerable amount of performance and patient data for the Federal Government and third-party payers. In addition, practices may collect information for other quality improvement work going on at the practice. Therefore, practices may be resentful if you try to impose new data collection on them that is seen as duplicative.
Leverage the data the practice is already collecting whenever possible. Sources of assessment and monitoring data include:
- Disease registries: patient characteristics, quality of care metrics, possible use as population management tool.
- Electronic health records: patient characteristics, quality of care metrics, possible use as population management tool, utilization.
- Patient surveys: patient experience.
- Health Resources and Services Administration (HRSA) Uniform Data System (UDS): quality of care and clinical outcome metrics.
- Reports required by health plans: quality of care and clinical outcome metrics, utilization, other process indicators specific to plan.
- Existing QI reporting: various metrics.
- Data collected for prior research or quality improvement efforts: various metrics.
- Workflow maps.
- Staff surveys: various metrics.
Be sure to take an informal inventory of data sources before recommending any new data collection. This inventory should include the resources listed above and extend beyond them to data collection required by their different payers and projects they may be participating in with local researchers. For example, HRSA’s UDS is a requirement for grantees of HRSA Primary Care Programs. A variety of data elements are included, such as patient demographics, services provided, staffing, clinical indicators, utilization rates, costs, and revenues. Go to the Module 6 Appendix for a sample data inventory form.
Using Assessment Tools To To Stimulate Reflection and Discussion
The MacColl Center for Health Care Innovation designed an assessment tool, the ACIC, that you can use to stimulate productive discussions about the needs and improvement goals of the practice. Table 6.2 summarizes the elements of the ACIC.
Building Practice Capacity for Data Collection and Use in a Practice
While you will most likely collect and analyze data early in an improvement intervention, from the very beginning you will need to plan how you will build capacity in the practice to continue producing performance measures over the long term. Consider the following questions:
- What information systems do they have or need to support this effort?
- How can you help them develop the systems they lack and learn to mine data from those they have?
- Who in the practice will do this?
- What data will they collect? How often?
- What skills will they need and how can you help them develop these skills?
- What systems and software will they need to analyze and interpret the data for use in QI work? What will the workflow be for staff who will collect and analyze these data?
- How long will it take to complete this task each reporting period?
- Can this activity piggyback off of other required reporting activities, such as reports to health plans?
- Will leadership provide protected time to staff for these activities?
- What factors are likely to prevent staff from completing this key activity?
- How can this get written into their job descriptions?
- How will new staff filling this role in the future be trained?
- What systems will be put in place to hold them accountable for completing these tasks?
- What schedule will they follow for collecting the data and reporting them to the QI team?
- How will the data be displayed so it is meaningful and actionable to the QI team?
- Can they “automate” parts of this process to make it easier for staff to obtain data and produce periodic reports?
As you work with the data, you will need to begin to work on the answers to these and other relevant questions aimed at building internal capacity in the practice to do the things with data that you are doing now and to sustain this work over time.
Kretzmann J, McKnight, J. Building communities from the inside out: a path toward finding and mobilizing a community’s assets. Evanston, IL: Asset-Based Community Development Institute, Northwestern University; 1993.