Module 8. Collecting Performance Data With Chart Audits

One of the most important functions of a facilitator is to help practices obtain, present, and interpret data in a meaningful and compelling way and translate the findings into action. Data collection, however, is a laborious task. Facilitators frequently spend most of their time with a practice creating systems to access reliable data and building capacity in the practice to use these data in their improvement work.

With data being key to quality improvement (QI), it is important that you feel comfortable collecting, analyzing, and reporting data. Once data have been collected, they will need to be cleaned, analyzed, and presented to both the practice team involved with the project and to practice staff, providers, and leadership. The use of data and feedback systems allows practices to see improvements during an intervention, make adjustments, and stay engaged.

Considerations When Collecting Clinical Performance Data

Depending on the practice, you may gather data through hand abstraction or by downloading data from an electronic health record (EHR) or registry. In general, an audit of 10 to 30 patient records seen during the target time periods is sufficient to generate usable performance data for a practice. You will need to collect data multiple times so the practice can track its progress.

For the initial performance audit, it is most effective to conduct an audit of the previous 12 months and organize these data by quarter to show fluctuations in performance over the time period. Fluctuations can be a valuable source of information about factors that may be affecting clinical performance. During active improvement work, monthly performance audits of patients seen during that time period can help a practice monitor its progress toward improvement goals and make adjustments to processes and procedures when progress has not occurred.

When a practice is engaged in a PDSA (Plan-Do-Study-Act) cycle, daily performance audits may be needed to assess how effective the modification is in improving the targeted performance metric, and for deciding if a modification is ready for wider spread in the practice or organization. For a practice that has achieved an improvement goal, quarterly audits can be used to help them ensure that the improved performance is maintained. They also can alert the practice to the need for adjustments when performance unexpectedly declines.

 To conduct the actual audit, you will need to create a paper or electronic extraction form, or a performance data request for staff in charge of the practice’s EHR. Figure 8.1 contains an example of an abstraction spreadsheet. If you use paper records, you can enter the data directly into the spreadsheet. If you request the data from the practice information technology (IT) staff or the individual in charge of access to the EHR, ask for the data to be output to the spreadsheet. Spreadsheet data can be exported into statistical software for further analysis.

Procedures for Electronic Health Record Audits

While in the past most audits were conducted using paper abstraction forms and paper medical charts, with the increased use of EHRs, many audits are now conducted by accessing electronic health data. If your practice uses EHRs, you may be able to get the system to generate a report with the data you need. This function, however, might require new programming; depending on the way you want the data arrayed, it could be beyond the functionality of the EHR system. It is worth a significant investment of your time to learn as much as you can about how to coax data from the system. Developing a relationship with those who are in charge of the IT system and can reconfigure reports to meet your needs will also have a high payoff.

With electronic patient data, you and the practice staff can create standing reports on key performance metrics that can be run repeatedly over time. These reports make it easy for the practice to continue performance reporting after the active facilitation intervention is finished. Equally important is to train staff to develop their own reports and modify existing reports so they can easily add new performance metrics or change the parameters of old ones.

In addition to providing a list of the performance variables you want included in the data pull, inclusion criteria for the patient records that will be queried, and time period for the data, you will also need to specify the format for receiving the data. The advantage of performance audits using data from EHRs is that you can often pull data on the entire population of patients seen during the specified time period, rather than limiting the audit to a subset of 10 to 30 patient records. Provide the IT staff, or whoever will pull the data, precise written descriptions of the criteria for inclusion and exclusion. A sample of instructions for IT for a performance audit data pull are provided in Module 8 Appendix C.

Procedures for Paper Chart Audits

Unlike with electronic data, where you should be able to collect data on the universe of patients in your target population, you will have to sample patients when doing audits using paper records. For performance audits, a random sampling of 30 to 60 charts or patient records for the initial performance audit can be sufficient to provide information on the practice’s performance. Smaller samples are too vulnerable to random variability.

Another approach can be to sample 10 percent of eligible charts or to take a convenience sample from a single day of patients who meet inclusion criteria. For monthly performance monitoring, an audit of the records of 10 patients seen during that month can be sufficient for a practice to evaluate progress toward an improvement goal.

You will need to obtain a list of patient records that you want to review. These lists can be generated using billing data with diagnostic codes and information on other inclusion/exclusion criteria. The patient record numbers then need to be given to medical records staff, who can pull the charts and provide them to you for audit.

You will need to work closely with the QI team and practice manager to ensure that you do not create an undue burden on medical records staff and that you do not pull and retain charts of patients being seen that day whose medical records will be needed.

Privacy and Data Security

All data collected from a practice are highly sensitive. Whether the data are from patient records or staff surveys, the practice facilitator must keep data secure at all times. A number of measures can be taken to protect confidential information. As a rule, never take identified patient data offsite from a practice.

Electronic data are particularly difficult to secure, especially in the era of cloud computing. Any data transmitted to or stored on your computer, tablet, or laptop should be deidentified with all personal health information (PHI) removed. A list of what is considered protected PHI can be found in the Health Insurance Portability and Accountability Act descriptions.

NEVER take identifiable patient data (data with patient names or other identifiers) from a practice or store data with PHI on your computer. Lost or stolen laptops are a common cause for data breaches.

A key code connecting patient PHI, including medical record number, to data you maintain on your computer or any that you are transporting offsite will need to be created to allow you to reidentify data if needed. This key code should be housed at the practice and never taken offsite. In addition, you will need to set the security on your laptop to require a password to access any practice information stored on it. Any data transmitted through email or stored on cloud applications should similarly be deidentified, with the master code maintained only at the practice.

You will need to be familiar with and make sure you comply with all regulations of the Health Insurance Portability and Accountability Act as it relates to performance data and access to patient data. In addition to protecting sensitive patient information used in assessing clinical performance, you also need to be concerned about privacy and confidentiality of a practice’s performance data.

Assessing clinical performance can be a threatening and sensitive process for a practice. While sharing performance data and best practices across practices is a critical part of the facilitation process, and of quality improvement in general, you will need to confirm that you have a practice’s permission to share information about their performance and improvement work before you do this. You will also need to clarify the conditions under which this is acceptable to the practice. Typically, these discussions will occur with practice leadership and your program director, and will be clarified at the start of an improvement intervention, but you will need to remain sensitive to these issues as you work across your practices and with other facilitators.

Page last reviewed May 2013
Internet Citation: Module 8. Collecting Performance Data With Chart Audits. May 2013. Agency for Healthcare Research and Quality, Rockville, MD. http://www.ahrq.gov/professionals/prevention-chronic-care/improve/system/pfhandbook/mod8.html