Transcript: Analyzing Your IQI and PSI Rates
Speaker: Amelia Haviland, Associate Professor of Statistics and Health Policy, H. John Heinz III College of Public Policy and Management, Carnegie Mellon University
Interviewer: Donna Farley
Date: April 2012
Audio format: Analyzing Your IQI and PSI Rates (MP3 audio file; 16 min., 40 sec.)
Interviewer: On behalf of the Agency for Healthcare Research and Quality, I'd like to welcome you to a series of interviews designed to orient and educate users of the AHRQ Quality Indicators Toolkit for Hospitals. This toolkit was designed to support hospitals seeking to improve their performance on the AHRQ Inpatient Quality Indicators, or IQIs, and the Patient Safety Indicators, or the PSIs.
The topic for this interview is tools that offer information and guidance for hospital teams to calculate QI rates for use in their hospital's quality improvement processes. As we start this discussion, we note that AHRQ has established and maintains the IQIs and the PSIs, as well as several other sets of quality indicators, including specifications for the indicators, how they were developed, and validation work that they've performed on them. AHRQ also has developed software packages that hospitals can use free of charge to calculate their rates for the IQIs and the PSIs. The Web site for the AHRQ QIs, which is located at http://www.qualityindicators.ahrq.gov, provides all of this comprehensive information for organizations that use the QIs in their quality programs.
The tools we are discussing here are designed to provide hospitals additional guidance for how best to work with the QI rates. And they serve as a complement to the resources already available on the QI Web site. In particular, we'll be addressing tool B1, which provides information about the different rates calculated by the AHRQ software and how to use them, and B5, which discusses methods and issues for assessing indicator rates using trends and benchmarks. You can find these tools at www.ahrq.gov/qual/qitoolkit/. On this Web page, click on the roadmap link to open it and select the tools labeled B1, Applying the QIs to Hospital Data, and B5, Assessing Indicator Rates Using Trends and Benchmarks.
Today I'm interviewing Amelia Haviland, who will talk about several considerations involved in analyzing and working with your IQI and PSI rates. Dr. Haviland is associate professor of statistics and health policy at the H. John Heinz III College of Public Policy and Management at Carnegie Mellon University. She was formerly a statistician at RAND and she helped to develop many of the tools related to working with the PSIs and the IQIs. Amelia, we're very pleased to have you with us today.
Amelia: Thank you, Donna. I'm glad to be here.
Interviewer: Let's start with the first obvious basic question, and that is, when working with the rates, how should hospitals be using the rates that are calculated by the AHRQ software to assess their current performance on the quality indicators?
Amelia: That's a great question, Donna. The AHRQ software can calculate several different rates for each quality indicator. Two of these rates are particularly useful for assessing current performance. Before I describe those, the quality indicators are all expressed as rates. It's important to understand what information is being used in the numerator and in the denominator in order to calculate these rates. This is the same for every quality indicator. The numerator is the count of the number of events recorded, and the denominator is the number of people eligible for that event. So each quality indicator has a specific eligible population that is represented in the denominator: for example, all surgery patients. This group is the reference population when looking at the rates for each QI.
This leads us to the first rate calculated by the AHRQ software that we'll want to look at to assess current performance. It's called the observed rate. To take an example, IQI number 14 is hip replacement mortality rate. In the observed rate, the denominator for this IQI is the number of discharges in near current time period for adult patients with a code indicating a partial or full hip replacement, plus some more technical details for inclusions and exclusions. The numerator for the observed rate for this IQI is the number of deaths that occur for patients during those hip replacement hospital stays that were identified in the denominator.
So this rate, the observed rate alone, tends to have limited usefulness in assessing current performance or providing guidance for setting priorities for quality improvement. It only tells you how often these events were flagged as having happened in your hospital relative to how many eligible patients you have. To be most useful, a hospital team also needs to know how their hospital is doing relative to some standard, and this standard is provided by a rate also calculated by the AHRQ software for your hospital and for each QI that's called the expected rate. For each quality indicator of interest to you, the idea then will be to compare your hospital's current observed rate to your hospital's current expected rate.
For each QI, this expected rate is calculated by applying national event rates calculated for each type of eligible patient onto your hospital's specific current mix of eligible patients. So a given estimate of what your rate would be, if you provided your eligible mix of patients with the same care as the national average hospital. In the expected rate then, performance is standardized and applied to your particular patient mix. This rate then serves as the standard that your observed rates can be compared to in order to assess current performance. Specifically, you can make a ratio of your current observed rate to your current expected rate on a particular quality indicator, and it will tell you whether your rate is higher or lower than expected given your patient mix.
In our IQI example of hip replacement mortality rates, a ratio of 1.1 would mean that your mortality rate for your patients was 10% larger than expected, which is a negative finding. If instead you had a ratio of 0.9, this would mean that your mortality rates for these patients was 10% lower than expected, which is a positive finding. So calculating the ratio of observed to expected rates for each quality indicator allows the hospital team to identify areas of strength and priorities for quality improvement efforts.
Interviewer: Thanks. That's extremely helpful, a good comparison of the two rates. A lot of hospitals also want to take a look at how they are doing over time and compare themselves to other hospitals. Which rates are best for those hospitals to use if they want to do those trend and benchmarking analyses?
Amelia: Both of those activities are really key additional activities that the QI rates can be extremely useful in. And for these purposes, hospital teams will want to use a third type of rate that the software automatically calculates for each QI, and this rate is called the case-mix-adjusted rate. People might ask why do we need a different rate for this purpose than the two we just discussed? The reason is that we want to be sure that we're comparing apples to apples.
When examining observed QI rates for a hospital that may appear to improve over time, two things could be behind this improvement. One is that hospital procedures really improved, but the other possibility is that the mix of eligible patients changed while hospital procedures remained the same. And the same issue holds for comparisons across hospitals, in which different mixes of eligible patients could lead to incorrect performance comparisons. So by applying case-mix adjustment to your rates, which is carried out by the AHRQ software, you create QI rates that are based on your hospital's performance applied to a standard population instead of your specific patient mix.
When this procedure is applied to your own hospital's discharges from a different time period, or to a different hospital's discharges, all the rates being compared are for performance on the same standardized patient mix. This tells you that any differences, whether over time or between hospitals, are not due to different mixes of patients but are due to other factors such as hospital performance.
While we're on the topic of comparing QI rates over time or across hospitals, another measurement issue for these rates is worth mentioning. When the denominators for any of these rates is small—in other words, the discharges eligible for a particular QI are infrequent—small changes in event counts can create large fluctuations in the rates. This is more likely to happen for small hospitals or for quality indicators that have relatively small denominators regardless of hospital size.
These seemingly large fluctuations that are associated with small denominators may not be a meaningful measure of underlying changes in quality. Instead, with small numbers of eligible patients, large changes in the rates could be due to other factors besides your improvement process, including random fluctuations. Under these circumstances, an apparent reduction or increase in a QI rate would be unlikely to persist in later time periods.
So the AHRQ software provides a final rate for each QI that's called a smooth rate, which is designed to address this measurement issue by providing information to help distinguish between random fluctuations and systematic trends. This smooth rate uses the statistical procedure called shrinkage that pulls outlying rates closer into the national average rate to an extent that's relative to the size of the denominator. The idea here is that with few patients eligible for a QI, there's much less information about hospital performance on that QI than if there were many patients who were eligible.
A good way to determine whether your case-mix-adjusted rate may be unstable due to infrequent eligible cases for a particular QI is to compare it with your smoothed rate on that QI. If they are similar, then the case-mix-adjusted rate is stable, whereas if they differ, then the case-mix-adjusted rate is likely to be unstable and not persist over time.
If this is the case, then a hospital team may seek alternatives. To address these unstable rates, there are a few options. One is to pool several years of data in order to increase the number of eligible cases and obtain a less up-to-the-minute but more stable performance measure. Another option is to use the smoothed rate, which will provide a conservative estimate. And a final option is to refrain from drawing conclusions about performance on a particular QI where this measurement issue arises.
Interviewer: Amelia, what alternative ways are there to measure performance on the IQIs and PSIs other than using rates?
Amelia: I'm glad you asked this, Donna. For some audiences and for some QIs, rates may not be the most useful measure. An alternative is to track counts of events, that is, just the numerators for the QIs. In particular, some hospitals have found that clinicians tend to relate better to counts of events than the rates. And for low frequency severe events, such as mortality, they could be addressed as sentinel events that merit attention even if only one of them occurs.
Interviewer: How can hospitals be sure that they have useful information for trends in their QI rates?
Amelia: To obtain accurate trend information, a hospital team needs to pay attention to two things. One is to measure their hospital's QI rates the same way in every year. And the second is to case mix adjust the rates to a standard patient population to remove the effects of changes in a hospital's patient population over time. Those two steps will go a long way toward measuring the true underlying hospital quality status by eliminating other sources of changes over time.
This issue is especially pertinent for monitoring progress and tracking sustainability of quality improvement efforts. Now, a little more detail on both of those two things. In terms of applying case-mix adjustment, all that is required is to use the case-mix-adjusted QI rates that are provided by the AHRQ software. But in terms of measuring QI rates the same way each year, there's a relevant issue, which is that AHRQ has periodically changed the methods for calculating the indicators by updating the software over time. These changes in methods correspond to changes in the QI definitions and changes in diagnosis and procedure codes used in hospital discharge records. So for accurate trending, the same AHRQ software version should be used to calculate rates for all the years in the time period being trended. This will ensure the consistency in measurement necessary for making comparisons across those time periods.
So the question, then, is which version of the software to apply to all years in the trending time period. When making this choice, a good general rule is use the most recent version of the software and apply it backward to your earlier years. However, in some circumstances, the hospital team may decide to use an earlier version of the software instead. This would typically be preferred when the current version has eliminated a quality indicator that is important for your purposes or implemented a redefined indicator that has been less relevant to your needs.
Interviewer: How should hospitals be applying the QIs to hospital data tools that are in this toolkit to work with their IQI and PSI rates?
Amelia: Hospitals tend to find that they can use the rates for the quality indicators throughout the improvement process. To start with, the rates are used when diagnosing where performance problems exist and for setting priorities for action, then for tracking changes in performance during their implementation process and then finally, during ongoing monitoring after implementation is done to assure sustainability.
Interviewer: Thank you so much, Amelia. You've really provided some valuable insights about how hospitals can best use the different rates for the quality indicators that are generated by the AHRQ software. This includes the importance of consistency in measurement when making comparisons and in examining trends and performance over time.
As I mentioned earlier, you can download the tools for applying the QIs to your hospital data from the AHRQ Quality Indicators Toolkit by going to www.ahrq.gov/qual/qitoolkit/. Click on the link for the roadmap, and select the tool labeled B1, Applying the QIs to Hospital Data, and the tool labeled B5, Assessing Indicator Rates Using Trends and Benchmarks. Both of these tools together provide a great deal more information along the line of what Amelia has shared with us today.
In fact, the roadmap is a great way to orient yourself to all of the tools in the toolkit. I suggest that you also check out a video recording that you can use to introduce the toolkit to quality improvement teams and other staff. You can find the video and other materials from a Webinar about the toolkit by clicking a link at the bottom of the toolkit page or by going directly to www.ahrq.gov/qual/qitoolkit/webinar0215/index.html.
Again, this is one in a series of audio interviews about the use of specific tools in the quality improvement process, so please check the toolkit pages for additional interviews and watch for announcements from AHRQ. Thank you very much.