Measure Criteria Expert Meeting
The following is a transcript of the Pediatric Health Care Quality Measures Program Measure Criteria Expert Meeting, held on February 24-25, 2010. The meeting was held at the Agency for Healthcare Research and
Quality (AHRQ) Conference Center, 540 Gaither Road, Rockville, MD.
Select for a list of participants.
February 24, 2010
February 25, 2010
February 24, 2010
Dr. Dougherty: Good morning. Welcome to AHRQ everybody. I'm Denise Dougherty, I'm the senior adviser for Child Health and Quality Improvement at the Agency for Healthcare Research and Quality. Francis, you're up here. My boss, Francis Chesley, who's the Director of the Office of Education, Research, and Priority Populations is coming to the table.
So, first things first. I put this together and put the finishing touches on it this morning before I got in my car and heard the weather.
If you haven't heard the weather forecast close your ears, you don't need to know this. But if you are already thinking about the weather, there is some prediction of some snow, maybe some wind, whatever, overnight. I don't have the exact details, but some people do, and they have already asked if they can leave tonight. So here's what we're going to be doing. We don't want you to spend your time on your laptops, looking for the weather forecasts or which airline is canceling. We will keep an eye on the weather and the airlines for you and provide updates.
If you really are certain that you need to leave tonight, that's fine, and we can give you a number so you can call in and join the conversation tomorrow, if that's your wish. Or you can stay and we will see how the weather goes tomorrow and we can end early so people can get out of here if they need to, or if we somehow get stuck, you can stay overnight tomorrow night.
I want to thank everybody for agreeing to participate in this important meeting and for doing the pre-work that you did that we sent you many, many E-mails and documents, even during the last snowstorm as Nora reminded me, and then for participating today and tomorrow. So we really appreciate your being here and thanks so far for the work you've done.
We have a very busy agenda and a lot of work to do today. You are intentionally a very diverse group, so here we have clinicians who may not be measurement experts. We've got lots of measurement experts, some are child measurement experts, some are what we call generic measurement experts. We have users of measures who may not be measurement experts—and who else do we have. If you're not in one of those categories, you must be bureaucrats. We've got lots of those, yes.
So, this is going to be a challenge. The nature of the task is a challenge and maybe having conversations with all these different languages going on may be a challenge, so we ask you to be patient with each other and with us. We did this intentionally, but we don't know how it's going to turn out, and we would like to know what you think. So we will be sending you an evaluation or feel free to E-mail or call me or whatever if you have any comments, or come up to us during the break and let us know what you think.
So I'm just going to give a little bit of the purpose of the meeting and the overview of the day just to get us oriented. You have your briefing books with the agendas in them. So why are we here? We're here for improving health care quality for children. and since CHIPRA has been referred to as a model for perhaps other measurement development and quality improvement road maps, this may be applicable to other populations, other settings, other situations.
So here's the model. We measure, we examine the measures—which doesn't always happen, you know—examine the data, then act to improve the situation if there are quality gaps or disparities in quality, then re-measure, continuously re-measure to see how we're doing with those quality improvements and eventually we succeed, and we have a healthy population. So, a healthy population with lots of high-quality health care. So we're in part of that road map, that model, and one CHIPRA goal is to have an improved core set of children's health care quality measures. So, we started with an initial core set of Medicaid and CHIP measures, and you're going to hear about how we did that and the challenges we faced from Rita Mangione-Smith a little bit later this morning. So that's over here in 2009. Now we have the 2010-2013 period, and there's lots of stuff going on with measurements and with quality improvement activities.
We're getting public comments on the initial core set, and we also will be learning from experience the way we always do. CMS has just announced its quality demonstration awards to 10 States, some of which are multi-State programs, and a number of those proposals said that they would work on improving the initial core set or testing the initial core set and building additional measures and identifying additional needs. So that will be going on, and what we do here today will be relevant to that quality measurement activity. Then we have in the legislation—it's very unusual that we have this big effort for quality measurement and improvement and money to actually do it, especially for kids. So we have the Pediatric Quality Measurement Program of grants and contracts, and AHRQ is taking the lead on that, working very collaboratively with CMS.
Then there will be other measurement developments. So when we get to the point in 2013 when the CHIPRA legislation calls for the improved core set of children's health care quality measures, all of that will be going into our learning to identify the improved core sets. So we'll probably have a public call for other measures that have been developed while these other CMS activities are going on and perhaps another subcommittee-like group to determine the consensus on the improved core set.
So that's basically why we're here today, because to have an improved core set that is usable by public payers, private payers, providers, and patients across the whole spectrum of children's health care, including perinatal care, we really need to have some consistency in what all these folks are doing. The CMS demo people, the Pediatric Quality Measurement Program people and whoever else wants to use the core set. We can't force anybody to use these criteria elsewhere, but we certainly can force it—sort of force it, I hate to use that word when I'm a government employee—but strongly encourage and provide technical assistance to people to use it in the CMS demos and in our grants and contracts program.
Our target audience for what we are doing here today is really these awardees. The states who have received the CMS quality demonstration awards and the forthcoming awardees of the Pediatric Quality Measurement Program, but the announcement is not out on the streets yet, so don't ask me for it now.
So our questions are, when we started out here and said wow, how are we going to get to a consistent set of core measures that are applicable across all these programs the way CHIPRA wants was to ask if existing measurement criteria are sufficient and specific enough to guide future developers and enhancers of measures. And that means that the criteria could be used prospectively by these awardees, and they would know for example that they would have something to go the National Committee for Quality Assurance (NCQA) with or go to the National Quality Forum (NQF) with. So are the measurement criteria specific enough to give them guidance so they don't spend 3 years working on measures and then find out that they're not going to pass any of the current criteria? And if they're not specific enough, what additional recommendations would experts make? And you all are the experts who are going to give us additional recommendations today. And then assuming we will not get down to the nitty-gritty of specific criteria and confidence intervals and stuff for every criteria domain and sub-domain today, we want you to tell us what next steps we should be taking.
So here's a summary of what we'd like from you. Needed changes to the current measure criteria, particularly in the context of the CHIPRA requirements, and the more specific and targeted to our audiences and awardees the better, and then thoughts and recommendations on next steps that we should take.
How are we going to do that in these 2 days? Our agenda today is that after we go around and say our names and affiliations very quickly, we're going to have some framing by AHRQ and CMS leaders, and that will be followed by some stage-setting about criteria. We're going to have Rita Mangione-Smith talking about the National Advisory Council on Healthcare Research and Quality Subcommittee on Quality Measures for Children's Healthcare in Medicaid and CHIP (SNAC) CHIPRA process, the identification of initial core measures. Ernest Moy is going to talk about the National Healthcare Quality and Disparities Reports' criteria. Helen Burstin will join us by phone, we have her slides here, to talk about—to do some updates on what NQF is doing, and then I will give you the charge for the day for the breakouts and for tomorrow too.
We'll have two breakout sessions this afternoon, one on validity—both on validity. Tomorrow, if all goes well—these have not been updated due to weather conditions—we'll have three more breakouts and an overall synthesis of where we are and then additional thoughts about next steps. Just an FYI, I heard Gil Scott-Heron has a new album out, and he's the one who said the revolution will not be televised, so it stuck in my head. So this evolution of criteria, however, will be recorded. It will be posted on the AHRQ CHIPRA Web site as a slightly edited transcript of the meeting and then possibly as an audio for more accessibility. That would be true of the plenary sessions only, not the breakout sessions. We don't have enough equipment for that, and you need some time when you're not being listened to, I assume. So if you do want your name associated with comments you should speak into the microphone and say who you are before you make a comment. So we'll be calling on you for Q&A in a little while.
Now I'd like to actually see if anybody has any—no? Okay, we're going to then I think go to Carolyn. Carolyn Clancy has some welcoming and framing remarks for us. She's coming to us technologically via DVD. There she is.
Dr. Clancy: Good morning and welcome to the Agency for Healthcare Research and Quality. You can probably see just a few remnants around the building of "Snowmageddon" as President Obama called the blizzards that hit us a few weeks back. It was a very interesting time in Washington. I can't recall another time when weather caused the Federal Government here to shut down for 4 consecutive days. We also had the President's Day Federal holiday and 2 days for which Federal employees could use leave to avoid getting caught in the storm.
So basically our schedule was affected from February 5 through February 15. When we came in last Tuesday for the first time since the blizzards, the snow was plowed so high around the parking lot that you couldn't see the cars when you came through the gates. Fortunately, most of that is behind us now. I personally cannot wait for warm weather and sunshine, and we can get back to a more normal schedule. Unfortunately my schedule did not permit me to be with you today, so I very much appreciate the opportunity to share some of my thoughts on the work that you're about to begin.
But first, I want to thank each and every one of you in advance for coming together to offer your expertise for this extremely important project. Obviously, the key to our success in this endeavor is having standardized measures for use in the improved core measure set, the Children's Health Insurance Program Reauthorization Act of 2009, or CHIPRA, calls for. Your knowledge will be critical to ensuring that we're able to accomplish this goal because as we all know, health care quality is still an emerging science. Unlike other fields of science it has no well-tested theories and few criteria that can be used prospectively by new entrants in the field.
Until now, we haven't had substantial funding to support new entrants into health care quality measurement development and testing. Now that Federal funding has been made available, your work will give us a head start. As I've said ever since its passage a little over a year ago, CHIPRA gives us a road map and a model for linking health care quality measurement to quality improvements, and as so often happens, our child health colleagues are leading the way. Here we have an unprecedented opportunity to develop and enhance children's health care quality measures and improve on the initial core set that has already been published for public comments. We also have a deadline, January 1 of 2013.
We're holding this meeting so that those who will be working to develop and enhance children's health care quality measures under CHIPRA programs will have a common set of criteria from which to work. Only with a common set of criteria can we hope to have a standardized set of core measures.
Standardization can be a scary word, but many State programs and health care providers have come to see the wisdom of consistency across measures. As we work to build this consistency, here is our challenge for today. The health care quality measures and the child health communities have made enormous strides in identifying and often agreeing on key criteria domains of validity and feasibility, but we've let thousands of flowers bloom in very different kinds of soil. We remain consensus-based and not as transparent about the application of criteria as we could be, and this means we can't give measurement developers and enhancers a clear set of rules that will let them know whether their new or enhanced measures will pass the test we've created. This is the challenge we'll begin to overcome today and tomorrow with some very hard work.
Of course, 2 days is probably not enough time to develop quantitative and universal criteria for every aspect of quality measurement, but we hope it's time enough for you to get us off to a good start. We're also very interested in getting your ideas on how we should move forward to achieve our goals. So here's to a very productive meeting. I'm looking forward to hearing about it and seeing the results. Thank you.
Dr. Dougherty: Obviously that was recorded before the latest weather forecast. Okay, now we're going to hear from Victoria Wachino who is going to give us some framing comments and some guidance from CMS, our partner in this effort.
Ms. Wachino: Good morning. It's really exciting for me to be here with you all this morning. I'm Vicki Wachino. I'm the director of Family and Children's Health Programs at CMS, and my job and my group's job—many of you probably already know Barbara Dailey—is basically to manage all of the policy and operations and procedures and management for the Medicaid and CHIP programs, in partnership of course with the States. And that includes quality, and I wanted to talk with you a little bit this morning about quality and why the work you're doing is so important to all of our efforts at CMS.
I think the starting point for all of CMS' work in general and with respect to our quality efforts in particular is really fulfilling the commitment and the promise that the Medicaid and CHIP programs offer to low-income kids and families, and fulfilling the promise of giving them health care that meets their needs and improves their health status. I think that's a big priority, Federal priority in general, I think it's a big priority for this Administration, and I think it's clearly a big priority for the states.
CHIPRA and also the Health Information Technology for Economic and Clinical Health (HITECH) Act create fantastic new opportunities to bring to bear on your work and your thinking to move quality forward. And I think as Carolyn said, it's really quite unprecedented. And since we recently passed the 1-year mile marker of the enactment of those laws I wanted to reflect a little bit on CMS' accomplishments, particularly as they pertain to CHIPRA which I think is what brought you all here today.
If you look at CHIPRA, it did a lot of different things, but its efforts really concentrate around two areas. One of them is expanding coverage to as many eligible low-income kids as possible, and it does that through a combination of creating new options for States and new incentives for States to enroll eligible low-income kids. And so over the course of the past year we have worked closely at the Federal level and in partnership with States to really bring new focus and resources to bear on getting kids in the door. And Secretary Sebelius is very aware and committed to the fact that right now there are 5 million kids out there who are eligible for our programs but not yet enrolled, and CHIPRA brings a lot to bear to that effort of reaching those 5 million kids. And over the past year we've seen a number of States taking up options around eligibility expansions and making it easier for families to enroll by streamlining their programs, by rethinking the way they do things and get families in the door, and that's been extremely productive.
The second and related area that CHIPRA brings a ton of focus to is obviously quality and the reason you're all here today. And the reason I think about those two things in concert is I think it's both important to get all of the kids who need our programs and are eligible for them in the door, and once they're in the door, it's also critically important that our programs work as well for them as possible in terms of providing them with the quality of care that is designed to meet their health care needs. That's why you're here today, and that's why I'm excited to be here with you.
In my career, I haven't spent as much time on quality as I have over the past 3 months at CMS, and I was just saying that even though I'm reflecting on the past year of CMS' efforts, I can take credit for none of those things that I just described because I've only been there for 3 months. But it's been really striking to me as I've gotten to know and understand the CHIPRA quality provisions and as we've been implementing them in partnership with AHRQ. What an inspired piece of legislation it really is, and I say inspired for three reasons. First, because it really brings a level of commitment and focus to efforts around quality measurement and management for kids that I don't think has existed before. Second, it brings a level of Federal resources that I'm quite sure is unprecedented when it comes to quality of care for kids and families. It's $225 million over 5 years. And third, is the very strategic nature of the process that it laid out. It's really quite a thoughtful piece of legislation, starting with the development of the initial core measurement set, which as Denise and Carolyn said is out for public comment now. Then moving to the stage of the quality grants and making sure that CMS and AHRQ are working with States and moving along at the State level as fast as we can and as thoughtfully as we can around the adoption of quality measurement, trying to move towards consistency as Carolyn said, but recognizing that consistency across a 50-State program can be extremely challenging.
I'm inspired because in the next step of that process, even as we are finalizing and refining and polishing the initial core set, is what you all are doing today and starting to think about pediatric quality measures for the whole population. Obviously, that's not the end of the process, and I think one of the striking things about CHIPRA is the way it takes where we were a year ago and moves us so much closer by forcing us all to do this work to a model where we're really able to measure outcomes for kids.
And I will tell you that one of the very striking things about CHIPRA to me and one of my personally favorite parts of the legislation is that we start with Medicaid and CHIP, and that's a good place to start because we cover—at any point in the year a quarter of the kids in the United States come through our programs. So just by developing the initial core set for Medicaid and CHIP, we've made a starting point in improving quality of care for kids.
It's exciting to me that we start there and then build to the rest of the pediatric population because too often in the history of changes to the health care system or improvements to the health care system, we start with private insurance first, and we get as far as we can with that, and then we look back and we say, well what about Medicaid? You know, what about CHIP? What about the public programs? What are we going to do for them? And so it feels right to me both because of the size of the population, the kid population and in the sequencing that we deal with Medicaid first. So I think it's really exciting.
You have a really hard job today, and I will say I know there's snow coming, but I hope you all brought your snow boots and are intrepid and stick it out. I find defining what constitutes quality care and establishing measures and management to be extremely difficult. And I say that, and I don't have the background that a lot of you do, but just—I find thinking about quality in a meaningful way to be extremely challenging, so I don't envy you. As hard as that job is, I think I'm going to make it a little bit harder by telling you about some of the things that we at CMS would like you to think about as you think about quality. And there are just a few so I won't totally over-burden or overwhelm you.
The first is access to care, and making sure that access and quality are thought of together is extremely important. I think if we look at the experience of kids in Medicaid and CHIP, making sure they have access to the services they need is incredibly important.
The second thing is balancing consistency with State variation as I kind of said a little bit earlier. It is critical, and I think this is why everyone's doing this, to have consistent national measures. In Medicaid—it's challenging in any health program, but I think it can be especially challenging in Medicaid and in CHIP because it is a 50-State program, and states are at all different places. Some of them are leading the way, and some of them haven't had as much time to think about it. So really thinking about how we can accommodate State variation and work with States is very important.
The other thing I'm sure that none of you will lose sight of is the need to think about health disparities, disparities in care for different subpopulations, particularly racial and ethnic subpopulations. But I think underlying it all, and the single thing that's most important, is really thinking about quality and quality measures as tangibly as possible so that we at CMS and other health insurers and the States can really use quality measures as a management tool. I think that can be extremely challenging to move from the abstract to the particular, and to really establish—to find the right measures is hard, to define quality is hard, and then to think about the data and what we will do with the data over time as managers of this program and as purchasers—Medicaid purchases one-fifth of the health insurance and health care services in the United States. So we at CMS really think of the work you're doing as management tools, and as hard as it can be to be tangible, I would just urge you to take it to that level in your conversations to the degree that you are able.
I've said a lot about CHIPRA, but I also wanted to talk just a little bit about HITECH, which is also such an enormous opportunity. And as you're sitting here today I feel like we're standing at a turning point where we now have this level of investment to move from systems that have been designed primarily for payment, to pay providers, to information systems that are designed primarily to measure and gather information on the quality of care, and that is both unprecedented and extremely welcome. But I think it also presents some challenges.
One of the challenges for you all may be that the environment is changing so quickly, and who knows what these systems are going to look like. I think we have some idea, but it is a very big change for us. And the second challenge and one that I bring to all of my thinking about HITECH, although I say that and need to note that there are many people at CMS who do more thinking about HITECH than I do, but for me it's really making sure that the technology isn't driving the quality measurements. The goal of HITECH, health information technology (health IT), and electronic health records (EHRs) is to measure quality, and the quality measures should always come first, and we should always be thinking about how all of these new technological resources that are being brought to bear support the goals of quality measurement. So don't put the cart before the horse, don't put the technology before the quality measurement. I would urge that we do the opposite.
The second challenge, and this is a challenge I think for us at CMS and perhaps for AHRQ as well—and I will say that from what I know of our collaboration with AHRQ so far, it's been really fabulously successful and something we value a great deal—is really ensuring coordination at the ground level between CHIPRA and HITECH. I think we have staff that are trying to do that. I think one of the ironies is that after years of maybe not spending as much as we could on quality and certainly not spending as much as we could on health IT, suddenly we had these two huge pieces of legislation enacted almost at the same time, but to my thinking and reading, not a lot of thought about how they would relate to each other. So I think one of the things that we're doing at CMS is trying to bring them together and to make sure that those efforts are complementing and not duplicating, or worse, getting in the way of each other.
The last thing I wanted to say since I'm new to CMS is that we really, really want to know how it's going for you. We want to know how this effort is going, we want to know what more we can be doing to improve the quality of care for kids in this context, in measurement, and just in general. I know a lot of you have interactions with the Medicaid program outside of this context, so we're looking for feedback. And I kind of make that offer at my own peril because I don't want to over-promise and I'm not over-promising, but I can tell you that we won't be able to solve any problem that we don't know about. I say this to almost every audience I talk to—we really want feedback on how it's going for everyone that our program touches, including beneficiaries and providers. A number of you are extremely thoughtful, and senior researchers who thought about our programs for a long time, and we want to hear from you. So thanks so much for having me here, and thank you for all of the work you're going to do over the next 2 days. It's really important to us.
Dr. Dougherty: Thanks very much. I realize we skipped the part where we all go around and say who we are, but you'll have lots of time to do that during this meeting. We only have a few minutes for Q&A. Obviously Carolyn is not going to be able to answer any questions, but we have other people here from AHRQ, and Vicki, you'll be here for a few more minutes?
Ms. Wachino: I'll be here for the next few minutes.
Dr. Dougherty: Okay, great. So let's—anybody have any questions, comments for AHRQ? Yes, Judith.
Dr. Thierry: I just have a question probably for CMS, just a comment, is portability of insurance, interstate issues?
Ms. Wachino: Can you say a little bit more about that?
Dr. Thierry: Well, for American Indians/Alaska Natives, they go to off-reservation boarding schools. They may live in one State, have family in another State, need services, and their services aren't portable.
Ms. Wachino: I think that's an important point, and I don't know—I mean, Barb might know better than I do. We have Native American quality grants, specific Native American quality grants that we are reviewing the applications right now, and I don't know if there are opportunities there to address that issue, but we'll certainly look out for it.
Ms. Dailey: And also, I think the opportunities before us under HITECH, as well as the CHIPRA provision on EHRs, we're developing a program in a specific format for children. That is one of the areas where we're actually seriously taking a look at how those records will help support portability and sharing of information. So in that aspect, we are definitely focused on that.
Dr. Dougherty: Anybody else? Yes. Can you say your name?
Ms. Reuland: I'm Colleen Reuland from the Child and Adolescent Health Measurement Initiative (CAHMI), and I just want to make sure I'm crystal clear on the goals of the measures because as we think about the models your "uber" goal is probably the most important thing to drive how that model plays out. And so I'm hearing you say that the goal of the measures is to improve care and to drive improvements. And so I just want to make sure that that's the centering goal, because when I look at some of the legislation, it seems like there's also a goal to have standardization to be able to compare care across States and to be able to have benchmarks about—to be able to have like a statement about what's the quality of care that children receive. That could lead you in a different direction than trying to have measures that drive improvement, because some measures may allow you to be able to compare States but not be able to drive improvement.
So could you help clarify that dissonance in my head? Because it would affect how—it affects the validity, it affects the usability, it affects the feasibility conversation if the goal is slightly different or if the goal is maybe all of those.
Dr. Dougherty: Well, I would leave that to my CMS colleagues, but I kind of disagree with you that measures for accountability cannot be used to drive improvement because there's nothing like seeing that some State is better than you at something and then trying to figure out why that is and improve your care. But as to the general issue of comparability, Barbara or Vicki?
Ms. Dailey: Actually, part of the CHIPRA program also does require annual quality reporting, and one of the roles for CMS is to develop procedures and a format for States to submit information so that the Secretary of Health and Human Services (HHS) publishes an annual report that is transparent and provides information in terms of what is going on with the States in terms of improving care, and I think ultimately, too, to demonstrate that we're improving health outcomes. So that is also a component of that. So in terms of driving comparability, I think the first part is the focus on improving care and then, through this transparent annual reporting, identifying ways if we need to as a next step if we haven't gotten to the point we can do comparability, if the States are still saying we're struggling, we haven't been able to meet the specifications.
We're still exploring in terms of what the options are going to be for the States. That's one of the things we're very anxious about with these public comments on the initial core measures is what can they do and what can they not do with what's existing now. And then that information is going to feed into the quality measures program. So ultimately at the next set of initial core measures we're hoping that there will be some sort of degree of comparability, but that's going to be part of our learning process in the next 2 years as part of this. I hope that answers your question.
Return to Contents
Proceed to Next Section