February 25, 2010 (continued)
Dr. Dougherty: Okay, I think we can get you all out of here by 2 pm. So who is in the room? Jose Gonzalez, do you want to report out?
Dr. Gonzalez: Oh, no, I have a great reporter.
Dr. Dougherty: A great reporter, okay. Last but not least, you know there are some States that have less than 50 percent of their population, sometimes pretty much less, are truly non-Hispanic whites. So I think there are seven States with that situation. So we're an increasingly diverse population, and some are more advantaged than others. But we do want to keep track of racial and ethnic disparities says Congress, not just me.
So with that in mind, I think Jose—are you ready? Oh, Kerri, sorry.
Ms. Fei: So we had medical home. I just have some general comments about this. Overall there needs to be a better defined, standardized process for data collection. And it shouldn't be limited to race and ethnicity. It needs to include socioeconomic status and the children with special health care needs. Then we said, for example, using the Office of Management and Budget (OMB) for race and ethnicity, poverty levels for socioeconomic status. And then additionally, specifically to the medical home, it may actually be a little bit easier to stratify each measure of disparities within a medical home.
Dr. Dougherty: Why? Sorry.
Ms. Fei: Well, if you think about it, depending on the definition or the concept of medical home, it is going to be a more discrete, more—I don't want to say confined—but defined population of folks that are assigned to the medical home. So it might not be as fluid as a regular general practice or something along those lines.
Okay. So under the next one, which is stratification of vulnerable populations, we had this as a criterion to keep. But we left it as desired but not required. And then we had discussion around, again, like we said before, that the criteria would need to be better defined, what is meant by vulnerable, how is that vulnerable defined?
Are we talking about—is it different from what we've already defined? So the race, ethnicity, and socioeconomic status in children with special health care needs, and then we had a little bit of discussion about considering language proficiency, which would then lead into health literacy where there are some issues as well.
Stratification by racial groups, again, a standardized approach with preference to the OMB categories. And we kept that as a criterion—had that as a criterion to keep.
For the next three, which were validity of underlying scientific soundness, validity of measure properties, and feasibility, we have these as criteria to delete because they were covered in other discussions. We didn't feel they were necessary to rehash them here.
And then under settings and types of care beyond traditional health care settings, we also had this as a criterion to delete, and we felt that it should be more of an overarching idea that may be covered in a proposal but setting of care need not be specific—may not be a specific criterion. And that was it.
Dr. Dougherty: Okay, thank you. Next? Who wants to go next? Inpatient? Who is inpatient? Okay, Mark?
Dr. Antman: So, for the inpatient setting, first of all, we began the discussion by acknowledging that inpatient could be defined in different ways. So for our discussion, we limited inpatient to acute care only. And I don't know if that was the intent in identifying inpatient as a category. But that was our decision.
With regard to stratification by vulnerable populations, I think as Kerri just said for the other group, we noted that vulnerable needs to be well defined.
And we talked about three different categories—at least three categories. Are they medically vulnerable? Financially vulnerable? Socially vulnerable? The thinking being that whatever the vulnerable categories that are defined, they should be defined based on national standards. So in other words, not selected—not categories selected by the individual grantees.
And in general, we noted that in the inpatient setting, gathering this information may be challenging because—well, not that gathering information will be challenging, but there will be a problem of a small number in the inpatient setting for any of the individual subpopulations.
And the grantees should be required to note if they are going to stratify by the vulnerable populations how they are going to do so. And we also noted that this, in fact, may not be relevant to each and every measure.
As to stratification by racial groups, again, as I think that Kerri reported for the previous group, there should be a national standard for how the groups are identified. I think you said the OMB categories. We talked about the definitions of the Institute of Medicine (IOM) and the Census categories. Whatever the choice is, it has to be some national standard so there is standardization there.
Underlying scientific soundness, we interpreted this to mean—we interpreted this as a questioning whether or not there is evidence for disparities across these different groups. And certainly there is. But that being the case, it seems that perhaps gathering the information may be more appropriate for some measures than for others. So, for example, measures obviously relevant to diseases such as sickle cell or Tay-Sachs or other conditions that are limited to ethnic populations or racial populations, obviously they are going to be more relevant to those groups.
Dr. Dougherty: Right. So as Ernest was saying, you know, do we make sure that we have a measure for a particularly vulnerable population as opposed to taking standard measures and cutting them by race and ethnicity. Is that what you meant yesterday?
Dr. Moy: Yes.
Dr. Antman: Okay. So as to measure properties, which we took as a reference to validity, reliability, feasibility, all the properties that we've talked about in the last day-and-a-half, I guess the question is, is the measure or would the measures under development by the grantees be equally—would they have those same properties for each of the ethnic or racial populations defined? That reliability, or variability, or feasibility may vary across the different groups. And that being the case, the grantees would hopefully document how well any one of these measures apply to each of these groups. And if they don't apply, say so.
And in particular, we noted in the inpatient setting, collecting the data—again, with regard to the feasibility of data collection, collecting these racial and ethnic data may be more challenging. On the other hand, we also talked about the fact that this may be motivation or this may spur some changes in processes in the inpatient setting or changes in documentation to be sure that in the admission process or somewhere in inpatient documentation, they provide the means to record that information. So that may be some—that may be a benefit of adding that.
For feasibility, we noted that, again, in the inpatient setting, we noted that re-admissions may raise some feasibility issues, particularly because kids may be discharged from one facility and readmitted to another. So that would be a challenge.
And then lastly with regard to settings and types of care beyond the traditional health care delivery settings, possibly not relevant to this topic in that we were talking about inpatient. But we also talked about the fact that the care that kids receive in these other settings may influence whether or not they are, in fact, admitted or re-admitted. So there is a relationship between—and that also ties to disparities, of course. So might the disparities that we see across these groups affect their access to admissions or re-admissions? And I think that covers it. I'll defer to any of the group if I've missed something.
Dr. Dougherty: Anybody else want to add something? Subtract something? No? Okay, and next we have Glenn.
Dr. Takata: Okay. We talked about health outcomes, and our discussion sort of revolved around the same issues as the other groups, but I'll go ahead and report.
We felt that stratification by vulnerable populations should be kept as a criterion with the knowledge that there may be do-ability issues. But, indeed, a need to define the vulnerable populations in detail, as has already been said. Special health care needs children, the homeless, other groups such as infants born in a low-quality hospital—someone brought up the issue that if you start out in a low-quality hospital, it can have impact on future health outcomes. The need for standard definitions, for example, from OMB, as has already been mentioned.
Also, if it is a known disparity, that it really should be included in the analysis. And if it is a known disparity, as has already been said, the measure could apply just to that population.
Our discussion around racial groups was quite similar in the need for a detailed definition and standard definitions. And also, was the intent to include ethnicity as well as racial group in this question?
With regard to underlying scientific soundness, since we were talking about outcomes, we wanted to have outcomes measured that are linked to a process that would lead to improvement. So is it a modifiable outcome?
In terms of measure properties, the same discussion about validity and reliability. And we would expect that there would be some field testing done so that the report would include information on those performance aspects of the measure because it may not be available up front but certainly should be by the end of the project.
In terms of feasibility, yes, that was also a criterion to keep. The data sources should be detailed, including information on the vulnerable populations' race and ethnicity. In a parent or guardian or patient survey or questionnaire, how will translation be provided and literacy issues be addressed? How will the proposal deal with combined other race or no response categories, and particularly if those categories are large, provide information on those specific groups if it is known up front?
The comments were that it may be difficult to pick up children in the combined category, and again, that the other, or combined, or no response categories that you are trying to measure may be large.
In terms of settings beyond traditional health care delivery settings, we thought, as I believe the previous group said, that that should not be an absolute criterion. So we would delete it. It would be nice if we could study differences in outcome in different settings, but we felt the feasibility might be difficult. So to require that as a criterion might not be a good idea. Does the rest of the group—
Dr. Dougherty: Anything from the rest of the group? No, okay. Thanks very much. Just a point of information that the Medical Expenditure Panel Survey (MEPS) and the Healthcare Cost and Utilization Project (HCUP), which are two big data sources, do pick up this other or unknown or multiple races information. And when you look at the quality measures, they typically have the worst quality reporting. So who knows what that means. So, okay, Rita?
Dr. Mangione-Smith: Okay. So our group went fiercely rogue in this round. We did not stick to the table at all. I'm just warning you.
Okay. So we'll start with what we finished with, which was at the very end, Ernest did make some points about meta measures and disparities that again do not go along with the table, but I think are good points. And so we'll make those first.
In terms of disparities and meta measures, grantees should discuss how they plan to measure disparities if they have a composite measure or a measurement set. Will they use OMB criteria? IOM criteria? Or some other locally defined criteria?
And we felt it is okay to use locally defined criteria for groups that might be of special interest in your local area for quality measurement, but those groups needed to be able to be rolled up into one of the bigger national criteria sets.
An example that was given was there was a strong interest in the Hmong population in San Francisco for a certain area of quality measurement. And they could look at that group specifically for improvement reasons. But then that group could be rolled up into the Asian category for one of the bigger nationally defined ways of looking at disparities.
A grantee should also give a rationale for what subgroups they plan to look at and why. Rather than look at—so disparities, this was an interesting point. So should we be developing our quality measures and then doing stratified analyses to look at different groups where there may be disparities? Or should we be developing measures of disparities themselves? And the example that was given was self-report measures such as, do you think you got worse care because of your race or ethnicity? That was just more of a question than a criterion.
And then the other type of measure that was mentioned was we tend to look at disparities of one group versus another referent group. And apparently there are now measures available where you can summarize all differences across all groups and get a score. And that might be a way that some people may—Ernest, I don't know if you want to say anything more about that because you brought that up.
Dr. Moy: Yes, I think it was as much something to think about as it was a specific recommendation or criteria. But there are different scoring techniques to try to summarize disparities across multiple groups within a population. And they might have some interest when you are comparing different States or different geographic regions.
Dr. Mangione-Smith: So back to the beginning of our conversation, which I think was really very interesting. These were more kind of overarching thoughts about the last couple of days.
Alright—no one who applies for one of these grants will meet all of the criteria we've said they need to meet. And the criteria that will be important will really be quite dependent on the type of measure they are proposing to develop.
Some criteria will only apply to some measures. And we felt that it was important that the criteria not just be met by individual grantee, but that the grant portfolio needs to meet all of the criteria that have been laid out today.
Dr. Dougherty: So going back to nobody would be able to meet all the criteria, but you would see that the grant portfolio might—
Dr. Mangione-Smith: The composite of the grants will meet the criteria. There we go. It's a meta measure. See, we were on topic.
Excellent, okay. Then we got into the whole health information technology (health IT) thing. That was a very interesting conversation. Our group was a little bit—I don't want to say we're split. I think in some ways we were actually all on the same page, but we were saying it in different ways. I expressed some concern that the health IT—potential health IT requirement might limit the types of measures that can be developed if you require that the measure be ready to be deployed with health IT today.
We think it is very important moving forward as these people develop measures that health IT be very much in the front of their thinking in terms of how the measure might later be implemented when we have better health IT and better electronic health records (EHRs), and it would help to inform the enhancement of EHRs to do this kind of measurement.
One thing that came up towards the end, but I think it is a really good example of this, is the whole idea of patient self-reported measures or parent-reported measures. And with any current EHR in use today, these don't exist. But in pediatrics, it can be a really important part of measurement. So we wouldn't want to not have grantees put forward those types of measures because they can't be measured with an EHR.
I think that was about—does that summarize it? Okay, that's it.
Dr. Scholle: Actually one more thing. If the focus is on health IT measures, the approach to testing the measures could be really different. I mean if you want a single—start off by saying okay, here are some measures. Could they be in an EHR environment, then it is not likely to look like a field test. It would look more like a case study about feasibility in different systems. And what would be the workflow implications and the implication for the technology?
Dr. Dougherty: Okay, thanks. That's a good point. Okay, well Barbara and I have a couple of things to say. And you may have some things to say to us, which we would like to hear.
So I think this has been beyond our expectations. You know we certainly have more work to do. And you've given us—we'd like to hear more ideas on what next steps we should take.
I think CHIPRA has given us this enormous opportunity to create a science of quality measurement basically, and though that was not what it was intended to do only, it is creating the science—and this is the diverse group here—creating a science that is actually useful and was planned to be useful from the beginning.
So we're not just asking grantees to follow all these criteria because we like to dictate what they need to do. It is really because these are the criteria that are going to be useful—this is the information that is going to be useful to the States, to providers, to the Centers for Medicare & Medicaid Services (CMS), so they can understand where the measures are coming from, what their advantages, disadvantages, and limitations are, and how they can actually be used.
I think this is one of the most exciting opportunities in my career of working on child health and health care quality, which goes back to 1986. So let me just say a couple of things about what our plans are after this meeting.
First of all, we have a transcriptionist who is taking down notes, and we will get that transcription of this meeting—at least the plenary sessions early next week. And our editor here—our Web site editor at AHRQ has promised to turn it around very quickly. So it will be posted by about a week from Tuesday, something like that, about 10 business days.
So the next step is releasing the funding opportunity announcement for these awards. And by posting the transcription quickly, we can—we won't be able to put all these criteria into the the funding opportunity announcement (FOA), but the transcript will be available for whoever wants to apply to take a look at to see what the thinking of this group was.
Now this—the thinking of this group, there are some discrepancies among different folks and some issues that still need to be worked out. So AHRQ and CMS will be working together with our other Federal partners, the Federal Quality Workgroup, two members of the workgroup are over there—working out how we can turn this into a guidance document basically for awardees. Both the CMS demo awardees and the Pediatric Quality Measurement Program awardees.
So as is clear, we have made enormous progress here today. But there are still some issues, the details of which will need to be worked out and will probably not be worked out by us before these awards are made. So we will be counting on the awardees to help us come to where we need more consensus around what the guidance should be. We will be working with them just as we've worked with all of you. It is a quality improvement (QI) cycle to be sure, which will take quite some time.
The other thing is when I announced in the beginning, we'll be sending you emails for an evaluation of some comments on what we could have done better at this meeting, what you liked about it, because we will be doing more quality measures work, and to get more thoughts that you may have about—why didn't I say that when I was there, you know, or you go back to your office and you say well here is an important issue that we didn't, you know, that I have to deal with right now that we didn't deal with at this meeting. But that we should know about it. So I will be doing that.
And I want to thank you—yes, Patrick?
Dr. Romano: I just wanted to make one other sort of general comment which is that I think we had a very robust discussion about some of the issues that are particular to child health measures and CHIPRA. But, again, from my perspective anyway as a general measurement person and as a primary care physician who works with both adults and kids, I think it is important to keep in mind that the commonalities are just as important or more important than the things that may be unique to this population. So again, I would encourage people to look back at the work that has already been done with the National Quality Forum (NQF) guidelines and so forth and to avoid, you know, creating additional criteria where those criteria have already been well specified by NQF or by other organizations that are active in the field.
So it just strikes me that a lot of the issues that we've discussed, some are particular to kids, and those need to be carefully dealt with. But others are just general issues of measurement.
Dr. Dougherty: Okay. So, Barbara, before you make your closing comments, should we get idea like that from the rest of the group?
Ms. Dailey: Sure.
Dr. Dougherty: I think that's an excellent point and where we started out. Certainly we don't want to reinvent the wheel just for kids because then that's always a difficulty because people will continue to say well, kids are too different and too unique, and we'll deal with them later or that kind of thing.
So, are there other thoughts about next steps that we could take? We're posting the transcript. We'll get ourselves together. We will be working with whoever the grantees are and the CMS grantees and the States, continuing to work.
Are there other things we should be doing? Should we have—or should we be asking for resources for specific enhancements to this criteria specification goal? I mean I assume one of the reasons why we don't have a big guidebook of how to specify a measure, all the different kinds of measures, is because there have never been any resources for that kind of thing.
And, okay, Jeff, I think you were first, then Colleen, and then Nora.
Dr. Thompson: So this is Jeff Thompson. Just one, be kind to us, for the States. That's all I ask. But then the other one, we keep sort of skirting around the issue of length of eligibility in a plan and how that will be treated. When will you have a final—whether it is 1 month, 6 months, 12 months? When will you decide that for either all or any of the measures?
Dr. Dougherty: I would expect that would be one of the charges of—correct me if I'm wrong—of whoever these grantees are—awardees to figure out what is the best way to do that.
And I'm not sure it is going to be, you know, whether it is 6 months versus 12 months as maybe some kind of algorithm to include all kids but to adjust for the necessary duration of enrollment. You are going well beyond my level of expertise on how to actually do scoring and weighting. But I'm assuming that could happen.
Ms. Dailey: And more specific to the initial core measure set that is out for public comment, that's one of the areas of information that we were looking for comments on. Under the CHIPRA law, we're required to provide procedures and approaches that we recommend to States to do voluntary reporting. And that's due by February 1st.
Our initial thoughts—
Dr. Dougherty: March 1st.
Ms. Dailey: March 1st, excuse me. Our initial thoughts were to put out some recommendations, maybe even do some brainstorming with people or with States and/or small groups that have had experience with the ground measures to find out what the specific challenges are before we release the final set of recommended procedures and approaches.
Dr. Thompson: Well, and this is Jeff Thompson again—I'll put in my two cents, you know, in writing. But, you know, continuous eligibility of a sustained time for an effect would be something that I would like to see, so then it is not confusing for, you know—I didn't have him long enough to do that effect.
And so the churn and medicate is actually, depending upon the State, 6 months or 12 months of required eligibility tracking can be pretty darn high. So I think it is something—you know, I would like to see a lot of due diligence and a lot of sort of background about if you are going to say 1 month, I'd kind of like to see what the rationale is.
Dr. Dougherty: Okay. Let me ask Sarah, because I know you've—not all of the National Committee for Quality Assurance (NCQA) measures but many of them have different continuous enrollment criteria. So has NCQA ever tested that actual part of the measure? Like what would happen if we—
Dr. Scholle: Yes. So we've tested it, and in the set of measures that we're using, the confidence of well care measures that we just tested, we do have data on—we're comparing 6 to 12 months enrollment period to at least 12 months enrollment period.
And then there is also some information in the literature about this as well. So, in fact, we haven't analyzed the data yet from our field test, but I think we can help to inform that question from our field tests and from some of our prior work.
Dr. Dougherty: Okay. Yes, there was Colleen, and then Nora, and then Kim. And then was there somebody—Beth, were you—did you have your hand up?
Dr. McGlynn: Yes.
Dr. Dougherty: Okay.
Ms. Reuland: I may have misinterpreted your call for resources, but I know just in terms of parents' surveys, we've had a really hard time trying to get funding to develop and validate our surveys for non-English-speaking populations and for the different racial/ethnic groups.
So educating the quality measurement world that it is not just translation but is it culturally sensitive and appropriate, and are we really getting at the same concepts would be great because it has been an uphill battle to try to get funding for it.
So when you have the requirement that the measures be sensitive to it, as a measure developer, we're not finding a lot of funders who are willing to support that work, really important needed work.
Dr. Dougherty: Thanks. Nora?
Ms. Wells: Well, I know this is a comment I keep repeating, but I'm just wondering in these grant—these RFAs [Requests for Applications] that are going out, what the requirement is for involvement of consumers in the development stage. But also a plan for how there is going to be the involvement of consumers at every other stage of the use of this—of these measures? So I am talking about really thoughtful consideration of the kind of education that will need to happen and the way that these measures might be used in partnership with communities to improve care.
So I don't know whether that's already in there. And if it's not, it is a cooperative agreement. And maybe it can sneak in.
Dr. Dougherty: Okay, thank you. Kim, I think you were next. No? Okay. Beth?
Dr. McGlynn: I may have misheard you, so please correct me if I'm wrong, but you said something that sort of raised a red flag on for me, which is that you are creating a science of quality measurement.
I just would argue that one already exists, and I think the point of the CHIPRA effort is that it is no longer acceptable to ignore the fact that there is a science of quality measurement.
And in terms of the kinds of resources that I think would be useful, it struck me that one of the unique opportunities here is that—at least my read of kind of the initial—not that there is the potential for different kinds of grantees. Measure developers who often get accused of spending too much time in petri dishes, people who are actually running programs who may not have a lot of patience for sort of the science piece behind the measurement, and then the people to whom these measures are being applied who mostly feel overburdened as it is. They often feel that the importance of doing the work is not so clear to them.
So it seems to me that one of the resources you might want to ask for is some sort of, you know, cross cultural education amongst these groups—some way of bringing everybody into some alignment with the concerns and perspectives of each of these groups because they all kind of need to be able to walk together to really take advantage of this, what I think is a pretty unique opportunity.
They all have important perspectives. But I think often each group thinks their thing is the most important. And they don't really have ways of understanding the perspectives of the others.
It looks to me like you could end up with a group of grantees in this set who don't really have to talk to the other kinds of grantees. And so if there is kind of a way to create that, I think that would be terrific.
The second thing is, the other red flag that I heard, having been on the bad end of this more than once, is we can't figure it out, so we're going to throw it out there, and we'll figure it out as we go along. And I have to tell you that it is almost never the case that a grant has enough flexibility for the grantee to allow the grantor to come up with new things in the process of doing the work.
So I guess I would urge you to be pretty clear and come to consensus on the absolute critical must haves. And then think about what the sort of learning organization approach would be to learning things where there may be less clarity but not so much with the idea that you're going to give people different driving directions as they are trying to do this work. But more that you use it to inform the next round of grants. And that it is really worth being clear before you release the RFAs about what the must haves are.
I feel like kind of to a point that Rita made earlier that there is a pretty—I was imagining grant applications that would have to be 175 pages long in order to satisfy the criteria and actually also say what you were planning to do.
So I do think that being really kind of clear and focused about the must haves and then having what I suspect will be a much longer list of nice to have or, you know, might be something that would discriminate one grant over another if the grantee was able to sort of get into the nice-to-have area, but to kind of be careful not to just sink the whole enterprise by trying to be overly inclusive.
So that's my guidance de jour.
Dr. Dougherty: Yes, we'll go back—thank you—we'll go back to grounded, intermediate, and aspirational as we did with the—okay, who was next? Cathy Hess? Or did you want to say something exactly on that point? Barbara?
Ms. Dailey: Oh, the only other point I was going to make related to that is we actually just went through some of that experience with the quality demonstration process that we had for the quality grants. With $100 million being dedicated to looking at four specific categories that CHIPRA outlined, it also did give us an option to do an innovation category.
So we went through some of that exact area that you were talking about in terms of what are things that we explicitly do want to recommend that people look at versus what would be nice to have and where you may get bonus points if you consider this.
And an example—Cindy Mann just joined us in June, and she was pretty clear that when we were looking at the demonstrations with the initial core measure set, were we going to look at all of them? Or just some of them? And she was explicit that there is a reason that the initial core set was released. And so we required the grantees to be able to look at all of them.
The other area in terms of the innovativeness, there was a section on provider delivery models, but it wasn't specific to medical home. So we made sure we put into the solicitation that you may want to consider looking at medical home specifically. And additionally we mentioned early and periodic screening, diagnosis, and treatment (EPSDT) specifically because that wasn't highlighted as a specific area. But that is an area we know needs improvement. And so I appreciate that comment very much. And we're taking some of our lessons learned through the quality demonstrations, and we will be working with AHRQ in that respect.
Return to Contents
Proceed to Next Section