Appendix B: Case Study: The Cincinnati Health Collaborative Physician Dashboard

Private Performance Feedback Reporting for Physicians: Guidance for Community Quality Collaboratives

To gain an indepth understanding of how a CVE-sponsored private report might be enhanced to better meet the needs of community providers, we engaged in a case study project with the Health Collaborative (HC) of Greater Cincinnati aimed at developing recommendations for improving the utility of its private "physician dashboard" report. The case study involved a series of interviews with HC staff that led to a decision to obtain feedback directly from Cincinnati physicians and physician practice quality improvement managers. We sought feedback regarding their needs for performance information reporting, as well as suggestions for improving the content and functionality of HC's current physician dashboard.

The following sections present a brief description of the Collaborative's public and private reporting Web sites, the process we followed for obtaining feedback from Cincinnati physicians and practice leaders, and a summary of our findings and recommendations based on this input.

HC Public and Private Reporting Web Sites

HC is a nonprofit organization founded in 1992 that brings together multiple stakeholders in the Cincinnati region including physicians, employers, health plans, hospitals, and community groups to work on collaborative ways to improve the quality of care. A major focus of HC's work has been the development of a public reporting Web site (http://yourhealthmatters.org).

Launched just over 3 years ago, YourHealthMatters (YHM) presents comparative health care quality information regarding the treatment of diabetes (based on the D5 measures) and cardiovascular disease and screening rates for colon cancer. Users can access quality ratings for participating primary care practices related to how well they achieve specific treatment goals with their patients in these three areas.

By emphasizing health goals associated with these conditions, the Collaborative hopes to encourage patients and doctors to work together to achieve better health care results. The Web site also includes educational information about patient experience surveys, and HC plans to add ratings for these and other topics in the future.

The data for YHM come from voluntarily participating medical practices that annually submit the clinical information needed to report these quality measures. Currently, 135 medical practices from 20 medical groups are participating, representing 443 primary care physicians. Participating practices submit medical record data electronically according to detailed submission guidelines through a separate, secure Web site or "data portal." On this secure data portal, participating practices and physicians also can get access to detailed data, called a "physician dashboard," related to the publicly reported measures so they can use this information as a tool for quality improvement.

In addition to the measures publicly reported on YHM, the private portal includes data related to patient body mass index, as well as some additional process measures related to diabetes and cardiovascular disease. These additional measures are submitted to comply with reporting requirements of both the National Committee for Quality Assurance (NCQA) and Bridges to Excellence (BTE) practice recognition programs.

Similar to the data publicly reported on YHM, the private dashboard presents scores for each of these measures at the individual practice site level. However, unlike YHM, data on the private portal also can be viewed either rolled up to the appropriate medical group or system levels, or drilled down to the individual physician level assigned to a given practice site. (Note that scores for individual physicians can be viewed only by users affiliated with the physician's medical group.)

For practices participating in a special diabetes quality improvement collaborative, the private portal displays additional charts and graphs related to practice site performance on the D5 measures. Currently 24 practices are involved in this collaborative, which was initiated as part of a patient-centered medical home project sponsored by the regional health information exchange known as HealthBridge.

Despite the progress made by HealthBridge to create a communitywide infrastructure to support the electronic transfer and collection of health care data, many health systems in Cincinnati still face barriers in using HealthBridge to exchange standardized data. When it became clear that HealthBridge could not support the pooling of data needed for the diabetes improvement collaborative, HC's private portal provided an alternative solution. Practices participating in the diabetes collaborative submit monthly uploads of data to the HC portal for a sample of patients, and the HC private dashboard displays run charts as well as scatter plot charts for each measure at the practice site level.

Process for Obtaining User Feedback on the HC Physician Dashboard

To develop useful recommendations for HC on ways to enhance the current private physician dashboard to better meet the needs of its intended audience, in consultation with HC staff, we decided to seek empirical feedback directly from users. Our process for gathering this feedback consisted of two major steps:

  • Focus groups with physicians. We conducted two focus groups with primary care physicians (PCPs) recruited from the group of PCPs who participate in the YHM public reporting Web site. Focus group participants included a mix of gender, specialty, years in practice, health system representation, and practice settings.
  • Interviews with quality improvement managers. In addition to the physician focus groups, HC staff determined that it was important to obtain feedback and perspectives separately from the quality improvement managers of the major health systems. Separate one-on-one conversations with these managers were considered helpful for promoting an open dialogue that might otherwise be constrained by the presence of the physicians they work with or representatives of other health systems in the market. One-hour interviews were held with a representative group of managers responsible for participating in the YHM public reporting process, including the submission of required data through the private portal.

Key Findings

Information Needs of Physicians

Individual physicians expressed the need for information that will help them improve the care they provide to patients. Physicians said that information should be provided at the patient level and indicate which patients need followup reminders to get services they require to manage their conditions. To be useful, such information for managing patient care should also be timely and up to date:

"If data are more than 24 hours old, it's old hat at this point."

A few participants described their access to such information systems within their own organizations:

"I have access to a data warehouse and can pull real-time information on my diabetic patients. I can also look up other docs. It's a good tool to be more proactive, to reach out to patients that may be deficient in areas we want to target."
"We can see up-to-date information on a patient level. It's very helpful."

Physicians are also mindful of the need to monitor their performance on selected measures:

"I get reports every 3 months on diabetes, cholesterol, hypertension, etc., from corporate. It helps us to know where we stand."
"I want my numbers to look good."

Another key need expressed by physicians is for practical strategies and "best practices" for improving their management of the conditions and topics that are the focus of public reporting:

"I've been to many of the steering committee meetings and...the amazing thing to me is the lack of the "how do we do it better"; discussion... how do we improve? That discussion has actually gone backwards, because we've diluted down the measurement goals; for instance, the initial goals for diabetes have been backed off."
Where I see this [the HC physician dashboard] being helpful is to be able to see where other practices might be doing better, and then reach out to them and ask, what are you guys doing that's successful? Ideally we could do this systemwide. Use it as a springboard for best practices."

One physician who participates in the diabetes quality improvement collaborative expressed strong interest in seeing data at the practice level over time. Of note, he was not aware that the HC private portal was the platform supporting this information display:

"It is very helpful to see what other sites are doing, then reach out to find out how to improve."

A major information need not currently being met for any of these physicians is the ability to access patient compliance information when the patient receives care or services outside of their organization:

"The one thing that would be helpful, that I hear from my docs, is they want data from other places that their patients go to. We have in our system... only what we have in our system. But if an eye exam was done [someplace outside of our system] we won't have a record of it. Or if we were lucky and we got a report [from that place] then we'd have it. The health plans have a lot of data that we don't have, and we have data they don't have, but never the twain shall meet."
"The biggest obstacle we face is getting data on patients that don't show up. If they have not been in for 12 months, we don't get that information. We need to find the patients who are not there for their appointment."

These last two comments point out the need for cross-organizational exchange and sharing of data on a population basis. Perspectives on the role of HC in meeting this need are noted below.

Concerns Expressed by Physicians

In addition to identifying specific information needs, focus group participants expressed a variety of concerns related to public reporting and the demands placed on them for complying with internal and external performance goals.

Several physicians expressed frustration over being held accountable for what they view as patient behavior beyond their control:

"Why should I get penalized because my patient is obese and noncompliant, when I try as hard as I can to call them back and yell and scream, and their A1c is still at 10 and their blood pressure is still off the wall. This health grading... the whole concept... I just don't get it."
"I know what I should be doing for my patients. Don't tell me what to do with my patients. Tell the patient what they need to do!"

Concerns were also expressed about the accuracy of the data being reported, in part based on their own experience with the challenges of accurate coding and entry of data on their patients in new electronic medical record (EMR) systems. Other data concerns are related to small numbers and a multitude of factors they perceive as contributing to differences in reported measures that are not attributable to physician performance. These include lack of resources to support patient compliance (e.g., certified diabetes educators), patient characteristics that influence scores, and a shift to mandatory use of generic drugs that requires a long-term process of bringing patients up to speed on medication adherence.

Some physicians complained about not having enough compensated time built into their days to actually understand and use internal information systems. Some are relying on care coordinators to do this work for them.

Other frustrations expressed by physicians relate to the lack of incentives for public reporting and quality improvement and consumer disinterest in the performance measures that are reported:

"No one is paying for quality in town. BTE is only paying for diabetes. No patient looks at the YHM site to find out how we're doing, even those who pay a premium to come to [a concierge practice]."
"Patients don't understand quality; they may understand service. But even health plans and payers don't pay attention to quality. They are concerned about use and cost."
"We need incentives. We are doing this quality reporting now out of the goodness of our heart. What we get paid for is to push patients out the door."
"We also need to turn around payment and compensation internally [to focus on quality]."

Information Needs of Quality Improvement Managers

In addition to supporting the information needs of physicians described above, a major need expressed by quality improvement managers is to be able to respond to external reporting requirements of health plans and certifying organizations such as NCQA and BTE, as well as Centers for Medicare & Medicaid Services meaningful use and Physician Quality Reporting Initiative standards. The other major information need facing these managers is to support internal quality improvement goals related to physician performance.

To meet these internal and external information and reporting needs, most managers are investing substantial amounts of time and resources transitioning to new EMR systems and working with practice site managers and physicians to adjust to new data coding and entry protocols. Although the EMR transition process is challenging, getting physicians to focus on data accuracy in charting has benefits. For example, engaging physicians in the process of correcting data can help invest them in using the information to provide better care:

"Getting physicians to verify and clean up the data is a very useful step to take because now you have physician buy-in."

Many quality managers expressed frustration with the inability of most standard EMR systems to generate the reports they need for both internal and external purposes. Some are purchasing add-on systems to generate needed reports or are working with their information technology (IT) departments to develop customized solutions.

Awareness and Use of the HC Physician Dashboard To Meet Identified Needs

Most physicians are not aware of the physician dashboard and do not personally access the information:

"I didn't know we could [access the portal]. I didn't know there was a behind-the-scenes report. There are certain people in our group—IT and quality people—who run the reports and keep us all in check."
"Personally, I never have used the site. There are people in the business office that do."
"I doubt if any of our docs are accessing the site... to be honest, the only reason I go is to see how we rank against everyone else. I go once a year when we put the data up there."

Several physician focus group participants commented that there is very little difference between the private dashboard and the public reporting site:

"But you don't really have to go into the portal to see that...you can get it from the public Web site...even though I can get a little more detail on the back end [through the portal]."

Quality managers are all aware of the portal because they use it to upload data for public reporting. Even so, most of the managers were not familiar with the capabilities of the physician dashboard. They all expressed skepticism that their physicians were aware of the dashboard:

"I don't think our physicians have logged in to look at their personal stuff; I don't know how much they are encouraged to do that. What they care about is what the public can see [on YHM]."
"There's not a reason for individual doctors to look at the portal or their reports; they have no reason to do it when the practice managers are handing them the results."
"I don't think they [physicians] are finding value in the portal because we are already giving them the data quarterly, at least on the D5."
"The private portal is not viewed by physicians but by administrative staff that pull data from the community to graph comparisons for leadership."

One of the biggest drawbacks noted about the HC dashboard is the lack of timely and current data:

"It's just annual; it's old data."
"They [physicians] can't use it for process improvement because they're not gonna wait that long to see if a process is going to improve an outcome."
"A one-time snapshot each year is helpful for community comparisons but not for patient management."

All of the quality managers are using their own internal EMR and related information systems to create the reports they need. Although some systems are more advanced than others in terms of capability and timeliness, even the small, unaffiliated practices have developed reporting systems. Some systems are updated every night and can be accessed in real time to look up complete registries of patients for multiple conditions. More commonly, quality managers are developing monthly or quarterly reports that are shared with practice managers and quality committees and may also be sent directly to physicians as an email attachment.

One current feature of the HC private portal that was widely cited as extremely valuable by quality managers (as well as the few physicians who were aware of it) is the component of the site supporting the diabetes quality improvement project described earlier. The features of this dashboard component that make it especially useful include its monthly updating of measures and the ability of practice sites in one system to view their performance relative to sites in other systems, a capability that no single organization has on its own:

"The run charts are great. I want to see trends and drill down to specific D5 components."

However, to make the HC data portal even more useful, some managers expressed interest in uploading not just a sample of patients manually, but a complete set of records for all patients through their EMRs:

"I would rather upload a complete patient census than manually abstract a sample, which is very time consuming. This would need to be EMR driven and would work for all the systems that have EMRs. We could then compare to other practice sites and track for QI purposes."

Another aspect of the diabetes quality improvement component of the HC portal that was widely praised goes beyond the reporting function itself to the value of the learning collaborative that HC has convened to support practices in identifying and testing process improvement methods:

"It's not just the numbers; it's the interaction. It [the learning collaborative] brings us together in a different spirit."
"Internal benchmarking is certainly useful but if we are only looking at potential for improvement in isolation, we're not challenged by what another practice might be demonstrating or trending or making a major improvement. So we benefit from that competition on one level and then certainly, more altruistically, what we can learn from each other."

Role of HC Moving Forward To Support Physician Information Needs

Given that all practices in the Cincinnati market, whether part of a large health system or unaffiliated as small, independent offices, appear to be meeting most of their performance information and reporting needs on their own, opinions were mixed regarding the future role of HC in this arena.

Most quality managers acknowledged the important foundational role that the HC public and private reporting Web sites played in moving physician practices to focus on their internal reporting systems:

"The [HC dashboard] was actually my jumping off point for what I did...as far as what do we want to report...if this data is going on a public Web site, then we need to be looking at it every month and not just once a year when it's time to do the data submission... to be able to anticipate what we're going to be reporting."

Moving forward, most focus group participants and quality managers expressed skepticism that HC could create a community-level information system that would complement what organizations are doing internally by centrally linking together each EMR:

"How can we get this to the community level? I would need to give you data out of my EMR. Clearly that's what I'd love to see happen but knowing the political issues behind it, I struggle to see how that would ever happen. We struggle now to share basic data [through HealthBridge], let alone that level."
"Getting systems on the same platform with their EMRs seems unreal."
"From the system perspective, the Collaborative needs to go whole hog or get out of the business. Because it [the current HC dashboard] is just not useful. We're all doing something... to do exactly this [what the dashboard does]. They [HC] would have to get us all to agree to put the data in and have one system we use for looking at it, or just don't bother."
"To move things forward, there has to be a commitment to either make it work as a community or to let each system go on its own way. A lot of tension exists among [the health systems]. It's time to say we have an integrated EMR across the community or go our separate ways."
"The Health Collaborative provides value by generating ideas and catalyzing action. Now it's up to us to make this happen in our own systems. This not a sustainable effort [for HC] since you need real-time data and internal operations to support it."
"The idea of a future communitywide data warehouse has some appeal but the downside is that systems will remain competitive and unlikely willing to share all of their EMR data even if technically they can connect."

One area that almost all participants agreed would be a useful value-added function for HC is to provide both the data reports and facilitation expertise to support specific quality improvement projects, modeled after the successful diabetes improvement collaborative.

"We spend too much time pulling the data, not enough time working the data."
"The Collaborative should focus less on being the community database, and focus more on skill building for QI."
"The convening function is a good role for the Collaborative. The diabetes collaborative was a good experience. It could be expanded to other areas."
"Physician practices are notoriously not experienced in any kind of process improvement.
Patients are being called back and examined in office settings exactly as they were 30 years ago.... Getting them to change... in the medical home model...or even having office staff working at the top of their licensure to take away some things that physicians don't need to be doing... is not simple. If we want to have them change, we have to have a way to measure and show them the improvement in a pretty rapid period of time."

One final area noted by many participants is the unique ability of HC to provide community-level benchmarks that no single system can create on its own. However, several quality managers noted that such community benchmarks should go beyond the mean or median to include percentile distributions and to provide trend data over time.

Recommendations

Our recommendations for HC were built directly on the very clear and compelling feedback obtained through the physician focus groups and interviews we conducted with quality managers.

Although originally conceived to seek input on ways to improve the design of the HC physician dashboard reports, the comments and perspectives gathered from the target audience for these reports made it clear that the major issue HC faces is not one of report design. Rather, it is a more fundamental set of issues related to the strategic positioning of HC in a rapidly evolving health care information marketplace.

All of the physician practices we spoke with are meeting all or most of their performance information and reporting needs on their own. Moreover, the likelihood of HC building an integrated EMR at the community level for exchanging real-time information needed for improvement appears remote at best. Therefore, we recommended that HC focus its limited resources and unique collaborative role in the Cincinnati market in the following ways:

  • Report the private data needed for specific quality improvement initiatives. The clear value provided by HC in supporting the diabetes quality improvement project suggests that the private data portal could be used for similar projects, thereby filling an important niche. The current limitations of HealthBridge in meeting the data exchange needs across multiple practices provides an opportunity for HC to fill this role at least in the near future, or to work in concert with HealthBridge to overcome the data exchange constraints that limit the electronic uploading of data needed to monitor progress on a monthly or quarterly basis.
  • Provide enhanced support for the quality improvement activities of participating practices. It is clear that the requisite experience and skill set for designing and maintaining quality improvement processes within Cincinnati physician practices are in extremely short supply. This is a need that is apparent even within the largest and most technically advanced health systems. We recommended that HC fill an important community need by expanding the current quality improvement consulting, facilitation, and training services they provide to reach a larger number of practices. Such an expanded consulting role could also serve as an important business line for helping to ensure sustainable HC operations.
  • Provide community benchmark data for health systems and practices to import into their internal reporting systems. Another unique role for HC is to continue creating and disseminating communitywide benchmark data that no single organization can develop on its own. However, it's not clear that a separate physician dashboard is needed to support this function, since the YHM public report includes this information. As noted above, expanding the metrics provided through this benchmarking (to include the full distribution of scores and other helpful comparative statistics) would enhance the value of these community benchmarks even further.

Our interviews and focus groups revealed strong support for HC as an important community resource for facilitating health care improvement among Cincinnati physician practices. By listening closely to the information needs of this key audience and identifying those products and services that HC is in a unique position to supply, we concluded that the Collaborative will be poised to continue providing value for its multiple stakeholders while helping to maintain its own organizational viability.

Page last reviewed November 2012
Internet Citation: Appendix B: Case Study: The Cincinnati Health Collaborative Physician Dashboard: Private Performance Feedback Reporting for Physicians: Guidance for Community Quality Collaboratives. November 2012. Agency for Healthcare Research and Quality, Rockville, MD. http://www.ahrq.gov/professionals/clinicians-providers/resources/privfeedbackgdrpt/privfeedbackgdrptapb.html