Page 1 of 1

Challenges of Measuring Care Coordination Using Electronic Data and Recommendations to Address Those Challenges

Prospects for Care Coordination Measurement Using Electronic Data Sources

Key Challenge Area 3: Unknown Clinical Data Quality in Electronic Data Sources

Panelists noted that since few measures to date have relied on clinical data extracted from EHRs or other health IT sources, the quality of these data—including the accuracy of the information itself, as well as the process for extracting the data from electronic records—have not yet been fully assessed. Another panelist noted that the quality of clinical data, whether obtained from an electronic or paper medical record, is generally not considered as robust as inpatient claims data because there is no auditing of clinical data. However, a different panelist commented that because clinical data are used in providing patient care, there is an important incentive to maintain accuracy, even without auditing. Another panelist reported that within an HIE, they found the greatest accuracy in attributing patients to particular primary care providers when using both claims and clinical data. When providers were given lists of patients who had been attributed to them using an algorithm that relied on both data sources, providers agreed with the attributions about 90% of the time. Having the ability to attribute patients is fundamental to measuring coordination between the primary care provider and other providers or settings (such as a specialist or hospital). It is also critical for care coordination measurement (or any quality measurement) at the provider level.

Over the course of discussions with panelists, we supplemented their commentary by consulting sources they mentioned that have investigated the accuracy of information extracted automatically from EHRs compared to traditional chart review extraction methods.7-11 Although none of these studies assessed measures directly related to coordination of care, they demonstrate some successes using Health IT in quality indicator construction. However, they also highlight that inadequate documentation of measure exclusions (i.e., patients who should be excluded from the numerator or denominator) is a key challenge in such endeavors. Information needed for measure exclusions was typically recorded as unstructured data, which was captured during manual chart review but not when data were automatically extracted, resulting in lower measure performance when using automatically extracted data. Until such information is regularly captured in structured fields, this limitation in EHR data is likely to impact many measures of care processes that rely on data automatically extracted from EHRs, including measures of care coordination.

Any use of health IT data for quality measurement will need to be accompanied by an assessment of data accuracy, reliability, and quality. NQF has developed a Quality Data Model that provides a framework for assessing the quality of data elements. This framework provides a useful starting place for such evaluation (go to Appendix D for links to additional sources).

The completeness of data also must be considered. For example, few HIEs today have participation from all providers or hospitals in a particular region. Information on care or services received at non-participating sites will be missing, which will impact the reliability of any quality measurement using that data source, and is particularly salient to assessments of coordination of care across settings. Panelists noted that the problem of incomplete data should improve as HIEs mature and more data sources are linked (e.g., APCDs and EHRs). However, this development will take time.

Recommendations to Address Key Challenge Area 3: Unknown Clinical Data Quality in Electronic Data Sources

The gaps identified by panelists in knowledge about the quality of clinical data contained in health IT systems, in particular data pertaining to coordination processes, suggest the need to include an evaluation of the reliability and accuracy of any health IT data as part of measure development for any indicators based on information from health IT systems. Although the investigations of health IT system data noted by panelists do not pertain directly to care coordination processes, they suggest that particular attention should be paid to documentation of measure exclusions, as evidence suggests that current documentation structures and practices within EHRs are inadequate for many of the quality measures that have been tested using health IT data. Support for data quality investigations and dissemination of those results, would further quality measurement using health IT data.

Return to Contents

Key Challenge Area 4: Limitations in Linking Data

Data sources, such as all-payer claims databases or health information exchanges, that link data from multiple sources provide a key advantage for care coordination measurement because they provide a view of care received across the health care system, rather than focusing on care at a particular site or setting. In addition, they offer insight into the various care transitions that patients experience, including the care received before, during, and after each of those transitions. Panelists commented that linking claims data with clinical data would provide a particularly rich picture of service use and clinical context, while linking data from multiple providers or settings (e.g., hospitals, primary care practices, multispecialty clinics, behavioral health centers, long-term care, and home health) would provide a view of interactions with a wide range of health care providers.

Panelists identified policies limiting collection or use of identified data in response to privacy concerns as an ongoing challenge in linking data across sources, whether clinical data from EHRs or claims data from hospitals, pharmacies and ambulatory care settings. Such policies complicate, and in some cases seriously limit, the ability to link data across sources. Though the Federal Health Information Portability and Accountability Act (HIPAA) and other privacy regulations allow data sharing for treatment, payment, and operations, the complicated nature of privacy statutes requires careful planning, operational structures (e.g., written business agreements or data use agreements), and technological protections (e.g., encryption, data security) prior to sharing protected information. Adding to the complexity, each State has its own privacy laws, further complicating any data linkage efforts that cross State boundaries.

One panelist commented that privacy policies and regulations limiting exchange of patient information pose a challenge for the execution of care coordination, and, by extension, measurement of coordination processes. Nevertheless, panelists noted that the value of linked data is becoming widely recognized, which is lowering policy barriers to this process.

Overall, panelists anticipated that comprehensive data sets linking clinical and claims data will become more widely available in the future, but to date such sets are available from only a limited number of States or regions. A recent report on APCDs states that currently, five States with existing or developing APCDs are collecting patient identifiers that would allow linking the dataset to other outside data sources, such as clinical data from EHRs or an HIE, four States do not currently allow patient identifiers to be collected, and five States are either examining the issue legislatively or are unable to disclose whether or not they are collecting patient identifiers.12 Panelists noted that established HIEs and APCDs where these challenges have been overcome can serve as models to ease creation of new linked datasets in other regions or States.

Panelists also noted limitations in the completeness of data contained within linked data sources such as APCDs and HIEs. For example, even well-established HIEs typically cover only a particular region or a subset of providers within a State. Similarly, APCDs vary in the percent of the patient population included, both with respect to patients with commercial insurance (e.g., only patients covered by insurers with a minimum number of lives covered, or that cover a certain share of a particular market), and also with respect to entire groups of patients (e.g., the uninsured; Medicare beneficiaries; and patients with other Federal health coverage such as through the Department of Veterans Affairs, the military, or Indian Health Services). This has implications for implementation of denominator definitions for any quality measures that use such a data source, as well as potential measures' utility and interpretation.

Recommendations to Address Key Challenge Area 4: Limitations in Linking Data

Panelists emphasized that efforts should be made to communicate the value of linked data to policymakers and the public. They also highlighted the importance of developing, demonstrating, and sharing strategies for overcoming privacy barriers. They noted that established health information exchanges and States and regions that allow collection of identified patient information within APCDs will be well-positioned to provide such demonstrations.

Return to Contents

Key Challenge Area 5: Technical Hurdles to Accessing Data

Several panelists noted that it can be very difficult and costly for practices to extract data from their EHRs for use in quality measurement or quality improvement. One panelist with extensive experience extracting data from different EHRs noted that much site-specific work is required to identify and extract the necessary data elements and that often how information is recorded varies by patient condition within single practices. This would create a significant challenge if trying to scale up to measures of care coordination applicable across a wide range of conditions, rather than disease-specific measures. Another panelist estimated that extracting data for a single quality report might cost $100,000. Yet another panelist characterized EHRs as "data sinks," rather than data repositories.

In one example of the problems in extracting data, a panelist noted that at a five-physician practice using a single EHR, they discovered 136 combinations of where and how colorectal cancer screening was documented within the EHR. This variation resulted from differences in terminology used (i.e., lack of standardization in how information was coded or recorded) and differences in where within the EHR the information was stored (i.e., lack of structured data fields, as well as variation in the clinical workflow and use of EHR technology). Panelists emphasized that identifying all of the ways and locations in which a single piece of information can be recorded and developing algorithms to extract and normalize that data require considerable health IT resources, which is why many practices currently have trouble using or simply are unable to use data from their EHR systems for quality improvement or quality reporting. The logical extension of this problem is that it may be indicative of the challenge to a practice in actually coordinating care (e.g., right care delivered at right time in right setting) in the current health care environment, much less assessing whether the constellation of activities required occurred in the most efficient way possible.

Our panel of experts included several involved in the design or administration of databases that use information from EHRs. Approaches to obtaining that data varied across these systems. One database uses third-party companies to extract data directly from the databases that underlie EHR systems. This administrator noted that the difficulty in extracting data varies considerably by the EHR system. Another panelist explained that a software program designed to aggregate information across patient encounters relies on individual health care sites to export data from their EHRs, which are then integrated into the software and standardized. When possible, the software maker offers some guidance to sites on how to identify and export data, but the burden of data extraction falls on the individual practices, which often then turn to the EHR vendor for help extracting data. Another panelist noted that even when data are compiled from users of products from a single vendor, the high degree of customization performed when implementing EHRs at individual sites complicates data extraction. The ability to efficiently coordinate and measure care may depend partly on finding an appropriate balance between system customization, which may help improve coordination by adapting technology to meet local needs, and standardization, which facilitates comparative measurement.

HIEs have been suggested as potential data sources for care coordination measurement because they aggregate information across many different parts of the health care system. However, panelists emphasized that HIEs themselves do not typically store data. Many are just channels for transmitting information with all the data housed in the original systems (e.g., the ambulatory clinic EHR or hospital EHR). Panelists emphasized that without a data repository or underlying database that stores information from the various health care entities that participate in an exchange, an HIE is not a data source. One panelist with experience working with an HIE noted a cultural challenge in having the need recognized for such a repository underlying HIEs.

Recommendations to Address Key Challenge Area 5: Technical Hurdles in Accessing Data

Discussions with panelists led us to the following recommendations for improving the ease of access to data within health IT systems:

  • Consider the accessibility of data to end users when designing systems. Panelists emphasized that vendor design is highly responsive to user demand, highlighting the importance of having users and purchasers understand the need for easily accessible data and of communicating that need to health IT vendors.
  • Carefully consider the resources required to extract data from health IT systems when choosing a product.
  • As demonstrated by the Meaningful Use incentive program, vendors are also highly responsive to certification requirements and incentive programs because such programs drive demand among users and purchasers who benefit from those incentives. ONC and its health IT committees may wish to consider whether any additional EHR certification requirements could help improve the ease of extracting data from within EHRs.
  • Some data elements of potential interest may be extracted from health IT systems using a free, open source software service from ONC, called the popHealth tool. The popHealth tool is designed to help EHR vendors and health care providers extract data elements required to inform all 44 Meaningful Use Stage I quality measures from their Continuity of Care records (CCD or CCR). It is geared toward simplifying the standardization process for EHR users and is designed to assist users that do not have programs in place to extract the necessary data elements themselves. More information on the popHealth tool is available in Appendix D.

Expanding the popHealth tool to facilitate extracting data elements for Stage II, and eventually Stage III, measures (when those measures use data elements not required in Stage I) would further increase the availability of health IT data for quality measurement, particularly if new quality measures are developed using Meaningful Use data elements. Eventually incorporating other key data elements into popHealth or similar data extraction tools would further facilitate access to health IT data, to the extent that standard definitions of concepts critical for a wide range of quality measures are developed (go to Recommendations to Address Key Challenge Area 2.

Return to Contents

Key Challenge Area 6: Business Models That Facilitate Competition Rather Than Cooperation

Another challenge noted by several panelists as limiting HIE development and exchange of clinical information in general is the fact that exchanging information among competing health care institutions and health IT vendors runs counter to current business models. Makers of health IT products are generally reticent to share information about the design of proprietary software, complicating efforts to achieve interoperability and standardization across the health IT sector. In addition, health care delivery entities may be wary about sharing information with their competitors. Some use EHR deployment strategically, such as a hospital supporting installation of their vendor's EHR system into primary care practices within their market reach, so that these physicians have a workflow incentive to refer their patients to the hospital with a compatible information system. One panelist observed that good care coordination often means less money for health care institutions. For example, ordering a repeat test generates revenue, while obtaining results from a test performed at another institution does not. Indeed, the resources required to obtain results from an outside source (whether through institutional investment in an HIE or time spent by individual providers to seek out information from other sites) and to integrate them within the receiving EHR (ranging from scanning documents to developing mapping algorithms that recode lab results from one system to another) generally increase non-reimbursable costs for a health care entity. Another panelist commented that hospitals will have to rethink their business models to maintain financial stability if hospital admissions decline as a result of improved care coordination or other health reforms. Thus, although from the patient and societal perspectives any activities, such as care coordination, that are expected to decrease hospital admissions are seen as valuable, organizations delivering care have disincentives for spending resources implementing changes that may undermine their current business models.

Despite these challenges, one panelist with experience administering an HIE was optimistic about the prospects for increased information sharing. He noted that, at a meeting of leaders of competing institutions who are participating in an HIE, discussion focused around the realization that information sharing that helped one institution ultimately helped others as well. He emphasized that helping stakeholders realize the mutual benefits of information sharing will be key to overcoming the obstacles of a competitive health care marketplace, and pointed to the example set by successful HIEs as an important demonstration of those benefits.

The ability to share information across health systems has important impacts on the ability to coordinate care, and by extension to measure coordination. Similarly, barriers such as business models that impede information sharing also often impede the standardization necessary for quality measurement using health IT data.

Recommendations to Address Key Challenge Area 6: Business Models That Facilitate Competition Rather Than Cooperation

To address some of these business model barriers, panelists suggested supporting and widely disseminating projects that demonstrate the value of information sharing. They also highlighted the need for evidence that can demonstrate any cost savings for institutions that result from information sharing or other care coordination activities. In addition, panelists emphasized the importance of bringing leaders of competing health care organizations together to facilitate dialog and encourage information sharing. Finally, some expressed optimism that financial incentives will become better aligned between payers and health care providers as alternative models of health care delivery and payment evolve, particularly through initiatives related to accountable care organizations and patient centered medical homes. Supporting such initiatives may help overcome some business model barriers that have hindered information sharing and care coordination and by extension make data about coordination more readily available for measurement purposes.

Return to Contents
Proceed to Next Section

Page last reviewed March 2012
Internet Citation: Challenges of Measuring Care Coordination Using Electronic Data and Recommendations to Address Those Challenges: Prospects for Care Coordination Measurement Using Electronic Data Sources. March 2012. Agency for Healthcare Research and Quality, Rockville, MD. http://www.ahrq.gov/research/findings/final-reports/prospectscare/prospects1a.html