Care Coordination Measures Atlas Update

Appendix II. Identifying Measures

Main Indicator Sources

  1. Literature Search. A measure-specific care coordination search was conducted to identify published literature related to the development, validation, and testing of measures of care coordination. The search strategy is outlined below.
  2. Care Coordination EPC Report. As part of a previously published care coordination report ("Closing the Quality Gap: A Critical Appraisal of Quality Improvement Strategies"; Volume 7: Care Coordination), background research and a systematic review identified care coordination indicators in published studies.
  3. Panelist Calls. A series of panel calls were held in order to obtain information regarding additional measures of care coordination and ongoing research and development in the field. Panel participants had backgrounds ranging from research and evaluation in care coordination to clinical practice. For a list of panel participants, please go to Appendix III: Advisory Group Participants.
  4. NQF Draft Report. NQF evaluated a list of 77 candidate measures and recommended a set of preferred practices across five domains of care coordination: 1) health care home, 2) proactive care plan, 3) communication, 4) information systems, and 5) transitions. The final report was released in October, 2010, shortly before completion of the Atlas.16

Literature Review Search Strategy

The final measure search was performed on July 13, 2010, using the following strategy:

[ ("(("healthcare" or "healthcare" or care) adj3 (coordinat* or "co-ordinat*" or integrat*)).tw.") And ("(rated or rating or indicator* or measure* or valid* or reliab* or outcome* or model* or scale* or subscale* or questionnaire*).tw. or methods.fs. or exp Questionnaires/") ] Not [ ("exp geographic locations/ not exp united states/") ]

The search was limited to English language publications. Details of the search strategy development are included in the box below.

In addition, publications by known key researchers involved in care coordination measurement were also searched. Bibliographies of particularly relevant included references were also reviewed for any further sources of information.

We compared our search strategy to RAND's ACOVE-3 search strategy post-hoc, and we found no additional terms, phrases, or combinations that were not captured in the strategy outline above.

Details of Search Strategy Development

With the help of a research librarian, a literature search was conducted using Ovid MEDLINE® In-Process & Other Non-Indexed Citations and Ovid MEDLINE® 1950 to Present. Several sets of search terms were used in combination to net articles describing care coordination measures and measurement strategies. Search sets 1-7 (shown below) represent the concept of 'care coordination'. Search sets 8-10 represent the concept of 'measures'. Search sets 11 and 12 represent the concept of 'outpatient' and 'medical home'. Search set 13 represents any citation indexed with a non-United States country subject heading and is used with the Boolean operator "NOT" to narrow the size of other search sets. Search set 14 is a high-precision title search using only the most relevant terms to "catch" obviously relevant citations the other searches might have missed.

Searches using combinations of the above sets were conducted. Searches were checked for article inclusion compared to a list of 10 highly relevant articles. The most effective search strategy was chosen based on inclusion rate and the total number of search results. The team determined that the final search strategy should yield no more than 4000 results, while simultaneously including as many of the 10 "test articles" as possible. The final search strategy used was: ((7 and 10) not 13) and A search of the database through April 5, 2010, using this search set yielded 3306 publications and included 8 of the 10 test articles. The measure search was updated on July 13, 2010, to capture any additional measures indexed in MEDLINE® after the original search. The updated search yielded 8 new measures from among 142 new publications.

Search Sets:

Search Set 1: "exp "Continuity of Patient Care"/" (10856 results)

Search Set 2: "exp *"Continuity of Patient Care"/" (5213 results)

Search Set 3: "disease" (6824 results)

Search Set 4: "exp case management/ or "case manager*".mp. [mp=title, original title, abstract, name of substance word, subject heading word, unique identifier]" (7613 results)

Search Set 5: "patient centered" (6153 results)

Search Set 6: "exp Delivery of Healthcare , Integrated/" (6135 results)

Search Set 7: "(("healthcare" or "healthcare" or care) adj3 (coordinat* or "co-ordinat*" or integrat*)).tw." (8073 results)

Search Set 8: "exp "Outcome and Process Assessment (Healthcare )"/ or exp Quality Indicators, Healthcare / or exp treatment outcome/ or exp quality of healthcare /" (3711934 results)

Search Set 9: "exp "Outcome and Process Assessment (Healthcare )"/ or exp Quality Indicators, Healthcare /" (484437 results)

Search Set 10: "(rated or rating or indicator* or measure* or valid* or reliab* or outcome* or model* or scale* or subscale* or questionnaire*).tw. or methods.fs. or exp Questionnaires/" (4889524 results)

Search Set 11: "exp ambulatory care/ or or or (visit* adj3 (clinic or clinics)).mp. [mp=title, original title, abstract, name of substance word, subject heading word, unique identifier]" (156296 results)

Search Set 12: "("medical home" or pcmh).mp. [mp=title, original title, abstract, name of substance word, subject heading word, unique identifier]" (532 results)

Search Set 13: "exp geographic locations/ not exp united states/" (1661486 results)

Search Set 14: "((measure* or valid* or reliab* or outcome* or model* or scale* or subscale* or method* or questionnaire* or rated or rating or quality or indicator*) and ((coordinat* or "co-ordinat*" or integrat*) and (care or healthcare or "health-care"))).ti." (553 results)

10 Relevant Articles:

  1. Antonelli RC, Stille CJ, Antonelli DM. Care coordination for children and youth with special healthcare needs: A descriptive, multi-site study of activities, personnel costs and outcomes. Pediatrics 2008;122:e209-16.
  2. Rittenhouse DR, Casalino LP, Gillies RR, et al. Measuring the medical home infrastructure in large medical groups. Health Aff 2008;1246-58.
  3. McGuiness C, Sibthorpe B. Development and initial validation of a measure of coordination of healthcare. Int J Qual Health Care 2003;15(4):309-18.
  4. Coleman EA, Eilertsen TB, Magid DJ, et al. The association between care coordination and emergency department use in older managed care enrollees. Int J Integr Care 2002;2:e03.
  5. Cooley WC, McAllister JW, Sherrieb K, et al. The medical home index: Development and validation of a new practice-level measure of implementation of the medical home. Ambul Pediatr 2003;3:173-80.
  6. Zillich AJ, Doucette WR, Carter BL, et al. Development and initial validation of an instrument to measure physician-pharmacist collaboration from the physician perspective. Value Health 2005;8(1):59-66.
  7. Malouin RA, Starfield B, Sepulveda MJ. Evaluating the tools used to assess the medical home. Manage Care 2009 18(6): 44-8.
  8. Coleman EA, Smith JD, Frank JC, et al. Development and testing of a measure designed to assess the quality of care transitions. Int J Integr Care 2002;2:e02.
  9. Bethell CD, Read D, Brockwood K. Using existing population-based data sets to measure the American Academy of Pediatrics definition of the medical home for all children and children with special healthcare. Needs 2004;113:1229-537.
  10. Peikes D, Chen A, Schore J, et al. Effects of care coordination on hospitalization, quality of care, and healthcare expenditures among Medicare beneficiaries. JAMA 2009;301(6):603-18.

Measure Selection

Measures for inclusion in the Atlas were identified in two steps. First, a list of potential measures was compiled from the search strategies outlined above, which yielded a total of 3448 unique measure sources. Measure sources were included if they featured any relation to measurement or evaluation of care coordination or of any of the care coordination domains included within our measurement framework, with an emphasis on specific instruments or measures. Although the ambulatory setting is the focus of this project, we did not exclude sources discussing measurement of care coordination in nonambulatory settings. Validity, testing, or feasibility of measures were not considered during this review phase.

A single reviewer compiled the potential measures list after reviewing titles and abstracts of all search results. A second reviewer provided input on measure sources for which inclusion was unclear and a decision was made through discussion.

Based on this preliminary review, 149 potential measure sources were identified to consider for inclusion in the Atlas. It is important to note that in some cases multiple sources related to a single measure, and in other cases a single source discussed multiple measures.

In the second step, we reviewed the full text of all articles on the potential measure list and made decisions about whether they should be included within the Atlas. Measures were excluded if, in the opinion of the reviewer, they did not meet all of the following criteria:

  1. Clear relevance to care coordination or at least one of the care coordination measurement framework domains. Measures that did not include at least one instrument item that mapped to at least one framework domain were not included. Measures that contained only 1 or 2 minimally relevant items within a large instrument unrelated to care coordination were also excluded.
  2. A clearly defined and reproducible measure yielding quantitative data. Examples of evaluations that did not meet this criterion were interview guides, focus group reports, or free-response questionnaires yielding textual data that required content analysis; quality improvement guides designed to walk users through a process of self-evaluation without yielding measurable data; and evaluations of specific programs or interventions tailored to the subject of study in such a way as to make use in any other situation very difficult without major modification.
  3. Information available demonstrating some valid measurement properties or that the measure was developed in association with a logic model that has evidence of causal linkages between the activities measured and outcomes desired. Measures that underwent testing and were shown to have poor validity or reliability were not included in the Atlas.

In many cases, additional sources were consulted to address the testing criteria. When the decision about whether to include a measure seemed unclear, the primary reviewer consulted with additional team members and a decision was made through discussion.

Of the 149 potential measure sources identified, 70 were excluded. Of these, 31 were excluded due to lack of relevance (criterion 1); 34 were excluded because they were not a clearly defined, quantitative measure (criterion 2); and 38 were excluded due to unknown or poor validation or testing (criterion 3). Thirty-five potential measure sources met more than one exclusion criteria. In addition, 7 measures sources were excluded because we could not identify information necessary to assess suitability for inclusion in the Atlas. Our attempts to obtain the missing information from the developers of these measures were unsuccessful at the time of publication. A further 18 measure sources were not unique; that is, multiple sources pertained to a single measure. These sources were used to create the profiles but did not themselves contribute a unique measure.

In all, we include 61 measures in the Atlas, which are detailed in 78 profiles. The number of profiles is greater than the number of measures because for measures with multiple versions, we created separate profiles for versions with substantially different question items. In instances where the only difference between versions was a minor wording change to reflect a different population, setting, or year, we created just one profile to represent all versions.

Atlas Update Measure Selection

A similar process of measure selection was utilized during the Atlas update search. In total, we screened 1346 articles from the peer reviewed literature. Of these, we reviewed the full text for 157 articles, of which 19 yielded at least one potential measure. In addition, we reviewed a further 57 secondary sources, such as articles included within a systematic review. Together, these sources yielded 32 potential new measures that met the Atlas inclusion criteria. A further 133 potential measures were identified through the environmental scan, of which 65 were eligible for inclusion in the Atlas. The same Atlas measure selection criteria were used, with the following modifications:

  • Measures that were currently endorsed by NQF at the time of review, or that were based on evidence-based guidelines, were considered to meet the validity criterion.
  • Only measures applicable to the primary care setting were included in this update. This includes measures that are not setting specific, or that are applicable in primary care as well as other settings. Primary care was selected as a focus given its often central role in coordinating care across settings, particularly as accountable care and patient-centered medical home delivery models are more widely implemented. Furthermore, this focus aligns with the original scope of the Atlas that centered on measures that might reasonably be applied in the ambulatory care setting.
  • Measures tailored towards individuals with a particular disease or condition were not included in the update. This exclusion reflects the desire to focus on measures that are broadly applicable. Measures applicable to patients with a range of chronic diseases were included, but those applicable to only a particular disease were not.

In addition, measures relying exclusively on chart review or administrative claims data (i.e., not utilizing survey methods) were not included in the Atlas update. However, EHR-based measures are reviewed in the chapter on emerging trends in care coordination measurement, although individual profiles for these measures are not included.

In total, the Atlas update includes 20 additional measures in 22 profiles. A further 26 EHR-based measures are reviewed in Chapter 4.


Although we attempted to identify as many potential measures of care coordination as possible through our various search strategies, we relied primarily on published instruments available in the public domain. Instruments not published in journals were identified chiefly through suggestions from our review panels. This method omits an unknown number of potential measures that were not published in the literature, not identified by our search terms, or were not recommended for review by our advisory panels. When a potential measure of care coordination was reported in the literature without including the measure instrument, we contacted the article author to request a copy of it. We were also limited in our ability to provide information on the feasibility and cost of using measures by what was reported in the literature; few studies describe these aspects of measurement.

16National Quality Forum. Preferred practices and performance measures for measuring and reporting care coordination: a consensus report. Washington, DC: National Quality Forum; 2010.

Return to Jump Start

Page last reviewed June 2014
Page originally created June 2014
Internet Citation: Appendix II. Identifying Measures. Content last reviewed June 2014. Agency for Healthcare Research and Quality, Rockville, MD.