Appendix II. Identifying Measures
Main Indicator Sources
- Literature Search. A measure-specific care coordination search was conducted to identify published literature related to the development, validation, and testing of measures of care coordination. The search strategy is outlined below.
- Care Coordination EPC Report. As part of a previously published care coordination report ("Closing the Quality Gap: A Critical Appraisal of Quality Improvement Strategies"; Volume 7: Care Coordination), background research and a systematic review identified care coordination indicators in published studies.
- Panelist Calls. A series of panel calls were held in order to obtain information regarding additional measures of care coordination and ongoing research and development in the field. Panel participants had backgrounds ranging from research and evaluation in care coordination to clinical practice. For a list of panel participants, please go to Appendix III: Advisory Group Participants.
- NQF Draft Report. NQF evaluated a list of 77 candidate measures and recommended a set of preferred practices across five domains of care coordination: 1) health care home, 2) proactive care plan, 3) communication, 4) information systems, and 5) transitions. The final report was released in October, 2010, shortly before completion of the Atlas.15
Literature Review Search Strategy
The final measure search was performed on July 13, 2010, using the following strategy:
[ ("(("healthcare" or "healthcare" or care) adj3 (coordinat* or "co-ordinat*" or integrat*)).tw.") And ("(rated or rating or indicator* or measure* or valid* or reliab* or outcome* or model* or scale* or subscale* or questionnaire*).tw. or methods.fs. or exp Questionnaires/") ] Not [ ("exp geographic locations/ not exp united states/") ]
The search was limited to English language publications. Details of the search strategy development are included in the box below.
In addition, publications by known key researchers involved in care coordination measurement were also searched. Bibliographies of particularly relevant included references were also reviewed for any further sources of information.
We compared our search strategy to RAND's ACOVE-3 search strategy post-hoc, and we found no additional terms, phrases, or combinations that were not captured in the strategy outline above.
Details of Search Strategy Development
With the help of a research librarian, a literature search was conducted using Ovid MEDLINE® In-Process & Other Non-Indexed Citations and Ovid MEDLINE® 1950 to Present. Several sets of search terms were used in combination to net articles describing care coordination measures and measurement strategies. Search sets 1-7 (shown below) represent the concept of 'care coordination'. Search sets 8-10 represent the concept of 'measures'. Search sets 11 and 12 represent the concept of 'outpatient' and 'medical home'. Search set 13 represents any citation indexed with a non-United States country subject heading and is used with the Boolean operator "NOT" to narrow the size of other search sets. Search set 14 is a high-precision title search using only the most relevant terms to "catch" obviously relevant citations the other searches might have missed.
Searches using combinations of the above sets were conducted. Searches were checked for article inclusion compared to a list of 10 highly relevant articles. The most effective search strategy was chosen based on inclusion rate and the total number of search results. The team determined that the final search strategy should yield no more than 4000 results, while simultaneously including as many of the 10 "test articles" as possible. The final search strategy used was: ((7 and 10) not 13) and eng.la. A search of the database through April 5, 2010, using this search set yielded 3306 publications and included 8 of the 10 test articles. The measure search was updated on July 13, 2010, to capture any additional measures indexed in MEDLINE® after the original search. The updated search yielded 8 new measures from among 142 new publications.
Search Set 1: "exp "Continuity of Patient Care"/" (10856 results)
Search Set 2: "exp *"Continuity of Patient Care"/" (5213 results)
Search Set 3: "disease management.de" (6824 results)
Search Set 4: "exp case management/ or "case manager*".mp. [mp=title, original title, abstract, name of substance word, subject heading word, unique identifier]" (7613 results)
Search Set 5: "patient centered care.de." (6153 results)
Search Set 6: "exp Delivery of Healthcare , Integrated/" (6135 results)
Search Set 7: "(("healthcare" or "healthcare" or care) adj3 (coordinat* or "co-ordinat*" or integrat*)).tw." (8073 results)
Search Set 8: "exp "Outcome and Process Assessment (Healthcare )"/ or exp Quality Indicators, Healthcare / or exp treatment outcome/ or exp quality of healthcare /" (3711934 results)
Search Set 9: "exp "Outcome and Process Assessment (Healthcare )"/ or exp Quality Indicators, Healthcare /" (484437 results)
Search Set 10: "(rated or rating or indicator* or measure* or valid* or reliab* or outcome* or model* or scale* or subscale* or questionnaire*).tw. or methods.fs. or exp Questionnaires/" (4889524 results)
Search Set 11: "exp ambulatory care/ or outpatient.mp. or ambulatory.mp. or (visit* adj3 (clinic or clinics)).mp. [mp=title, original title, abstract, name of substance word, subject heading word, unique identifier]" (156296 results)
Search Set 12: "("medical home" or pcmh).mp. [mp=title, original title, abstract, name of substance word, subject heading word, unique identifier]" (532 results)
Search Set 13: "exp geographic locations/ not exp united states/" (1661486 results)
Search Set 14: "((measure* or valid* or reliab* or outcome* or model* or scale* or subscale* or method* or questionnaire* or rated or rating or quality or indicator*) and ((coordinat* or "co-ordinat*" or integrat*) and (care or healthcare or "health-care"))).ti." (553 results)
10 Relevant Articles:
- Antonelli RC, Stille CJ, Antonelli DM. Care coordination for children and youth with special healthcare needs: A descriptive, multi-site study of activities, personnel costs and outcomes. Pediatrics 2008;122:e209-16.
- Rittenhouse DR, Casalino LP, Gillies RR, et al. Measuring the medical home infrastructure in large medical groups. Health Aff 2008;1246-58.
- McGuiness C, Sibthorpe B. Development and initial validation of a measure of coordination of healthcare. Int J Qual Health Care 2003;15(4):309-18.
- Coleman EA, Eilertsen TB, Magid DJ, et al. The association between care coordination and emergency department use in older managed care enrollees. Int J Integr Care 2002;2:e03.
- Cooley WC, McAllister JW, Sherrieb K, et al. The medical home index: Development and validation of a new practice-level measure of implementation of the medical home. Ambul Pediatr 2003;3:173-80.
- Zillich AJ, Doucette WR, Carter BL, et al. Development and initial validation of an instrument to measure physician-pharmacist collaboration from the physician perspective. Value Health 2005;8(1):59-66.
- Malouin RA, Starfield B, Sepulveda MJ. Evaluating the tools used to assess the medical home. Manage Care 2009 18(6): 44-8.
- Coleman EA, Smith JD, Frank JC, et al. Development and testing of a measure designed to assess the quality of care transitions. Int J Integr Care 2002;2:e02.
- Bethell CD, Read D, Brockwood K. Using existing population-based data sets to measure the American Academy of Pediatrics definition of the medical home for all children and children with special healthcare. Needs 2004;113:1229-537.
- Peikes D, Chen A, Schore J, et al. Effects of care coordination on hospitalization, quality of care, and healthcare expenditures among Medicare beneficiaries. JAMA 2009;301(6):603-18.
Measures for inclusion in the Atlas were identified in two steps. First, a list of potential measures was compiled from the search strategies outlined above, which yielded a total of 3448 unique measure sources. Measure sources were included if they featured any relation to measurement or evaluation of care coordination or of any of the care coordination domains included within our measurement framework, with an emphasis on specific instruments or measures. Although the ambulatory setting is the focus of this project, we did not exclude sources discussing measurement of care coordination in nonambulatory settings. Validity, testing, or feasibility of measures were not considered during this review phase.
A single reviewer compiled the potential measures list after reviewing titles and abstracts of all search results. A second reviewer provided input on measure sources for which inclusion was unclear and a decision was made through discussion.
Based on this preliminary review, 149 potential measure sources were identified to consider for inclusion in the Atlas. It is important to note that in some cases multiple sources related to a single measure, and in other cases a single source discussed multiple measures.
In the second step, we reviewed the full text of all articles on the potential measure list and made decisions about whether they should be included within the Atlas. Measures were excluded if, in the opinion of the reviewer, they did not meet all of the following criteria:
- Clear relevance to care coordination or at least one of the care coordination measurement framework domains. Measures that did not include at least one instrument item that mapped to at least one framework domain were not included. Measures that contained only 1 or 2 minimally relevant items within a large instrument unrelated to care coordination were also excluded.
- A clearly defined and reproducible measure yielding quantitative data. Examples of evaluations that did not meet this criterion were interview guides, focus group reports, or free-response questionnaires yielding textual data that required content analysis; quality improvement guides designed to walk users through a process of self-evaluation without yielding measurable data; and evaluations of specific programs or interventions tailored to the subject of study in such a way as to make use in any other situation very difficult without major modification.
- Information available demonstrating some valid measurement properties or that the measure was developed in association with a logic model that has evidence of causal linkages between the activities measured and outcomes desired. Measures that underwent testing and were shown to have poor validity or reliability were not included in the Atlas.
In many cases, additional sources were consulted to address the testing criteria. When the decision about whether to include a measure seemed unclear, the primary reviewer consulted with additional team members and a decision was made through discussion.
Of the 149 potential measure sources identified, 70 were excluded. Of these, 31 were excluded due to lack of relevance (criterion 1); 34 were excluded because they were not a clearly defined, quantitative measure (criterion 2); and 38 were excluded due to unknown or poor validation or testing (criterion 3). Thirty-five potential measure sources met more than one exclusion criteria. In addition, 7 measures sources were excluded because we could not identify information necessary to assess suitability for inclusion in the Atlas. Our attempts to obtain the missing information from the developers of these measures were unsuccessful at the time of publication. A further 18 measure sources were not unique; that is, multiple sources pertained to a single measure. These sources were used to create the profiles but did not themselves contribute a unique measure.
In all, we include 61 measures in the Atlas, which are detailed in 78 profiles. The number of profiles is greater than the number of measures because for measures with multiple versions, we created separate profiles for versions with substantially different question items. In instances where the only difference between versions was a minor wording change to reflect a different population, setting, or year, we created just one profile to represent all versions.
Although we attempted to identify as many potential measures of care coordination as possible through our various search strategies, we relied primarily on published instruments available in the public domain. Instruments not published in journals were identified chiefly through suggestions from our review panels. This method omits an unknown number of potential measures that were not published in the literature, not identified by our search terms, or were not recommended for review by our advisory panels. When a potential measure of care coordination was reported in the literature without including the measure instrument, we contacted the article author to request a copy of it. We were also limited in our ability to provide information on the feasibility and cost of using measures by what was reported in the literature; few studies describe these aspects of measurement.
15National Quality Forum. Preferred practices and performance measures for measuring and reporting care coordination: a consensus report. Washington, DC: National Quality Forum; 2010.
Return to Contents
Return to Jump Start Guide
Proceed to Next Section