Appendix I. Measure Mapping Strategy
Measure Mapping Procedures
Before beginning the mapping process, the research team developed domain definitions and the Measure Mapping Table (go to Table 4).
To begin the mapping process, all measures included in the Atlas were reviewed by one of two reviewers. Reviewers read through each measure, keeping in mind the specific components of care coordination that the measure addresses. Using the measure mapping table and the domain definitions, the reviewers identified the specific domains that correspond to the components of care coordination that each instrument measures. Reviewers also determined the perspective of measurement. When questions arose about appropriate mapping, the reviewers discussed and ultimately agreed upon a perspective and set of domains corresponding to each measure.
To assess inter-rater reliability of the measure mapping, reviewers selected at random 6 measures (approximately 10 percent of the total included in the Atlas) from among 31 measures that were planned for inclusion within the Atlas at the time the reliability testing was performed. The 19 measures included in an earlier draft Atlas were not considered for reliability testing because their mapping was discussed during development of the draft. The reviewers also did not consider reliability testing on those measures for which a final decision had not yet been made about whether it would be included in the Atlas, or measures that were missing key information (such as the measure instrument) at the time of reliability testing. Three measures were selected randomly from among those mapped by reviewer 1 (n=9 measures total) and three were selected randomly from among those mapped by reviewer 2 (n=22). Reliability was assessed before any discussion among the reviewers regarding the selected measures.
Across the 6 measures, there were 169 individual measure items (e.g., survey questions). Agreement about whether a specific item mapped to any domain was 86 percent (146/169), with a kappa of 0.694 (p<0.001). Conventionally, a kappa >0.67 is considered sufficient for drawing some conclusions. Therefore, we believe that the observed kappa of 0.69 is sufficient for the purposes of the measure mapping, which is intended to facilitate identification of relevant measures.
We also assessed reliability of mapping to the 3 perspectives: patient/family, health care professional(s), and system representative(s). Across 6 measures and 3 perspectives, there were 18 possible perspective mappings. (Each measure may be mapped to multiple perspectives). Reviewers agreed on all but one combination, resulting in 94 percent agreement. We did not calculate a kappa statistic because it is not an appropriate statistic when more than one mapping is possible for each measure.
Reliability of mapping to the framework domains was also assessed. To assess agreement of domain mapping across measurement items, only items that were mapped by both reviewers (n=101) were considered. Subdomains (e.g., Interpersonal Communication and Information Transfer) were considered as distinct domains for the purposes of reliability assessment.
Domain mapping agreement was examined in two ways. First, we examined agreement by domain. That is, what proportion of the 101 measure items did both reviewers agree should be mapped to each domain? Agreement in mapping to domains was good, ranging from 80 percent (Communicate) to 100 percent (Facilitate Transitions as Coordination Needs Change; Health Care Home; Health IT-Enabled Coordination).
Reliability of domain mapping was also assessed by comparing mapping across measure items. That is, how similar were each reviewer’s mappings for each item? For this comparison, the denominator was calculated by multiplying the total number of items mapped (n=101) by the total number of possible mappings (17 domains). Agreement was excellent. The reviewers agreed on 1604/1717 possible mappings, or 93 percent. As was the case for the perspective reliability assessment, a kappa statistic was not calculated because it is not an appropriate statistic when more than one mapping is possible for each measure.
Examples of Measure Item Mappings
The following list provides sample items (and their measure source) that were mapped to each care coordination domain on the measure mapping table. Copies of the measure instruments will be added to Appendix IV: Care Coordination Measures, currently under development. Appendix IV will be updated regularly.
Establish Accountability or Negotiate Responsibility
- I clarify whether the nurse or I will have the responsibility for discussing different kinds of information with the patient. [Measure #7b, item 10 (CPS)]
- How often were you confused about the roles of different providers? [Measure #6. item 9 (CPCQ)]
- Across health care teams or settings – How effective is one-to-one communication between ICU staff and members of other units? [Measure #12a. item VIIB.f (ICU Nurse-Physician Questionnaire)]
- Between health care professional(s) and patients/family – How often does your service provider talk with you about your future care? [Measure #6, item 27 (CPCQ)]
- Within teams of healthcare professionals – I discuss areas of agreement and disagreement with nurses in an effort to develop mutually agreeable health goals. [Measure #7b, item 5 (CPS)]
- Across health care teams or settings – Medical record transfer: IF a person age 75 or older is transferred between emergency rooms or between acute care facilities, THEN the medical record at the receiving facility should include medical records from the transferring facility, or should acknowledge transfer of such medical records. [Measure #2, item 11 (ACOVE-2 Quality Indicators)]
- Within teams of health care professionals – It is often necessary for me to go back and check the accuracy of information I have received from nurses in this unit. [Measure #12b, item I-4 (ICU Nurse-Physician Questionnaire)]
- Did your primary care provider (PCP) or someone working with your PCP help you make the appointment for that visit (referred to specialist)? [Measure #17a, item E9 (Primary Care Assessment Tool-Child Edition (PCAT-CE))]
As Coordination Needs Change
- In preparation for transition (to adulthood), does your provider have a process to share information with the adult care provider including: transition plans, medical records, key health issues, and current family and youth roles in managing care? [Measure #11a, item 4.2E (FCCSAT-Family Version)]
Assess Needs and Goals
- Before I left the hospital, the staff and I agreed about clear health goals for me and how these would be reached. (Y/N) [Measure #9b, item 1 (CTM-15)]
Create a Proactive Plan of Care
- When I left the hospital, I had a readable and easily understood written plan that described how all of my health care needs were going to be met. [Measure #9b, item 1 (CTM-15)]
Monitor, Follow Up, and Respond to Change
- In the past 3 months, how often have service providers responded appropriately to changes in your needs? [Measure #6, item 10 (CPCQ)]
- Diagnostic test followup: IF the outpatient medical record documents that a diagnostic test was ordered for a person age 75 or older, THEN the medical record at the followup visit should document 1 of the following: result of the test, test was not needed or reason why it will not be performed, test is still pending. [Measure #2, item 6 (ACOVE-2 Quality Indicators)]
- Does your partnership with your provider change over time as your experiences, knowledge, and skills change? [Measure #11a, item 1.8 (FCCSAT-Family Version)
Support Self-Management Goals
- When I left the hospital, I clearly understood the warning signs and symptoms I should watch for to monitor my health. (Y/N) [Measure #9b, item 6 (CTM-15)]
- In the past 3 months, how often did someone on your diabetes care team teach you how to take care of your diabetes? [Measure #21, item 7 (RSSM)]
Link to Community Resources
- Linking patients to outside resources: 1) is not done systematically; 2) is limited to a list of identified community resources in an accessible format; 3) is accomplished through a designated staff person or resource responsible for ensuring providers and patients make maximum use of community resources; or 4) is accomplished through active coordination between the health system, community service agencies, and patients. [Measure #1, item 7 (ACIC)]
Align Resources With Patient and Population Needs
- Do you and your staff: Offer trained interpretation (foreign language or sign)? [Measure #11b, item 13.1C (FCCSAT-Provider Version)]
- Is your facility able to change health care services or programs in response to specific health problems in the communities? [Measure #17c, item J4 (PCAT-FE)]
Teamwork Focused on Coordination
- When problems arise regarding the care of ____ patients, do care providers in these groups work with you to solve the problem? [Measure #46, item 4 (RCS)]
- Overall, our unit functions very well together as a team. [Measure #12a, item V.9 (ICU Nurse-Physician Questionnaire)]
Health Care Home
- Is there a doctor or place that you usually take your child if s/he is sick or you need advice about his/her health? [Measure #17b, item A1 (PCAT-AE)]
- Does anyone help you or coordinate [CHILD’S NAME]’s care among the different doctors or services [he/she] uses? (asked for children who used more than two services) [Measure #51, item K5Q20 (NSCH)]
- The pharmacist and I negotiate to come to an agreement on our activities in managing drug therapy. (Y/N) [Measure #18, item 7 (PPCI)]
Health IT-Enabled Coordination
- What is the policy timeframe for clinicians to respond to patient PHR emails?14 [Measure #34, item 10 (PHR)]
*Note 1: When the mode of communication was not clear, measures and measure items were mapped to the less specific Communicate domain rather than to either of the subdomains (Interpersonal Communication and Information Transfer).
†Note 2: We were able to map all measures related to transitions to one or the other of the subdomains specifying transition type (Facilitate Transitions Across Settings and Facilitate Transitions as Coordination Needs Change). Therefore, no measures or measure items were mapped to the less specific Facilitate Transitions domain.
14 PHR = Personal Health Record.
Return to Contents
Return to Jump Start Guide
Proceed to Next Section