CAHPS Patient Narrative Item Sets

The CAHPS Patient Narrative Item Sets—also referred to as Narrative Elicitation Protocols—are sets of open-ended questions that prompt survey respondents to tell a clear and detailed story about their healthcare experiences. Patients’ narratives provide a valuable complement to standardized survey scores, both to help clinicians understand what they can do to improve their care and to engage and inform patients about differences among providers. The CAHPS narrative items generate insights into the topics addressed by the survey’s measures as well as other important aspects of patient experience that may not be captured by closed-ended questions.¹

Currently available:

• Narrative items for the Clinician & Group Survey.
• Narrative items for the Child Hospital Survey (Child HCAHPS).  

Why Narrative Item Sets?

Over the last decade, there has been a tremendous growth in user-generated content about healthcare experiences. At the same time, Americans are increasingly seeking out online reviews of health care services. A study conducted by CAHPS researchers found that patient comments have become the form of physician quality information that Americans are most likely to see online.³ However, the unscientific way in which patients’ comments are commonly collected, whether by health care systems or provider review sites, tends to result in information that is neither representative of patients’ experiences nor a full account of those experiences.

Collecting comments as part of a standardized patient experience survey is one way to address some of these concerns because the comments, like the survey data, are collected from a random sample of confirmed patients in a systematic and structured way. Thus, AHRQ has been funding the development and testing of narrative items sets to support health care organizations in eliciting short, salient narratives from patients about their experiences with care. The goal of this work has been to develop a method for collecting patient narratives that is as scientifically grounded and rigorous as the CAHPS closed-ended questions that are used to gather standardized data on patient experience.

Researchers affiliated with the Yale School of Public Health, RAND, and the University of Wisconsin-Madison have been working collaboratively to develop and test open-ended questions that meet the standards of and are compatible with CAHPS surveys. In the initial phase of the research, the team focused on specifying the characteristics of a “high-quality” patient narrative and developing strategies for measuring those characteristics. Drawing on the literature of narrative medicine, the development emphasized the following characteristics:

• Completeness: The narratives create a full picture of the experiences that matter to each patient.
• Balance: They reflect both positive and negative aspects in proportion to each patient's experiences.
• Meaningfulness: They help the reader picture the patient’s experience with the clinician.
• Representativeness: They include experiences from patients across a range of health status and socio-demographic characteristics.

For more information about this work and the methods used to develop the items, please refer to:

• Grob R, Schlesinger M, Parker AM, et al. Breaking Narrative Ground: Innovative Methods for Rigorously Eliciting and Assessing Patient Narratives. Health Serv Res 2016 Jun;51 Suppl 2:1248–72.
• Schlesinger M, Grob R, Shaller D, et al. Taking Patients’ Narratives about Clinicians from Anecdote to Science. N Engl J Med 2015 Aug;373(7):675–679.

Using Comments from the CAHPS Narrative Items to Improve Care and Inform Consumers

Patient narratives can provide valuable information for both improving care and supporting consumer decisionmaking. For providers, narratives can reveal what is driving CAHPS survey scores and what specific processes and behaviors can be addressed to improve patient experience. For consumers, narratives can communicate information beyond CAHPS scores regarding what a patient’s experience with a clinician and their office staff is like, and how that compares to the patient’s own values and preferences.

Using Narratives to Inform Providers

Many health systems that are currently collecting open-ended responses as part of their patient surveys disseminate the verbatim comments to medical practice leaders and individual clinicians along with their survey scores.

The CAHPS team conducted research to identify and better understand useful ways of reporting patient narrative information to clinicians and administrative staff. In collaboration with New York-Presbyterian's Patient Experience Team and Ambulatory Care Network, the team assessed the feasibility, value, and use of the open-ended CAHPS items in selected ambulatory care practices. The research questions for this study included the following:

• How feasible is the collection of narratives using the CAHPS Narrative Item Set in routine patient experience survey operations?
• What is the added value of the Narrative Item Set compared to conventional open-ended questions?
• How can narrative information be reported to practice leaders, clinicians, and administrative staff in ways that are easily understood and useful for improving patient experience?

Learn more:

Learning from Patient Narratives Through Innovative Feedback Reporting Methods (Webcast) (May 31, 2023)

Learning from Patient Narratives Through Innovative Feedback Reporting Methods (Webcast)  (1:01:36)

The Power of Patient Stories for Improving the Patient Experience (Webcast) (May 12, 2022)  

The Power of Patient Stories for Improving the Patient Experience (Webcast) (1:00:26)

Using Narratives to Inform Consumers

The CAHPS team conducted an experiment to explore whether and how including patient narratives in consumer reports might enhance consumers’ understanding of standardized measures of quality, better engage consumers in health care decisionmaking, and more effectively convey patient-reported experiences. Findings from this research suggest that narratives have the potential to increase consumers’ attention to and engagement with reports on physician quality. But narratives can also reduce consumers’ attention to standardized measures and lead to suboptimal doctor choices based on those measures.

Learn more: Kanouse DE, Schlesinger M, Shaller D, Martino SC, Rybowski L. How patient comments affect consumers’ use of physician performance measures. Medical Care 2016 January; 54(1):24-31.

In a second round of experiments, the CAHPS team sought to determine whether tagging patient narratives with short labels to indicate their content helps consumers to better integrate narratives and quality scores when making decisions about physicians. The team also explored the impact of providing access to a navigator, an individual trained to assist with the decision-making process. As in its first round of experiments, the team found that the narratives enhanced engagement with information on physician quality but led to a decline in decision quality. Labeling comments helped mitigate the decline in decision quality, although consumers’ choices were most consistent with their stated preferences when a navigator was present. Engagement with the quality information and satisfaction with available choices were likewise highest when a navigator was present.

Learn more: Martino SC, Grob R, Davis S, et al. Choosing Doctors Wisely: Can Assisted Choice Enhance Patients’ Selection of Clinicians? Med Care Res Rev 2017 Nov 25. Epub.

Analyzing Patients’ Comments

While patients’ comments have great value when read verbatim, the collection of large volumes of narrative responses requires some type of processing to more efficiently extract the core meaning of the information. Two basic types of processing can be used: (1) qualitative analysis methods using human coders, and (2) methods using computerized, automated analyses such as natural language processing (NLP).

In testing the CAHPS Narrative Items, the CAHPS team used a relatively labor-intensive qualitative method to analyze the narrative data collected.² This method worked well for initial testing, and a number of health systems have used a similar approach to coding and labeling themes in their comments. However, at a large scale, this method may prove too costly or cumbersome to administer in a way that ensures consistent coding.

The CAHPS team has been studying the potential of NLP methods as a more cost-effective way to analyze patients’ comments. In October 2018, the CAHPS team convened a meeting of a Technical Advisory Group (TAG) to inform the team's research. One outcome of this meeting was an initial set of key principles and considerations regarding the automated narrative analysis of patient comments. In addition, the team conducted a study to explore and assess NLP approaches for purpose of analyzing narrative comments. In this study, the team used various NLP and machine-learning methods to recognize in a set of real-world patient narratives the presence or absence of a specified list of concepts.

Read a summary of the team’s study on using NLP to analyze patient comments.

^1. Martino SC, Shaller D, Schlesinger M, et al. CAHPS and Comments: How Closed-Ended Survey Questions and Narrative Accounts Interact in the Assessment of Patient Experience. J Patient Exp 2017 Mar;4(1):37-45.

^2. The CAHPS team’s method involved the development of a coding scheme for identifying key content domains. (The coding scheme used in testing is available upon request to the CAHPS Help Line at cahps1@westat.com.) Responses to the five narrative questions were aggregated to create a single narrative. Trained researchers then coded the narrative on several dimensions, including the overall valence of the narrative and the number of times certain aspects of care were mentioned, and counted the number of words used. Reliability was established through agreement among the trained coders.  

^3. Schlesinger MJ, Rybowski L, Shaller D, et al. Americans' growing exposure to clinician quality information: Insights and implications. Health Aff (Millwood) 2019 Mar, 38(3): 374-382. https://pubmed.ncbi.nlm.nih.gov/30830827