CAHPS Patient Narrative Item Sets

The CAHPS Patient Narrative Item Sets—also referred to as Narrative Elicitation Protocols—are sets of open-ended questions that prompt survey respondents to tell a clear and detailed story about their healthcare experiences. Patients’ narratives provide a valuable complement to standardized survey scores, both to help clinicians understand what they can do to improve their care and to engage and inform patients about differences among providers. The CAHPS narrative items generate insights into the topics addressed by the survey’s measures as well as other important aspects of patient experience that may not be captured by closed-ended questions.¹

AHRQ has released a Narrative Item Set for the Clinician & Group Survey. The CAHPS team is currently developing and testing Narrative Item Sets for two other CAHPS surveys:

• Child Hospital Survey.
• Health Plan Survey.

View the infographic: Narrative Item Set for the Clinician & Group Survey (PDF, 201.5 KB)

Clinician & Group Narrative Item Set

Originally known as the Narrative Elicitation Protocol, the first Narrative Item Set asks patients about their experiences in ambulatory care settings. The items were developed to be used with the CAHPS Clinician & Group Survey (CG-CAHPS) but they can also be used on their own. The original beta version of these items was released in December 2016. In June 2021, AHRQ released:

• A Narrative Item Set for the Clinician & Group Survey 3.0, which asks patients about their experiences over the last 6 months: These supplemental items are available in English and Spanish for both the Adult and Child versions of the survey.
• A Narrative Item Set for the Clinician & Group Visit Survey 4.0 (beta), which asks patients about their most recent visit: These supplemental items are available in English for the Adult version of the survey.

Questions in the Narrative Item Set for the Clinician & Group Survey

The CG-CAHPS Narrative Item Set has five questions in the Adult version and six questions in the Child version:

• The first question asks respondents to say what they are looking for in a health care provider and his or her staff.
• The second question asks the respondent to comment on the provider and his or her staff vis-à-vis those characteristics.
• The third and fourth questions ask the respondent to elaborate on what went well or could have been better.
• The fifth question asks about the relationship between the patient and the provider.

In the Child version, the fifth question asks specifically about the parent and provider's relationship; a sixth question asks about the relationship between the child and the provider.

Review the CG-CAHPS 3.0/3.1 Narrative Item Set for adults.

Review the CG-CAHPS 3.0/3.1 Narrative Item Set for children.

Review the CG-CAHPS Visit Survey 4.0 (beta) Narrative Item Set for adults.

Because of the sequential design of the questions, they are meant to be used together in the order specified. For example, the first question is designed to prompt respondents to state the most important things they look for in a health care provider so that they have those things in mind when answering the subsequent questions. The CAHPS team strongly recommends against reordering the questions or selecting a subset of the questions for partial use.

Administering the Clinician & Group Narrative Item Set

Organizations interested in using the CG-CAHPS Narrative Item Set can consult Administering the CAHPS Clinician & Group Narrative Item Set (PDF, 295 KB). This guidance is based on the testing of these items in real-world conditions and addresses the following issues:

• Placing the items in the survey.
• Using the items on their own.
• Introducing the items.
• Editing the items.
• Drawing a sample.
• Collecting the data.
• Invitations and reminder letters and emails.
• Anticipated response time.
• Anticipated length of comments.
• Oversight of vendors.

Using Comments from the CAHPS Narrative Items to Improve Care and Inform Consumers

Patient narratives can provide valuable information for both improving care and supporting consumer decisionmaking. For providers, narratives can reveal what is driving CAHPS survey scores and what specific processes and behaviors can be addressed to improve patient experience. For consumers, narratives can communicate information beyond CAHPS scores regarding what a patient’s experience with a clinician and their office staff is like, and how that compares to the patient’s own values and preferences.

Informing Providers:  Many health systems that are currently collecting open-ended responses as part of their patient surveys disseminate the verbatim comments to medical practice leaders and individual clinicians along with their survey scores.

The CAHPS team is conducting research to identify and better understand useful ways of reporting patient narrative information to clinicians and administrative staff. In collaboration with New York-Presbyterian's Patient Experience Team and Ambulatory Care Network, the team is assessing the feasibility, value, and use of the open-ended CAHPS items in selected ambulatory care practices. The research questions for this study include the following:

• How feasible is the collection of narratives using the CAHPS Narrative Item Set in routine patient experience survey operations?
• What is the added value of the Narrative Item Set compared to conventional open-ended questions?
• How can narrative information be reported to practice leaders, clinicians, and administrative staff in ways that are easily understood and useful for improving patient experience?

Informing Consumers: The CAHPS team conducted an experiment to explore whether and how including patient narratives in consumer reports might enhance consumers’ understanding of standardized measures of quality, better engage consumers in health care decisionmaking, and more effectively convey patient-reported experiences. Findings from this research suggest that narratives have the potential to increase consumers’ attention to and engagement with reports on physician quality. But narratives can also reduce consumers’ attention to standardized measures and lead to suboptimal doctor choices based on those measures.

Learn more: Kanouse DE, Schlesinger M, Shaller D, Martino SC, Rybowski L. How patient comments affect consumers’ use of physician performance measures. Medical Care 2016 January; 54(1):24-31.

In a second round of experiments, the CAHPS team sought to determine whether tagging patient narratives with short labels to indicate their content helps consumers to better integrate narratives and quality scores when making decisions about physicians. The team also explored the impact of providing access to a navigator, an individual trained to assist with the decision-making process. As in its first round of experiments, the team found that the narratives enhanced engagement with information on physician quality but led to a decline in decision quality. Labeling comments helped mitigate the decline in decision quality, although consumers’ choices were most consistent with their stated preferences when a navigator was present. Engagement with the quality information and satisfaction with available choices were likewise highest when a navigator was present.

Learn more: Martino SC, Grob R, Davis S, et al. Choosing Doctors Wisely: Can Assisted Choice Enhance Patients’ Selection of Clinicians? Med Care Res Rev 2017 Nov 25. Epub.

Learn More About the Clinician & Group Narrative Item Set (Elicitation Protocol)

• Webcast: Implementing the New CAHPS Protocol for Obtaining Patient Comments About Their Care (October 2018)
• Webcast: Introducing a Protocol To Obtain Patient Comments Using the CAHPS Clinician & Group Survey (January 2017)

Development of Additional Narrative Item Sets

AHRQ’s CAHPS team is currently developing and evaluating Narrative Item Sets that will be suitable for use with the CAHPS Child Hospital Survey and the CAHPS Health Plan Survey.

• Child Hospital Narrative Item Set: Like the closed-ended questions on the Child Hospital Survey, these items are designed to be answered by the parent or guardian of the child. This work could inform the future development of narrative items for other institutional settings as well as settings where proxies report on patient experience (such as hospice).
• Health Plan Narrative Item Set: These items are designed to be answered by health plan enrollees to provide detailed feedback on their experiences with the plan and its providers.

Analyzing Patients’ Comments

While patients’ comments have great value when read verbatim, the collection of large volumes of narrative responses requires some type of processing to more efficiently extract the core meaning of the information. Two basic types of processing can be used: (1) qualitative analysis methods using human coders, and (2) methods using computerized, automated analyses such as natural language processing (NLP).

In testing the CG-CAHPS Narrative Items, the CAHPS team used a relatively labor-intensive qualitative method to analyze the narrative data collected.² This method worked well for initial testing, and a number of health systems have used a similar approach to coding and labeling themes in their comments. However, at a large scale, this method may prove too costly or cumbersome to administer in a way that ensures consistent coding.

The CAHPS team has been studying the potential of NLP methods as a more cost-effective way to analyze patients’ comments. In October 2018, the CAHPS team convened a meeting of a Technical Advisory Group (TAG) to inform the team's research. One outcome of this meeting was an initial set of key principles and considerations regarding the automated narrative analysis of patient comments. In addition, the team conducted a study to explore and assess NLP approaches for purpose of analyzing narrative comments. In this study, the team used various NLP and machine-learning methods to recognize in a set of real-world patient narratives the presence or absence of a specified list of concepts.

Read a summary of the team’s study on using NLP to analyze patient comments.

Brief History

Over the last several years, there has been a tremendous growth in user-generated content about healthcare experiences. At the same time, Americans are increasingly seeking out online reviews of health care services. A study conducted by CAHPS researchers found that patient comments have become the form of physician quality information that Americans are most likely to see online.³ However, the unscientific way in which patients’ comments are commonly collected, whether by health care systems or provider review sites, tends to result in information that is neither representative of patients’ experiences nor a full account of those experiences.

Collecting comments as part of a standardized patient experience survey is one way to address some of these concerns because the comments, like the survey data, are collected from a random sample of confirmed patients. Thus, AHRQ is funding the development and testing of narrative items sets to support health care organizations in collecting patients’ comments in a systematic and structured way. The goal of this work has been to develop a method for collecting patient narratives that is as scientifically grounded and rigorous as the CAHPS closed-ended questions that are used to gather standardized data on patient experience.

Researchers affiliated with the Yale School of Public Health, RAND, and the University of Wisconsin-Madison have been working collaboratively to develop and test open-ended survey items that can be incorporated into the CAHPS Clinician & Group Survey. These researchers are now collaborating to develop and test items for the CAHPS Child Hospital Survey and the CAHPS Health Plan Survey.

Learn more about the development of the CAHPS Patient Narrative Elicitation Protocol.

For additional information, refer to:

• Grob R, Schlesinger M, Parker AM, et al. Breaking Narrative Ground: Innovative Methods for Rigorously Eliciting and Assessing Patient Narratives. Health Serv Res 2016 Jun;51 Suppl 2:1248–72.
• Schlesinger M, Grob R, Shaller D, et al. Taking Patients’ Narratives about Clinicians from Anecdote to Science. N Engl J Med 2015 Aug;373(7):675–679.

^1. Martino SC, Shaller D, Schlesinger M, et al. CAHPS and Comments: How Closed-Ended Survey Questions and Narrative Accounts Interact in the Assessment of Patient Experience. J Patient Exp 2017 Mar;4(1):37-45.

^2. The CAHPS team’s method involved the development of a coding scheme for identifying key content domains. (The coding scheme used in testing is available upon request to the CAHPS Help Line at cahps1@westat.com.) Responses to the five narrative questions were aggregated to create a single narrative. Trained researchers then coded the narrative on several dimensions, including the overall valence of the narrative and the number of times certain aspects of care were mentioned, and counted the number of words used. Reliability was established through agreement among the trained coders.  

^3. Schlesinger MJ, Rybowski L, Shaller D, et al. Americans' growing exposure to clinician quality information: Insights and implications. Health Aff (Millwood) 2019 Mar, 38(3): 374-382. https://pubmed.ncbi.nlm.nih.gov/30830827