Development of the CAHPS Patient Narrative Elicitation Protocol
AHRQ funded the development of the Patient Narrative Elicitation Protocol by researchers affiliated with the Yale School of Public Health, RAND, and the University of Wisconsin-Madison. The goal of this work has been to develop and test a scientifically rigorous design for open-ended survey items that can be administered as part of the CAHPS Clinician & Group Survey to elicit short, salient narratives from patients about their experiences with clinicians.
Design phase. In the initial phase of the research, the team focused on specifying the characteristics of a “high-quality” patient narrative and developing strategies for measuring those characteristics. Drawing on the literature of narrative medicine, the development emphasized the following characteristics:
- Complete: The narratives create a full picture of the experiences that matter to each patient.
- Balanced: They reflect both positive and negative aspects in proportion to each patient's experiences.
- Meaningful: They help the reader picture the patient’s experience with the clinician.
- Representative: They include experiences from patients across a range of health status and socio-demographic characteristics.
Development of the initial protocol. The first version of the Elicitation Protocol was developed based on previous patient narratives research. The goal was to design a sequence of questions applicable to and readily understood by patients with more or less complex health care experiences, as well as those with various levels of health literacy. Initial versions of the questions were tested with roughly 30 respondents, and wording was revised accordingly.
Experimental testing of the protocol. To assess the performance of the initial item set, the team conducted an experimental test by administering the CAHPS Clinician & Group Adult Survey in English with the short set of open-ended questions to a sample of 443 respondents using both telephone and Web-based response modes. The sample was diverse in terms of both socio-demographics and health status. To evaluate the narrative information collected by the survey questions, the team compared the content to information obtained during hour-long interviews with a subsample of 50 survey respondents who were interviewed 2 to 3 weeks later. By carefully coding all of the content, the research team was able to assess the degree to which the information in the open-ended survey responses matched the "real story" obtained through the interviews.
Based on the results of this first experiment, the team revised the Elicitation Protocol. To improve the flow of the narrative, the team restructured the questions into a more "storytelling" mode, starting with a question on what patients expect or seek from their health care providers. To promote more detail, particularly in the written responses, additional prompts to the third and fourth questions were added in the sequence. Finally, to remind respondents to comment on the nature of their relationships with clinicians (and staff) as well as the events that comprised their medical experiences, the team revised the fifth question to focus explicitly on how patients relate to their provider.
In a second round of experimental testing, the team assessed responses to these revised items. This testing was conducted with 332 respondents, with about a third having an ongoing health problem, a third having a recent serious health problem, and a third having neither problem. As in the first round, a subsample of 50 respondents participated in intensive interview by phone. The changes to the item set resulted in an increase in the completeness of elicitations from roughly 40 percent to approximately 60 percent of the content gleaned from the hour-long interviews. Information about both patient expectations and their communication with clinicians was substantially deepened using the revised protocol.
More information on the development of the Elicitation Protocol is available in Grob R, Schlesinger M, Parker AM, et al. Breaking Narrative Ground: Innovative Methods for Rigorously Eliciting and Assessing Patient Narratives. Health Serv Res 2016 April; 51:1475-6773.
Field testing. To assess how well the Elicitation Protocol performs outside of the restrictions of an experiment and better understand how the responses can be used, the CAHPS team field-tested the protocol in a pilot project conducted by the California Healthcare Performance Information System (CHPI) and Massachusetts Health Quality Partners (MHQP). This pilot found that:
- 18–25% of respondents agreed to answer open-ended responses in a scenario in which they had to actively opt-in to do so after the closed-ended survey questions.
- The additional questions did not exhaust respondents’ enthusiasm for participating in the survey. Virtually all of those respondents completed the full set of five items; response rates for those items declined only 0.5 percentage points from the first question to the fifth.
- The five-question protocol performed substantially better at capturing patient experiences than a shorter, alternative sequence of three open-ended questions.
Beta version. The beta version of the CAHPS Narrative Elicitation Protocol described in this document was released in early 2017. The CAHPS team is continuing to investigate ways to refine the questions, strategies for encouraging survey participation, and methods for most effectively analyzing and reporting the information obtained through comments to make them useful to clinicians and other patients.
Page originally created December 2016