Developing the Consumer Assessment of Healthcare Providers and Systems (CAHPS®) for Cancer Care Survey

Select to access the PowerPoint® presentation (355 KB).

Slide 1

Developing the CAHPS® for Cancer Care Survey

Steven Garfinkel, American Institutes for Research
Kathleen Yost, Mayo Clinic

AHRQ Annual Conference
Bethesda, Maryland
September 10, 2012

Project Cosponsored by the National Cancer Institute and the Agency for Healthcare Research and Quality under AHRQ Contract# HHSA290200600019I, Task Order# HHSA29032003T TO#9

Slide 2

Why Develop a CAHPS for Cancer Care Survey?

• Uniformly measure cancer care by providers from a patient perspective.
• Compare the care provided by different providers.
• Provide information for quality improvement.

Slide 3

CAHPS Development Process

Image: A chart shows the following process:

Formative Research:

• Call for Measures.
• Inventory items.
• Focus groups.
• Expert review.

→

Develop Draft:

• Draft domains.
• Draft items.
• Cognitive testing.
• Expert review.
• Revise survey.

→

Test and Finalize:

• Field test.
• Analyze data.
• Test report domains.
• Expert review.
• Finalize survey.
• Disseminate.

Slide 4

Formative Research

Slide 5

Item Inventory

• 1781 items:
  • CAHPS surveys.
  • Cancer surveys.
• All items coded into hypothesized domains:
  • e.g., affective communication, shared decisionmaking.
• Three independent reviewers of initial domains and items.

Slide 6

Focus Groups

16 focus groups to assess how consumers identify cancer providers and quality of care

• Stage 0/I (solid and common tumors).
• Stage II/III (solid and common tumors).
• Stage IV (any cancer).
• Rare cancers.
• Hematologic cancers.
• Spanish speaking.
• Family and Caregiver.

Image: A drawing shows three people having a discussion around a table.

Slide 7

Focus Group Topics

• Who are the key providers that patients focus on when they think about quality of care?
• What are key factors that patients use to determine quality of care?
• Are domains developed by the team relevant to patients?
• Which domains are most important?

Slide 8

Focus Group Findings

• Cancer care providers:
  • Oncologists and surgeons most often.
  • Other clinical staff: nurses, nurse practitioners, physician assistants, primary care physicians, and technicians.
  • Non-clinicians: support groups, family members, chaplains, and even a therapy dog.
• Perceived care coming from a provider and a team or teams.

Slide 9

Focus Group Findings, continued

• Top two domains of quality:
  • How providers interact with patients:
    • "They care about you. There's nobody else in the world but you."
    • "He's always been straightforward. He didn't sugarcoat."
  • Technical communication skills:
    • "My doctor explained to me that chemo is very hard on the body. He didn't want to jump in and do the chemo. When he found another lump in my neck then he said we have to treat it."

Slide 10

Focus Group Findings, continued

• Consumers also interested in:
  • Care coordination.
  • Access to information.
  • Providers who are up-to-date:
    • "There is so much research and so many new drugs coming out that you have to be up on it and pay attention to it because the more targeted the chemo the better."
  • Shared decision making.

Slide 11

Stakeholder Interviews

• Technical Expert Panel of clinicians and patients.
• Interviews with representatives of 5 clinical professional organizations.
• Interviews with representatives of 5 community cancer centers.
• Discussions with Mayo Clinic Oncology staff.

Slide 12

Stakeholder Findings

• Most cancer care is delivered through cancer centers:
  • Over 1,500 accredited by ACS Commission on Cancer.
• Survey will be most useful if it provides scores by major treatment modalities:
  • Medical, radiation, and surgical oncology.

Slide 13

Cognitive Testing

• Detects unexpected problems.
• Provides an underlying model of the questionnaire item response process.

Slide 14

Cognitive Testing Findings

• Need to define "team."
• Medical oncology = drug therapy.
• Specify "cancer" surgery to distinguish focal surgery from surgery for other conditions received by cancer patients.

Slide 15

Major Design Decisions

• Use the cancer center as the unit to assess.
• Focus on post-diagnosis and active treatment.
• Include inpatient and ambulatory experiences.
• Use 3-month reference period.
• Sample equally from medical, radiation, and surgical oncology patients.
• Parallel questionnaires for medical, radiation, and surgical oncology.

Slide 16

Example Item

• In the last 3 months, how often did your radiation therapy team listen carefully to you?
• In the last 3 months, how often did your cancer surgery team listen carefully to you?
• In the last 3 months, how often did your drug therapy team listen carefully to you?

Slide 17

Field Test

Slide 18

Field Test Activities

1. Site recruitment.
2. Eligibility and sampling.
3. Response rate and response bias analysis.
4. Evaluation of mode effect.
5. Comorbidity Index Study.
6. Item functioning.
7. Analysis to determine structure of composites.
8. Case-mix adjustment.

Slide 19

Field Test Design and Preliminary Results on Survey Operations

Slide 20

Purpose of Field Test

• To develop an assessment of the quality of cancer care that would obtain the CAHPS trademark:
  • Assess psychometric properties of survey:
    • Item non-response (including failed skip patterns).
    • Mode effects (mail vs. phone).
    • Composites.
    • Case-mix adjusters.
  • Assess ability of survey scores to discriminate quality of cancer care across sites.
  • Evaluate feasibility of administering the survey in different treatment settings:
    • Challenges of enumerating a sample.
    • Unit non-response.

Slide 21

Site Recruitment

• Initially intended to work with a clinical trials cooperative group.
• American College of Surgeons Commission on Cancer (CoC) expressed interest in possibly sponsoring the survey:
  • 6 CoC member organizations were recruited.
  • Large enough case volume to provide 750 eligible patients (250 per treatment modality) within data collection timeframe:
    • 300 completes per site (100 per treatment modality)
    • Assumed 40% response rate.
  • Geographic representation.
• Sampling frames sent to Mayo Clinic Survey Research Center:
  • Waiver of written HIPAA authorization at 5 sites.
  • Exception of Site 3.

Slide 22

Site Recruitment

Slide 23

Patient Eligibility

• Age 18 or older at the time of the recruitment material mailing.
• Received treatment for cancer in the 10 weeks prior to being selected for the sampling frame.
  • Treatment setting can be inpatient or outpatient.
  • Treatment includes surgery, medical oncology and radiation therapy.
  • Medical oncology includes chemotherapy, hormonal therapy, immunotherapy, and any other treatment for cancer delivered as a drug.
  • Treatment intent can be curative or palliative.
  • For radiation and medical oncology care, an additional criterion is that the patient had at least two visits to your organization for cancer treatment in the previous 10 weeks.
• In one of the following three phases of the cancer control continuum: receiving treatment for a new diagnosis, receiving treatment for a recurrence, or completed cancer treatment within the sampling window and is transitioning to survivorship.
• Not receiving care for squamous cell or basal cell skin cancer.

Slide 24

Data Collection

• Mayo Clinic Survey Research Center.
• Mail arm:
  • Goal—1500 completes.
  • Initial recruitment packet with letter and survey.
  • Second mailing to non-respondents after 3 weeks.
  • 3 attempted reminder phone call to non-respondents after 3 weeks.
    • No survey administration by phone if contact was made.
• Phone arm
  • Goal—300 completes.
  • Prenotification letter.
  • Phone interviews begin 2 weeks after prenotification.
  • 6 attempts—2 weekday day, 2 weekday night, 2 weekend across 2 weeks.
• Embedded study of comorbidity questions
• Double data-entry.

Slide 25

Data Collection Protocol for All Sites

Image: A chart shows the data collection protocol; phone and mail notifications are sent for each topic and 12 survey conditions are embedded.

Slide 26

Misidentified Cases

• Site 2:
  • 2 cases in cancer surgery arm self-identified as ineligible—did not receive surgery for treatment of cancer.
  • Unanticipated Problems Involving Risk to Subjects or Others (UPIRTSO).
    • Unanticipated risk of emotional harm.
  • Suspension of recruitment of cancer surgery patients at Sites 1 and 2 pending chart review.
  • Chart review found 1 additional case.
  • Follow-up protocol not completed at Site 2 due to IRB delay.
• Sites 3 - 6 changed IRB protocols to acknowledge misidentification as an anticipated risk.

Slide 27

Eligibility Determination Among Respondents

Slide 28

Response Rate

                Number of completed surveys
______________________________________________
total number respondents selected − (deceased + ineligible)

¹ Chi-Square

Slide 29

Distribution of Demographic and Cancer Type Among Responders and Nonresponder

Response Bias
Omitting observations from Site 2/Surgery due to break in protocol

¹ Chi-Square
² Kruskal Wallis

Slide 30

Quality of Care Questions That Differ by Mode of Survey Administration

Slide 31

Mode of Survey Administration (cont.)

¹ Chi-Square

Slide 32

Embedded Study of Comorbidity Questions

• 2 versions of survey:
  • Version 1: "Are you currently being treated by a health professional for..."
    • Diabetes/high blood sugar, high blood pressure/hypertension, heart condition, chronic lung disease/COPD/emphysema/chronic bronchitis, arthritis/joint problems, depression/anxiety.
    • Number of conditions reported:
      • 0 = 43.4%.
      • 1 = 30.2%.
      • 2 = 18.5%.
      • 3+ = 7.9%

Slide 33

Embedded Study

• Version 2:
  • Duration of care:
    • "In the last 12 months, did you get health care 3 or more times for a condition or problem?
    • Is this a condition or problem that has lasted for at least 3 months?"
  • Duration of prescription medication:
    • "Do you now need or take medicine prescribed by a doctor?"
    • "Is this medicine to treat a condition that has lasted for at least 3 months?"
  • Indices were defined as shown below:

Slide 34

Embedded Study

Version 2 indices results:

Slide 35

Composite Measure Analysis

1. Create analytic files.
2. Confirmatory test of original composite structure.
3. Identification and evaluation of alternative composite structures.
4. Evaluation of validity of best alternative.

Slide 36

Create Analytic Files

• Recode to facilitate interpretation:
  • For some questions higher numbered response indicated better care; for others, a higher numbered response indicated worse care.
  • Reverse-scored questions where higher indicated worse care so that higher always indicates better care.
• Impute missing values:
  • Multivariate analysis requires complete data for all variables in the analysis.
  • Used CAHPS-adopted method of multiple-imputation.

Slide 37

Original Composite Structure

*All questions in this domain are also listed under other domains.

Slide 38

Revised Composite Structure

Slide 39

Preliminary Composite Results

Slide 40

Affective Communication

• In the last 3 months:
  • How often did your radiation therapy team treat you with courtesy and respect?
  • How often did your radiation therapy team sHow respect for what you had to say?
  • How often did you feel your radiation therapy team really cared about you as a person?
  • How often did your radiation therapy team listen carefully to you?
  • How often was your radiation therapy team direct and straightforward when talking with you about your cancer and radiation therapy?
  • How often did your radiation therapy team spend enough time with you?

Slide 41

Shared Decision Making

• Since your cancer was diagnosed, did a doctor or other health care professional at [NAME OF CANCER CENTER]:
  • Clearly explain the advantages of each choice for cancer treatment, including the treatments you did not get?
  • Clearly explain the disadvantages of each choice for cancer treatment, including the treatments you did not get?
  • Ask for your opinion about each choice of cancer treatment, including the treatments you did not get?
  • Involve you in decisions about your cancer treatment as much as you wanted?

Slide 42

Communication About Therapy

• Since it was decided that you would have radiation therapy:
  • Did your radiation therapy team encourage you to contact them with questions between visits?
  • Tell you to call them immediately if you have certain symptoms or side effects?
  • Give you clear instructions about how to contact them outside of regular office hours?
  • Clearly explain how your cancer and radiation therapy could affect your normal daily activities?
• In the last 3 months, did your radiation therapy team:
  • Seem up-to-date about how to treat your type of cancer?
  • Tell you what the next steps in your radiation therapy would be?
  • Explain test results in a way that was easy to understand?
  • Explain what that medicine was for in a way that was easy to understand?

Slide 43

Enabling Patient Self-Management

• In the last 3 months, did your radiation therapy team:
  • Advise you about or help you deal with these changes in your energy levels?
  • Talk about any emotional problems, such as anxiety or depression, related to your cancer or radiation therapy?
  • Advise you about or help you deal with these emotional problems?
• In the last 3 months, did you and your radiation therapy team:
  • Talk about additional services to manage your cancer care at home, such as home health care, special medical equipment, or special supplies?
  • Talk about things you can do to maintain your health during cancer treatment such as what to eat and what exercises to do?

Slide 44

Evaluation of Composites

Slide 45

Questions Significantly, Uniquely Related to Overall Care
Net of 4 Composites

Q16. ...did your therapy team encourage you to contact them between visits?
Q18 ....did your therapy team give you clear instruction about how to contact them outside of regular office hours?
Q19. ...did your radiation therapy team clearly explain how your cancer and therapy could affect your normal daily activities?
Q34. How often were the blood tests,... scans, or other procedures scheduled to be done as soon as you thought you needed?
Q36. ...how often did your team explain test results in a way that was easy to understand?

Slide 46

Next Steps

• Continue data collection at 4 sites and analysis with additional support from the California HealthCare Foundation.
• Revise analysis.
• Final recommended composite structure.
• Final recommended questionnaire.

Slide 47

Challenges

• Extremely challenging project:
  • Focuses on a single clinical condition across multiple care sites.
• Choosing unit to be assessed:
  • Maximizing potential for adoption and sustainability.
  • Defining the care team—all or modality-specific.
• Sampling feasibility:
  • Determining patient eligibility.
  • Identifying sampling frames.

Slide 48

Acknowledgements

• AHRQ: Judy Sangl, Chris Crofton, Elma Chowdry.
• NCI: Neeraj Arora, Steve Clauser.
• CoC: Steve Edge, David Winchester, Andrew Stewart.
• Donna O’Brien and Arnold Kaluzny.

Slide 49

Preliminary Case Mix Adjusters

• Age.
• Education.
• Self-rating of Mental Health.
• Race: Other‡.
• Treatment/Services Received: Surgery (q58_3).
• Treatment/Services Received: none/ w&w (q58_6).
• Number Visits to Cancer Center for Appts (q21).
• How long have you been treated for cancer (q2).