Slide Presentation from the AHRQ 2008 Annual Conference
On September 9, 2008, Donna Farley, made this presentation at the 2008 Annual Conference. Select to access the PowerPoint® presentation (128 KB; Plugin Software Help).
Evaluating Consumer Assessment of Healthcare Providers and Systems (CAHPS®) Quality Improvement (QI) Demonstrations
Senior Health Policy Analyst, RAND
September 9, 2008.
Overview of the Presentation
- Goals for evaluating CAHPS® QI demonstrations.
- Conceptual Framework to guide evaluation.
- Process evaluation approach and methods.
- Outcome evaluation issues and options.
Goals for Evaluating CAHPS® QI Demonstrations
- Generate information on implementation experiences:
- Use by implementing organization to improve.
- Use by other organizations in their QI work.
- Assess effects of QI interventions:
- CAHPS® scores.
- Other outcomes and stakeholders.
- Understand which factors contribute to observed effects (or not).
- Compare results across demonstrations.
Major Evaluation Components
- Process Evaluation:
- Document and analyze QI intervention and implementation process.
- Identify factors influencing progress in achieving desired process changes.
- Outcome Evaluation
- Analyze effects of QI interventions on outcomes of interest to implementing organization.
How the Evaluation Addresses the Evaluation Goals
- Evaluation Goal—Evaluation Component.
- Goal 1: experiences—Process evaluation.
- Goal 2: effects—Outcome evaluation.
- Goal 3: factors for effects—Process & Outcome.
- Goal 4: comparison—Standard Methodology.
Conceptual Framework—An Evaluation Guide
Framework: CAHPS® Quality Improvement
The slide shows two nested rectangles. The inner rectangle represents "Organization Philosophy and Capacity." This involves:
- Executive Leadership.
- Implementation of QI Interventions:
- Team Leads.
- Involved Staff.
- Other Units.
The outer rectangle represents the "External Environment."
- Core activities:
- Change methods used.
- Process changes & cycles.
- Monitoring and feedback.
- Implementation synergies.
- Implementation experiences.
- Changes to clinical and operational processes (expected and actual).
Framework: Key Stakeholders
- Implementation team—champion, facilitator, team members.
- Higher level (e.g., organization leaders).
- Horizontal (e.g. other departments, services that coordinate with intervention).
- Directly affected or involved.
- Implementers—physicians, nurses, other clinical staff, administrative staff.
- End-users—patients, family members.
Framework: Organizational Philosophy
- Formal policies.
- Human resource practices.
- Decision-making authority.
- Reporting responsibilities.
- Role expectations.
- Culture of excellence.
- Patient-centered focus.
- Management approach and style.
Framework: Organizational Capacity
- Support service.
Framework: External Environment
- Laws and regulations.
- Credentialing policy.
- Reporting policies.
- Payment incentives.
- Perceived quality, costs, access.
- CAHPS® credibility.
- Public reports.
- Patient experience (CAHPS®).
- Organizational change.
- Program change.
- Employee effects.
Process Evaluation Methods
Types of Data Collected
- Descriptive (factual) data:
- Organizational environment.
- External environment.
- Decision process leading to the QI interventions.
- Strategy used to implement the interventions.
- Timeline of the implementation processes.
- Experiential data:
- Differing views of stakeholders.
- Perceptions of progress of the QI interventions.
- How QI interventions affecting them.
Data Collection Instruments
- Checklist of descriptive data to collect:
- Structured according to the framework.
- Multiple sources of data—written materials, discussions with QI leads, interviews.
- Implementation timeline form:
- Shows planned implementation schedule.
- Updated as QI work proceeds.
- Standard protocol for stakeholder Interviews.
Interview Grid for Comparative Data on Stakeholders Perspectives
The slide shows the framework for a table.
- Framework Component:
- Organization Philosophy.
- Organization Capacity.
- External Environment.
- Effects on Outcomes.
- Actual Progress: Successes/Challenges.
- Effects on You.
- Effects on Others.
Outcome Evaluation Issues and Design Options
Challenges in Measuring Effects of CAHPS® QI Interventions
- Difficulty in "moving" CAHPS® scores:
- Scores are composites of several items.
- QI interventions often address only some items.
- Time required to make practices change.
- Time required to change patients' perceptions.
- Difficulty in attributing effects to QI intervention:
- Many initiatives are in just one organization.
- Others are in many (e.g. medical practices).
- External control groups may not be good controls.
- Need for process information to interpret effects.
Design Options for Outcome Evaluations
- Use control groups to control confounding factors.
- Allows attribution to intervention.
- Controls may not control for confounders.
- Differences by degree of implementation:
- Classify participating groups (e.g. practices) by degree of implementation and compare.
- May not measure implementation accurately.
- Compare each entity to itself over time:
- Control for confounders but not temporal changes.
- Small N for analysis and power needs.
- Need for multi-dimensional information leads to complex evaluation requirements.
- Ultimate goal is to learn how QI interventions affected patient experience, as measured by CAHPS® scores.
- But implementers also need feedback to improve intervention actions.
- Process evaluation must collect good comparative data to serve all these needs.
Current as of January 2009
Evaluating CAHPS® Quality Improvement Demonstrations. Slide Presentation from the AHRQ 2008 Annual Conference (Text Version). January 2009. Agency for Healthcare Research and Quality, Rockville, MD. http://www.ahrq.gov/about/annualmtg08/090908slides/Farley.htm