Results of ARRA-Funded CER Dissemination Activities

Text version of a slide presentation by Sari Siegel, Ph.D., representing the final report from the ARRA-funded evaluation contract with IMPAQ.

Contents

Slide 1. High-Level Data Analysis Presentation Slide Deck
Slide 2. Overview
Slide 3. Quarterly Metrics
Slide 4. Quarterly Metrics
Slide 5. Cumulative Exposures by Dissemination Strategy
Slide 6. Cumulative Exposures from All Strategies by Audience
Slide 7. Academic Detailing
Slide 8. Continuing Education
Slide 9. Media and Marketing
Slide 10. Dissemination Partnerships
Slide 11. Virtual Centers
Slide 12. Continuing Education Retention Test Analysis
Slide 13. Continuing Education Retention Tests
Slide 14. Continuing Education Retention Tests: Confidence in Applying PCOR
Slide 15. Continuing Education Retention Tests: Perceived Value of PCOR
Slide 16. Web and Clearinghouse Reports
Slide 17. EHC Program Web Site Visits
Slide 18. Total Web Site Visits Since Start of Dissemination Contracts
Slide 19. Clearinghouse Results
Slide 20. Distribution of Major Orders Placed Using Contractor Codes
Slide 21. Consumer and Clinician Surveys
Slide 22. Surveys: Background
Slide 23. Consumer Survey Findings (Wave 1 )
Slide 24. Consumer Survey Findings (Wave 2)
Slide 25. Consumer Survey Findings (Longitudinal)
Slide 26. Consumer Survey Findings (Longitudinal con't)
Slide 27. Clinician Surveys
Slide 28. Clinician Survey Findings (Wave 1)
Slide 29. Clinician Survey Findings (Wave 2)
Slide 30. Clinician Survey Findings (Longitudinal)
Slide 31. Clinician Survey Findings (Longitudinal con't)
Slide 32. Focus Groups
Slide 33. Focus Groups
Slide 34. Consumer Focus Group Findings
Slide 35. Clinician Focus Group Findings
Slide 36. Business Leader Findings
Slide 37. Summary
 


Slide 1: High-Level Data Analysis Presentation Slide Deck

Text Description is below the image.

October 15, 2013

Sari Siegel, Ph.D.
Project Director
ARRA CER Dissemination Evaluation

Return to Contents


Slide 2: Overview

Text Description is below the image.

  • Quarterly Metrics.
  • Continuing Education Retention Test Analysis.
  • Web and Clearinghouse Report.
  • Survey:
    • Consumer.
    • Clinician.
  • Focus Groups:
    • Consumer.
    • Clinician.
    • Business.
    • Leader/Purchaser.
  • Summary.

Return to Contents


Slide 3: Quarterly Metrics

Text Description is below the image.

Reach and Dissemination

Return to Contents


Slide 4: Quarterly Metrics

Text Description is below the image.

  • As part of the ARRA CER Dissemination Evaluation, IMPAQ/Battelle collects:
    • Quarterly metrics on dissemination activities and audience exposure to PCOR from the four dissemination contractors (Academic Detailing, Continuing Education, National Initiative, and Regional Offices).
    • Quarterly data on EHC Program Web site traffic and AHRQ Publications Clearinghouse orders.

Return to Contents


Slide 5: Cumulative Exposures by Dissemination Strategy

Text Description is below the image.

Image: A series of text boxes connected by arrows show dissemination activities and resulting exposures across the different strategies:

Academic Detailing → 10,203 Detailing Visits → 11,667 Exposures
Continuing Education → 45 CE Modules → 61,297 Exposures
Media and Marketing → 133 Completed Projects → 111,967 Exposures
Partnerships → 1,417 Partner Activities → 38,949,853 Exposures
Virtual Centers → 20 Virtual Centers → 1,870,663 Exposures

Return to Contents


Slide 6: Cumulative Exposures from All Strategies by Audience

Text Description is below the image.

Audience Total Exposures
Consumers 7,131,478
Clinical Decisionmakers 5,270,264
Health Systems Decisionmakers 2,022
Other Health Care Professionals 28,901
Unspecified 28,572,782
Total 41,005,447

Return to Contents


Slide 7: Academic Detailing

Text Description is below the image.

Image: This line graph shows "snapshots" of the Academic Detailing activities as reported each quarter through the final data collection period of this contract. The values are presented in the table below.

 

  FY11 Q2 FY11 Q3 FY11 Q4 FY12 Q1 FY12 Q2 FY12 Q3 FY12 Q4 FY13 Q1 FY13 Q2 FY13 Q3 Plus July
Unique Sites 98 443 429 661 678 685 753 680 900 1,483
Detailing Visits 98 443 471 1,099 1,060 1,196 1,437 1,147 1,145 2,107
Exposures 145 583 608 1,199 1,228 1,414 1,599 1,324 1,289 2,278

Return to Contents


Slide 8: Continuing Education

Text Description is below the image.

Image: This line graph shows "snapshots" of Continuing Education activities as reported each quarter through the final data collection period of this contract. The values are presented in the table below.

  FY11 Q2 FY11 Q3 FY11 Q4 FY12 Q1 FY12 Q2 FY12 Q3 FY12 Q4 FY13 Q1 FY13 Q2 FY13 Q3 Plus July
Modules Released 4 2 6 5 10 9 8 0 0 1
Learners This Period 0 1,061 1,745 2,685 3,307 3,076 3,348 3,296 3,297 3,805
Course Completions 0 1,776 2,951 5,187 6,779 8,280 8,672 9,402 8,902 9,348

Return to Contents


Slide 9: Media and Marketing

Text Description is below the image.

Image: Two line graphs show "snapshots" of media and marketing activities as reported each quarter through the final data collection period of this contract. The values are presented in the table below.

 

  FY11 Q2 FY11 Q3 FY11 Q4 FY12 Q1 FY12 Q2 FY12 Q3 FY12 Q4 FY13 Q1 FY13 Q2 FY13 Q3 Plus July
Completed Projects 1 6 9 15 4 16 19 15 18 30
Potential Exposures - - 4,086,519 9,465,714 1,874,995 4,307,154 4,637,516 4,723,930 3,835,529 66,303,384
Documented Exposures 46 1,625 5,032 18,040 6,523 13,567 13,094 10,481 8,566 34,993

Return to Contents


Slide 10: Dissemination Partnerships

Text Description is below the image.

Image: Two line graphs show "snapshots" of disseminating partnership activities as reported each quarter through the final data collection period of this contract. The values are presented in the table below.

 

  FY11 Q2 FY11 Q3 FY11 Q4 FY12 Q1 FY12 Q2 FY12 Q3 FY12 Q4 FY13 Q1 FY13 Q2 FY13 Q3 Plus July
Partners with Activity 0 9 71 78 69 90 115 142 101 218
Activities 0 9 120 103 92 139 152 233 144 425
Exposures 292 2,030,451 1,742,282 4,194,675 3,937,185 3,290,273 3,352,345 7,923,213 740,654 11,738,483

Return to Contents


Slide 11: Virtual Centers

Text Description is below the image.

  Fy13 Q2 FY
New Virtual Centers Launched this Period 9 11
Potential Exposures 11,870,401 57,868,397
Documented Exposures 637,204 1,233,459

Return to Contents


Slide 12: Continuing Education Retention Test Analysis

Text Description is below the image.

Continuing Education Retention Test Analysis

Return to Contents


Slide 13: Continuing Education Retention Tests

Text Description is below the image.

  • Learners were generally more likely to answer content questions correctly after taking a CE module.
  • More learners rated their confidence in applying PCOR as "high" after taking a module.
  • More learners rated their perceived value of PCOR as "high" after taking a module.

Return to Contents


Slide 14: Continuing Education Retention Tests: Confidence in Applying PCOR

Text Description is below the image.

Image: This set of pie graphs shows confidence in applying PCOR in continuing education retention tests for two periods: April 2011 through June 2012 and July 2012 through June 2013.

Return to Contents


Slide 15: Continuing Education Retention Tests: Perceived Value of PCOR

Text Description is below the image.

Image: This set of pie graphs shows perceived value of PCOR in continuing education retention tests for two periods: April 2011 through June 2012 and July 2012 through June 2013. The majority of respondents rated their perceived value of PCOR as "high" before taking a module, so there were fewer respondents who could "change their mind" and move from "low" to "high" perceived value.

Return to Contents


Slide 16: Web and Clearinghouse Reports

Text Description is below the image.

Web and Clearinghouse Reports

Return to Contents


Slide 17: EHC Program Web Site Visits

Text Description is below the image.

CY 2009* through CY 2013 Totals

Calendar Year Total Number of Visits
2009 (Nov-Dec) 86,739
2010 567,650
2011 858,922
2012 1,822,343
2013 (through July) 1,903,058

* The Eisenberg Center took over the EHCP Web site contract in November 2009; CY 2010 is the full first year of data available.

Return to Contents


Slide 18: Total Web Site Visits Since Start of Dissemination Contracts

Text Description is below the image.

Image: This graph displays the trend in Web site visits during each of the quarterly periods since the dissemination contracts (and activities) began. The values are presented in the table below.

Fiscal Year Number of Visits
FY11 Q2 172,582
FY11 Q3 171,281
FY11 Q4 227,102
FY12 Q1 287,957
FY12 Q2 356,218
FY12 Q3 353,407
FY12 Q4 466,152
FY13 Q1 646,566
FY13 Q2 764,458
FY13 Q3* 1,138,600

* This total includes visits from April-July 2013.

Return to Contents


Slide 19: Clearinghouse Results

Text Description is below the image.

  • Between January 2011 and July 2013:
    • Audience members placed 661 major orders (representing 56 percent of all major orders in this time frame) to the Publications Clearinghouse using contractor codes.
    • Result: 171,665 ordered units (53 percent of the product volume from large orders in this time frame).

Return to Contents


Slide 20: Distribution of Major Orders Placed Using Contractor Codes

Text Description is below the image.

Image: This slide presents a visualization of both the five regions of this CER dissemination effort, as well as the distribution of large Clearinghouse orders (200 or more units) placed using a dissemination contractor codes. The largest number of orders and the largest volume of units came from orders placed by audience members in the Atlanta area. Audience members used the Academic Detailing code most often, but ordered the largest volume of units using the Regional Offices code.

Return to Contents


Slide 21: Consumer and Clinician Surveys

Text Description is below the image.

Understanding, Knowledge, Use, and Benefits

Return to Contents


Slide 22: Surveys: Background

Text Description is below the image.

  • Team developed one instrument for consumers and one for clinicians.
  • Data collection was completed at two points in time one year apart (Wave 1 and Wave 2).
  • Surveys were designed to assess:
    • Levels of awareness, understanding, use, and perceived benefits of CER.
    • If and how levels of awareness, understanding, use, and perceived benefits of CER have changed.
    • Trends in awareness of AHRQ and its EHC Program.

Return to Contents


Slide 23: Consumer Survey Findings (Wave 1)

Text Description is below the image.

  • 18% of surveyed consumers reported awareness of CER when prompted with a definition.
  • Of those who indicated aided awareness of CER, just over half (56%) currently use the research to help make medical decisions.
  • The most common sources of information of CER were health care providers (29%), Web sites (26%), and print media (24%).
  • Very few surveyed consumers had heard of AHRQ (4%) or the EHC Program (4%); just under 1% had used its products.

Return to Contents


Slide 24: Consumer Survey Findings (Wave 2)

Text Description is below the image.

  • 21% of surveyed consumers reported awareness of CER when prompted with a definition.
  • Of those who indicated aided awareness of CER, just over half (52%) currently use the research to help make medical decisions.
  • The most common sources of information of CER were printed media (21%), Web Sites (19%), television/radio (19%) and health care providers (18%).
  • Very few surveyed consumers had heard of AHRQ (11%) or the EHC Program (7%); just under 3% had used its products.

Return to Contents


Slide 25: Consumer Survey Findings (Longitudinal)

Text Description is below the image.

  • Consumers' unaided and aided awareness of CER increased over time (61% to 65%, 18% to 21% respectively), although the increase was not statistically significant.
  • Consumers reported a statistically significant increase in awareness of AHRQ from wave 1 to wave 2 (4% to 11%).
  • Consumers reported an increased awareness of the EHC Program from Wave 1 to Wave 2 (4% to 7%), although this increase was not statistically significant.

Return to Contents


Slide 26: Consumer Survey Findings (Longitudinal con't)

Text Description is below the image.

  • Consumer interest in learning more about CER increased over time (37% to 51%) and was statistically significant.
  • There was a statistically significant increase in interest in learning about the EHC Program over time (44% to 58%).
  • Consumers reported an increased intention to use AHRQ's products or other studies to inform decisionmaking (39% to 45%), although the increase was not statistically significant.

Return to Contents


Slide 27: Clinician Surveys

Text Description is below the image.

Understanding, Knowledge, Use, and Benefits

Return to Contents


Slide 28: Clinician Survey Findings (Wave 1)

Text Description is below the image.

  • 18% of clinician respondents indicated awareness of CER when prompted with its name.
    • A larger portion of clinicians indicated awareness of PCOR (46%) or evidence-based medicine (93%).
  • The most common sources of exposure to CER were medical or science journal article (23%), conference or professional meeting (17%), and a continuing education course (10%).
  • One-third (33%) of surveyed clinicians were aware of AHRQ.
    • The vast majority of clinicians (92%) were unaware of the EHC Program.
  • Slightly over half of clinicians indicated interest in learning more about CER (54%) or the EHC Program (57%)

Return to Contents


Slide 29: Clinician Survey Findings (Wave 2)

Text Description is below the image.

  • 20% of clinician respondents indicated awareness of CER when prompted with its name.
    • A larger portion of clinicians indicated awareness of PCOR (49%) or evidence-based medicine (94%).
  • The most common sources of exposure to CER were medical or science journal article (24%), conference or professional meeting (17%), and a continuing education course (11%).
  • One-third (38%) of surveyed clinicians were aware of AHRQ.
    • The vast majority of clinicians (88%) were unaware of the EHC Program.
  • Slightly over half of clinicians indicated interest in learning more about CER (54%) or the EHC Program (57%).

Return to Contents


Slide 30: Clinician Survey Findings (Longitudinal)

Text Description is below the image.

  • Clinicians' aided awareness of CER increased over time (18% to 20%), although the increase was not statistically significant.
  • Clinicians reported a statistically significant increase in awareness of AHRQ from Wave 1 to Wave 2 (33% to 38%).
  • Clinicians also reported a statistically significant increase in awareness of the EHC Program (8% to 12%).

Return to Contents


Slide 31: Clinician Survey Findings (Longitudinal con't)

Text Description is below the image.

  • Interest in learning more about CER and about the EHC Program increased over time, although the change was not significant.
  • Of those who had heard of the EHC Program Web site, the number of clinicians who reported visiting the EHCP Web site increased statistically significantly between the two survey administrations (28% to 41%).
  • Clinicians reported a slight increase in their intention to use EHC Program clinician products in the near future, although the change was not statistically significant.

Return to Contents


Slide 32: Focus Groups

Text Description is below the image.

Understanding, Knowledge, Use, and Perceived Benefits

Return to Contents


Slide 33: Focus Groups

Text Description is below the image.

Focus groups were conducted with the following stakeholder groups:

  • Consumers/patients.
  • Clinicians.
  • Business Leaders/Purchaser.

Return to Contents


Slide 34: Consumer Focus Group Findings

Text Description is below the image.

  • A majority of participants expressed interest in CER and noted that they plan to use CER (from AHRQ and other reliable sources) to inform future medical decisions.
  • Aware participants used online medical information more frequently and robustly than unaware participants to inform medical decisions.
  • Unaware participants relied heavily on medical professionals for medical information and used online medical sources to supplement this information.
  • To help make medical decisions, participants most frequently desired an assessment of the pros, cons, and costs of available treatment options.
  • Almost all participants approved of the government's role in sponsoring CER and the EHC Program.

Return to Contents


Slide 35: Clinician Focus Group Findings

Text Description is below the image.

  • While clinicians rely on several information sources, they rely on those with which they have developed a comfort level and will continue to use that source, often to the exclusion of other new and potentially useful sources.
  • Clinicians are cautious about biases or conflicts of interest from information sources, including commercial and government-sponsored research.
  • The majority of clinicians had little to no experience with the EHC Program.
  • To increase interest in the EHC Program, clinicians suggested that AHRQ more visibly promote the benefits and credibility of EHC Program-sponsored CER, and then integrate the results and products into existing, easy-to-access sources of medical information, with special emphasis on packaging to support point-of-care decisionmaking.

Return to Contents


Slide 36: Business Leader Findings

Text Description is below the image.

  • All participants indicated awareness either specifically of CER or generally of research that compares treatment options.
  • Most focus group participants indicated awareness of AHRQ; several had heard of AHRQ's EHC Program.
  • Those participants familiar with CER attributed their awareness to the ACA, AHRQ, and business-focused coalitions.
  • Participants' organizations did not disseminate information on CER specifically; however, organizations did share general health care information with their employees and members.
  • Participants were interested in how employers can use CER to help prepare employees make medical decisions and affect outcomes for specific diagnoses (including arthritis, obesity, diabetes, cancer, and cardiovascular conditions).

Return to Contents


Slide 37: Summary

Text Description is below the image.

  • The findings from our multiple data sources (dissemination contractor metrics, Web and Clearinghouse metrics, and surveys and focus groups with target audiences) indicate interest in CER/PCOR and a growing engagement with AHRQ's brand of this research.
  • As the dissemination contractors increased their efforts over time, we saw corresponding increase in the number of exposures to the EHC Program. These exposures lead to measurable outcomes of engagement with EHC Program products, including the fact that the majority of large-volume Clearinghouse orders referenced a dissemination contractor code.
  • Survey and focus group data show that clinicians, consumers, and business leaders are interested in CER to inform medical decisions. Similarly, these audiences have had some exposure to AHRQ's EHC Program and are interested in learning more about and engaging with EHC Program materials.

Return to Contents

Page last reviewed December 2014
Page originally created December 2014
Internet Citation: Results of ARRA-Funded CER Dissemination Activities. Content last reviewed December 2014. Agency for Healthcare Research and Quality, Rockville, MD. http://www.ahrq.gov/cpi/about/mission/arra/index.html