Evaluation of the National Guideline Clearinghouse™ (Text Version)
On September 19, 2011, Michelle Tregear made this presentation at the 2011 Annual Conference. Select to access the PowerPoint® presentation (335 KB). Plugin Software Help.
Slide 1
AHRQ Annual Conference
September 2011
Evaluation of the National Guideline Clearinghouse™
Slide 2
Acknowledgements
Thank-you to:
- Survey respondents.
- Focus group participants.
- Key informants:
- AMIA and ACCP.
- AHRQ
- Mary Nix
- Judi Consalvo
- Gov Delivery Staff.
- Marjorie Shofer
- HIT Portfolio.
- AMA and AHIP.
- ECRI Institute.
- NGC Evaluation.
- PET
- Florence Chang.
- Belinda Ireland.
- Richard Shiffman.
- Katrin Uhlig.
- Cally Vinz.
Evaluation Project Team Members
AFYA, Inc.
- Michelle Tregear.
- Jenice James.
- Debra Dekker.
- Craig Dearfield.
- Ajay Bhardwaj.
- Robin Pugh-Yi.
The Lewin Group
- Carol Simon.
- Jaclyn Marshall.
- Jacob Epstein.
Slide 3
Evaluation Objectives
- Gain a better understanding of how NGC:
- Is used by its stakeholders (including awareness among key stakeholders).
- Supports dissemination of evidence-based clinical practice guidelines and related documents.
- Has influenced efforts in guideline development, implementation, and use.
- Can be improved.
Slide 4
Data Collection Methods
Key Project Milestones
- CIPP evaluation framework—Logic model to develop key questions.
- Developed instruments which were Informed by the PET.
- Received OMB clearance Feb 2011.
- Conducted survey, focus groups, and interviews (Mar—Jul 2011).
Mixed-Methods Approach
- Quantitative Data:
- Web-based Survey:
- Skip logic and Branching.
- Respondents solicited by E-mail lists for AHRQ, AMA, and AHIP.
- Web-based Survey:
- Qualitative Data:
- Focus groups (4):
- Stakeholder-specific.
- Key informant interviews (26):
- Mix of stakeholders.
- Focus groups (4):
Slide 5
Survey Sample Characteristics
Referral Source
Source | Count |
---|---|
Total | 9,389 |
| 9,298 (99%) |
| 42 |
| 49 |
Survey Sample
- Majority familiar with AHRQ (99%).
Respondent Demographics
Survey Reach
- Majority from the U.S. (87%).
Occupation Mix
- 56%—Providers, clinicians, nurses.
- 13%—Researchers, librarians, or similar.
- 12%—Consultants, managers, administrators.
- 19%—Other Respondent Demographics Referral Source.
Slide 6
Survey Respondent Counts
Image: A graph of Survey Respondent Counts.
Section / Module | Count |
---|---|
Total | 9,389 (100%) |
NGC awareness and use, demographics, other guideline source questions | |
NGC Unaware | 2,075 (22.1%) |
NGC Aware | 7,314 (77.9%) |
NGC questions | |
Non NGC User | 1,395 (19.3%) |
NGC User | 5,828 (80.7%) |
Stakeholder-specific questions | |
| 1,076 (18.5%) |
| 3,271 (56.1%) |
| 204 (3.5%) |
| 292 (5.0%) |
| 1,219 (20.9%) |
| 1,219 (20.9%) |
| 351 (6.0%) |
Slide 7
NGC Awareness, Use, and Other Guideline Sources
Slide 8
Results—NGC Awareness
Key Notes:
- Majority of respondents derived from AHRQ e-mail invitation.
- Awareness of NGC greatest among AHRQ and AHIP respondents.
- AHRQ Opportunity to increase physician awareness of NGC.
- Awareness of NGC in 2011 substantially higher than in 2001 evaluation
To the right of the notes are 2 bar charts.
Top Chart: Awareness by Survey Source
AHRQ: 78% (N=9,289)
AHIP: 83% (N=42)
AMA: 46% (N=49)
Bottom Chart: Awareness Over Time
2001:
Aware: 30%
Unaware: 70%
2011:
Aware: 79%
Unaware: 22%
Slide 9
Results—Guideline Sources Used
- Survey Findings:
- Most use multiple sources to find CPGs (3-5 most common):
- PubMed, government sources, general search engines, medical/professional societies.
- Most use multiple sources to find CPGs (3-5 most common):
- Qualitative Findings:
- NGC often cited as "first go-to source".
Bar chart showing percentages of respondents who are satisfied. The majority said about equally satisfied. (N=5,828).
Key Finding
Most NGC users satisfied or more satisfied with NGC compared to other guideline sources.
Slide 10
Results—Degree to Which NGC Fulfills Users' Specific Needs When Using NGC
Image: A graph of Results—Degree to which NGC Fulfills Users' Specific Needs when Using NGC.
Key Finding
NGC is doing well on these needs.
Slide 11
Results—NGC Compared to Other Sources in Meeting Needs for Guidelines
Image: A graph of Results—NGC Compared to Other Sources in Meeting Needs for Guidelines.
Finding
NGC does better or about equal compared to other sources. This is supported by qualitative findings .
Slide 12
NGC—Trustworthiness and Inclusion Criteria
Slide 13
Results—Trustworthiness
- Survey Findings:
- 75% - very good or good:
- 80%—would definitely, very likely, or probably recommend NGC to colleague.
- No differences by key stakeholder group.
- 75% - very good or good:
- Qualitative Findings:
- Differences by key stakeholder group:
- Guideline developers and informaticians less trusting.
- Others note that they "trust" the content because "it comes from AHRQ".
- Differences by key stakeholder group:
Slide 14
Results—NGC Inclusion Criteria
- Survey Findings:
- 63%—appropriate:
- By length-of-use: longer users more likely to find the criteria too loose.
- No differences by key stakeholder group.
- 63%—appropriate:
- Qualitative Findings:
- Differences by key stakeholder group:
- Guideline developers/informatics specialists said they are 'too loose'.
- Policymakers, medical librarians, researchers generally satisfied.
- Differences by key stakeholder group:
Slide 15
Results—NGC Inclusion Criteria
AHRQ Opportunity to revisit NGC's Inclusion Criteria.
- Qualitative Findings (Guideline Developers and Informaticians):
- "...the quality of the criteria was good when NGC started out but it has gotten more complicated..."
- "...If the goal is to be 'all inclusive,' then the criteria are fine. [but]....there needs to be some other ways to separate the wheat from the chaff."
- "They could raise the bar."
- "I'd say too loose.... I think there's a belief that: A) NGC creates these guidelines...and then, B) there's also a belief that NGC somehow has a very rigorous process for only allowing certain guidelines in, or certain types of very high quality guidelines, or that it's an endorsement of these guidelines. And it isn't."
Slide 16
Results—NGC Age Criterion
- Survey Findings:
- ~Equal # say 5 years is "appropriate" or "too long":
- Reduce to 3 years most common selection.
- No differences by key stakeholder group.
- ~Equal # say 5 years is "appropriate" or "too long":
- Qualitative Findings:
- Common theme: the age criterion is too long:
- AHRQ Opportunity to revisit NGC's Age Criterion.
- Common theme: the age criterion is too long:
Slide 17
Stakeholder-Specific
- NGC Influences.
Slide 18
NGC Influences—Stakeholders
Key Take-Aways from Survey Findings
- Respondents, by group, indicated NGC greatly influenced:
- Guideline developers' ability to identify guidelines, and develop and use quality measures.
- Health providers' ongoing learning activities, clinical decision-making processes, and ability to identify guidelines.
- Medical librarians' ability to meet their client's needs.
- Medical librarians' and researchers' ability to identify current and high quality guidelines.
- Measure developers' measure development activities and approach to identifying guidelines.
- Policymakers' and purchasers' ability to identify guidelines and convert clinical information.
Slide 19
NGC Influences—Stakeholders
Key Take-Aways from Survey Findings
- Respondents, by group, indicated NGC greatly influenced:
- Guideline developers' ability to identify guidelines, and develop and use quality measures.
- Health providers' ongoing learning activities, clinical decision-making processes, and ability to identify guidelines.
- Medical librarians' ability to meet their client's needs.
- Medical librarians' and researchers' ability to identify current and high quality guidelines.
- Measure developers' measure development activities and approach to identifying guidelines.
- Policymakers' and purchasers' ability to identify guidelines and convert clinical information.
Slide 20
NGC Influences—Guideline Developers Participating in NGC
- Survey Findings (n=199):
- NGC guideline submitters reported greater NGC influence for guideline updating frequency, and how organizations document or report their guidelines.
- Qualitative Findings (n=24):
- NGC serves primarily as a source for locating guidelines.
- NGC's age criterion applies some "pressure" to stay current.
- NGC has had little influence on how guideline developers do their work� e.g., methodology, reporting.
AHRQ Opportunity to increase knowledge among guideline developers about how to create and report trustworthy guidelines.
Slide 21
NGC Influence—Dissemination of CPGs
Survey Findings for Participating Developers
- Noteworthy finding:
- 65% said excellent or good.
- 21% were neutral.
- 14% said fair or poor.
AHRQ Opportunity to identify additional efforts to enhance the dissemination of guidelines.
Slide 22
NGC Enhancements
Slide 23
Likely Use of Potential Enhancements
Survey Findings
- Enhancements
- Two thirds said they would "definitely," "very likely," or "probably" use the following NGC enhancements if available:
- Ratings of guideline quality.
- Subject-specific e-mail alerts.
- Two thirds said they would "definitely," "very likely," or "probably" use the following NGC enhancements if available:
- Enhancements for providers:
- 72%—having NGC content at point-of-care would be useful.
- 66%—would take CME if available..
- Enhancements for informaticians:
- 52%—NGC Summaries as XML file according to the Guideline Elements Model (GEM) or other similar.
- Examples of Potential Enhancements.
- Ratings of guideline quality.
- Subject-specific E-mail alerts.
- Access to archived guidelines.
- Additional data download options and XML formats AHRQ Opportunity to invest in major enhancements that will increase the value of NGC.
AHRQ Opportunity to invest in major enhancements that will increase the value of NGC.
Slide 24
Additional Enhancements—Common Themes
Qualitative Findings
Additional Common Themes
- Guideline Developers:
- Commentary/responses to guidelines from users.
- Guideline developer conferences / methodology workshops.
- Informatics Specialist:
- Assessment of the executability of guidelines.
- Medical Librarians:
- Integration with other Web sites (PubMed).
- Quicker access to "new" guidelines.
- Researchers:
- Assessment of attributes in IOM standards for guidelines.
- More information about treatment in multi-morbidity or comorbid conditions.
Slide 25
Conclusions and Recommendations
Slide 26
Key Take-Aways for Expanding NGC's Use and Impact
- AHRQ Opportunities include:
- Build on NGC's user base, including with health providers.
- Revisit the NGC inclusion criteria.
- Revisit the guideline age criterion.
- Increase knowledge among guideline developers about how to create and report trustworthy guidelines.
- Enhance guideline dissemination efforts.
- Invest in major enhancements to the NGC Web site that will provide significant added value.