Evaluation of the National Guideline Clearinghouse™ (Text Version)

Slide Presentation from the AHRQ 2011 Annual Conference

Evaluation of the National Guideline Clearinghouse

Slide Presentation from the AHRQ 2011 Annual Conference

On September 19, 2011, Michelle Tregear made this presentation at the 2011 Annual Conference. Select to access the PowerPoint® presentation (335 KB). Plugin Software Help.

Slide 1

AHRQ Annual Conference
September 2011

Evaluation of the National Guideline Clearinghouse™

Slide 2

Acknowledgements

• Survey respondents.
• Focus group participants.
• Key informants:
  • AMIA and ACCP.
• AHRQ
  • Mary Nix
  • Judi Consalvo
  • Gov Delivery Staff.
  • Marjorie Shofer
  • HIT Portfolio.
• AMA and AHIP.
• ECRI Institute.
• NGC Evaluation.
• PET
  • Florence Chang.
  • Belinda Ireland.
  • Richard Shiffman.
  • Katrin Uhlig.
  • Cally Vinz.

Evaluation Project Team Members
AFYA, Inc.

• Michelle Tregear.
• Jenice James.
• Debra Dekker.
• Craig Dearfield.
• Ajay Bhardwaj.
• Robin Pugh-Yi.

The Lewin Group

• Carol Simon.
• Jaclyn Marshall.
• Jacob Epstein.

Slide 3

Evaluation Objectives

• Gain a better understanding of how NGC:
  • Is used by its stakeholders (including awareness among key stakeholders).
  • Supports dissemination of evidence-based clinical practice guidelines and related documents.
  • Has influenced efforts in guideline development, implementation, and use.
  • Can be improved.

Slide 4

Data Collection Methods

• CIPP evaluation framework—Logic model to develop key questions.
• Developed instruments which were Informed by the PET.
• Received OMB clearance Feb 2011.
• Conducted survey, focus groups, and interviews (Mar—Jul 2011).

Mixed-Methods Approach

• Quantitative Data:
  • Web-based Survey:
    • Skip logic and Branching.
    • Respondents solicited by E-mail lists for AHRQ, AMA, and AHIP.
• Qualitative Data:
  • Focus groups (4):
    • Stakeholder-specific.
  • Key informant interviews (26):
    • Mix of stakeholders.

Slide 5

Survey Sample Characteristics

Referral Source

Survey Sample

• Majority familiar with AHRQ (99%).

Respondent Demographics

Survey Reach

• Majority from the U.S. (87%).

Occupation Mix

• 56%—Providers, clinicians, nurses.
• 13%—Researchers, librarians, or similar.
• 12%—Consultants, managers, administrators.
• 19%—Other Respondent Demographics Referral Source.

Slide 6

Survey Respondent Counts

Image: A graph of Survey Respondent Counts.

NGC awareness and use, demographics, other guideline source questions.
NGC questions.
Stakeholder-specific questions.

Slide 7

NGC Awareness, Use, and Other Guideline Sources

Slide 8

Results—NGC Awareness

Key Notes:

• Majority of respondents derived from AHRQ E-mail invitation.
• Awareness of NGC greatest among AHRQ and AHIP respondents.
• AHRQ Opportunity to increase physician awareness of NGC.
• Awareness of NGC in 2011 substantially higher than in 2001 evaluation N=9,289 N=42, N=49.

Slide 9

Results—Guideline Sources Used

• Survey Findings:
  • Most use multiple sources to find CPGs (3-5 most common):
    • PubMed, government sources, general search engines, medical/professional societies.
• Qualitative Findings:
  • NGC often cited as "first go-to source".

Key Finding

Most NGC users satisfied or more satisfied with NGC compared to other guideline sources.

Slide 10

Results—Degree to Which NGC Fulfills Users' Specific Needs When Using NGC

Image: A graph of Results—Degree to which NGC Fulfills Users' Specific Needs when Using NGC.

Key Finding

NGC is doing well on these needs.

Slide 11

Results—NGC Compared to Other Sources in Meeting Needs for Guidelines

Image: A graph of Results—NGC Compared to Other Sources in Meeting Needs for Guidelines.

Finding

NGC does better or about equal compared to other sources. This is supported by qualitative findings .

Slide 12

NGC—Trustworthiness and Inclusion Criteria

Slide 13

Results—Trustworthiness

• Survey Findings:
  • 75% -very good or good:
    • 80%—would definitely, very likely, or probably recommend NGC to colleague.
  • No differences by key stakeholder group.

• Qualitative Findings:
  • Differences by key stakeholder group:
    • Guideline developers and informaticians less trusting.
  • Others note that they "trust" the content because "it comes from AHRQ".

Slide 14

Results—NGC Inclusion Criteria

• Survey Findings:
  • 63%—appropriate:
    • By length-of-use: longer users more likely to find the criteria too loose.
  • No differences by key stakeholder group.

• Qualitative Findings:
  • Differences by key stakeholder group:
    • Guideline developers/informatics specialists said they are 'too loose'.
  • Policymakers, medical librarians, researchers generally satisfied.

Slide 15

Results—NGC Inclusion Criteria

AHRQ Opportunity to revisit NGC's Inclusion Criteria.

• Qualitative Findings (Guideline Developers and Informaticians):
  • "...the quality of the criteria was good when NGC started out but it has gotten more complicated..."
  • "...If the goal is to be 'all inclusive,' then the criteria are fine. [but]....there needs to be some other ways to separate the wheat from the chaff."
  • "They could raise the bar."
  • "I'd say too loose.... I think there's a belief that: A) NGC creates these guidelines...and then, B) there's also a belief that NGC somehow has a very rigorous process for only allowing certain guidelines in, or certain types of very high quality guidelines, or that it's an endorsement of these guidelines. And it isn't."

Slide 16

Results—NGC Age Criterion

• Survey Findings:
  • ~Equal # say 5 years is "appropriate" or "too long":
    • Reduce to 3 years most common selection.
  • No differences by key stakeholder group.
• Qualitative Findings:
  • Common theme: the age criterion is too long:
    • AHRQ Opportunity to revisit NGC's Age Criterion.

Slide 17

Stakeholder-Specific

• NGC Influences.

Slide 18

NGC Influences—Stakeholders
Key Take-Aways from Survey Findings

• Respondents, by group, indicated NGC greatly influenced:
  • Guideline developers' ability to identify guidelines, and develop and use quality measures.
  • Health providers' ongoing learning activities, clinical decision-making processes, and ability to identify guidelines.
  • Medical librarians' ability to meet their client's needs.
  • Medical librarians' and researchers' ability to identify current and high quality guidelines.
  • Measure developers' measure development activities and approach to identifying guidelines.
  • Policymakers' and purchasers' ability to identify guidelines and convert clinical information.

Slide 19

NGC Influences—Stakeholders
Key Take-Aways from Survey Findings

• Respondents, by group, indicated NGC greatly influenced:
  • Guideline developers' ability to identify guidelines, and develop and use quality measures.
  • Health providers' ongoing learning activities, clinical decision-making processes, and ability to identify guidelines.
  • Medical librarians' ability to meet their client's needs.
  • Medical librarians' and researchers' ability to identify current and high quality guidelines.
  • Measure developers' measure development activities and approach to identifying guidelines.
  • Policymakers' and purchasers' ability to identify guidelines and convert clinical information.

Slide 20

NGC Influences �Guideline Developers Participating in NGC

• Survey Findings (n=199):
  • NGC guideline submitters reported greater NGC influence for guideline updating frequency, and how organizations document or report their guidelines.
• Qualitative Findings (n=24):
  • NGC serves primarily as a source for locating guidelines.
  • NGC's age criterion applies some "pressure" to stay current.
  • NGC has had little influence on how guideline developers do their work� e.g., methodology, reporting.

AHRQ Opportunity to increase knowledge among guideline developers about how to create and report trustworthy guidelines.

Slide 21

NGC Influence�Dissemination of CPGs
Survey Findings for Participating Developers

• Noteworthy finding:
  • 65% said excellent or good.
  • 21% were neutral.
  • 14% said fair or poor.

AHRQ Opportunity to identify additional efforts to enhance the dissemination of guidelines.

Slide 22

NGC Enhancements

Slide 23

Likely Use of Potential Enhancements
Survey Findings

• Enhancements
  • Two thirds said they would "definitely," "very likely," or "probably" use the following NGC enhancements if available:
    • Ratings of guideline quality.
    • Subject-specific e-mail alerts.
• Enhancements for providers:
  • 72%—having NGC content at point-of-care would be useful.
  • 66%—would take CME if available..
• Enhancements for informaticians:
  • 52%—NGC Summaries as XML file according to the Guideline Elements Model (GEM) or other similar.
• Examples of Potential Enhancements.
  • Ratings of guideline quality.
  • Subject-specific E-mail alerts.
  • Access to archived guidelines.
  • Additional data download options and XML formats AHRQ Opportunity to invest in major enhancements that will increase the value of NGC.

AHRQ Opportunity to invest in major enhancements that will increase the value of NGC.

Slide 24

Additional Enhancements—Common Themes
Qualitative Findings

Additional Common Themes

• Guideline Developers:
  • Commentary/responses to guidelines from users.
  • Guideline developer conferences / methodology workshops.
• Informatics Specialist:
  • Assessment of the executability of guidelines.
• Medical Librarians:
  • Integration with other Web sites (PubMed).
  • Quicker access to "new" guidelines.
• Researchers:
  • Assessment of attributes in IOM standards for guidelines.
  • More information about treatment in multi-morbidity or comorbid conditions.

Slide 25

Conclusions and Recommendations

Slide 26

Key Take-Aways for Expanding NGC's Use and Impact

• AHRQ Opportunities include:
  • Build on NGC's user base, including with health providers.
  • Revisit the NGC inclusion criteria.
  • Revisit the guideline age criterion.
  • Increase knowledge among guideline developers about how to create and report trustworthy guidelines.
  • Enhance guideline dissemination efforts.
  • Invest in major enhancements to the NGC Web site that will provide significant added value.

Current as of December 2011

Internet Citation:

Evaluation of the National Guideline Clearinghouse. Slide Presentation from the AHRQ 2011 Annual Conference (Text Version). December 2011. Agency for Healthcare Research and Quality, Rockville, MD. http://www.ahrq.gov/about/annualconf11/nix_schiffman_tregear/tregear.htm