III. Background

Evaluation of the Use of AHRQ and Other Quality Indicators (PDF)


NORC examined the published and unpublished literature on the USPSTF, and reviewed documents that analyzed, commented on or evaluated the most relevant literature on the USPSTF from 1991 to the present. While not comprehensive, this brief review of literature presents a sample of the studies that have explored the utilization and impact of the USPSTF recommendations in health care settings. Research studies, observational studies, surveys, cross-sectional evaluations, qualitative analyses and other reports were identified and summarized from peer-reviewed clinical and policy journals. Literature was identified through a number of methods, including a search of published materials using Academic Search Premier, Medline, and other major health services research databases as well as the search tools of Web sites for major health policy journals such as Health Affairs, Annals of Internal Medicine, and the American Journal of Preventive Medicine. In addition to peer reviewed literature, we conducted a search of popular media and materials relevant to the topic of the USPSTF program published on the Internet or otherwise publicly available through use of search engines such as Google.com. Searches were conducted on Google Scholar to obtain access to the "gray" literature (e.g., reports and news sources not catalogued in electronic peer-reviewed literature databases but available online). As expected, many of the documents were found as a result of "snowballing," in which the citations from a source identified through traditional searches are examined to identify additional sources.


Review of the Literature on the USPSTF and Clinical Preventive Services

The USPSTF provides clinicians with a valuable service, gathering and analyzing the available literature on preventive medicine and transforming it into sound evidence-based recommendations. The Task Force's recommendations are regularly updated and made available to patients, physicians, and health plans via the USPSTF section of the Agency for Healthcare Research and Quality (AHRQ) Web site, various journal publications, and the Electronic Preventive Services Selector (ePSS) software tool for physicians to use on their personal digital assistants. Yet, despite the value and ease of attaining the USPSTF recommendations, patients are not receiving the recommended amount of preventive services.10-16

The delivery of clinical preventive services—those services recommended by the USPSTF and other organizations—is affected by factors at both the individual and systems levels, including: physician time; physician awareness and attitudes; patient attitudes and expectations; presence or absence of implementation tools; financial incentives or disincentives;17 and efforts designed to increase the methods to examine these factors, including case studies, patient and physician surveys, and medical record audits. The findings of this research are summarized below.

A study of 230 adults in family practice clinics by Medder et al. (1992) found that the average adult seeking outpatient care presented with 15.4 risk factors, making the patient eligible for 24.5 recommended services according to the USPSTF recommendations.18 This finding highlights the challenge confronting primary care physicians in busy practices to incorporate all of the USPSTF recommended services in the limited time available during routine care. Through an observational study, Zyzanski et al. (1998) found that high-volume practices19 had a significantly lower proportion of patients with up-to-date screening, counseling, and immunizations recommended by the USPSTF.20 A potential cause of the low proportion of up-to-date patients in high volume practices is that high volume practices schedule 33% fewer well-care visits.21 A survey of family physicians using a hypothetical case scenario found that the USPSTF recommendations were delivered more often during the well visit than the illness visit presented in the case scenario.22 The low rates of health maintenance visits, especially in underserved populations, decrease the number of opportunities for community physicians to administer recommended preventive services. Recognizing this concern, the USPSTF recommended the delivery of preventive services during illness visits in 1996.23

Soon thereafter, an observational study of community family physicians was conducted to evaluate the effectiveness of the Task Force's recommendation to provide preventive services during illness visits.24 Stange et al. (1998) found that preventive services were delivered 32.5% of the time, averaging 1.7 preventive services per visit.25 These preventive services were delivered in 39% of the chronic illness visits and 30% of the acute illness visits, and tended to be recommendations related to counseling on health habits rather than immunizations or screenings.26 One possible explanation may be the length of the visit: the illness visits with preventive services were, on average, two minutes longer than visits without.27 Some research has suggested that patients may be more likely to receive preventive services related to their presenting signs and symptoms during an illness visit as the illness may provide the impetus for prevention and/or a teachable moment.28 Other research seems to indicate that patients may not receive the clinical preventive services most appropriate for their signs and symptoms. A survey of psychiatric patients by Carney et al. (2002) found that those preventive services perhaps most critical to their conditions-counseling on firearm ownership and alcohol avoidance while driving-were received by significantly fewer patients than were screening tests and immunizations.29

The time constraints of family physicians are a widely recognized barrier to the delivery of USPSTF recommendations, as preventive services are just one of many competing demands on a physician's time. In a study to estimate the total time burden of implementing USPSTF recommendations, Yarnall et al. (2003) found that physicians would need 7.4 hours per day to implement the recommendations in every case that warranted a preventive service.30 Recognizing the need for physicians to prioritize the USPSTF recommendations, the Task Force evaluates each recommendation based on the burden of suffering from the target condition and the effectiveness of the preventive service. The Yarnall et al. study found that performing even just the "A" rated services would take a physician two hours per day to implement. The study concluded that preventive services are competing with each other for a physician's time. This has led a number of researchers to recommend that the Task Force develop a more comprehensive evaluation system.

Stange et al. (1994) recommends that physicians should prioritize preventive services based on the patient's risk factors, values, ability to pay, reason for the visit, and their own knowledge and skills.31

Other researchers feel that the USPSTF should base its evaluations on all available data, including analysis of cost effectiveness and clinically preventable burden, in order to rank its recommendations and increase their utilization by time-crunched primary care providers.32-33 The current USPSTF review process includes information from cost-effectiveness analyses in developing recommendations, although such information is generally only included for recommendations where relevant questions regarding cost-effectiveness exist.34

An alternative explanation for the low level of USPSTF recommended services provided is that physicians may not be aware of, or may not have confidence in, the recommendations. Several researchers have investigated this question and found that, counter to what might be expected, the relationship between a physician's attitude toward the USPSTF recommendations and his or her delivery of the services may not be correlated. A survey of family physicians in New York found that a large majority believe that evidence-based recommendations are effective for educating physicians, family practice residents, and students.35 Specifically, these same physicians are confident in the USPSTF recommendations, ranking them second only to the clinical recommendations published by their professional organization, the American Academy of Family Physicians.

A study from the Improving Preventive Services through Organization, Vision, and Empowerment (IMPROVE) initiative found that approximately 83% of primary care doctors and nurses surveyed believe that pneumonia and influenza vaccinations are important or very important; 90% believe tobacco screening and cessation counseling are important or very important; 95% believe that cholesterol screening is important or very important; and between 98 and 100% believe hypertension screening, breast examinations, mammograms, and Papanincolaou screenings are all important or very important.36 However, this same study found little evidence of a correlation between physician attitudes and delivery of preventive services. Only four preventive services that researchers examined had a statistically significant correlation with physician attitudes: cholesterol screening, tobacco use screening, smoking cessation counseling, and Papanincolaou screenings. Of those four, only tobacco use screening and providing smoking cessation counseling were positively associated with clinician attitude. Although clinicians appear to have confidence in the USPSTF recommendations and believe delivery of clinical preventive services to be important, these factors do not seem to actually affect the rates of delivery.

Systems-level factors may also play a role in the delivery rates of clinical preventive services. While the IMPROVE survey found a strong belief among nurses and physicians that clinical leaders are committed to the delivery of preventive services in their organization, several researchers believe that organizations may not be well-equipped to facilitate the delivery of the USPSTF recommendations. Researchers have suggested several factors—including manual or computer assisted reminder and tracking systems and more involvement of non-physician personnel—to increase the likelihood of the recommended services being delivered.37-42 One researcher found that the combination of a commitment to the delivery of preventive services and a flow sheet to ensure that preventive services are discussed with every patient could increase delivery rates significantly, without decreasing the number of patients seen per day.43 The flow sheet protocol added just over two minutes to each visit in which time, health habits counseling, screening, immunization, and follow-up visits were discussed.

Researchers have also remarked that adding preventive services systems will not only increase the number of USPSTF recommendations performed, but decrease the number of non-recommended services. In the survey of family physicians using the hypothetical case scenario discussed above, Stange et al. (1994) found that while several USPSTF services were performed during the health maintenance visit, so were an alarming number of non-recommended services performed.44 This satisfaction with a visit is not affected by the absence or addition of recommended preventive services.45 Exploring further, the authors found that the physicians delivering a high rate of USPSTF services and a low rate of non-recommended services were generally younger, less likely to be in solo practice, residency trained, and had significant exposure to the USPSTF recommendations. Understanding the characteristics associated with delivery of the recommended USPSTF services will help clinic organizers to best direct resources to achieve high delivery rates of recommended services.

While the majority of research on delivery of clinical preventive services has focused on the clinician and patient point of service, some researchers point to systems-level factors such as reimbursement and patient outreach as possibly affecting the rate of service delivery. In 1999, Merenstein et al. published a study in which 10 medical directors representing six types of health plans were interviewed about clinical preventive services at their health plans. Specifically, the medical directors were queried on the method of adoption of clinical preventive services, which services were paid for, frequency and age groups for services, and patient encouragement to obtain services. The authors concluded that while little difference existed between the plans in terms of which preventive services are recommended and covered, plans seemed to be influenced more by consumer demand and public opinion than by evidence.46 Perhaps further underscoring the need for a systems-level, evidence-based approach to the adoption and integration of clinical preventive services, the authors note that in all but one plan, a new service was considered only when a physician or patient initiates a claim for that service.47

Mehrotra et al. (2006) compared the quality of care provided to patients by physicians who are part of integrated medical groups (IMGs) [centralized organizations in which physicians are employees or participants in a partnership arrangement] and independent practice associations (IPAs) [decentralized groups in which physicians generally have nonexclusive contractual relationships and typically manage their offices independently]. In the study, quality was measured as the percentage of eligible patients receiving three clinical preventive services (mammography, Papanicolaou screening, or Chlamydia screening) or three chronic disease management measures (diabetic retinal examination, asthma controller medication, or ß-blocker use after acute myocardial infarction). The authors concluded that patients in IMGs generally received more recommended clinical preventive services than those in IPAs, but noted that neither the presence of an EMR nor the implementation of quality improvement (QI) strategies explained the differences in quality in their study.48 They caution, however, that compared with the period of their study [1999-2000], quality improvement processes and EMRs are now more established and may therefore have a stronger positive association with quality. However, support for quality improvement as a means of improving clinical preventive services remains mixed. Some studies have shown limited, inconsistent, or no benefits of quality improvement activities as a means of increasing the delivery of clinical preventive services,49-51 while other researchers have demonstrated a positive correlation.52-53

Measuring the effect of quality improvement processes and interventions on rates of clinical preventive services delivery creates many obstacles, often cited as study limitations by researchers. These include: the time lag between the initiation of a quality improvement activity—which may increase the accuracy of documentation of the targeted services—and the actual increase in delivery of those services; inconsistency between patient reports and medical record audits; the incomplete picture of preventive service delivery resulting from the use of claims or Health Plan Employer Data and Information Set (HEDIS) data to measure and monitor delivery rates; and physician and staff awareness that a particular service or services have been targeted for improvement. Kim et al. (1999) tested whether either provider education or comprehensive intervention, consisting of education, peer-comparison feedback "report cards," and academic detailing, would affect preventive services delivery rates. The authors found few differences between the education-only and comprehensive intervention groups, and, indeed discovered that delivery rates decreased for some services in the education-only group.54 The authors hypothesize that this decrease may have been related to physician attitudes and patient expectations—the services in question were mammography, clinical breast examinations, and influenza immunizations.

Clearly, the factors affecting the delivery of the clinical preventive services recommended by the U.S. Preventive Services Task Force are complex and often interrelated. The literature shows that physician time and the inability to prioritize recommendations are perhaps the greatest challenges to the delivery of clinical preventive services. The resultant delivery of clinical preventive services is often inconsistent, incomplete, and based on factors other than evidence-based recommendations like those established by the USPSTF. Such factors include patient symptoms, cost effectiveness, and other opinions about preventive services—such as those of the individual, the public, and of specialty societies.

While clinicians value the USPSTF recommendations and believe they are scientifically sound, evidence-based, and objective, the organizations in which clinicians practice may lack the tools, systems, and resources that facilitate the delivery of the clinical preventive services recommended by the USPSTF. Examples of these factors include implementation systems such as an electronic medical record, quality improvement strategies to increase the appropriate delivery of clinical preventive services, and a multitude of system characteristics that range from reimbursement to an organizational culture of prevention. The literature suggested that further research is needed to better understand the factors that affect integration and delivery of the USPSTF recommendations. Furthermore, policies and tools must be developed to overcome challenges to integration and improve the delivery of the USPSTF recommendations.

Return to Contents
Proceed to Next Section

Current as of December 2007
Internet Citation: III. Background: Evaluation of the Use of AHRQ and Other Quality Indicators (PDF). December 2007. Agency for Healthcare Research and Quality, Rockville, MD. http://www.ahrq.gov/research/findings/final-reports/uspstf/3.html