II. Study Methods

Evaluation of the Use of AHRQ and Other Quality Indicators (PDF)

 

NORC applied a tiered-qualitative approach to design and implement an assessment of the adoption and integration of the USPSTF recommendations. In the first major phase, NORC prepared an evaluation design for AHRQ that posed three key research questions to be explored further through qualitative methods and a case study approach. In the second phase, NORC conducted over 40 semi-structured phone and in-person interviews, reviewed relevant prevention materials provided by each of the four health plans sites, and prepared a review of the literature on the USPSTF and its impact on the delivery of clinical preventive services. During the final phase of the project, NORC developed five thematic overviews to further explore several key cross-cutting themes related to the adoption, integration, and delivery of the USPSTF recommendations in health plans. In the sections below, we describe the three major project phases in greater detail.

 

Evaluation Design

NORC designed an evaluation to assess the integration of the recommendations within and across these health plans. NORC worked with AHRQ to select four health plans that vary across a number of key characteristics. NORC addressed the following three research questions in the evaluation:

 

Exhibit 1: Research Questions

Research Question 1:

To what extent are clinical preventive services (CPS) integrated into the selected health care delivery systems?

Research Question 2:

What deficiencies exist in the delivery of clinical preventive services by these health care systems?

Research Question 3:

How can AHRQ contribute to the increased implementation of USPSTF recommendations among health care delivery systems?

For each research question, NORC explored a range of issues related to the adoption, integration, delivery, and dissemination of the USPSTF recommendations in health plans. Below, we present our research questions and several corresponding sub-topics for consideration.

Research Question 1: To what extent are clinical preventive services (CPS) integrated into the four selected health care delivery systems?

  • Are the USPSTF recommendations being delivered?
  • Are the health plan sites engaged in clinical or practice change activities?
  • Are health information technology systems utilized for CPS integration?
  • Does the reimbursement structure reward CPS delivery?
  • How is the delivery of CPS measured?
  • How is the delivery of CPS monitored?
  • Are patient outcomes evaluated relative to the delivery of recommended CPS?

Research Question 2: What deficiencies exist in the delivery of clinical preventive services by these health care systems?

  • What prevents the sites from adopting the USPSTF recommendations at the systems-level?
  • Are there specific types of recommendations that are easier to adopt and/or implement than others (e.g. depression screening vs. mammography)?
  • Are there fundamental issues with the recommendations themselves that prevent adoption and implementation by health care delivery sites?

Research Question 3: How can AHRQ contribute to the increased implementation of USPSTF recommendations among health care delivery systems?

  • How can AHRQ improve its dissemination of the USPSTF recommendations to improve adoption rates at the systems-level?
  • Are the USPSTF's prevention priorities aligned with other systems-level variables, such as payer expectations, industry quality indicators, and consumer demand?
  • Are the USPSTF's prevention priorities aligned with other HHS initiatives that the sites are required to follow?
  • Are there any specific tools or information AHRQ could provide to sites to improve integration of the recommendations at the systems-level?

NORC used existing data and primary data collection to explore these research questions. Four evaluation elements were used to guide the evaluation of health plans: (1) Evaluate the impact the USPSTF recommendations have had on improving the delivery of clinical preventive services among the study sites; (2) Evaluate how effectively the USPSTF's findings are disseminated among the study sites; (3) Evaluate whether the study sites believe the recommendations are adequate in terms of prevention priorities, completeness of the recommendations, and timeliness of the recommendation; and (4) Evaluate the extent to which the USPSTF recommendations align with other HHS and AHRQ priorities and the priorities of external stakeholders such as payers and consumers.

 

Exhibit 2: Evaluation Elements

Evaluation ElementsSources of Data
Health plan CPS manualsInterviews with health plan staffCPS reports and publicationsLiterature Review
A: Impact of the USPSTF recommendations on the delivery of clinical preventive servicesxx  
B: Dissemination of the USPSTF recommendationsxxxx
C: Adequacy of the USPSTF recommendationsxx  
D: Alignment of the USPSTF recommendations with other HHS and external stakeholder priorities xxx

Key: x = Main data source

Qualitative Interviews and Data Collection

The research questions and themes were used to develop the specific approach for qualitative interviews and data collection. NORC worked closely with AHRQ to select and secure participation from four different health plans: an open-panel health plan in which physicians are independent contractors to the plan, closed-panel health plan in which physicians are health plan employees, "hybrid" health plan in which some physicians are plan-affiliated employees and some are contractors, and a governmental health plan.

Health Plan Site Selection and Recruitment

Selected in concert with AHRQ, the four health plans were chosen based on their operational mix (e.g., open-panel plan, closed-panel plan, hybrid plan, and governmental plan) and willingness to participate in the study. To protect the confidentiality of the health plan sites and the respondents, throughout the report we refer to the four sites as "closed-panel plan," "open-panel plan," "hybrid plan," and "governmental plan." Selecting these four different types of delivery models provided a valuable look at the integration and delivery of the USPSTF recommendations for clinical preventive services in health plans with different organizational structures. To increase the generalizability of our study findings, we have included four health plans that differ in scope (number of lives covered and provider sites), service area (rural versus urban), organization and management structure (health plan model), and usage of health IT. The participating health plans encompass a variety of unique characteristics that are important to the delivery and integration of the USPSTF recommendations:

  • Closed-panel health plan. The closed-panel health plan is a centralized managed care organization with regional units across the U.S. Our efforts focused on one of the plan's regional units that covers over 500,000 members. The plan directly employs physicians on an exclusive basis to provide services. As a result, the physicians of the closed-panel plan are employees of only one health plan, and do not see patients from other managed care plans. Physicians of the closed-panel health plan practice in medical facilities that are owned and managed by the plan.
  • Open-panel health plan. The open-panel health plan is a decentralized preferred provider organization (PPO), operating through nonexclusive contractual relationships with private physicians and practices who deliver care independently in their offices. The open-panel plan is an independent, not-for-profit health plan that covers more than two million people statewide. In addition to a PPO option, the open-panel plan also operates a large Medicaid managed care program.
  • Hybrid health plan. The term "hybrid plan" has been used in recent empirical literature to describe a mixed-model plan that encompasses features of both open- and closed-panel health plans.9 The hybrid health plan is composed of a core group of physicians that are also associated with a parent health system, with the remaining providers operating under contract to the health plan. Approximately half of the hybrid plan's members seek health care services from providers that are directly employed by the health system, while the other half seek services from independent providers contracted by the system. This mixed-model dynamic has interesting implications for the delivery of clinical preventive services to members.
  • Governmental health plan. The governmental health plan is part of a larger public health care system that serves a population of approximately 25 million people nationwide. For the purposes of this study, we focus on one health plan facility under the umbrella of the larger governmental system. Formed as a result of an integration of two large medical center campuses in one state, the governmental health plan offers primary and secondary medical and surgical care services, chronic and acute psychiatry care, nursing home care, and extended care for its members.

The different health plan models were selected for several reasons. First, we hypothesized that the governmental plan and closed-panel plan could more effectively facilitate the dissemination of the USPSTF recommendations and delivery of clinical preventive services through the use of clinical decision support tools via system owned information systems as well as employer incentives. Such plans may be an ideal environment for efficient adoption and integration of the USPSTF recommendations. Including an open-panel plan provided information about how clinical preventive services are delivered through a more decentralized system whereby information systems may be less integrated than in the closed-panel system. By including a hybrid system, which encompasses features of both open and closed-panel plans, we were able to examine these issues within a single health plan in addition to our analysis across health plans.

Early in our interactions with the sites, we identified a key contact at each of the four health plan sites to serve as a resource for identifying the appropriate staff for data collection. NORC sent a letter of invitation to each health plan's Medical Director which included a description of the project, the project purpose, and an outline of the activities related to participation. The letter also conveyed the importance of the project to AHRQ and its potential for improving the utilization of the USPSTF's products by health plans. NORC mailed the invitation letters by Federal Express to highlight the importance of the project and ensure receipt. Within one week of mailing the letters, NORC's project director called the Medical Directors to further discuss participation in the project, including the need for the health plan site to supply prevention materials, clinical preventive services manuals, and other relevant data.

Health Plan Staff Interviews

After participation was secured, NORC worked with the Medical Director and site contact to request contact information for participants for interviews. NORC specifically asked to interview individuals in several different kinds of positions. The Medical Director and/or site contact worked with NORC to select the appropriate health plan staff participants to fulfill the following roles:

  • Medical Director.
  • Director of Quality Improvement.
  • Director of Health Information Technology.
  • Clinical Advisors.b
  • Quality Improvement Staff.
  • Health Information Technology Staff.

Health plan sites provided NORC with the names and contact information (phone numbers and email addresses) for the prospective study participants. Prospective participants were sent an initial email that explained the purpose of the USPSTF study and requested their participation as well as a follow-up phone call to schedule a time for a one hour phone or in-person interview. Health plan staff respondents were promised anonymity in this report, and will hereafter be referred to according to their designated titles and plan only (e.g., Director of Quality Improvement from the open-panel plan, Health IT staff from the closed-panel plan, etc). In the initial email, NORC also provided study participants with a web link to the USPSTF recommendations. Neither health plans nor individual participants were offered a financial or in-kind incentive to encourage participation.

NORC developed an interview guide that is organized into modules reflecting the perspective of the particular respondent type: Medical Director, Director of Prevention and Quality Improvement, Quality Improvement Staff, Director of Health IT, Health IT Staff, and Clinical Advisor. [The interview protocols are provided in the Appendix.] Given the different types of health plans and differences in the range of activities involved in the delivery of clinical preventive services, NORC peppered the interview guide with additional follow-up questions, when appropriate, that focused on topics or issues that were directly relevant to each health plan. Consequently, no two interviews followed the exact same protocol.

NORC gathered as much information about each health plan site as possible prior to the interviews, including information collected from public and private reports, electronic media, and the published and gray literature, to ensure that we were fully informed about each site. Phone interviews were conducted with representatives of the open-panel plan, closed-panel plan, and governmental plan, and in-person interviews were conducted with representatives of the hybrid plan. One senior and one intermediate-level NORC health research staff member facilitated the interviews, assuring appropriate coverage of the research areas of interest, but allowing interviewees to discuss other issues freely. An additional NORC researcher served as a note-taker for each interview.

NORC interviewed 42 health plan staff members from four different health plans. Exhibit 3 presents the distribution of the sample across health plans and respondents. Participants were administered one of five interview protocols, depending upon their role at the health plan (e.g., the Medical Director was asked a series of questions from the Medical Director protocol). As a result, while the Director of Quality Improvement protocol was used for 12 respondents, all of these individuals were not necessarily Directors of Quality Improvement; rather these respondents were selected by their health plans to provide perspectives for this particular module.

 

Exhibit 3: Distribution of Respondents by Protocols Used

Respondents by Protocols Used (n=42)Open-Panel Plan (n=10)Closed-Panel Plan (n=12)Hybrid (n=9)Governmental Plan (n=11)
Medical Director (n=3)20*0*1
Director of QI (n=12)**4512
Director of Health IT (n=7)**3121
QI Staff (n=4)0*121
Health IT Staff (n=6)0*222
Clinical Advisor (n=9)1323
Other (n=1)***0001

*We were not able to interview respondents in these categories.
**The large number of Directors of Quality Improvement and Health IT reflects that we administered these protocols to multiple respondents at each plan. The Directors of Quality Improvement and Health IT protocols contain several high-level questions that the corresponding staff protocols do not (refer to the Appendix).
***The interviewee is a senior-level manager at the headquarters office of the governmental health plan. The respondent was asked a combination of questions from more than one protocol.

Discussion Topics

We asked individual respondents about their familiarity with the USPSTF recommendations, the adoption, integration, and delivery of the USPSTF recommendations in their health plans, challenges or barriers they face, and specific ways that AHRQ can improve the dissemination of the USPSTF recommendations to increase adoption rates at the systems-level. In addition, respondents that hold positions related to quality improvement and health IT were asked questions about the monitoring and measurement of clinical preventive services and the level of integration using health IT. Discussion topics for respondents are described in further detail in Exhibit 4 and the accompanying text.

 

Exhibit 4: Discussion Topics for Respondents

Discussion TopicMedical DirectorDirector of QIDirector of Health ITQI StaffHealth IT StaffClinical Advisors
Familiarity with USPSTF recommendationsxxxxxx
Methods of adopting CPSxxx  x
Dissemination of adopted USPSTF recs in the systemxxx  x
Monitoring and measurement of CPS xxxx 
Systems-level integration of CPS using health IT  xxx 
Barriers to adoption, integration, and delivery of USPSTF recsxxxxxx
Alignment with systems-level variables and prioritiesxx   x
AHRQ's role in improving dissemination of USPSTF recsxxxxxx
  • Familiarity with USPSTF recommendations. We asked respondents about their familiarity with the Task Force recommendations.
  • Methods of adopting clinical preventive services (CPS) recommendations. We asked respondents about the process used to review clinical preventive services recommendations from the USPSTF and other sources. We also asked about the criteria used to select which recommendations will be adopted.
  • Dissemination of adopted USPSTF recommendations in the system. We asked whether all or some of the "A" and "B" USPSTF recommendations are being delivered and how those recommendations are disseminated among health plan providers.
  • Monitoring and measurement of CPS. We asked health plan staff whether their plans are engaged in practice change or quality improvement (QI) activities. We asked respondents to list and discuss major QI activities that are used to improve the adoption and integration of clinical preventive services. Finally, we asked about the monitoring and measurement of CPS (e.g., how is the delivery of CPS measured and monitored) and whether patient outcomes are evaluated relative to the delivery of recommended CPS.
  • Systems-level integration of CPS using health IT. We asked respondents whether health IT is utilized to integrate CPS recommendations from the USPSTF and other sources.
  • Barriers to adoption, integration, and delivery of recommendations. We asked health plan staff what deficiencies exist in the delivery of CPS by their health care systems and what prevents their sites from adopting the USPSTF recommendations at the systems-level. In addition, we asked whether specific types of recommendations are easier to adopt and implement than others (e.g., screening vs. counseling) and whether there are fundamental issues with the recommendations themselves that prevent integration and implementation in health plans.
  • Alignment with systems-level variables and prevention priorities. We asked whether the USPSTF's prevention priorities are aligned with other systems-level variables, such as payer expectations, industry quality indicators, and consumer demand. We also asked respondents whether the USPSTF's prevention priorities are aligned with other state or Federal initiatives, and how the USPSTF could better align the recommendations with other variables and initiatives.
  • The impact of reimbursement. We asked whether the reimbursement structure at each plan rewards the delivery of clinical preventive services.
  • AHRQ's role in improving dissemination. We asked health plan staff how AHRQ can improve dissemination of the USPSTF recommendations in order to increase adoption rates at the systems-level. Specifically, we asked respondents about their perceptions of the packaging of the USPSTF recommendations, and whether more information about the methodology would be useful. Finally, we asked whether there are any specific tools or information AHRQ can provide to improve integration of the recommendations at the systems-level.

Data Analysis

NORC compiled the qualitative findings from the phone calls and site visit discussions into a Microsoft Access database. Interview notes were regularly reviewed to highlight unanswered questions and raise areas for further consideration. In addition, during the data analysis phase of the project, NORC mapped the research questions to the interview notes to identify key themes and findings.

Our findings should be interpreted in light of the fact that the study methods employed for this project were inherently qualitative. The four health plans cannot be assumed to represent the entire universe of health plans integrating the USPSTF recommendations. Rather, the four health plans represent a convenience sample, as each agreed to participate in the study. Nevertheless, these plans do encompass of variety of characteristics that may affect the delivery and integration of the USPSTF recommendations. The plans are unique in many aspects, including service area, geography, demographics, and methods for integrating and delivering CPS recommendations. Additionally, within each plan, we interviewed a wide sample of respondents with different focus areas, or in the case of the Clinical Advisor respondents, different medical specialties.

We developed an interview protocol that contains a variety of open-ended questions in order to gain unique insights and information from respondents. Due to the open-ended nature of the research questions, it is not possible to precisely present the findings in a quantitative fashion. Rather, the results of this evaluation are discussed more generally in the body of the findings section.

Thematic Overviews

NORC prepared five stand-alone thematic overviews which explore several key cross-cutting themes related to the adoption, integration and delivery, and dissemination of the USPSTF delivery sites. Data from qualitative interviews with health plan respondents, expert interviews, and relevant reports and scholarly literature were used to produce the case studies.

The thematic overviews explore important issues related to the adoption, integration, and delivery of the USPSTF recommendations. Where appropriate, overviews present recommended next steps and opportunities to improve the adoption, integration, and delivery of the USPSTF recommendations. The first overview explores the impact of pay-for-performance on the delivery of the USPSTF recommendations. The second explores the role of health IT in the integration and delivery of the USPSTF recommendations. In the third overview, we suggest systems changes that are needed to improve the implementation of the USPSTF recommendations. The fourth explores the impact of health plan structures on the delivery and integration of the USPSTF recommendations. Finally, the fifth overview explores the impact of a rural health care setting, and the associated challenges of delivering the USPSTF recommendations to a rural patient population.


b. For the purposes of this study, Clinical Advisors are primary care physicians or specialty care providers, who also serve in a leadership or broader prevention role at their health plan.


Return to Contents
Proceed to Next Section

Page last reviewed December 2007
Internet Citation: II. Study Methods: Evaluation of the Use of AHRQ and Other Quality Indicators (PDF). December 2007. Agency for Healthcare Research and Quality, Rockville, MD. http://www.ahrq.gov/research/findings/final-reports/uspstf/2.html