Chapter 2. Methods
In this chapter we describe the methods used in this study. There are
three main components to our approach: an environmental scan to
identify and catalog users of the AHRQ QIs and developers or vendors of similar
projects; a series of interviews with individuals and organizations
who use AHRQ or other QIs to understand how and why QIs are used, and what
improvements, if any, users might like to see; and case studies to
illustrate how QIs are used in two particular geographic areas.
2.1. Environmental Scan
We conducted an environmental scan to identify four types of organizations:
(1) users of the AHRQ QIs; (2) "non-users," i.e., organizations
that are using an alternative to the AHRQ QIs; (3) developers of similar sets
of quality indicators; and (4) vendors of quality measurement products that
may or may not include AHRQ QIs. The following six types of sources were
- Databases of published literature.
- Conference presentation abstracts.
- World Wide Web search engines.
- Reviews of quality measurement activities conducted by third parties.
- AHRQ materials.
- Query of RAND Health research staff.
The specific sources and the search terms are listed in Tables 2.1 and 2.2.
Each organization identified in the search
was entered into a database together with the following information:
- Type of organization (e.g., hospital).
- List of quality indicators used.
- Primary use of quality indicators (research, public reporting, pay-for-performance,
quality improvement, vendor, developer).
- Description of other uses of quality indicators.
- Contact information.
- Method of identification.
The environmental scan was used to understand the range of ways in which the
AHRQ QIs are being used and to select interview candidates.
Return to Contents
2.2.1. Selection of interviewees
In order to select interview candidates, we first consulted with five members
of the AHRQ QI team, which includes both AHRQ staff and contractors. We
used information gained from these discussions, together with the results of
our environmental scan, to compile a list of potential interviewees. We
used the following method to select interviewees from this complete list:
- We cross-tabulated the environmental scan results by type of organization
(hospital association, state government, etc.) and primary indicator use
(pay-for-performance, public reporting, etc.) to create a table representing
the universe of potential interviewees.
- We selected the number of interviewees to be chosen in each type-use cell
in order to distribute interviews across cells and ensure multiple interviews
per cell where possible.
- Two of the researchers (PH and SM) reviewed the list of organizations
independently and chose interviewees for each cell based on the criteria:
(a) maximize the variety of uses and types of organizations within each
cell; (b) maximize the estimated impact of quality indicator activity. The
reviewers compared notes and reconciled differences to come to the final
list of suggested interviewees to forward to AHRQ.
- We met with AHRQ staff to discuss the suggested interviewees and incorporated
AHRQ staff comments to arrive at the final list.
We then began the process of identifying the most appropriate respondent(s)
for each organization. Identification of the most knowledgeable respondent
was a multi-stage process. First, we identified an initial contact through
available published materials or a telephone call to the organization. We
then requested that the individual we contacted forward our interview request
to the most appropriate individual(s). Prior to the interview, interviewees
were given a fact sheet on the project, information about how the data would
be used and assurances about confidentiality, and a list of sample questions
similar to those asked during the interviews. All procedures, including verbal
consent procedures, were approved in advance by the RAND Human Subjects Protection
Committee, RAND's Institutional Review Board.
2.2.2. Interview procedures
We created interview guides based on input of AHRQ QI team staff, our own
knowledge of the AHRQ QI program, and the results of the environmental scan.
Separate guides were created for users of AHRQ QIs, developers of similar products,
and vendors of quality measurement products. The interviews covered three
- How AHRQ QIs (and other quality indicators) have been used.
- Experiences (including impact of use and lessons learned) from quality
- Suggestions for future priorities for the AHRQ QI program.
Semi-structured interviews were conducted by telephone by one researcher accompanied
by a note-taker. Some interviews were also recorded using a digital
voice recorder. Interviewees were guaranteed that none of their responses
would be reported in an identifiable format. During the interviews, we made
note of any supporting documents mentioned by the interviewees and asked for
a copy of the documents at the close of the interview. Interview
notes were completed and edited in a timely fashion and data from the interviews
and supporting materials were coded into the dimensions in the interview guide
and entered into a database for analysis. Quotes published in this report
have been reconstructed from interview notes or recordings and thus in some
cases may differ slightly from the exact wording used by the interviewee.
2.2.3. International Users
We took advantage of the (non-project related) participation of one member
of the team (SM) in an OECD Expert Group meeting on patient safety. The
meeting took place on June 29 and 30, 2006, in Dublin, Ireland. At the
meeting, we interviewed five researchers who had used the AHQI QIs in countries
other than the United States.
Return to Contents
2.3. Case Studies
Based on the results of the first round of interviews, we identified two geographic
areas in which to conduct in-depth case studies on the use of AHRQ QI indicators
for public reporting. The case study is a research strategy that is used
in many settings (e.g., policy, political science, and public administration
research; organizational and management studies; planning research; etc.). It
is an empirical inquiry in which multiple sources of evidence are used to provide
data from which judgments can be made about the usefulness of an approach to
a problem and the generalizability of the findings to other sponsoring organizations
and markets. We selected two geographic areas (Boston, MA and Dallas-Fort
Worth, TX) and identified multiple organizations within each of these markets
to provide a full picture of the various perspectives on a common application
of AHRQ QIs.
These case studies were designed to refine our notions of the critical variables
present in AHRQ QI initiatives; obtain evidence regarding how successful organizations
were in developing and implementing QI initiatives; and to document the barriers
and facilitators to using AHRQ QIs in real-world health care settings. We
first identified organizations for case study interviews by (1) analyzing the
results of the environmental scan to identify users of the AHRQ QIs in the
two geographic areas; (2) asking for nominations from representatives of organizations
who had already been asked to participate; (3) taking suggestions from AHRQ
staff. Detailed notes were taken for each case study. These notes
as well as media and policy reports on health policy issues in the geographic
areas were used as the basis for the case study analysis.
Return to Contents
Proceed to Next Section