Chapter 8. Description of Ideal Evaluation Methods: Describing
Patient Safety Practices
One of the key issues in patient safety practice (PSP)
research and literature is adequately defining and describing PSPs. The key
goal in standardizing PSP descriptions is to provide sufficient detail on the
PSP and its implementation to assess: (a) whether it improves safety, (b) the
risks that random or systematic error influences the results, and (c) the
applicability to other situations for others to replicate or adapt it. Other
goals for standard descriptions include the ability to make biases transparent,
evaluate inclusion criteria and determine heterogeneity in literature reviews,
and improve the ability of journal editors and reviewers to assess the quality
of the information. These goals require some basic information on the PSP
itself and on key elements of what was done.
Describing the intervention in sufficient detail so that it can
be replicated is a requirement included in reporting guidelines such as SQUIRE
for quality improvement studies, and it also was endorsed by our technical
expert panel (TEP). The key challenge in PSP studies—in contrast to pharmaceutical
or surgical interventions that are concrete or highly standardized and can be
precisely defined, described, and delivered across settings—PSPs generally lack
sharp boundaries between the practice, implementation process, and context. Indeed,
as the science regarding effective implementation strategies and the
understanding of context increase, implementation and context often merge
together to become part of the intervention. PSP interventions are inherently
dependent on context and vice versa—some PSPs include influencing context as
part of the intervention. These interventions are often fluid, since some PSPs
incorporate into "the intervention" unforeseen necessary adaptations
or lessons learned during the implementation process. While this could be a
major problem for clinical outcome studies (for example, if physicians were
allowed to vary the dose of the study drug or add additional co-interventions
at their discretion), the TEP felt that such fluidity was welcome in PSP
studies as long as it was measured and described well. The intervention is
often iterative, evolving in response to outcome feedback and changing as
context changes through the implementation process. Finally, fully developed
PSPs are often complex and include multiple components. While the knowledge
regarding how to effectively implement a PSP and the effect of context on
implementing a PSP is growing rapidly, it is still a relatively immature
Existing descriptions of PSPs in the published literature vary
widely, are often incomplete, and criteria to guide researchers on how to
describe PSPs, the implementation strategies, or the contexts are lacking. Some
organizations have produced criteria for describing a quality improvement
intervention, which may have relevance to describing a PSP. For example, the SQUIRE
guidelines are an important step in describing quality improvement studies
generally aimed at directly changing provider behaviors.1 Table 2 contains
an excerpt of the SQUIRE guidelines (the full set of guidelines is in Appendix
M; see also http://squire-statement.org/).
The NQF also
lists requirements for describing quality improvement practices (Table 3). These
guidelines and requirements are generic for quality improvement interventions.
No criteria exist for how
to describe the key components of a PSP intervention. Thus, we developed
general principles of a PSP description based on theory and existing frameworks.
We also created two specific examples based on expert opinion from the project
team. Key concepts for describing a PSP and additional elements that can be
helpful are included in Box 1. In describing these concepts, we recognize that
the borders between implementation and context are not sharp, and that the
delineations may evolve as our understanding of important contextual factors
grows. Currently, the line between what is context and what is PSP is often not
clear. Likewise, the PSP and its implementation are usually too intertwined to
tease them apart. The criteria we present here are limited by the lack of
evidence to state what should be described about a PSP, so they should be
considered as general concepts that appear to be important based on principles
of behavior change, other guidelines, and the overall finding of this project.
Box 1—Key concepts in describing a
patient safety practice
Target: Patient safety problem practice is intended to
elements of the intervention.
detail of the implementation process to allow relatively comparable adaptation
to another entity.
or settings where practice applies; is the intervention intended to apply to a
single level of the organization or multiple levels, and if so, how many?
care professionals and administrators that were involved.
qualifications and additional staff training.
technology or tools.
Implementation detail should include:
plans for intervention.
explanation or diagram of intervention.
of elements key to association with improved outcomes.
the PSP implementation requires education for executive leaders, team leaders
- How the
study or implementation team ensured that the executive leaders, team leaders,
and staff executed the PSP implementation.
- How the
study or implementation team evaluated whether the actions of executive
leaders, team leaders, and staff made a difference.
barriers and facilitators of change and how these were addressed.
- Steps for
ensuring that the PSP was implemented as designed (or measuring how it was
actually implemented and why changes were made).
Other possible elements to include:
that contributed to choice of the specific intervention (e.g., analysis of
causes of dysfunction; other interventions that might have been considered).
used and which were required; feasibility.
that observed gains may weaken over time.
- Plans, if
any, for monitoring and maintaining improvement (or state that such planning
was not done).
for moving from study to maintenance.
- Ease of
incorporating PSP into clinical practice or systems.
of the elements of the intervention to other settings: unique features of the organization
or external environment that may have influenced impact of the PSP.
- Risk for
these generic criteria for describing the PSP, we provide here more specific
criteria for describing two of our five target PSPs: catheter-related
bloodstream infection prevention and CPOE. We solicited input from two members
of the project, Peter Pronovost and David Bates, who are international experts
in designing, implementing, and evaluating these two PSPs. As such, these
criteria are based mostly on expert opinion. Nevertheless, in the absence of
strong empirical evidence, the opinions of recognized experts can be a valuable
source of guidance for implementers and evaluators. Criteria for describing
interventions to prevent catheter-related bloodstream infections are presented
in Box 2, and criteria for describing CPOE interventions are presented in Box 3.
Box 2—Case example: Key
elements in describing PSP to prevent catheter-related bloodstream infections
- Items on
the checklist supported by strong evidence from the Centers for Disease Control
and Prevention and professional societies.
to ensure patients receive checklist items.
- Who is
intended to do what task when the PSP is implemented?
education and training.
there was local tailoring or an iterative process in the intervention
and involvement of leadership (unit and executive).
patient safety culture and teamwork.
evidence summaries and standardized measures.
barriers to comply with checklist.
central line carts that store all needed supplies.
hospital leaders to purchase central line kits that have chlorhexadine.
culture and teamwork through the Comprehensive Unit-based Safety Program.
Box 3—Case example: Key
elements in describing studies of CPOE
- To what
extent clinical and operational leadership were involved in building support.
- Staff education
and training over time.
of tailoring in implementation.
- Pace of
management during implementation.
time of application.
- Level of
clinical decision support implemented.
- What type
of process was put in place to identify issues with the application and correct
of alert frequency and responses to alerts.
Our Proposed Framework for a PSP Classification System
process of synthesizing existing conceptual frameworks and an expert panel
consensus process, we developed a conceptual framework for describing the
dimensions of PSPs. The framework includes 11 dimensions, as shown in Table 4. The
process of how this was developed is described in Appendix 2.
Reference for Chapter 8
F, Batalden P , Stevens D,
et al. SQUIRE Development Group. Publication guidelines for improvement studies
in health care: Evolution of the SQUIRE Project. Ann Intern Med 2008; 149(9):670-6.
Return to Contents
Proceed to Next Section