Chapter 8. Description of Ideal Evaluation Methods: Describing Patient Safety Practices

Assessing the Evidence for Context-Sensitive Effectiveness and Safety

One of the key issues in patient safety practice (PSP) research and literature is adequately defining and describing PSPs. The key goal in standardizing PSP descriptions is to provide sufficient detail on the PSP and its implementation to assess: (a) whether it improves safety, (b) the risks that random or systematic error influences the results, and (c) the applicability to other situations for others to replicate or adapt it. Other goals for standard descriptions include the ability to make biases transparent, evaluate inclusion criteria and determine heterogeneity in literature reviews, and improve the ability of journal editors and reviewers to assess the quality of the information. These goals require some basic information on the PSP itself and on key elements of what was done.

Describing the intervention in sufficient detail so that it can be replicated is a requirement included in reporting guidelines such as SQUIRE for quality improvement studies, and it also was endorsed by our technical expert panel (TEP). The key challenge in PSP studies—in contrast to pharmaceutical or surgical interventions that are concrete or highly standardized and can be precisely defined, described, and delivered across settings—PSPs generally lack sharp boundaries between the practice, implementation process, and context. Indeed, as the science regarding effective implementation strategies and the understanding of context increase, implementation and context often merge together to become part of the intervention. PSP interventions are inherently dependent on context and vice versa—some PSPs include influencing context as part of the intervention. These interventions are often fluid, since some PSPs incorporate into "the intervention" unforeseen necessary adaptations or lessons learned during the implementation process. While this could be a major problem for clinical outcome studies (for example, if physicians were allowed to vary the dose of the study drug or add additional co-interventions at their discretion), the TEP felt that such fluidity was welcome in PSP studies as long as it was measured and described well. The intervention is often iterative, evolving in response to outcome feedback and changing as context changes through the implementation process. Finally, fully developed PSPs are often complex and include multiple components. While the knowledge regarding how to effectively implement a PSP and the effect of context on implementing a PSP is growing rapidly, it is still a relatively immature science. 

Existing descriptions of PSPs in the published literature vary widely, are often incomplete, and criteria to guide researchers on how to describe PSPs, the implementation strategies, or the contexts are lacking. Some organizations have produced criteria for describing a quality improvement intervention, which may have relevance to describing a PSP. For example, the SQUIRE guidelines are an important step in describing quality improvement studies generally aimed at directly changing provider behaviors.1 Table 2 contains an excerpt of the SQUIRE guidelines (the full set of guidelines is in Appendix M; see also http://squire-statement.org/). 

The NQF also lists requirements for describing quality improvement practices (Table 3). These guidelines and requirements are generic for quality improvement interventions.

No criteria exist for how to describe the key components of a PSP intervention. Thus, we developed general principles of a PSP description based on theory and existing frameworks. We also created two specific examples based on expert opinion from the project team. Key concepts for describing a PSP and additional elements that can be helpful are included in Box 1. In describing these concepts, we recognize that the borders between implementation and context are not sharp, and that the delineations may evolve as our understanding of important contextual factors grows. Currently, the line between what is context and what is PSP is often not clear. Likewise, the PSP and its implementation are usually too intertwined to tease them apart. The criteria we present here are limited by the lack of evidence to state what should be described about a PSP, so they should be considered as general concepts that appear to be important based on principles of behavior change, other guidelines, and the overall finding of this project. 

Box 1—Key concepts in describing a patient safety practice

Target: Patient safety problem practice is intended to address:

  • Key elements of the intervention.
  • Sufficient detail of the implementation process to allow relatively comparable adaptation to another entity.
  • Population or settings where practice applies; is the intervention intended to apply to a single level of the organization or multiple levels, and if so, how many?
  • Health care professionals and administrators that were involved.
  • Personnel qualifications and additional staff training.
  • Required technology or tools.

Implementation detail should include:

  • Initial plans for intervention.
  • Step-by-step explanation or diagram of intervention.
  • Description of elements key to association with improved outcomes.
  • Whether the PSP implementation requires education for executive leaders, team leaders or staff.
  • How the study or implementation team ensured that the executive leaders, team leaders, and staff executed the PSP implementation.
  • How the study or implementation team evaluated whether the actions of executive leaders, team leaders, and staff made a difference.
  • Other barriers and facilitators of change and how these were addressed.
  • Steps for ensuring that the PSP was implemented as designed (or measuring how it was actually implemented and why changes were made).

Other possible elements to include:

  • Factors that contributed to choice of the specific intervention (e.g., analysis of causes of dysfunction; other interventions that might have been considered).
  • Resources used and which were required; feasibility.
  • Likelihood that observed gains may weaken over time.
  • Plans, if any, for monitoring and maintaining improvement (or state that such planning was not done).
  • Plans for moving from study to maintenance.
  • Ease of incorporating PSP into clinical practice or systems.
  • Generalizability of the elements of the intervention to other settings: unique features of the organization or external environment that may have influenced impact of the PSP.
  • Risk for unintended consequences.

To supplement these generic criteria for describing the PSP, we provide here more specific criteria for describing two of our five target PSPs: catheter-related bloodstream infection prevention and CPOE. We solicited input from two members of the project, Peter Pronovost and David Bates, who are international experts in designing, implementing, and evaluating these two PSPs. As such, these criteria are based mostly on expert opinion. Nevertheless, in the absence of strong empirical evidence, the opinions of recognized experts can be a valuable source of guidance for implementers and evaluators. Criteria for describing interventions to prevent catheter-related bloodstream infections are presented in Box 2, and criteria for describing CPOE interventions are presented in Box 3.

 

Box 2—Case example: Key elements in describing PSP to prevent catheter-related bloodstream infections (CRBI)

  • Items on the checklist supported by strong evidence from the Centers for Disease Control and Prevention and professional societies.
  • Efforts to ensure patients receive checklist items.
  • Who is intended to do what task when the PSP is implemented?
  • Staff education and training.
  • Internal incentives.
  • Whether there was local tailoring or an iterative process in the intervention implementation.
  • Selection and involvement of leadership (unit and executive).
  • Improving patient safety culture and teamwork.
  • Providing evidence summaries and standardized measures.
  • Identifying local barriers.
  • Removing barriers to comply with checklist.
    • Creating central line carts that store all needed supplies.
    • Asking hospital leaders to purchase central line kits that have chlorhexadine.
    • Improving culture and teamwork through the Comprehensive Unit-based Safety Program.

Box 3—Case example: Key elements in describing studies of CPOE

  • To what extent clinical and operational leadership were involved in building support.
  • Staff education and training over time.
  • Extent of tailoring in implementation.
  • Pace of implementation.
  • Project management during implementation.
  • Response time of application.
  • Level of clinical decision support implemented.
  • What type of process was put in place to identify issues with the application and correct them?
  • Measurement of alert frequency and responses to alerts.

Our Proposed Framework for a PSP Classification System 

Through a process of synthesizing existing conceptual frameworks and an expert panel consensus process, we developed a conceptual framework for describing the dimensions of PSPs. The framework includes 11 dimensions, as shown in Table 4. The process of how this was developed is described in Appendix 2.

Reference for Chapter 8

  1. Davidoff F, Batalden P , Stevens D, et al. SQUIRE Development Group. Publication guidelines for improvement studies in health care: Evolution of the SQUIRE Project. Ann Intern Med 2008; 149(9):670-6.
Page last reviewed December 2010
Internet Citation: Chapter 8. Description of Ideal Evaluation Methods: Describing Patient Safety Practices: Assessing the Evidence for Context-Sensitive Effectiveness and Safety . December 2010. Agency for Healthcare Research and Quality, Rockville, MD. http://www.ahrq.gov/research/findings/final-reports/contextsensitive/context8.html