Commentary: Fidelity Measurement, the CSI, and the Current Evidence Base

Current approaches to the measurement of fidelity are monochromatic—that is, the assessment is of a single program and focuses on provider behaviors. In reality, clients are multifaceted, receive multiple treatments from multiple providers concurrently, and may benefit from some services but not others. An article by Falloon and colleagues (1) in this issue of Psychiatric Services describes the Clinical Strategies Implementation Scale (CSI), which seeks to capture these real-world complexities.

The CSI raises several essential questions, three of which are represented by pairs of scale points: When are practices harmful? (points -2 and -1), What are the critical ingredients of a practice? (1 and 2), and What are the mechanisms of action of the practice? (3 and 4). Another question raised is How do we most optimally combine practices for a given individual? The CSI is an important first step toward integrating these issues into fidelity assessment. However, the current version obscures the importance of these issues and introduces psychometric problems by combining them into a single item. These important issues should be assessed independently and added only when the current evidence base supports their inclusion.

The negative points on the scale acknowledge that some practitioner behaviors can be harmful, such as confrontation of family members. However, these negative provider actions may represent a small percentage of behaviors overall, and the ones that are consistent with the practice may be predominant. The CSI includes both negative and positive clinician behaviors within a single item, making it impossible to rate such a program accurately.

The middle two scale points are closest to current fidelity assessment, measuring whether critical elements of a practice have been displayed by the practitioner. Unfortunately, there is no research to guide the selection of one single element, as done with the CSI. Many evidence-based practices for schizophrenia are bundled, with multiple components and techniques. Review papers and correlational studies have sought to identify critical ingredients of these practices, but the literature has not advanced past the point of speculation to identify a single necessary and sufficient program element. Until ample deconstruction studies have been done to parse out the most critical of program factors, we must resign ourselves to assessing multiple program domains.

Falloon and colleagues argue that measuring providers' behaviors is not sufficient and that to demonstrate optimal treatment there must be evidence that clients are implementing learned strategies. Thus high points are given on the CSI when families demonstrate problem-solving skills in daily practice. These authors thus suggest a mechanism of action, specifically that family psychoeducation is effective because families use problem-solving and interpersonal communication skills. To establish a mechanism, one must show a significant correlation between the treatment and the mechanism and between the mechanism and the desired outcomes. However, it is not yet established that acquisition of these skills results in distal outcomes. Should we achieve the laudable goal of understanding mechanisms behind specific practices, we will be able to better tailor treatments and optimize the change process. However, including presumed mechanisms in fidelity assessment is premature.

Understanding mechanisms will also help us understand how best to combine treatments. Embedded in the CSI is the assumption that client change accelerates when clients receive multiple services. However, many of these combinations have not yet been empirically tested. In general medicine, there is growing recognition that application of multiple treatment guidelines for complex patients can sometimes result in contradictory practices, and even harmful outcomes (2). We cannot simply assume that more is better.

It is an enormous challenge to create a simple, user-friendly scale to measure the complexity of real-life practice, and there will be no easy solutions. Falloon and colleagues have taken a bold initial step toward this worthy goal, thus beginning an important discussion about how best to define and assess optimal treatment.

Dr. Resnick is associate director of the Department of Veterans Affairs Northeast Program Evaluation Center and assistant professor in the department of psychiatry at Yale University School of Medicine in New Haven, Connecticut. Send correspondence to her at Northeast Program Evaluation Center, 950 Campbell Avenue, VA Connecticut Healthcare System (182), West Haven, Connecticut 06516 (e-mail, [email protected])