Continued Use of Evidence-Based Treatments After a Randomized Controlled Effectiveness Trial: A Qualitative Study

In recent years, there have been increased calls for the development and use of hybrid designs to examine simultaneously the effectiveness and the implementation of evidence-based treatments in real-world settings (1,2). An extension of the concept of “practical clinical trials” (3), effectiveness-implementation hybrid designs provide more rapid translational gains in clinical intervention uptake, more effective implementation strategies, and more useful information for researchers and decision makers, among other benefits (1,4–6). Such designs may give equal priority to the testing of clinical treatments and implementation strategies or give priority to the testing of the treatment effectiveness or the implementation strategy.

Increasingly, studies of implementation are embedded within randomized controlled effectiveness trials of specific treatments and practices (7–12). However, it is unclear whether such treatments are sustained once the effectiveness trial is concluded or whether they are abandoned because of lack of funding and external support or undergo adaptation to fit specific community needs, preferences, and support resources. Such information is critical to evaluating on the one hand, the value of hybrid designs, and on the other hand, the factors that promote or inhibit sustainment of evidence-based treatments.

Sustainment, defined as continued use of an innovation in practice (13), is included as the final stage in most models of evidence-based treatment implementation (5,13–15); yet, comprehensive models of factors that support maintenance or sustainment of evidence-based practices in public service sectors are lacking (15), and the factors that facilitate or impede sustainment—such as organizational culture, leadership, funding, and staffing—are poorly understood (15,16).

To address this lack of information, we examined the use of two alternative approaches to delivering evidence-based treatments subsequent to a randomized controlled trial (RCT) designed to evaluate their comparative effectiveness. The study sample was youths ages eight to 13 years being treated for anxiety, depression, and conduct problems. Embedded in the RCT was a qualitative study of the process of dissemination and implementation of the evidence-based practices. In a previous article (6), we reported that therapists anticipated they would continue using the evidence-based treatments upon conclusion of the RCT, but in a selective fashion—in other words, using some elements with all clients and all elements with some clients. In this article, we report on therapists’ continued use of the evidence-based treatments with nonstudy clients upon conclusion of the clinical trial. Our aim was to examine therapists’ reported patterns of use of these treatments for nonstudy clients, reasons for continued use, and reasons for treatment adaptation or modification.

Methods

The Child STEPS Effectiveness Trial

The qualitative study was part of the Child System and Treatment Enhancement Projects (STEPS) Effectiveness Trial (CSET), carried out by the Research Network on Youth Mental Health, which is funded by the MacArthur Foundation. The CSET focused on children ages eight to 13 who had been referred for treatment of problems involving disruptive conduct, depression, anxiety, or any combination of these. Ten clinical service organizations in Honolulu and Boston, 84 therapists, and 174 youths participated in the project. Youth participants were treated at one of these settings with the usual treatment procedures or with three selected evidence-based treatments: cognitive-behavioral therapy (CBT) for anxiety (17), CBT for depression (18), and behavioral parent training for conduct problems (19).

These evidence-based treatments were tested in two forms: standard manual treatment (standard treatment), in which therapists use full treatment manuals in the manner tested in previous research trials; and modular treatment, in which therapists learn all the component practices of the treatments but individualize their use for each child, guided by a clinical algorithm (20) and feedback on practices and clinical progress (21). Therapists assigned to the modular condition used a collection of modules that correspond to treatment procedures included in the standard treatment manuals (22). This Modular Approach to Therapy for Children, called MATCH, prioritizes a focus on the initial problem area identified by using a default sequence of modules outlined in a protocol flowchart. If interference arises—for example, a comorbid condition or stressor impedes use of the default sequence—the sequence is altered, with other modules used systematically to address the interference. For example, if treatment begins with a focus on depression but disruptive behavior interferes, the therapist may use modules from the disruptive-behavior section of the protocol to help parents manage that behavior, returning to depression treatment when the interference is resolved.

Therapists who consented to participate were randomly assigned to one of the three conditions: standard treatment, modular treatment, or usual care. We used a cluster randomization design (23), with therapists assigned to condition by using blocked randomization stratified by therapist educational level (doctoral versus master’s). Youths who met study criteria were randomly assigned to treatment delivered by one of these three groups of therapists (24). Therapists randomly assigned to standard or modular conditions trained together for specific treatment procedures at the same workshops. However, they were given different reference materials (MATCH instructions or the standard manual) and were separated to talk specifically about issues unique to their study condition. For example, MATCH therapists would discuss common structured adaptations and use of the flowcharts, whereas the standard group would discuss strategies to maintain fidelity of the planned sequence of sessions in the face of challenges. Both groups received weekly case consultation from project supervisors familiar with the protocols to help them apply the treatment procedures to their study caseload. Therapists providing usual care received no instructions or feedback about their practice. The usual-care sessions were audio taped, and a coded sample of the content revealed that only 8% involved procedures that were represented in either the modular or the standard protocol (24).

Mixed-effects regression analyses of weekly measures of a standardized brief problem checklist and a patient-generated top-problems assessment showed significantly superior outcome trajectories for modular treatment relative to usual care. After treatment, youths receiving modular treatment had significantly fewer diagnoses than youths who received usual care (24). In contrast, none of these outcomes were significantly different among youths in standard treatment versus usual care. Follow-up analyses of the weekly trajectory measures also showed significantly better outcomes for modular treatment than standard treatment. In general, the modular approach outperformed usual care and the standard approach on the clinical outcome measures, and the standard approach did not outperform usual care. Additional details of the RCT protocol and study results are available in the article by Weisz and colleagues (24).

Dissemination and implementation study

Participants.

Participants included 38 therapists (45% of the RCT participants), all six project supervisors from the RCT, and eight clinical organization directors or senior administrators (80% of the RCT participants). All of the therapists had master’s degrees or higher. A total of 76% (N=29) of therapists were female, and 79% (N=30) were Caucasian; their mean±SD age was 40.6±10.5 years, and they had 11.1±8.4 years of clinical experience. Among them were 14 (37%) social workers, 11 (30%) psychologists, and 13 (34%) other providers, such as licensed mental health counselor. There were no significant differences with respect to treatment assignment, demographic characteristics, or clinical training of the therapists who did or did not participate in the qualitative study. The protocol was approved by the institutional review boards of the University of Southern California, the University of Hawaii, and the Judge Baker Children’s Center, Harvard Medical School. After complete description of the study to participants, informed consent was obtained.

Data collection.

A series of semistructured interviews was conducted approximately three months after the conclusion of participation in the RCT. Interviews focused on the participants’ involvement with the CSET, their experiences of using either standard or modular approaches (depending on their assigned condition), and the continued use of the treatments after conclusion of the trial. Three focus groups, one with project clinical supervisors (N=3), one with therapists in the standard condition (N=3), and one with therapists in the modular condition (N=3), were conducted as a “member checking” procedure (25) to review validity of preliminary findings from the semistructured interviews and obtain additional information on practice patterns considered relevant by participants. Copies of the interview guides are available from the first author on request.

Data management and analysis.

All interviews and focus groups were digitally recorded and transcribed. All data were analyzed by “coding consensus, co-occurrence, and comparison,” a methodology outlined by Willms and others (26) and rooted in grounded theory (27). This analysis included a review of all data to develop a broad understanding of content and to identify topics of discussion and observation, open coding by two investigators to condense the data into analyzable units, axial coding to describe connections between categories and between categories and subcategories, discussion of codes to arrive at consensus regarding their use and comparison of the codes applied to sample texts (kappa=.89), use of NVivo 9 to generate a series of categories arranged in a treelike structure connecting text segments grouped by categories of code, or “nodes,” to further the process of using axial or pattern coding to examine the association between different a priori and emergent categories, and condensing of categories into broad themes through the process of constant comparison.

Results

Continued use of evidence-based treatments

During the clinical trial, all of the therapists assigned to the standard condition (N=13) and the modular condition (N=15) had attended training sessions in each of the three evidence-based treatments. Five of the ten therapists assigned to the usual-care condition received training in MATCH at the conclusion of the trial, but without follow-up clinical supervision of its use, and none had yet incorporated it into their practice.

Twenty-six of the 28 therapists (93%) who had been assigned to the standard or the modular condition reported using the techniques, for example, the fear ladder with clients with anxiety and homework with clients with disruptive conduct, with nonstudy clients subsequent to the conclusion of the trial. One therapist assigned to the standard condition reported that only one of his clients had been a study participant. Once the study ended, he returned to delivering services as usual, in part, because he felt unsure about whether he was applying the treatment correctly. Another therapist, assigned to the modular condition, reported never being assigned a client who was a participant in the study and thus, lacking the clinical supervision provided to participating therapists, did not feel competent to deliver the treatment as instructed.

Reasons for continued use

Reasons for the continued use of the three evidence-based treatments are illustrated in Table 1 with quotes from therapists. The primary reason for continued use among all 26 therapists was their personal experience with the effectiveness of the treatments, as demonstrated by improvements in their clients’ behaviors. Initial skepticism about the efficacy of CBT among therapists whose training reflected a more psychodynamic tradition and concerns about a lack of control over treatment were dispelled. Therapists also reported an improvement in morale because they were learning something new. Seven (27%) therapists also appreciated that the behavioral-problems modules were effective because they engaged parents in the treatment process, even though many therapists had no prior experience with eliciting such parental involvement in a child’s treatment.

All 26 therapists also cited the positive interactions and relationships with members of the research team as a reason for their continued use of the treatments. The research team was perceived as respectful and accommodating to the needs of the therapist and the organization. All 26 therapists in the standard and modular conditions stated during the interview that they valued the training and supervision and thought the researchers were helpful and accessible. However, the therapists assigned to the modular approach were more likely than therapists assigned to the standard approach to report that the approach they were assigned to allows for more accommodation and negotiation. Both therapists and supervisors felt that the modular approach gave them more “license” to negotiate with researchers with respect to circumstances in which the modules could themselves be modified or, more often than not, supplemented with additional materials and techniques acquired through experience with working with similar clients. As one therapist assigned to the modular condition observed, “I felt they gave me a lot of freedom within the protocols to kind of present things in a style that felt comfortable to me.”

A third reason for continued use, cited by 18 (69%) therapists in the modular and the standard conditions, was the structure of the treatments. Most therapists found the manuals and the training to be very concrete and easy to follow and considered the structure to be a useful tool for organizing their own treatment plans. Finally, 11 (42%) therapists reported that the treatment increased their confidence in treating clients because it is evidence based.

Patterns of use

Of the 26 therapists who reported using the treatments with nonstudy clients, 24 (92%) reported making some form of adaptation or modification. This group included all 14 therapists assigned to the modular condition, which included a coordinating framework for making informed adaptations of the protocols, and ten of the 12 therapists assigned to the standard condition, who were provided manuals with explicit instruction in how to use the treatments. Only two of the therapists assigned to the standard condition (17%) indicated they had continued using the treatments as they had been trained. The most typical pattern was to use selected components or modules of a treatment with all clients in need of that treatment (consistent with the modular approach) or all modules with some clients (consistent with the standard approach), but not the entire protocol with every client in need of treatment for a particular disorder, such as depression. A second pattern of adaptation, reported by 22 (85%) study participants, was to use the protocols with clients who did not meet the criteria specified in the clinical trial itself. A third pattern of use, reported by 19 (73%) therapists, was to make changes in the presentation of the materials, either by rearranging the order in which the components were delivered or by making changes in the tools used to facilitate the presentation of the modules. As explained by one therapist assigned to the standard condition, “I definitely loved the tools that were there. I just don’t necessarily use them in that order or necessarily all of them.”

Reasons for adaptations and modifications

Our analyses of the qualitative data revealed several reasons for adapting or modifying the protocols for nonstudy clients. These reasons were grouped into three broad categories: client centered, therapist centered, and organization centered. Illustrative quotations reflecting the reasons for adapting or modifying the protocols are provided in Table 2.

Client-centered reasons.

All of the therapists who reported the selective use of the treatment modules or components indicated that they did so because specific needs or circumstances of their clients limited therapists’ ability to use the treatments, especially the standard approach, as instructed. For instance, nine (35%) therapists reported difficulty in using the modules as instructed while attempting to respond to their clients’ frequent crises. Five (19%) therapists reported not using the treatments as instructed because either the client or the client’s parents were unwilling or unable to participate in treatment as directed by the protocols or because the clients took a particular interest in some specific protocol element. Four (15%) therapists used the treatments with clients who did not meet the age criterion of the clinical trial by making adjustments to the modules to fit the developmental stage of the child. Three therapists expressed a preference for making the treatment “client centered” by allowing clients to decide which elements of the protocol to use. Two (8%) therapists in Hawaii reported having to “translate” the modules to make them culturally appropriate to their clients by framing elements of the parent-training intervention in terms of traditional parent-child relations, avoiding technical jargon, and using cultural idioms of behavior and roles.

Therapist-centered reasons.

Almost all (N=35, 92%) therapists believed that flexibility in using the treatments was consistent with their approach to treatment in general. Eight (31%) therapists sought to integrate the new treatments with more familiar and time-tested therapeutic modalities. Four (15%) therapists reported concerns that using the protocols as instructed would interfere with the development of a therapeutic alliance with the client because the structure interfered with the normal flow of conversation and development of client trust in the therapist. Three (12%) therapists reported that exclusive use of scientifically validated treatments detracted from the “art” of treatment. Finally, two (8%) therapists reported that rigid adherence to strict protocols was inconsistent with their personality.

Organization-centered reasons.

Ten of the 24 therapists (42%) who reported making modifications to the standard or modular intervention were required to do so as a condition imposed by the work setting. These conditions included the priorities assigned to services delivery and related measures of performance. For instance, in school-based settings, academic performance is the primary indicator of performance—mental health treatments are viewed as important only to the degree that they help to improve academic performance. Thus use of both standard and modular approaches would be terminated as soon as there is evidence of improvement in academic performance, regardless of mental health status. In community clinic settings, where the primary indicator of performance is the number of clients seen, the standard approach of completing every module for every client was not as desirable as a modular approach that reduces the amount of time required for each client, thus enabling the therapist to see more clients. School-based policies determine when, where, and for how long therapists can work with students and parents. Community clinic policies prevent some therapists from engaging with clients outside the office, resulting in some modifications to the anxiety treatment module, which requires spending time with the client outdoors in the community.

Discussion

The results of this study suggest that engagement of therapists in a hybrid RCT of effectiveness and implementation of evidence-based treatment can and does lead to some form of sustainment in community-based settings. Almost all of the therapists interviewed in this study reported use of some or all of the three treatments for nonstudy clients after the trial.

There were four primary reasons for continued use: therapists came to appreciate the utility of the treatments after seeing for themselves the positive outcomes associated with their use; they valued the interaction and support from the researchers and treatment developers and trainers; they valued the structure and organization of the treatment protocols; and they valued the fact that the treatments are evidence based. Successful outcomes, positive interactions with the purveyors of evidence-based treatments, structure of treatment, and a foundation grounded in research appeared to be important to the continued use or sustainment of the treatment protocols.

Our results also suggested that continued use involved some form of modification or adaptation of the standard, manualized versions of the treatments. There were three patterns of modification or adaptation. The first and most common pattern was the use of some of the modules in each protocol with all of the clients or all of the modules with some of the clients, but rarely all of the modules with every client who met criteria for use as specified by the treatment developers. This was true for both the therapists assigned to the standard condition, which encourages completion of all modules for treatment of a specific mental health problem, and the therapists assigned to the modular condition, which explicitly encourages adaptation or modification of the model depending on the client’s needs or circumstances.

The second pattern was to use the modules with other types of clients, including youths with co-occurring disorders other than anxiety, depression, or conduct-related disorders, youths who did not meet the age criteria, and even some adults. The third pattern was to change the order or presentation of the modules to improve the flow or to work around more immediate issues. These patterns were not mutually exclusive.

Our results also indicated three predominant reasons for selective use, modification, or adaptation. The first reason was related to the reported willingness and ability of clients to work with therapists under the guidance of the protocols. Therapists made modifications to the protocols because they perceived that the protocols did not meet the needs of their clients; because they were compelled to deal with more immediate issues or crises; because of a desire to make the treatment client centered by allowing clients to choose the elements of the treatment with which to engage; and because the youths or their parents were either unable or unwilling to perform many of the activities as prescribed, usually for developmental (youths), logistical (parents), or cultural (both youths and parents) reasons.

The second reason for modification was the preferences and priorities of the therapist. Therapists deviated from the protocols as presented by the treatment developers and trainers because of an overriding conviction in the importance of flexibility, even with the modular approach; because of a desire to integrate the new techniques with usual and more familiar approaches to treatment; because of concerns that the protocols would interfere with the therapeutic alliance with the client; because of concerns that therapy should be as much an “art” as a “science”; and because the structure was inconsistent with their own personality.

The third reason was the resources and constraints imposed on treatment by the organizational context. Therapists in school-based settings, for instance, could not use the protocols in the recommended order of presentation or with all modules because of limitations imposed on them by the school schedule, the inability to visit the home, and the priority given to academic performance over mental health. Therapists in community-based settings were under pressure to see more clients, thereby reducing the time available to use the entire protocol of each treatment with individual clients.

The results of this study are consistent with previous studies that have documented organizational (28–31), provider-based (9,32–34), and client-based (9,35) barriers or constraints to the implementation of evidence-based treatments and practices. They are also consistent with previous studies that have focused on adaptations of evidence-based treatments to accommodate the needs of clients belonging to specific age (36) or racial-ethnic (37) groups, provider preferences (38), or organizational context (39).

This study also revealed distinctions between the adaptation of the content of the treatments and the adaptation of the process of using them. Content adaptations were made because organizational constraints limited the opportunities to use every module, therapists concluded that specific modules were not appropriate for the client, or the client (youth or parent) was unable or unwilling to use certain modules. Similarly, process adaptations were made because organizational constraints limited full implementation—for example, if teachers could not or would not visit parents in their homes, they met with them at school; therapists believed that altering the flow of module presentation or delivery would help build the therapeutic alliance, which some regarded as a greater priority than fidelity to the treatment; treatment was interrupted by crises; efforts were made to integrate the new protocols with usual care; and clients were unwilling or unable to use certain modules. Adaptations of both content and process were made as a result of a process of trial and error by therapists, therapists’ desire to make the treatment “client centered” by giving the client a choice or role in deciding in which modules to engage, and therapists’ view that treatment was both an “art” and a “science.”

There are several additional important implications of the findings of this study. First, the naturally occurring adaptations of the evidence-based treatments for nonstudy clients were consistent with the modular approach to treatment use and may help explain why this approach was found by the RCT to produce significantly better outcomes than the standard approach or usual care (24). Such an approach gives the therapist greater flexibility in using the treatment and greater control over the treatment process and is consistent with usual practice.

Second, the findings point to limitations as well as strengths of evidence-based treatments. Variations in organizational and system contexts, therapist skills and preferences, and client needs and characteristics make it difficult if not impossible to use evidence-based treatments that provide little latitude for modification or adaptation. For these types of treatments, giving priority to fidelity may be inconsistent with the increased emphasis on a client-centered approach to treatment (40). Health care changes consequent to the Affordable Care Act are leading states to promote personalized medicine, patient-centered approaches, and outcome monitoring. Use of a modularized approach to treatment and of approaches that allow flexibility is consonant with these larger system changes, even though such use may be inconsistent with the expectations of treatment developers regarding fidelity. States are creating performance-monitoring metrics to encourage use of specific health care indicators to assess, track, and hold accountable health providers, including behavioral health providers, for client outcomes (41). Flexible and adaptable therapies that target specific outcome indicators, such as improvement in functioning and reduction of symptoms, will be used increasingly within this rapidly changing health care context. Similar approaches to using evidence-based treatments within systems of care have been illustrated elsewhere in the literature (42,43). Specifically, use of modular design principles and clinical models that allow structured adaptation of treatment in response to real-time feedback have shown promising results (44).

Several limitations to our study deserve mention. First, this study focused on the factors contributing to sustainment of the evidence-based treatments from the perspective of the therapists, who represented only one group of stakeholders involved in the process of implementation. Other factors known to influence implementation processes and outcomes, such as the availability of funding or broader sociopolitical support for such treatments—for example, consumer demand and government legislation—and the culture and climate of mental health services agencies (11–13), will be presented in a subsequent study. Second, this investigation relied on self-reports of use of evidence-based treatments for nonstudy clients. There are a variety of methodological challenges in assessing practice, and there are potential limitations of relying solely on therapist self-report. However, these reports were verified by both clinic directors and CSET clinical supervisors, who stated during their interviews that study therapists as a group were continuing to use the treatments as indicated in the therapist interviews. Third, the therapists participating in this study may not represent the broader population of therapists who participated in the CSET or otherwise engage in child and adolescent mental health services. Although we found no differences between CSET participants who did and did not participate in this study with respect to demographic characteristics, condition assignment, clinical training, and years of experience, there may have been other characteristics of therapists, such as attitudes toward evidence-based treatments and theoretical orientation, that distinguished the participants from the nonparticipants. Fourth, as a qualitative study, both the collection and the interpretation of data were susceptible to subjective bias and preconceived ideas of the investigators. However, the use of multiple groups of participants (therapists, CSET supervisors, and clinic directors) to achieve “triangulation” (45) was designed to minimize such bias.

Conclusions

Despite these limitations, the results of this study suggest that once trained in the use of a specific evidence-based treatment, community-based therapists were highly likely to continue using it and likely to use it selectively or to make modifications or adaptations by either using some parts with most or all clients or all parts with some clients, by reorganizing the order or presentation of modules, or by using the treatment with groups of clients who did not satisfy evidence-based criteria for use. These patterns of use were determined by the therapist to meet specific needs of individual clients and their families, to integrate them with their own clinical priorities and experience, and to address constraints imposed by the systems or organizations that employ them. Consideration of these factors would seem to be important both in the design of effective evidence-based treatments and in their successful implementation in community-based settings. Future research efforts are recommended to determine whether such consideration results in more effective and sustainable treatments.