Implementation science is an emerging field and is the “scientific study of methods to promote the systematic uptake of clinical research findings and other evidence-based practices into routine practice” (1). Implementation science addresses challenging questions concerning how to most effectively transport empirically supported treatments into community settings. Implementation of empirically supported treatments in the children's mental health system has only recently gained prominence and holds promise (2). Barriers to implementation of clinical innovation in the children's mental health system include poor attitudes toward innovation, difficulty embedding innovation into existing systems, and untrained providers (3). Perhaps the most daunting challenge to implementation of empirically supported treatments for youths in community settings is training clinicians to administer these complex treatments with fidelity (3). Thus one frequently used implementation strategy is training providers (2).
The most commonly used training strategy is a brief workshop (4). Workshops improve therapist knowledge and attitudes but are not effective in changing therapist behavior in providing patient care (5,6). This outcome is due to the one-time nature of training, or “training and hoping” (7), which does not allow for integration into existing practices. Two methods hypothesized to improve brief workshops are to construct workshops that are informed by the adult learning literature (8) and to follow workshops with ongoing consultation that allows providers to integrate clinical innovation into existing practices (6).
The creation of brief workshops informed by adult learning principles may result in improved therapist behavior outcomes (8). Most workshops are passively delivered despite research demonstrating that active learning, a process in which an individual acquires knowledge and skills through interaction and reflection, is preferable (9). Furthermore, it has been recommended that training focus on principles of a treatment rather than on one protocol because principles provide a foundational understanding and reduce the burden of learning multiple protocols (10). Finally, computer-assisted training holds promise (11,12) because it is well suited for limited-resource settings, such as community mental health clinics.
Improving the efficacy of brief workshops is desirable but may not be sufficient to change therapist behavior. Ongoing consultation after training may be the critical element in increasing effectiveness of training (6). For example, one study demonstrated that coaching and performance feedback on cases posttraining changed therapist behavior (13). The mechanism by which consultation works is unknown, but consultation likely provides therapists with a venue for clarification, and practice of concepts, learning concepts and practicing over time, case consultation, and using problem solving to overcome implementation barriers. The impact of adding consultation to brief training has not been examined as an implementation strategy in child mental health services and holds promise for effective provider behavior change.
This study evaluated the efficacy of three modalities of brief training and the impact of including ongoing consultation after this training. Cognitive-behavioral therapy (CBT) for treating anxiety among youths, an empirically supported treatment, was used as the exemplar. Participants were randomly assigned to a one-day workshop to examine routine training (training as usual), computer training (computerized version of training as usual), or augmented training (training that emphasized active learning and principles of CBT). After training, all participants received three months of ongoing consultation that included case consultation, didactics, and problem solving of implementation barriers specific to the local context. Outcomes included therapist adherence to CBT for child anxiety, skill in CBT for child anxiety, knowledge about CBT for child anxiety, and satisfaction with training. The study design was prospective and experimental.
+
Recruitment and screening
All procedures were approved by the Temple University Institutional Review Board. Participants were recruited from the community via professional Listservs, training programs, and word of mouth. After complete description of the study to participants, written informed consent was obtained. Recruitment began in May 2009 and was completed by September 2009. Assessments, training, and consultation began in June 2009 and were completed by January 2010.
Criteria for inclusion in the study were working in the community with children ages eight to 17 with anxiety disorders, having training in mental health, reading and speaking English, having access to a computer or telephone for consultation, and participating voluntarily. Exclusion criteria were having eight or more hours of prior training in CBT for child anxiety (13).
Before recruitment, we conducted a power analysis, which indicated that a sample size of 38 participants in each group (N=114) would result in 80% power to detect a moderate effect (Cohen's d=.6) at an alpha level of .05.
We recruited 115 participants, who were randomly assigned to one of three training modalities. Using equal allocation concealment, random assignment occurred at the level of training date. An unpredictable randomization sequence was generated that determined random assignment of training dates to condition. Participants were not informed of which training condition they received.
Participants were randomly assigned to one of the three training conditions, all of which were provided in person. After training, all participants received weekly consultation by telephone or over the Internet for three months.
This condition, involving 41 participants, included a six-hour workshop that focused on session-by-session administration of the Coping Cat program (14). The workshop was designed to follow the format of continuing-education workshops and did not emphasize active learning. Didactic instruction included a PowerPoint presentation and viewing of videotapes of representative youths receiving treatment.
This condition included a six-hour computer-based, self-guided training for 34 participants (15). It closely resembled routine training, with the exception that it was delivered via computer and was therefore self-paced. The training provided step-by-step instructions for each session of the Coping Cat program and examples and materials for the therapist to peruse (14). Active learning in this condition included videos of treatment sessions, therapist tips, and links to research articles. Although training was via computer, participants received training in a computer lab to ensure that they completed an amount of training equivalent to that provided in the other two modalities.
Forty participants attended a six-hour experiential workshop focusing on core CBT principles: coping modeling, identifying somatic cues and relaxation, identifying and challenging anxious self-talk, problem solving, and behavioral exposure (16). This training emphasized active learning through the use of behavioral role-play (17) and interteaching (18). Behavioral role-play is a technique where therapists role-play implementing a treatment and receive feedback on their performance (11), whereas interteaching is a behaviorally based instructional technique in which individuals engage in small-group discussion with guided questions (18). Core principles were introduced didactically, modeled by the instructor, and then followed by small-group activities.
Participants from all three conditions were provided weekly consultation via the WebEx virtual conferencing platform for three months after training. Participants could call in via telephone or computer to attend the one-hour weekly virtual group meeting. Those who opted to consult via computer were able to view a whiteboard and the consultant via Web camera. The consultation curriculum was designed with participant input and included case consultation, didactic topics (such as treating a client with comorbid depression), practice with concepts (such as relaxation), and assistance in implementation of the treatment within context (in a psychiatric clinic or school, for example).
+
Clinician Demographics and Attitudes Questionnaire.
This 15-item questionnaire assesses demographic characteristics and prior experience with CBT for youth anxiety (19).
+
Training Satisfaction Rating Scale.
This 12-item questionnaire assesses training satisfaction on a Likert scale from 1 to 5 (20). The score was the total of the ratings on the 12 items. Possible scores range from 12 to 60, with higher scores indicating higher satisfaction.
This 20-item test assesses knowledge of CBT for youth anxiety and was previously used in CBT training (19,21). The total score was the total of correct items.
+
Adherence and Skill Checklist.
This checklist was developed to measure therapist fidelity to CBT for child anxiety (19). Adherence, defined as the utilization of the procedures of a protocol in the treatment of a client (22), was assessed by coding the presence of six CBT competencies: identification of somatic symptoms, identification of anxious cognitions, relaxation, coping thoughts, problem solving, and positive reinforcement. The score could range from 0 to 6, with higher scores indicating better adherence. Skill, the level of competence shown by the therapist in the delivery of treatment (22), was evaluated on a Likert scale that ranged from 1 to 7. Coders were blind to hypotheses, training condition, and time point of the assessment. Rated adherence (intraclass correlation=.98) and skill (intraclass correlation=.92) demonstrated outstanding interrater reliability.
+
Performance-based role-play
A structured role-play was used to assess participants' adherence and skill in a simulated clinical setting. Research assistants blind to condition were trained to portray anxious youths. Therapists were asked to prepare the client for a treatment session that included an exposure task, a competency central to CBT. The role-plays were digitally recorded and later independently coded with the Adherence and Skill Checklist. Role-plays were conducted pre- and posttraining and at three-month follow-up.
All analyses were conducted with SPSS, version 17.0. Four outcomes were examined: adherence, skill, knowledge, and satisfaction. Analyses were conducted to isolate the impact of training modality from pre- to posttraining on the outcome variables and the impact of consultation at the three-month follow-up on the outcome variables. To test the main effects (training and time) and the interaction (training × time) on the outcome variables (adherence, skill, and knowledge), repeated-measures analysis of variance was used. Satisfaction was analyzed with one-way analysis of variance. To test the impact of consultation on outcome variables, linear regression was used to model the effect of number of consultation hours on adherence, skill, and knowledge.
Adherence, skill, and knowledge were analyzed with reference to a clinically significant criterion (12). A 70% cutoff score was used to indicate being trained to criterion for adherence and knowledge. This value was selected to be consistent with the 80% training criterion used in other evaluations of child anxiety CBT (21) but is less conservative because the participants were community clinicians naïve to CBT as a treatment for child anxiety. A skill score of 3.5 was used as the cutoff for being acceptably trained to criterion and was consistent with past trials evaluating the efficacy of CBT (5,12,23). Chi square analyses were used to investigate differences in training to criterion by condition.
+
Demographic characteristics and retention of participants
Participants were primarily female (90% of sample), and the mean±SD age was 35.9±11.4 years. Table 1 provides additional information about demographic characteristics and professional background.
Individuals who completed the pretraining assessment composed the intent-to-train sample. Of the 115 intent-to-train participants, 113 (98%) completed the posttraining assessment and 100 (87%) completed the follow-up assessment.
Mean adherence scores significantly improved over time from pretraining (1.6±1.2) to posttraining (3.0±1.4) (F=83.21, df=1 and 111, p<.001), but there was no significant effect of training or interaction of time and training (Table 2).
Participant skill significantly improved over time from pretraining (2.9±1.2) to posttraining (3.8±1.5) (F=69.06, df=1 and 111, p<.001), but there was no significant effect of training. A significant interaction of time and training was identified (F=4.55, df=2 and 111, p<.01) such that participants in the augmented training (pretraining, 2.7±1.2; posttraining, 3.7±1.5) improved their skill more from pre- to posttraining compared with participants in the computer training (pretraining, 2.9±1.3; posttraining, 3.7±1.5) (Table 2).
Participant knowledge scores significantly improved over time from pretraining (15.1±2.2) to posttraining (17.2±1.9) (F=96.74, df=1 and 112,p<.001), but there was no significant effect of training or interaction of time and training (Table 2).
There were no significant differences across training modalities in percentage of participants trained to criterion (Table 3). Note that these analyses were based on data from only the participants who completed the study, not the intent-to-train sample. Overall, from pretraining to posttraining, the number of participants trained to criterion increased on adherence (32%), skill (37%), and knowledge (15%). From posttraining to follow-up, the number of participants trained to criterion increased on adherence (23%), skill (20%), and knowledge (1%). On average, participants received 13 hours of training (six hours of training and seven hours of consultation). After receiving that dosage, approximately 96% of participants were trained to criterion in knowledge, 61% were trained to criterion in adherence to CBT for child anxiety, and 85% of participants were trained to criterion in skill.
Differences in satisfaction across training modalities were observed (F=7.22, df=2 and 112, p<.001). Planned comparisons identified that participants in routine training (53.7±5.4, t=−3.78, df=112, p<.001) and augmented training (55.5±4.7, t=2.40, df=112, p<.05) were more satisfied than participants in computer training (50.8±5.9).
On average, the 108 consultation sessions lasted 52.6±10.8 minutes and had 7.8±4.5 participants per session. The mean number of cases discussed per call was 2.7±1.9. Each participant used a mean of 7.2±3.2 consultation hours. Number of consultation hours significantly predicted participant adherence (F=20.67, df=1 and 113, p<.001, β=.39, R2=.16) and skill (F=11.46, df=1 and 113, p<.001, β=.30, R2=.09) at the three-month follow-up but did not significantly predict participant knowledge.
Two main findings relevant to the implementation of empirically supported treatments in child mental health services emerged from this study. First, a brief workshop was not effective in changing therapist behavior to implement CBT into practice. All three training modalities (routine, computer, and augmented training) resulted in limited gains in therapist adherence and skill—even brief training emphasizing active learning. Second, ongoing consultation after a brief workshop was effective; it significantly improved therapist adherence and skill and allowed therapists to solidify concepts and skills learned in the brief workshop for embedding CBT within their clinical practices.
There was no effect of training modality on therapist adherence, skill, or knowledge, although satisfaction varied across modalities. The null finding with regard to therapist adherence and skill is consistent with previous research showing that, even at its best, training by itself is not a sufficient implementation strategy (12,13). Consistent with previous findings, knowledge of CBT for youth anxiety increased for all participants from pretraining to posttraining (11,24). Inconsistent with previous studies (11,12), participants were more satisfied with the face-to-face training conditions than with computer training. However, satisfaction was high across training modalities. Given limited resources in community mental health settings and high rates of therapist turnover, computer training may be the most viable training strategy given that it is as effective as instructor-led training.
Ongoing consultation after a brief workshop significantly improved therapist fidelity to an empirically supported treatment. This important finding corroborates theory (6) and suggests that consultation should be included as a core element in implementation. Each hour of consultation after training improved therapist adherence by .4 point and skill by .3 point. These increases suggest a sizable return on investment. The virtual delivery of consultation was especially innovative because a diverse set of clinicians could interact at times that suited their schedules. Virtual consultation seems ideal for limited-resource settings, such as community mental health clinics (25). Furthermore, consultation provided an experiential laboratory where therapists could learn from peers, receive feedback from an expert consultant on actual child cases, practice skills, tailor the treatment for context, and resolve implementation difficulties. Ongoing support made it possible for therapists to embed the new clinical practice into their existing repertoire of skills.
With regard to clinical significance, after a six-hour workshop, approximately 38% of participants were trained to criterion in adherence and 65% were trained to criterion in skill. Ongoing consultation of approximately seven hours over three months brought those numbers up further, to 61% and 85%, respectively. This finding is particularly noteworthy given that other studies have demonstrated only minimal to moderate gains in adherence and skill after training and often a decline at follow-up when training was not followed by consultation (11–13,17). These findings suggest that a relatively limited dosage of training, approximately 13 hours, can result in meaningful behavior change for therapists when consultation is included.
Despite several strengths of this study, there were limitations. The focus on an empirically supported treatment for child anxiety may not be generalizable to training in other treatments. However, a number of child mental health service settings were represented (including schools, outpatient clinics, hospitals, and community mental health clinics) and resulted in high generalizability across settings. The sample was self-referred, which may not be generalizable to therapists mandated to receive training. In addition, the therapists identified highly with CBT, had moderate knowledge of CBT, were primarily female and Caucasian, and had a master's-level education, which may not be generalizable to other samples. Finally, because our study endeavored to control the number of training hours in each workshop, it was necessary for participants to receive the computer training in a computer lab. This was a limitation because in a real-world setting, computer training would occur in the home or office. Further, the computer training was not highly interactive; thus these findings may not generalize to computer training that includes more interactive design.
Although a one-day workshop can be effective in changing provider knowledge, it is not effective in changing provider behavior. We recommend that training no longer be used as a stand-alone implementation strategy. However, with the combined implementation strategies of brief training and ongoing consultation, community clinicians can be effectively trained in clinical innovations that are complex and multistep (26). Brief training may be best provided through computer delivery given that it is a permanent product that an agency can purchase that is effective, accessible, and cost-efficient and can be used in situations where there is high therapist turnover (8). Most important, consultation merits as much emphasis as brief training, and we suggest that whatever limited resources are available be used to provide consultation. Unfortunately, little empirical work has been completed on consultation as an implementation strategy. Research on supervision (27) and coaching teachers (28) can inform this emerging and exciting area. Investigation of how to train consultants, what type of consultation is most effective (individual versus group), and the merit of internal versus external consultation is needed.
This study was funded by grant F31-MH083333 from the National Institute of Mental Health and by the Shipley Research Prize awarded to Dr. Beidas. The authors thank Robert Fauber, Ph.D., Philip Hineline, Ph.D., and David Mandell, Sc.D., for comments on drafts of this article. The authors thank the following individuals for their help as consultants: Kathleen Carroll, Ph.D., Wendi Cross, Ph.D., and Kimberly Hoagwood, Ph.D. The authors thank the following research assistants: Stacey Austin, Kayla Brehm, Mamatha Chary, Margaret Downey, Philip Fizur, Kyle Peer, and Anna Slavachevskaya.
Dr. Marcus has received grant support from Ortho-McNeil Janssen and Forest Research Institute and has served as a consultant to AstraZeneca. Dr. Kendall reports receiving royalties from sales of materials related to the treatment of anxiety among youths. The other authors report no competing interests.