Using Technology to Train Clinicians in Evidence-Based Treatment: A Systematic Review

There is a critical shortage of clinicians trained in evidence-based treatments (EBTs), and this shortage is a major public health concern because it limits patients’ access to effective mental health treatment (1, 2). Increasing clinician access to professional training on EBTs—a potential solution to the shortage—has been named a priority by the National Institute of Mental Health (2). Currently, the most common methods for training clinicians in EBTs remain workshops, therapy manuals, and live consultation or supervision (3).

Workshop-only methods are credited with increasing knowledge about EBTs, but they have been criticized for producing insignificant gains in attitudes, application of knowledge, and skills (4, 5). Manual-based training has been shown to be suboptimal compared with multicomponent training modalities (6, 7). Another common method of training clinicians in EBTs involves a two-step process in which the clinician completes a specialist training workshop given by an expert. The clinician is then supervised while providing treatment by someone experienced at delivering the treatment. This approach has yielded better outcomes, such as increased adherence and competence (1, 5). However, because of its high cost (8) and a lack of people qualified both to conduct the workshops and to provide clinical supervision, this method is incapable of meeting the demand for training (9).

Fairburn and Cooper (10) highlighted the need for new forms of training that are more cost-effective and scalable, and technology may be well suited for this purpose. The Internet, video conferencing, mobile applications, and other technologies provide a rare opportunity to increase accessibility to specialized training and enhance traditional face-to-face training while reducing training cost. Over the past 2 decades, there has been increased use of technology to provide clinicians with training in EBTs, particularly the use of Web-based training methods (11).

Two reviews to date have examined the use of Web-based training methods for clinicians. Calder and colleagues (12) conducted a systematic review of Web-based training methods for substance abuse counselors. Because of the small number of included studies, the authors were unable to draw definitive conclusions, although their findings suggested that Web-based training might be effective under certain conditions. Jackson and colleagues (13) also conducted a systematic review of Web-based training, in this case to train behavioral health providers in evidence-based practice across various disorders. They found that Web-based training may result in greater posttraining knowledge and skill acquisition compared with scores at baseline. However, their review included a mix of studies, ranging from case studies to randomized controlled trials (RCTs), which may have limited the conclusions, given that potential biases are likely to be greater for nonrandomized studies compared with RCTs (14).

This systematic review extends the work of Calder and colleagues (12) and Jackson and colleagues (13) by examining whether other types of technology, such as mobile applications and social media, have been used to train clinicians in EBTs and by including studies published after October 2017. The review uses a standardized quality assessment to rate bias in order to be able to draw more definitive conclusions regarding the effectiveness of using technology to train clinicians in EBTs.

Methods

Articles were identified through a search of the PsycINFO, PubMed, Medline, Web of Science, CINAHL, and Cochrane Library databases for articles published on or before June 30, 2018. The following key search terms were used: “Internet OR Internet-based OR web-based OR mobile app* OR smartphone app* OR technology” AND “therapist OR clinician” AND “training OR education.” The addition of an asterisk to a term captures all derivatives of the term (e.g., “app*” captures application and apps). (A sample search strategy is available in an online supplement to this review.)

The title, abstract, and/or full paper were assessed to determine which studies met the following inclusion criteria: sample comprised mental health workers, focus on technological intervention to train clinicians, outcome measures related to training, a comparison group included, and publication in an English-language, peer-reviewed journal. Single case reports, editorials, reviews, abstracts, and protocol papers were excluded. All potential studies were independently assessed by both authors. The reference sections of included articles were also hand-searched to identify other relevant studies. Additionally, the authors of the included articles were contacted and asked whether their research groups had published any additional articles. We then extracted and summarized information from the remaining articles using a data extraction sheet developed on the basis of the Preferred Reporting Items for Systematic Review and Meta-Analysis Protocols (PRISMA-P) (15).

With the exception of studies describing secondary data analyses, between-group effect sizes (ESs) for the primary outcomes of each study were estimated by using Hedges’ g. A variation on Cohen’s d, this ES corrects for biases due to small sample sizes (16). For cases in which the primary outcomes were not specified or in which multiple measures of the same construct were examined, only the first outcome described in the Methods section of the article was reported. Hedges’ g ES may be interpreted with Cohen’s convention (17) for small (ES=0.2), medium (ES=0.5), and large (ES=0.8) effects. Negative results were adjusted to be positive for ease of interpretation. Given the heterogeneous quality of studies and the difficulty in extracting ESs from some of the data descriptions, a meta-analysis was not conducted.

Each study was assessed for risk of bias by using the Quality Assessment Tool for Controlled Intervention Studies from the National Heart, Lung, and Blood Institute (NHLBI) (18). This tool assesses bias in controlled trials by using 14 criteria, including the method of randomization, whether outcome assessors were blind, and an evaluation of participation rate. The criteria are rated as yes, no, cannot determine, not reported, or not applicable, and an overall rating of quality is provided for the study (good, fair, or poor). Both authors independently rated the quality of all 24 studies. To assess interrater reliability, Cohen’s kappa was used. The kappa coefficient obtained in the study was 0.81, which represents a high level of agreement (19). This systematic review adhered to the PRISMA-P guidelines (see online supplement).

Results

The database search resulted in 7,767 potentially relevant citations. Review of titles and abstracts resulted in 122 full-text articles to be considered for possible inclusion. Fifteen articles met the inclusion criteria. Nine additional articles from the hand search met inclusion criteria. (A PRISMA-P flow diagram of our search history is available in the online supplement.)

Description of Studies

Most articles included a randomized controlled design in which assessments were conducted before and after randomization. The sample sizes across the 24 studies ranged from 35 to 363. Most studies included mental health professionals, such as psychologists, psychiatrists, nurse practitioners, counselors, and social workers. Sixteen studies recruited participants from the community, two from university settings, three from medical centers, two from outpatient clinics, and one from an addiction unit. The largest number of studies were conducted in the United States (N=19), followed by the United Kingdom (N=2), Russia (N=2), and Australia (N=1). The main study characteristics are summarized in Table 1 (3, 6, 7, 20–40).

Methodological Quality

Each study was assessed for risk of bias by two reviewers using the NHLBI’s Quality Assessment Tool for Controlled Intervention Studies (18). Eighteen studies were rated poor (3, 7, 20–22, 24, 26, 27, 29–34, 36–38, 40), five were rated fair (6, 23, 25, 28, 35), and one was rated good (39) (Table 2). According to the NHLBI guidelines, good-quality studies include strict adherence to most NHLBI criteria. Poor-quality studies have one or more “fatal flaws” indicating high risk of bias, such as high overall or differential dropout rates and absence of intent-to-treat (12) or other suitable analyses. Fair-quality studies have limitations such as use of unreliable or invalid measures, dissimilar groups at baseline, and low adherence to intervention protocols.

Training Programs

Studies on depression and anxiety.

Two studies examined using technology to train clinicians in treating depression and anxiety. Both were rated as having poor quality. Bennett-Levy and colleagues (20) used PRAXIS CBT for Common Mental Health Problems, a 12-week, 30-module online training (OLT) program consisting of 60-minute modules for training clinicians in rural and remote areas of Australia in cognitive-behavioral therapy (CBT) for depression and anxiety disorders. Clinicians were randomly assigned to PRAXIS CBT alone (independent training) or PRAXIS CBT plus 15 minutes of supervision by telephone or Skype (supported training). There were no significant group differences in CBT knowledge scores at postassessment and at follow-up. Completion rates were significantly higher among those in supported training (96%) compared with independent training (76%).

Similarly, Hubley and colleagues (21) used an OLT program to train clinicians in behavioral activation (BA) principles and treatment strategies. Participants were randomly assigned to receive either BA OLT or a placebo titled “DBT Validation Strategies” (control OLT). BA OLT consisted of 81 screens organized into six modules on BA principles and took 90 to 120 minutes to complete. Control OLT participants received dialectical behavior therapy (DBT) “validation strategies,” which instructed them on how to validate a client in therapy and were comparable to BA OLT in quality, length, and design elements. BA OLT participants scored significantly higher than control OLT participants on a BA knowledge test at postassessment and follow-up, with large ESs. BA OLT participants rated the training course as both relevant and usable.

Studies on anxiety-related disorders.

Seven studies examined use of technology to train clinicians in treating anxiety-related disorders. Five studies were rated as having poor quality, and two were rated as fair. Chu and colleagues (3) compared the effectiveness of various low-cost extended support methods following initial participation in “Evidence-Based Treatment for Anxiety Problems: Cognitive Behavioral Strategies,” a 6.5-hour online workshop on CBT for anxious youths. Following the workshop, clinicians were randomly assigned to either 10 weeks of streaming content with an expert (i.e., weekly video of an expert providing supervision to trainees), peer consultation (i.e., 1-hour weekly peer-led groups to discuss their current caseload), or weekly review of a one- to two-page fact sheet. No significant group differences were observed on a CBT knowledge test. Notably, scores on the knowledge test and self-reported beliefs about knowledge and skill decreased from pre- to postassessment. There were also no significant group differences in satisfaction ratings.

Ehrenreich-May and colleagues (22) examined the effectiveness of a 12-week online program to train clinicians in CBT for adolescent panic disorder on the basis of the Mastery of Anxiety and Panic for Adolescents treatment manual (TM) (MAP-A) (41). Clinicians were randomly assigned to receive either the MAP-A manual (TXT); the manual plus MAP-A OLT (TXT+OLT); or the manual plus OLT and a learning community (LC) (TXT+OLT+LC), which included weekly group conference calls and online discussions facilitated via Twitter. There were no significant differences between groups in knowledge scores at post- and follow-up assessment. TXT participants were significantly less satisfied with their training than their counterparts.

Two studies examined the use of FearFighter, a nine-session, self-help online program for training clinicians to treat people who have panic disorder and specific phobias. McDonough and Marks (23) compared computer-assisted instruction with face-to-face teaching in training third-year medical students in exposure therapy (ET). All participants first received a 20-minute lecture on CBT before being randomly assigned to computer-assisted instruction or the face-to-face tutorial. Those in the computer-assisted group used a shortened computer version of FearFighter for 90 minutes. Participants in the tutorial group received a 90-minute tutorial in ET. There were no significant group differences at postassessment. Tutorial-group participants reported significantly higher satisfaction ratings than participants in the computer-assisted group. Gega et al. (24) randomly assigned mental health nursing students to 1 hour of training in either FearFighter or a lecture group. After participants completed postassessments, they crossed over to the opposite group and completed an additional hour of training and postassessments. There were no significant group differences in knowledge scores at either postassessment point. Additionally, there were no group differences in satisfaction ratings.

Two studies examined the effectiveness of Foundations of Exposure Therapy (FET), a 10-hour online program to train clinicians in ET for anxiety disorders. Harned and colleagues (25), testing an early version of the program, randomly assigned clinicians to ET OLT alone, ET OLT plus motivational interviewing (MI) (ET OLT+MI), or placebo OLT titled “DBT Validation Strategies” (control OLT). Participants in the ET OLT+MI group received one to two brief phone calls based in MI to reduce ambivalence about adopting ET. The active training groups had significantly higher knowledge scores than the control OLT group at posttraining and at 1-week follow-up, with large ESs, but they did not differ significantly from each other. Participants in both active conditions rated their training comparably, and they found training significantly more acceptable than did control OLT participants.

Harned and colleagues (26) further elaborated on these results by randomly assigning a larger group of participants to either FET OLT alone, FET OLT plus motivational enhancement (ME) (FET OLT+ME), or FET OLT+ME plus a Web-based LC (FET OLT+ME+LC). Those assigned to FET OLT+ME received an additional motivational enhancement intervention to address attitudinal barriers to using ET. ME was a two-phase intervention that included watching a 5-minute video and having simulated conversation with a virtual ET consultant following completion of FET OLT. Participants assigned to FET OLT+ME+LC were allowed to join a Web-based LC that provided support during and after FET OLT+ME training. It consisted of eight 1-hour meetings held via an online conferencing platform over 12 weeks. Participants in the FET OLT+ME+LC group had significantly higher knowledge scores than their counterparts at postassessment and at follow-up, with large ESs. There was no significant difference between the FET OLT and OLT+ME groups. There were also no significant group differences in satisfaction with the FET program.

Finally, Rakovshik and colleagues (27) compared Internet-based training in CBT for anxiety disorders with and without supervision among a sample of Russian and Ukranian clinicians. This training consists of 20 hours of online presentations from the Oxford Cognitive Therapy Center that are completed across 3 months. Participants randomly assigned to the Internet-based training plus consultation worksheets group completed one translated version of Padesky’s consultation worksheet per month during training. Those assigned to Internet-based training plus Skype supervision completed one consultation worksheet and received three 30-minute individual supervision sessions per month. Participants in the nontraining control group received no training during a 3-month wait period. Participants with Skype supervision had significantly higher CBT competence scores at posttraining than consultation worksheet participants and participants in the control group, who did not differ significantly from each other, with medium and large ESs, respectively.

Studies on substance use disorders.

Five studies examined using technology to train clinicians in treating substance use disorders. Four studies were of poor quality, and one was of fair quality. Four studies examined use of traditional versus technology-based formats of the National Institute on Drug Abuse (NIDA) manual for training clinicians in CBT for substance use disorders (42). Sholomskas and colleagues (7) conducted a nonrandomized trial comparing three training methods. Participants in the manual condition (manual) spent 20 hours studying the NIDA manual. Participants in the manual plus Web condition (Web) had access to the NIDA manual and spent 20 hours working with an interactive online program based on the manual that included multiple-choice questions and virtual role-plays. Last, participants in the manual plus seminar and supervision condition (seminar+supervision) had access to the NIDA manual and attended a 3-day didactic seminar to review the manual. Additionally, participants in seminar+supervision practiced CBT skills with their patients over the next 3 months and submitted audiotaped CBT sessions for review by supervisors. Seminar+supervision group participants had significantly greater improvement than their counterparts on objective ratings of skills and adherence, whereas Web group participants had significantly greater improvement than participants in the manual group. ESs ranged from small to large.

Weingardt et al. (28) also compared three conditions for training in treatment of substance use disorders. Participants randomly assigned to the Web-based training (WBT) group completed a 60-minute online version of the Coping With Craving module from the NIDA manual. Those randomly assigned to the face-to-face (FTF) workshop group completed a 60-minute, expert-led workshop presenting the same content provided to the WBT group. Participants randomly assigned to the delayed-training control group watched an unrelated video for 60 minutes. At postassessment, participants in both active training groups had significantly higher knowledge scores than those in the delayed-training group, although the differences were small, with no significant differences between the active training groups.

Building on this literature, Weingardt and colleagues (29) conducted an RCT comparing two Web-based training models, each using an eight-module OLT course based on the NIDA manual. The training models varied in the adherence required and flexibility allowed. The high-fidelity group covered eight modules in a month, was instructor led and didactic, and was followed by structured group supervision. The low-fidelity group allowed participants to cover topics at random, was peer led and interactive, and provided supervision with a flexible agenda. There were no significant group differences in CBT knowledge scores at postassessment.

Larson and colleagues (30) examined the effectiveness of “Technology to Enhance Addiction Counselor Helping” (TEACH), an eight-module Web course based on the NIDA manual that is designed to increase clinicians’ use of CBT skills. Participants were randomly assigned to either TEACH-CBT or a manual-based training group and participated in monthly supervision phone calls. Participants in the manual-based group received the NIDA TM covering the same content as TEACH-CBT. No significant group differences were found in adequate adherence to CBT delivery at postassessment.

Finally, Sholomskas and Carroll (31) assessed the efficacy of two methods of training clinicians to implement the Twelve-Step Facilitation (TSF) manual, which approximates the approach of the 12 steps of Alcoholics Anonymous. Participants who were randomly assigned to the manual group received the TSF manual (43). Those randomly assigned to the CD-ROM+manual group received the manual and a seven-module computer program based on the manual that included role-plays, vignettes, and other interactive tasks to promote learning. Participants in manual groups were asked to spend 10 hours reading the manual while those in the CD-ROM+manual group were required to spend 10 hours working with the computer program for 3 weeks. CD-ROM+manual group participants saw significantly greater gains than manual group participants in their ability to demonstrate TSF skills at postassessment, with large ESs. Both groups reported a moderately high level of satisfaction with the manual and spent comparable time reading it. However, CD-ROM+manual group participants spent an average of 9.3 additional hours working with the computer program.

Studies on substance use and suicidality.

Three studies examined use of technology to train clinicians in treating substance abuse problems or suicidality with DBT. One study received a quality rating of fair, whereas two received poor ratings. Dimeff and colleagues (6) compared the efficacy of three methods of training clinicians to treat suicidal and substance-dependent clients with the DBT Skills Training Manual (44). Participants randomly assigned to the manual group received a copy of the DBT skills manual and a study guide. Participants randomly assigned to the OLT group were asked to use a five-module OLT program based on the manual for 20 hours. The modules covered mindfulness, distress tolerance, emotion regulation, interpersonal effectiveness skills, and skills coaching. Finally, participants randomly assigned to an instructor-led training (ILT) group attended a 2-day, expert-led workshop and were given the PowerPoint slides used during training. OLT participants reported significantly greater rates of change in knowledge than those in the ILT and manual groups at postassessment and follow-up, with small and medium ESs, respectively. OLT and ILT participants reported greater satisfaction with the learning objectives and with practical knowledge gained than those in the manual group. No significant group differences were found in adherence to the skills taught.

Dimeff and colleagues (32) evaluated the efficacy of three methods of training clinicians in DBT distress tolerance skills. Participants were randomly assigned to either a manual-alone condition (manual), in which they received the distress tolerance module of the DBT skills manual; an e-learning course or a CD-ROM that covered the same content as the manual (e-DBT); or a placebo e-learning course (“Care of the Client With Borderline Personality Disorder”) (e-control), a simulation of treatment for a client with borderline personality disorder in an inpatient setting. Manual and e-DBT participants had significantly higher knowledge scores than those in the e-control group at postassessment, which took place immediately after the respective training, and 15-week follow-up, with large ESs. The e-DBT group significantly outperformed the manual group at 15-week follow-up but not at postassessment, with a large ES. The manual and e-DBT conditions were also rated as significantly more acceptable than the e-control condition at postassessment and follow-up. Finally, participants in e-DBT spent significantly greater time with the course material than their counterparts.

Dimeff and colleagues (33) built upon that study by testing the efficacy of three methods of training clinicians in DBT chain analysis and validation strategies. Participants randomly assigned to OLT completed online courses in DBT chain analysis and DBT validation strategies for 8 and 4 hours, respectively. Participants randomly assigned to ILT attended a 2-day, 12-hour workshop. Those in the TM group received a 133-page manual covering DBT chain analysis and a 59-page manual on DBT validation strategies along with a study guide. OLT participants had significantly higher knowledge scores than their counterparts at postassessment and follow-up assessment, with large ESs. ILT participants found their training to be significantly more satisfactory than OLT and TM participants at postassessment, with medium and large ESs, respectively. It is noteworthy that a significantly higher percentage of participants dropped out of the OLT group (34%) than the ILT (5.5%) and TM (6.5%) groups.

Studies on other disorders.

The remaining seven studies trained clinicians in treating a variety of mental health problems, such as posttraumatic stress disorder (PTSD), bipolar disorder, and autism, as well as in general CBT skills. Five studies were rated as poor, one as fair, and one as good.

Ruzek and colleagues (34) tested the effectiveness of a three-module, Web-based program for training Veterans Health Administration clinicians in treating veterans with PTSD. The program incorporated elements from many CBT treatment protocols for PTSD and related disorders and focused on ME, goal setting, and behavior task assignment. Participants were randomly assigned to either Web-based training (Web), Web-based training plus consultation (Web+consult), or a no-training control group (control). Web+consult group participants received up to six weekly telephone-based, small-group consultation sessions, each lasting approximately 45 to 60 minutes. Compared with the control group, participants in the active training groups experienced significantly greater improvement in skills acquisition scores for the ME and behavioral task assignment modules at postassessment, with medium to large ESs. No significant group differences were found for the goal-setting module. Additionally, at postassessment, Web+consult group participants showed significantly greater skill acquisition than Web group participants on the ME module, with a medium ES.

Stein and colleagues (35) examined the effectiveness of a 12-hour online program to train clinicians in interpersonal and social rhythm therapy (IPSRT) for bipolar disorder. Participants were randomly assigned to either OLT (e-learning) with hour-long telephone supervision once a month or a 2-day, 12-hour, in-person workshop with weekly local supervision. Those in e-learning joined an implementation team, which participated in a learning collaborative focusing on quality improvement, implementation, and skills assessment. However, there were no significant group differences in the use of IPSRT techniques at any assessment point.

Cooper and colleagues (36) compared the effectiveness of two modes of Web-centered training in increasing clinicians’ competence in using enhanced CBT (CBT-E) (9, 45) for eating disorders. Web-centered CBT-E training consists of an 18-module online course that includes an expert description of how to implement CBT-E as well as handouts, learning exercises, video recordings of role-plays, and tests of knowledge with feedback. Participants were randomly assigned to either an independent training group, where they received the online course alone, or a supported training group, where they received the online course and up to 12 30-minute telephone calls from research assistants over the 20-week training period. Calls were designed to be supportive and encourage program completion. No significant group differences were found on measures of competence at postassessment or follow-up. There were also no significant group differences in training completion.

Granpeesheh and colleagues (37) evaluated an e-learning tool designed to train clinicians in academic knowledge of applied behavior analysis (ABA) treatment for children with autism. Participants randomly assigned to the e-learning group had access to a 10-hour, self-paced computer program that included topics ranging from an introduction to autism and ABA to antecedent-based and consequence-based interventions. They also attended a 2-hour discussion with an in-person trainer following program completion. Participants in the in-person training (standard) group received 16 hours of training over 2 days covering similar content through PowerPoint presentations, role-plays, and discussions. Standard group participants had significantly higher knowledge scores than the e-learning group at postassessment.

Rather than focusing on a specific disorder, Mullin and colleagues (38) trained clinicians in MI to help facilitate behavior change in their patients. A small group of clinicians chose to receive 22 hours of MI training through the “Certificate of Intensive Training in Motivational Interviewing” course. Spread over 3 to 5 months, the training was provided through an online or in-person workshop and was followed by 2 hours of individual MI practice and feedback. The course content for both workshops was grounded in the eight tasks of learning MI, as described by Miller and Moyers (46). No significant group differences in MI skills were found at postassessment.

The final two studies compared OLT for general CBT skills with other modes of training. Rakovshik and colleagues (39) randomly assigned fifth-year students from master’s-level clinical psychology programs in Russia to either a 3-hour Internet-based CBT training program (immediate) spread over a month or a DT control group (delayed), in which participants received access to the same training program after a 1-month wait. The immediate training provided instruction in CBT theory, assessment, and formulation and included videos of didactic lectures and role-plays and simultaneous display of associated PowerPoint presentations with Russian subtitles. The immediate training group scored significantly higher than the DT group on measures of CBT competence at postassessment, with a large ES. No significant group differences in satisfaction ratings emerged.

German and colleagues (40) compared expert-led training to Web-based training in the use of general CBT skills. A cohort of community mental health clinicians received in-person, expert-led (IPEL) training, which consisted of a 22-hour, in-person CBT workshop, followed by weekly, 2-hour group consultations with experts for 6 months. The consultations focused on applying CBT, including review of audio-recorded sessions. The next cohort participated in Web-based, trained-peer (WBTP) training. The Web-based training was based on the in-person core curriculum and added videotaped role-plays, on-screen activities, and quizzes to improve engagement. The Web-based training was followed by peer consultation with the initial cohort and regular consultations with an instructor. No significant group differences were found between the two cohorts in CBT competency at postassessment; however, participants in Web-based training were less likely than participants in in-person training to complete the course.

Studies on supported training.

Out of the 24 included studies, support with technology-based training was provided in 14 of the studies, which included supervision or engagement interventions. Supervision was aimed at promoting learning and use of therapy skills and generally included answering questions, review or discussion of training session content, and case feedback by experienced supervisors or clinicians. Nine studies examined the effect of supervision provided either individually or in small groups through face-to-face or technological modalities such as Twitter, telephone, and video calling platforms, such as Skype. The results were mixed, with some studies finding that supervision had no effect on primary outcomes (35, 37, 40). Other studies, however, reported improvements in CBT competence (27), skills acquisition (34), skills competence (7), and program completion rates (20) among those who received supervision compared with those who did not.

Additionally, five studies paired technology-based training with engagement interventions, which were mainly supportive in nature and were generally led by peers or research assistants. Although some studies did not find that the engagement interventions had a significant effect on primary outcomes (30, 36), others found some benefit. For example, Harned and colleagues (25) found that the addition of phone calls with a basis in brief MI significantly improved attitudes toward ET among clinicians. Notably, when ME was provided through a computerized intervention rather than individually, these results did not hold. Indeed, Harned and colleagues (26) found that clinical attitudes significantly improved only when ME was provided in conjunction with an LC.

Discussion

Effectively disseminating EBTs to the mental health workforce is a significant challenge in the field. This systematic review aimed to gain a better understanding of how technology has been used to train clinicians in EBTs by providing a comprehensive summary of the literature on how technology can aid in training clinicians. After a thorough literature search, we found 24 articles that met the inclusion criteria. These were subsequently categorized by the content area in which training was provided and were independently coded and assessed for risk of bias by two reviewers.

It is noteworthy that of the 24 studies reviewed, only one met criteria for good quality, which points to the limitations and challenges inherent in this field of research. Furthermore, it should be noted that the quality and interactivity of e-learning interventions vary widely, and shortcomings in these areas may have affected some of the individual study findings. As such, all interpretations should be made in the context of these limitations.

Clinicians were trained in some form of CBT in all of the studies reviewed, with the exception of studies by Sholomskas and Carroll (31), Stein and colleagues (35), and Mullin and colleagues (38), in which clinicians were trained in Twelve-Step Facilitation, IPSRT, and MI, respectively. Nineteen of the 24 studies used OLT, whereas five used computer software or CD-ROMs. Anxiety-related disorders were the focus of more studies than any other disorder (N=7), followed by substance use disorders (N=5). Nine studies also examined the addition of supervision, which included use of the Internet, social media, and video conferencing. Despite the proliferation of freely available mobile applications for smartphones, no app for training clinicians in EBTs was identified in our database search.

Ten studies compared technology-based training with technology-based training plus support or an attention control. Although important, these comparisons do not further our understanding of whether OLT is as effective as traditional training methods (i.e., in-person or manual-based training). Of the seven studies that compared technology-based training with in-person training, six found no significant difference between the modalities in gains in therapy knowledge and skills at postassessment.

Two studies compared OLT with manual-based training and concluded that participants in both conditions made similar gains in knowledge (32) and adherence scores at postassessment (30). Two studies examined the combination of technology and manual-based training, with one study finding no significant difference between OLT alone and OLT plus a TM (22). The other study found that a CD-ROM plus a TM was superior to a manual alone (7).

Finally, three studies compared OLT with both manual-based and in-person training. Arguably, such studies allow us to draw the most definitive conclusions regarding how technology-based training fares in comparison with traditional training. However, these studies made heterogeneous comparisons and had mixed results. For example, two studies found face-to-face training to be superior to manual-based training and OLT in improving participants’ scores on primary outcome measures (7, 33). One study found OLT to be more effective compared with the manual and face-to-face conditions (6). Replicability of such studies is of utmost importance to unequivocally infer the effectiveness of technology-based training.

With the exception of three studies, technology-based training was judged as or more effective than manual-based or in-person training. Across a majority of studies, participants receiving technology-based training improved their knowledge, skills, and competence and were more satisfied than comparison groups with their training. This result is consistent with previous systematic reviews that have found that Web-based training methods have a positive effect on training outcomes of mental health professionals (13, 47, 48).

This review also included studies examining the impact of supported training in the form of supervision or engagement interventions. Many studies compared technology-based training alone with technology-based training plus support (20, 22, 25, 27, 36). Others compared two types of in-person consultation (e.g., expert-led versus peer-led consultation) (40) or two forms of technology-based support (e.g., computerized ME and ME plus a Web-based LC) (26). Overall, findings regarding the utility of supervision and engagement interventions were mixed. This outcome may be due partly to the distinctive comparisons made, variation in dosage (e.g., 30 minutes versus 2 hours), frequency (e.g., weekly versus every 6 weeks), and duration (e.g., 12 weeks versus 20 weeks or 12 months) of support and differences in who provided the support (e.g., experts versus peers). Ongoing support may improve clinician knowledge and connection with peers and trainers (49). However, the heterogeneity in the included studies makes it difficult to draw clear conclusions on the effect of support for technology-based training.

Limitations of the Literature

Because only one study met criteria for a rating of good quality, the findings need to be interpreted in the context of the studies’ limitations, of which selection, information, and measurement bias were most notable. A majority of studies used convenience sampling to recruit participants and were conducted in the United States with predominantly white, female samples. Similarly, a disproportionate number had very small sample sizes and were statistically underpowered, which further limits our ability to discern meaningful group differences and draw definitive conclusions. Future research may mitigate such issues by using larger and more representative samples and employing systematic sampling techniques. Three studies did not use random assignment, and six failed to report the method of randomization. Three other studies were also found to utilize inadequate randomization methods, such as failing to use a randomly generated assignment.

Studies were also limited by their data collection approach. Most studies used participant self-reports to assess primary outcomes, such as knowledge and skill acquisition. Previous studies have found that clinicians tend to be rated as more skillful when the ratings are based on self-report rather than behavioral observations (5, 50). Although using behavioral observations can be expensive and time-consuming, such measures generate more objective and accurate results. Future studies may benefit from using objective strategies to assess outcomes, such as session recordings and role-plays assessed by blinded experts on reliable and valid scales. Studies also differed by intensity of training, including the number of training hours and treatment fidelity required across various conditions. This makes drawing conclusions from the observed results challenging because alternative explanations may be used to justify the findings.

Thirteen studies obtained satisfaction ratings from participants. Overall, most studies found that participants assigned to technology-based training groups were as satisfied or more satisfied with training compared with those assigned to manual-based or in-person training. However, only six studies reported on program completion rates, three of which found significantly lower completion rates among OLT groups (20, 33, 40). Program completion can have a significant impact on training outcomes, such as knowledge and skills acquisition. Future research should collect user experience data to ensure that programs are acceptable to participants, which will increase the likelihood of participant program completion.

Limitations of the Study

This systematic review had several limitations. First, given that the field has only recently begun to examine specific technology for training clinicians, there is a lack of consistency in identifying training methods. Therefore, some studies that did not match our search terms may have been unintentionally omitted. Second, our decision to include only studies with a comparison group may have restricted findings from the review. Third, although we calculated ESs to quantify the magnitude of between-group differences, we were unable to conduct a meta-analysis because information was missing in some of the included studies. Finally, only studies in the published literature were included in this review. This concern was addressed by contacting authors of included studies and inquiring about other related research. None of the authors contacted reported unpublished studies with null findings.

Despite its limitations, this systematic review provides a novel examination of how technology has been used to train clinicians. Although previous systematic reviews have examined Web-based training methods for clinicians, earlier efforts did not assess risk of bias and did not determine interrater reliability (13). Both methods allow for reduction of subjectivity and provide objective interpretations of the findings reported in this synthesis of studies.

Conclusions

Overall, our findings suggest that technology-based training is a promising avenue for training larger numbers of clinicians in EBTs. Providing face-to-face instruction can be expensive and time-consuming. Most of the technology-based training interventions identified in this review were self-paced, thereby affording clinicians more flexibility and independence. Finally, technology-based training can help disseminate information in a standardized manner so all trainees receive the same quality of instruction. Future research is needed to establish the long-term effects of technology-based training on clinician skills and knowledge as well as on patient outcomes. Finally, future research should conduct economic analyses to assess whether technology-based training is a cost-effective option for training clinicians.