The American Psychiatric Association (APA) has updated its Privacy Policy and Terms of Use, including with new information specifically addressed to individuals in the European Economic Area. As described in the Privacy Policy and Terms of Use, this website utilizes cookies, including for the purpose of offering an optimal online experience and services tailored to your preferences.

Please read the entire Privacy Policy and Terms of Use. By closing this message, browsing this website, continuing the navigation, or otherwise continuing to use the APA's websites, you confirm that you understand and accept the terms of the Privacy Policy and Terms of Use, including the utilization of cookies.

×
ArticlesFull Access

Improving Therapist Fidelity During Implementation of Evidence-Based Practices: Incredible Years Program

Published Online:https://doi.org/10.1176/appi.ps.201200177

Abstract

Objective

The aim of the study was to extend research on the potential benefits of adding ongoing feedback, coaching, and consultation to initial therapist training workshops to ensure fidelity of delivery of evidence-based practices, specifically for the Incredible Years parenting program.

Methods

A randomized controlled trial compared two models for training therapists to deliver the parenting program for children at high risk of developing conduct problems. Therapists (N=56) from ten community-based mental health service organizations in California were trained in either a three-day workshop model (N=25), based on active, experiential, self-reflective, principle-based learning, video modeling, and manuals, or an enhanced training model (N=31) that included all elements of the workshop model plus ongoing expert coaching, video review of and feedback on group sessions, and consultation for therapists and agency supervisors.

Results

Overall fidelity across both conditions was rated >3 on a 5-point scale in seven of eight domains measured. Therapists in the condition that received ongoing coaching and consultation were significantly stronger in four of the domains: practical support, collaboration, knowledge, and skill at mediating vignettes.

Conclusions

Consultation and expert coaching for training therapists beyond the standard three-day training enhanced skills and therapists’ adherence to the model.

Understanding factors that influence implementation fidelity in the delivery of evidence-based practices is an important national health care priority (1). Fidelity refers to treatment adherence to core program features (2) and is predictive of positive outcomes across several evidence-based practices (3,4). Research with the Incredible Years Parenting Program (the program evaluated in this study) corroborates the link between fidelity and outcomes (5,6).

Unfortunately, research shows that fidelity and positive program outcomes are often compromised when interventions are implemented in routine service settings (7,8). Therapist training and supervision are identified as core implementation components (9); training methods, amount of training, and quality of training make a difference in therapist skill and adherence (10,11). Short, passive, lecture-style training methods are largely ineffective, whereas longer, active, self-reflective, and principles-based training leads to more positive outcomes. However, these positive outcomes may not be sustained without ongoing support (9,10). Research examining the link between ongoing expert supervision and therapist fidelity and treatment adherence is rare, with a few exceptions (12). For example, Henggeler and colleagues (13) found that providing extensive training and supervision for community-based therapists increased their proficiency and skills with Multisystem Family Therapy (3) and that high-quality coaching was related to therapist adherence to the intervention protocol and outcomes for youths (14). Lochman and colleagues (15) found that training including clinical supervision resulted in higher therapist fidelity to the evidence-based program Coping Power than did training without supervision. Ongoing contact with an experienced coach appears to be important because of the length of time needed to build proficiency in a new practice (12).

This study examined the potential benefits of adding ongoing coaching and consultation to the Incredible Years workshop training model to ensure fidelity of delivery of this parenting program. This evidence-based program has been shown in randomized trials to promote positive parenting skills and reduce child behavior problems (16). The program has proved effective in community settings with diverse populations (1719). There are extensive materials to facilitate the delivery of the program with standardized, practice-based leader training.

This randomized controlled trial compared two models for training therapists to deliver the parenting program for children at high risk of developing conduct problems. Therapists from community-based mental health service organizations in California received either a three-day workshop or enhanced training that included the workshop plus ongoing coaching and consultation.

The research questions of interest were these: How effective is the workshop in ensuring intervention fidelity among trained therapists? And do ongoing coaching, consultation, and video feedback enhance therapists’ fidelity in implementing the intervention?

Methods

Sites

Participating sites were ten publicly funded community mental health agencies in California that provide children’s mental health and child welfare services. The study was carried out between 2005 and 2010.

Procedure

All participating agencies were recruited through the California Alliance of Child and Family Services, a statewide association of 130 private nonprofit child and family service agencies. Chief executive officers of all 130 member agencies received information about the study. Thirty-five agencies indicated interest and participated in a conference call describing the study; ten of the 35 agencies agreed to proceed with the study. These ten sites agreed to random assignment to training conditions and to pay for program materials and staff time (training was paid for by the study). In both conditions (workshop only or workshop plus coaching), group leaders receiving training agreed to complete questionnaires regarding program delivery and to have the parenting group sessions videotaped. Group leaders and parents provided voluntary consent to participate in the study and to be videotaped. The institutional review board at Rady Children’s Hospital, San Diego, reviewed and approved the work conducted.

Incredible Years parenting intervention

The basic parenting intervention (2001 version, for parents of children ages three to six years) was used for this project. It was conducted in 12–14 two-hour group sessions, led by two trained therapists employed by participating organizations. Program materials include a treatment manual, a set of DVD vignettes of parent-child interactions, therapist protocols, and handout materials for parents. Treatment methods include parent discussion, video modeling, role-play practices, and home assignments. Topics include parent-child relationship building, child-directed play and coaching, praise and encouragement, tangible rewards, consistent limit setting, use of timeouts, and problem solving. Therapists use a collaborative, self-reflective style rather than a didactic expert leadership style.

Training models

Workshop model.

The standard three-day workshop for therapists is based on research showing that performance-based, active experiential training methods, including self-reflection, behavioral role plays, clinical principles, and collaborative group sharing, result in learning and behavior change that are superior to results from verbally based, didactic, lecture-based training approaches (10,2022). The three-day training, taught by certified trainers, emphasizes clinical methods and therapeutic processes because content is embedded in the DVD materials and manuals and can be studied by therapists as they deliver the program, whereas clinical practices are more difficult to convey in a training manual. Training includes live modeling by the trainer as well as videotaped modeling showing standardized examples of group sessions.

Enhanced expert coaching and consultation model.

Therapists assigned to the intervention plus coaching received the workshop described above, plus ongoing coaching and consultation in the form of weekly individualized phone calls between therapists running the groups and expert coaches. The expert coach provided group leaders with support and reinforcement, helped them cope with therapeutic challenges, helped leaders set personal goals, reviewed session protocols, and helped tailor program material to the needs, culture, and goals of parents within the group. In addition, expert coaches provided detailed written feedback on video recordings of at least two group therapy sessions during the course of each group.

Fidelity monitoring of therapists and agencies

We took several steps to ensure consistency of training and consultation. First, expert trainers followed the standardized three-day training protocol by using training checklists. For each of the two training conditions, therapists attended training together. In addition, trainers were blind to intervention condition during the basic training. Ongoing consultation was provided by expert and certified coaches, who were supervised by the program developer.

Fidelity monitoring of Incredible Years group delivery

Therapists in both conditions were monitored to ensure that they were completing checklists for each group session, submitting attendance information, and videotaping groups. Outcomes were selected to provide both observational and self-report data on fidelity with regard to program content and therapeutic methods and processes. Measures are described below and include therapist report, parent report, and observational coding of session videos. Assessments were identical for the two study conditions.

Therapist report: session protocol checklists for therapists.

Therapists completed a checklist after each session, including information about vignettes, home activities assigned, topics discussed, role-play practices, brainstorms, parents’ goals, and parent-to-parent buddy calls. Therapists also rated parent participation and engagement.

Parent report: satisfaction and engagement.

After each session, parents completed a four-item satisfaction form rating therapists’ skill and the usefulness of the vignettes, role plays, and group discussion (scores ranged from 1 to 4; higher scores indicated more satisfaction).

Observational coding.

Independent observers (certified trainers) blind to assigned study conditions completed the therapist implementation–parent group leader rating scale, a 92-item coding scheme, as they observed videos of group sessions. For reliability, a second coder independently rated 48% of the sessions. Each response item used a 5-point Likert scale that ranged from 1 to 5; higher scores indicated greater implementation skill or adherence. Observers rated items in eight domains: practical support (offering participants child care and food), collaborative approach, content knowledge, leadership skill, relationship building, skill in reviewing completed home activities, skill in conducting role plays, and methods of mediating parents’ discussion of video vignettes.

Statistical analysis

Analyses examined the characteristics and structure of therapist fidelity in delivery of the parenting program and differences between the two therapist training conditions. Descriptive statistics summarize averages and variability in fidelity across and between study conditions. Two-level random-effects models, with treatment sessions (level 1) nested within sites (level 2), were used to test for condition differences in the fidelity dimensions. A site-level random effect was included to account for the clustering of treatment sessions within sites. A fixed-session order predictor was also included at level 1 to account for possible systematic improvements (or drift) that might occur over time. Tests of condition differences are reported for individual fidelity dimensions.

Results

Characteristics of Incredible Years group leaders

Table 1 provides descriptive information about the 56 therapists delivering the parenting program. Data on race-ethnicity were not analyzed. No significant differences emerged between the two groups.

Table 1 Characteristics of Incredible Years group leaders in the workshop-only and expert consultation conditions
Workshop only (N=25)
Workshop plus consultation (N=31)
CharacteristicN%N%
Degree
 Doctorate016
 Master’sa18721963
 Bachelor’s17722
 Associate’s31113
 Unknown31116
Experience (M±SD years)7.5±5.98.6±5.7
Training (M±SD years)
 Child development3.4±.53.4±.6
 Social learning theory3.2±3.23.7±.7
 Facilitating groups3.0±3.03.3±.8
Type of experience (M±SD years)
 Children and families4.0±.03.8±.5
 Child skills groups2.6±1.13.1±.9
 Parent therapies2.7±2.32.7±2.3
 Parent training groups2.5±1.22.8±1.0
 Family therapy3.0±1.02.6±1.2

a Included M.A., M.F.T., and M.S.W. degrees

Table 1 Characteristics of Incredible Years group leaders in the workshop-only and expert consultation conditions
Enlarge table

Variability in Incredible Years program fidelity

Therapists in the enhanced coaching and consultation condition received a mean±SD of 9.40±3.99 supervision calls for a total of 4.66±2.16 hours. Therapists in the workshop-only condition received no supervision calls, but therapists in both groups were monitored by research assistants to ensure completion of checklists and videotaping.

Program fidelity was measured along eight dimensions through observation of group sessions. As illustrated in Table 2, good interrater reliability was present across dimensions, with intraclass correlation coefficients on each dimension ranging between .53 (content knowledge) and .87 (skill in reviewing home activities), with most above .65. Differences in overall levels of fidelity were present across fidelity dimensions. In both training conditions, therapists were observed to demonstrate the highest levels of fidelity in implementing practical requirements for parents (offering child care and food), knowledge of program content, and building positive relationships with clients. Across sites (in both conditions), domains in which fidelity appeared to be weakest concerned use of active learning methods (including mediation of discussions about video vignettes, live modeling, and role-play practices), which are a core method in the parenting program.

Table 2 Therapists’ fidelity to Incredible Years program, as assessed by independent raters
Fidelitya
DomainNbN of itemsICCcNMSDMinMaxExample
Practical support322.70324.86.463.005.00Provides child care and food
Collaborative approach5311.85533.62.742.184.82Asks open-ended questions, reinforces parents’ self-learning, encourages problem solving, fosters learning from other parents, facilitates group, identifies families’ strengths, values every group member, creates atmosphere where parents are decision makers, adopts collaborative instead of expert model
Knowledge535.53534.16.553.005.00Has accurate knowledge of cognitive-behavioral principles, explains rationale, integrates parents’ ideas with program content, has accurate knowledge of child development
Leadership5213.67523.67.641.504.77Starts and ends on time; explains agenda; imposes structure to group; prevents sidetracking; predicts therapy process, including potential difficulties, long-term goals, and relapses; balances discussion of cognitive, affective, and behavioral domains
Relationship building538.65533.94.672.384.88Builds rapport with each participant, encourages participation, creates feeling of safety, uses humor, validates feelings, shares personal experiences
Skills
 Home activities review508.87503.85.622.715.00Reviews completed assignments with each parent, highlights key principles, explores barriers, takes responsibility for misunderstandings about assignments, stays on topic
 Role plays457.79452.661.091.005.00Moves from talk to practice, defines child behavior, solicits suggestions about parental role before starting, discusses parent and child roles, gives descriptive praise
 Vignettes497.86493.75.812.205.00Uses open-ended questions to discuss vignettes, acknowledges parents’ responses, writes key points on board, moves to next vignette after appropriate discussion, uses vignettes to trigger discussions and cover key concepts, refers to parents’ personal goals

a Possible scores range from 1 to 5, with higher scores indicating better fidelity and stronger skills.

b In some cases, items were not observable for all 56 therapists because rater or parents were in a different room or video quality was poor.

c Intraclass correlation coefficient

Table 2 Therapists’ fidelity to Incredible Years program, as assessed by independent raters
Enlarge table

Relationships among fidelity dimensions

Table 3 shows the correlations among the eight fidelity dimensions, which fell in the moderate range, between .30 and .70, with some aspects of fidelity highly correlated with other dimensions (using the collaborative group process and building strong relationships with participants) but also with identifiable differentiation between program aspects (between skill in reviewing home activities and most other dimensions, or between skill in using role plays and in building relationships with participants). All correlations among the characteristics of fidelity were positive, indicating a general tendency toward higher or lower fidelity across dimensions, but variability in the strength of the correlations suggests that the parenting program has distinct dimensions on which separate improvement may be possible.

Table 3 Intercorrelations between fidelity dimensions
Dimension12345678
1. Practical.43*.26.41*.39*.27.30.31
2. Collaborative.54***.68***.84***.44**.41**.67***
3. Knowledge.59***.42**.32*.61***.70***
4. Leadership.70***.37*.55***.57***
5. Relationship.40**.37*.52***
6. Skill: home activities.18.32*
7. Skill: role plays.59***
8. Skill: vignettes

*p<.05, **p<.01, ***p<.001

Table 3 Intercorrelations between fidelity dimensions
Enlarge table

Differences in fidelity between the two conditions

Observational reports of fidelity dimensions.

Individual comparisons were made between each fidelity dimension to understand aspects of the parenting program in which enhanced training support affected fidelity of delivery. Significant differences were present in four of the eight fidelity domains: practical support (child care and food), collaborative approach, knowledge of content, and skill in mediating discussion of vignettes. Effect sizes in these domains were relatively large, with effects (d) ranging between two-thirds and one standard deviation in magnitude (see Table 4). Other subdomains (conducting role plays, skill in reviewing home activities, and group leadership features) also trended in the expected direction, and none trended opposite the predicted direction. Individually, however, these differences were not statistically significant, and effect size estimates were correspondingly smaller.

Table 4 Fidelity ratings and effect size differences for therapists assigned to workshop only and to workshop plus consultation, by fidelity dimensiona
Workshop only
Workshop plus consultation
DimensionMSEMSEdp
Practical4.59.165.06.151.11.004
Collaborative3.44.223.88.22.65.040
Knowledge3.84.154.34.151.00<.001
Leadership3.66.193.86.19.31.279
Relationship3.99.224.05.21.09.809
Skill: home activities3.61.203.82.19.33.235
Skill: role plays2.76.353.17.33.38.234
Skill: vignettes3.48.244.13.23.90.007

a Possible fidelity scores range from 1 to 5, with higher scores indicating higher fidelity.

Table 4 Fidelity ratings and effect size differences for therapists assigned to workshop only and to workshop plus consultation, by fidelity dimensiona
Enlarge table

Leader and participant reports of fidelity dimensions.

Leaders and participants provided information about group delivery (Table 5). On average, parenting groups in both conditions met for 12 sessions. Group length was slightly longer in the workshop-only condition than in the enhanced condition, at just over two hours per session. The conditions were similar with regard to logistics, with most groups providing participating parents with food and child care but not transportation to the sessions. Therapists in each group reported using basic intervention components (agenda, brainstorms, role plays, and home activities review) in 70%−80% of sessions. Group leaders in the enhanced coaching condition showed significantly fewer vignettes than those in the workshop-only condition. Because expert raters reported that group leaders in the enhanced condition were significantly more skillful at mediating the vignette discussions, this difference likely reflects the additional discussion time for each vignette in this training group. Mothers in groups in which therapists had the enhanced training also completed significantly more home assignments than those in the workshop-only condition.

Table 5 Leader- and participant-reported characteristics of Incredible Years group delivery in workshop-only and expert consultation conditions
Workshop only
Workshop plus
consultation
CharacteristicMSDMSD
Group
 Number of sessions11.32.012.62.0
 Session length (minutes)127.315.4118.06.2
 Families in group6.42.86.02.3
Logisticsa
 Child care provided88226447
 Food provided9614992
 Transportation provided19341325
Participationb
 Mothers’ participationc2.72.222.73.24
 Mother completed home activitiesd.60.30.87.87
Mothers’ satisfaction ratingse
 Content3.55.283.71.21
 Leader3.45.353.64.28
 Vignette3.64.233.74.17
 Discussion3.56.273.69.23
 Role play3.43.343.54.25
Intervention componentsa,f
 Agenda86228417
 Brainstorming81287624
 Role plays71296019
 Home activities review with parents82217118
 Prior material completed85278718
 Assigned home activities89208319
Average number of vignettes per session6.022.744.311.92

a Percentage of sessions leaders reported providing these elements of the program

b Some fathers participated in the groups, but data were collected for mothers only.

c Rated by therapists on a 3-point scale ranging from 1, not at all involved, to 3, very involved

d Proportion of time that mothers completed home activities; p<.05 for comparison between groups

e Mothers’ group ratings could range from 1 to 4, with higher scores indicating more satisfaction with Incredible Years program.

f Values are percentages and percentage standard deviations.

Table 5 Leader- and participant-reported characteristics of Incredible Years group delivery in workshop-only and expert consultation conditions
Enlarge table

Discussion

Fidelity outcomes for both training conditions were of interest. The extent to which the basic three-day workshop training prepares leaders to run the groups with fidelity is an important outcome. Therapists in both conditions showed outcomes above 3.5 on a 5-point scale in the areas of child development, practical support of group participants (child care and food), group leadership, building relationships, and reviewing home activities, indicating a moderate degree of proficiency with key program components. This proficiency provides some support for the active training model used in this study rather than didactic teaching, which has been shown to be ineffective (9). However, it could also be argued that these training areas reflect skills and knowledge that are directly transferable from other therapy or group leadership experience. Without a materials-only comparison control group in which therapists delivered the program without receiving an intensive workshop, the unique value of the workshop in improving therapist implementation of the intervention cannot be determined.

Average fidelity scores were higher in the enhanced coaching group for almost all dimensions, indicating the added benefits of this condition. This finding supports previous research findings that concern training of therapists in evidence-based programs (22). These differences were significant for the practical aspects of program delivery, using a collaborative approach, knowledge, and skill in leading and in mediating discussion of vignettes. These areas may reflect methods that many therapists do not use in usual practice. Typically parental training is more didactic and does not use vignettes or self-reflective and collaborative group discussion, nor does it tailor material to individual goals or offer role-play practices. Thus the phone calls that offered coaching may have supported therapists in their use of these core components of the program.

Therapists in the two training conditions did not differ on several outcomes: leadership, relationship building, skill in review of home activities, and skill in role-play practices. On dimensions where fidelity was already relatively high (leadership, relationships, and skill in review of home assignments), this lack of difference from the workshop-only condition is likely not an area for concern. However, the use of role plays and experiential practice, where fidelity was lower, needs further attention and training. Conducting role plays was the most commonly reported area of difficulty and resistance for group leaders. Consequently, they sometimes did not introduce the role plays at all or did so in a way that made parents very unlikely to participate. In telephone supervision, group leaders were encouraged to think about effective strategies for engaging parents in the role play and how to structure the role play to support maximal learning. However, group leaders sometimes were not able to work through these issues until an in-person consultation day, often by engaging in role plays themselves. Thus the overall lower skill level in running role plays in this study may reflect that this skill is one of the more challenging skills for group leaders to learn and one that is difficult to adequately address in a telephone consultation.

Prior studies with evidence-based programs have shown that therapists’ proficiency and program fidelity are linked to improved client outcomes and reductions in antisocial child behavior (5,22). The findings of this study are limited in that they do not present parent or child outcomes; thus conclusions about what is “good-enough” fidelity are not possible. In addition, the study did not include a “no training” control group, so it is not known what level of fidelity therapists would have demonstrated if they had used the program materials without a training workshop. Finally, therapists in the workshop-only condition were monitored closely to ensure that they completed checklists and videotaped their groups. Arguably, these requirements of research study participation likely enhanced fidelity in both conditions. Further research will also examine other agency and therapist variables that influence fidelity. Therapist education and theoretical orientation—in addition to training and supervision variables, such as agency support and costs of consultation—are likely to play a role in fidelity to the program model.

Conclusions

Therapists assigned to the three-day workshop-only condition implemented the program with fidelity scores above 3.5 (out of 5) in several areas: content knowledge, group leadership, and relationship skills. The enhanced coaching and supervision condition was effective in increasing therapists’ proficiency in providing practical support to group participants and in their collaborative therapy process, knowledge of cognitive-behavioral principles and child development, and effective mediation of vignette discussions. Further supervision, perhaps in the form of in-person group consultation and practice, may be necessary for improving therapists’ skill in leading and promoting successful role plays and live modeling of practice experiences.

Dr. Webster-Stratton is professor emeritus of the School of Nursing and Dr. Reid is with the Department of Psychology, both at the University of Washington, Seattle (e-mail: ). Ms. Marsenich is with Western Implementation Research and Evaluation, Eugene, Oregon.

Acknowledgments and disclosures

This research was supported by grant MH744797-05 from the National Institute of Mental Health.

Dr. Webster-Stratton has disclosed that she disseminates the Incredible Years interventions and stands to gain from a favorable report. She has therefore voluntarily agreed to distance herself from certain critical research activities (recruiting, obtaining consent from participants, primary data handling, and analysis). The University of Washington has approved these arrangements. Dr. Reid reports that she is a paid trainer for the Incredible Years program. Ms. Marsenich reports no competing interests.

References

1 Weisz JR, Hawley KM, Pilkonis PA, et al.: Stressing the (other) three Rs in the search for empirically supported treatments: review procedures, research quality, relevance to practice and the public interest. Clinical Psychology: Science and Practice 7:243–258, 2000CrossrefGoogle Scholar

2 Schoenwald SK, Garland AF, Chapman JE, et al.: Toward the effective and efficient measurement of implementation fidelity. Administration and Policy in Mental Health and Mental Health Services Research 38:32–43, 2011Crossref, MedlineGoogle Scholar

3 Henggeler SW, Schoenwald SK, Liao JG, et al.: Transporting efficacious treatments to field settings: the link between supervisory practices and therapist fidelity in MST programs. Journal of Clinical Child and Adolescent Psychology 31:155–167, 2002Crossref, MedlineGoogle Scholar

4 Wilson SJ, Lipsey MW: School-based interventions for aggressive and disruptive behavior: update of a meta-analysis. American Journal of Preventive Medicine 33(suppl):S130–S143, 2007Crossref, MedlineGoogle Scholar

5 Scott S, Carby A, Rendu A: Impact of Therapists’ Skill on Effectiveness of Parenting Groups for Child Antisocial Behavior. London, University College London, King’s College, Institute of Psychiatry, 2008Google Scholar

6 Eames C, Daley D, Hutchings J, et al.: Treatment fidelity as a predictor of behaviour change in parents attending group-based parent training. Child: Care, Health and Development 35:603–612, 2009Crossref, MedlineGoogle Scholar

7 Schoenwald SK, Hoagwood K: Effectiveness, transportability, and dissemination of interventions: what matters when? Psychiatric Services 52:1190–1197, 2001LinkGoogle Scholar

8 Hoagwood K, Burns BJ, Weisz J: A profitable conjunction: from science to service in children’s mental health; in Community-Based Interventions for Youth With Severe Emotional Disturbances. Edited by Burns BJHoagwood K. New York, Oxford University Press, 2002CrossrefGoogle Scholar

9 Fixsen DL, Naoom SF, Blase KA, et al: Implementation Research: A Synthesis of the Literature. FMHI pub no 231. Tampa, University of South Florida, Louis de la Parte Florida Mental Health Institute, National Implementation Research Network, 2005Google Scholar

10 Sholomskas DE, Syracuse-Siewert G, Rounsaville BJ, et al.: We don’t train in vain: a dissemination trial of three strategies of training clinicians in cognitive-behavioral therapy. Journal of Consulting and Clinical Psychology 73:106–115, 2005Crossref, MedlineGoogle Scholar

11 Beidas RS, Kendall PC: Training therapists in evidence-based practice: a critical review of studies from a systems-contextual perspective. Clinical Psychology: Science and Practice 17:1–30, 2010Crossref, MedlineGoogle Scholar

12 Herschell AD, Kolko DJ, Baumann BL, et al.: The role of therapist training in the implementation of psychosocial treatments: a review and critique with recommendations. Clinical Psychology Review 30:448–466, 2010Crossref, MedlineGoogle Scholar

13 Henggeler SW, Schoenwald SK, Pickrel SG: Multisystemic therapy: bridging the gap between university- and community-based treatment. Journal of Consulting and Clinical Psychology 63:709–717, 1995Crossref, MedlineGoogle Scholar

14 Schoenwald SK, Sheidow AJ, Letourneau EJ: Toward effective quality assurance in evidence-based practice: links between expert consultation, therapist fidelity, and child outcomes. Journal of Clinical Child and Adolescent Psychology 33:94–104, 2004Crossref, MedlineGoogle Scholar

15 Lochman JE, Boxmeyer C, Powell N, et al.: Dissemination of the Coping Power program: importance of intensity of counselor training. Journal of Consulting and Clinical Psychology 77:397–409, 2009Crossref, MedlineGoogle Scholar

16 Webster-Stratton C, Reid MJ: A multifaceted treatment approach for young children with conduct problems; in Evidence-Based Psychotherapies for Children and Adolescents, 2nd ed. Edited by Kazdin AEWeisz JR. New York, Guilford, 2010Google Scholar

17 Gardner F, Burton J, Klimes I: Randomised controlled trial of a parenting intervention in the voluntary sector for reducing child conduct problems: outcomes and mechanisms of change. Journal of Child Psychology and Psychiatry, and Allied Disciplines 47:1123–1132, 2006Crossref, MedlineGoogle Scholar

18 Hutchings J, Gardner F, Bywater T, et al.: Parenting intervention in Sure Start services for children at risk of developing conduct disorder: pragmatic randomized controlled trial. British Medical Journal 334(7595):1–7, 2007Crossref, MedlineGoogle Scholar

19 Reid MJ, Webster-Stratton C, Beauchaine TP: Parent training in Head Start: a comparison of program response among African American, Asian American, Caucasian, and Hispanic mothers. Prevention Science 2:209–227, 2001Crossref, MedlineGoogle Scholar

20 O’Dell SL: Progress in parent training. Progress in Behavior Modification 19:57–108, 1985Crossref, MedlineGoogle Scholar

21 Webster-Stratton C, Herbert M: Troubled Families—Problem Children: Working With Parents: A Collaborative Process. Chichester, United Kingdom, Wiley, 1994Google Scholar

22 Lyon AR, Stirman SW, Kerns SE, et al.: Developing the mental health workforce: review and application of training approaches from multiple disciplines. Administration and Policy in Mental Health and Mental Health Services Research 38:238–253, 2011Crossref, MedlineGoogle Scholar