Therapist- and Client-Level Predictors of Use of Therapy Techniques During Implementation in a Large Public Mental Health System

The past 50 years have resulted in a proliferation of evidence-based practices (EBPs) for treatment of psychiatric disorders for a range of client populations and the patients’ presenting problems (1,2). Despite the development and validation of these treatments, most youths do not have access to EBPs in their communities (3,4). Although the need to translate EBPs from research to practice settings has been identified (5), progress to do so has been slow and hampered by a lack of understanding of community-based usual care (6–8). The limited research to date examining usual care has focused on characterizing the types of treatment techniques used by community providers (9), that is, the procedures used during the intervention process (such as role playing and relaxation training [10]). Although this has helped elucidate usual care, little is known about variables that may influence the use of specific therapeutic techniques delivered by community therapists.

Studies focused on usual care of disruptive behavioral disorders have demonstrated that community therapists report using a wide range of techniques, some evidence based and some not (11). Observational studies have shown that community therapists generally deliver a broad range of techniques at relatively low intensities (12,13) in the treatment of youths with externalizing disorders. Studies that have examined therapist self-report of usual care of youths with internalizing disorders have similarly shown a tendency of therapists to use a variety of therapeutic techniques at low intensities (14,15). Observational studies by McLeod and Weisz (16) and Southam-Gerow and colleagues (17) have demonstrated that community therapists use a variety of techniques in their treatment of youths with internalizing disorders, generally favoring client-centered approaches. Weisz and colleagues (18) found that therapists delivering usual care to youths with depression were observed to use more family and psychodynamic techniques than cognitive-behavioral therapy (CBT) techniques.

Increasing the use of EBPs in usual care is an important objective for mental health services research. The literature suggests that children receiving community-based usual care do not clinically improve at the same magnitude as youths receiving EBPs (19–21). In direct comparisons, EBPs generally outperform usual care. For example, in a series of meta-analyses by Weisz and colleagues (21,22), differences in effect sizes between usual care and EBPs were approximately .29 (Cohen’s d). Youths receiving usual care in publicly funded systems have worse treatment trajectories than youths in managed-care systems (20). Taken together, these data suggest considerable room for improvement in usual community care practices.

To date, few studies have examined contextual predictors of use of evidence-based therapy techniques in usual care. Therapist variables (knowledge and attitudes) and client variables (presenting disorder) may be important predictors of whether a particular therapist uses evidence-based techniques in usual care settings (23,24). In one recent study, therapist factors (knowledge and attitudes) were found to be more predictive of use of psychodynamic techniques, but organizational factors (such as culture and climate) were found to be more predictive of use of CBT and family techniques (25). A limitation of the literature to date is that client factors have been largely ignored as possible predictors of use of therapy techniques. Understanding both therapist and client factors that predict use of therapy techniques is important because it allows for the identification of factors that can be targeted to change therapist behavior.

This study allowed for exploration of predictors of therapist self-report of techniques that fall into cognitive, behavioral, psychodynamic, and family domains within the context of a systemwide effort to improve usual care through implementing EBPs (26). The community mental health therapists in this sample provide care to youths in a large, urban, publicly funded system—circumstances that support the generalizability of our study. In addition, we were able to examine usual care within the unique context of a system in which providers implement multiple EBPs. Specifically, in the publicly funded mental health system in Philadelphia, therapists are receiving training and consultation in a number of EBPs, including cognitive therapy (27), trauma-focused CBT (28), prolonged exposure (29), and dialectical behavior therapy (30). This study examined whether community-based therapists in this system report using cognitive, behavioral, psychodynamic, and family techniques and if therapist- or client-level factors predict use of these techniques. Given the limited data on predictors of usual care practices, this study was exploratory in nature and was not based on any a priori hypotheses.

Methods

Setting

The Department of Behavioral Health and Intellectual disAbility Services (DBHIDS) in the City of Philadelphia (26) has engaged in pilot CBT implementation initiatives in the public mental health system since 2007. Implementation efforts are supported by a full-time city employee who coordinates the initiative. The training and ongoing consultation provided to therapists closely follows the recommendations of the respective treatment developers and includes posttraining consultation. For example, outpatient providers enrolled in the cognitive therapy training participated in 22 hours of didactic workshops followed by six months of weekly group consultation (31). Other initiatives have similarly intensive procedures.

Participants

Purposive sampling was used to recruit the 29 largest agencies within the over 100 community mental health agencies in Philadelphia that provide outpatient services to youths (personal communication, Community Behavioral Health, 2012). These agencies together serve approximately 80% of Philadelphia youths receiving publicly funded mental health care. Of these 29 agencies, 18 (62%) agreed to participate. An additional agency involved in EBP efforts approached us to participate, resulting in a final sample of 19 agencies representing 23 sites, 130 therapists, 36 supervisors, and 22 executive administrators. There were no exclusion criteria for therapist participation, and approximately 60% of therapists from the 23 sites participated. Of participating therapists, 123 (95%) completed all measures on one occasion. Of the agencies enrolled, 16 had participated in DBHIDS-sponsored EBP initiatives (mean±SD years of participation=2.90±2.70); of the therapists enrolled, 54 (41%) had participated in DBHIDS-sponsored EBP initiatives. Table 1 provides demographic information about therapists and clients.

Procedure

All procedures were approved by and conducted in compliance with the appropriate institutional review boards. A complete description of the procedures of the larger study within which these data were collected has been published elsewhere (25). Participants provided informed consent and were compensated $50 for participation. The principles outlined in the Declaration of Helsinki were followed. Data were collected between March 1 and July 25, 2013. Therapists were informed that their data would be kept confidential and that only aggregated and deidentified information would be shared with agency leadership and DBHIDS.

Predictor Variables

Therapist characteristics.

Therapist demographic characteristics were assessed with the Therapist Background Questionnaire (32), a 21-item questionnaire that obtains information on personal characteristics, such as age, gender, and licensure status. In addition, therapists indicated on the Evidence-Based Practices Training Survey whether they had participated in any of the four DBHIDS EBP initiatives (cognitive therapy, trauma-focused CBT, prolonged exposure, and dialectical behavior therapy). We asked specifically whether they had received training and a year of consultation to ensure that they formally participated in the city-sponsored initiative.

Client characteristics.

Therapists identified a representative client, defined as a client of their choosing who was most like a typical client on their caseload, and reported on that client’s age, race-ethnicity, and primary diagnosis. Childhood primary diagnoses, when necessary, were coded into DSM-IV disorders (for example, depression was coded as major depressive disorder). To maximize power for statistical analyses, we subsequently coded diagnoses as internalizing, externalizing, and other, given that there is considerable overlap in evidence-based techniques for internalizing disorders as well as for externalizing disorders (33). Therapists then reported on the therapeutic techniques used with the representative client they identified.

Dependent Variable: Therapy Techniques

The Therapist Procedures Checklist–Family Revised (TPC-FR [15,34]) is a 62-item self-report therapist checklist that assesses therapeutic techniques from the following modalities: cognitive, behavioral, family, and psychodynamic. The TPC-FR informed the development of the Therapy Process Observational Coding System (TPOCS [16]), a gold-standard observer-rated measure of therapy fidelity. The TPC-FR is a revised version of the TPC (34) developed to also query about family therapy techniques (15). Therapists selected a recent representative client about whom they reported, on one occasion, on which techniques they had used with that client over the course of therapy to date. The rating for each subscale is a mean of the items that fit within that factor, measured on a continuum from 1 to 5, with 1 indicating rarely; 2, seldom; 3, sometimes; 4, often; and 5, most of the time. Higher scores indicate greater self-reported use of the set of techniques. Good internal consistency has been reported for the TPC-FR (15) and the factor structure has been confirmed (35). In our sample, subscale alpha values ranged from .84 to .94.

Data Analytic Plan

Four mixed-effects linear regression models examined the relationships of the set of variables with self-reported use of specific therapy techniques (specifically, cognitive, behavioral, family, and psychodynamic), as measured by the TPC-FR. Mixed-effects models included random intercepts for organization to account for therapists nested within organizations. Predictor variables included therapist predictors (age; gender, coded as male or female; clinical experience; licensure status, coded as yes or no; EBP initiative participation, coded as yes or no) and childhood predictors (primary diagnosis, coded as externalizing, internalizing, or other; age, coded as early childhood, ages three to seven; middle childhood, coded as eight to 12; or adolescence, ages 13 to 18). These predictor variables were included because treatment recommendations vary somewhat for older versus younger youths and for those with internalizing versus externalizing disorders and because therapists may or may not have participated in training opportunities in EBPs. Missing data were minimal (<10%). Series means were imputed for missing predictor variables. Analyses were conducted with PROC MIXED in SAS 9.0. Effect sizes (Cohen’s f²) were computed with the procedures outlined by Selya and colleagues (36): .02 represents a small effect, .15 a medium effect, and .35 a large effect (37).

Results

Specific therapeutic techniques reported were assessed via the TPC-FR (Table 2 shows model parameters and intraclass correlation coefficients). With regard to therapist-level predictors, therapists who had participated in EBP initiatives were more likely to report use of cognitive (f²=.05) techniques, whereas therapists who were not licensed were more likely to report use of behavioral (f²=.02) and psychodynamic (f²=.07) techniques. With regard to client-level predictors, therapists working with clients with a primary externalizing disorder were more likely than therapists working with youths with primary internalizing diagnoses to report using behavioral (f²=.13) and family (f²=.02) techniques over the course of therapy. Therapists working with early childhood–age youths were more likely than those working with adolescents to report using behavioral (f²=.08) techniques; likewise, therapists working with adolescents were more likely than those working with youths in early childhood to report using cognitive (f²=.05) and psychodynamic (f²=.05) techniques and more likely than those working with youths in middle childhood to report using psychodynamic techniques (f²=.05).

The structure of the analyses required the use of a reference group for dummy-coded variables. Thus report of techniques used could not be compared between externalizing disorders and other disorders in our model. When the reference group was changed from a primary internalizing disorder to a primary other disorder, the results were largely unchanged. Therapists working with clients with a primary externalizing disorder were more likely than therapists working with youths with primary other diagnoses to report using behavioral techniques over the course of therapy (B=.39, p=.04, f²=.13). A significant difference in reported use of family therapy was not observed.

Discussion

This study was the first to examine the relationship between therapist- and client-level predictors of community-based therapists’ self-report of techniques that fall into cognitive, behavioral, psychodynamic, and family domains within the context of implementation efforts. Results indicate that both therapist- and client-level variables predicted self-reported use of therapy techniques. Overall, these findings are largely encouraging and suggest that therapists in this system are self-reporting use of evidence-based usual care practices, such as cognitive and behavioral techniques, in certain circumstances. This may be attributed to the systemwide emphasis on implementation of EBPs; indeed, the finding that therapists participating in initiatives were more likely to report use of cognitive techniques strengthens this interpretation. This study provides preliminary support for the idea that community therapists can self-report ascribing evidence-based techniques to client needs.

Therapist factors, namely therapist licensure status and participation in EBP initiatives, predicted self-reported use of therapy techniques. Specifically, compared with licensed providers, therapists who were unlicensed were more likely to report using behavioral and psychodynamic techniques. Given that we controlled for therapist age and years of experience, it is unlikely that these findings were driven by these factors. Unlicensed providers are supervised by licensed providers, so perhaps supervisors in this system are emphasizing behavioral and psychodynamic techniques with supervisees. Understanding the impact of supervisor support on use of therapy techniques is an important area for future inquiry (38). Therapists who had participated in EBP initiatives were more likely to report use of cognitive techniques. Given that there is a long-standing cognitive therapy training initiative in the system (31), this result is not surprising and suggests a positive impact of this EBP initiative on the system.

That client-level variables also predicted use of therapy techniques is noteworthy. Therapists working with youths with primary externalizing disorders were more likely than those working with youths with primary internalizing diagnoses to report use of behavioral and family techniques. This finding is consistent with the evidence base that suggests that at least some behavioral and family-based techniques are more effective for youths with externalizing disorders (39,40). Also of note, therapists were more likely to report use of cognitive and psychodynamic techniques with adolescents than with younger children (those in early childhood) and more likely to describe using behavioral techniques with this younger group versus adolescents. Cognitive and psychodynamic techniques both involve insight (41), and therapists may be more comfortable using these techniques with older youths.

Although therapists reported using a range of evidence-based cognitive and behavioral therapy techniques, they also reported using non–evidence-based techniques, particularly psychodynamic techniques. This finding is consistent with previous work showing that therapists tend to be eclectic in their approach to therapy and use a variety of therapy techniques (8,42). Deimplementation, or exnovation, refers to an organization divesting itself of a previously adopted innovation (43). This represents an area ripe for future research. Given that unlicensed providers report using psychodynamic techniques more frequently compared with licensed providers, unlicensed providers may be a group worth targeting for exnovation.

These findings are preliminary because we relied on therapist report for two important variables: use of therapy techniques and client’s presenting diagnosis. First, studies on the concordance of therapist self-report with observation are equivocal. For example, a study conducted by Hurlburt and colleagues (44) casts doubt on concordance, but deeper examination of their article suggests that the demand characteristics of the task differed for observers and therapists, calling into question the applicability of these findings to our work. Specifically, therapists reported on their behavior several days after the session, whereas observers scored every minute of treatment sessions. Furthermore, therapists were not trained in how to rate the intensity of their behavior, whereas coders had intensive training and detailed instructions on how to score intensity of behavior. On the other hand, a recent study in which therapists, youths, caregivers, trained raters, and treatment experts rated therapist adherence to a substance abuse treatment protocol found high concordance between self-reported therapist ratings and trained raters (which showed high concordance with treatment expert ratings). In that study, the methodology was more similar to ours in that therapists were asked to reflect on the overall techniques used in treatment over a longer period (one month) (45). Other studies have corroborated this finding (46). Second, therapists reported on their client diagnoses, and we did not independently verify these diagnoses with semistructured interviews; concordance between therapist diagnosis and semistructured interviews is low (47). However, for the particular question at hand, it is more important to capture the diagnosis which therapists report they are targeting in treatment, whether or not the diagnosis is consistent with research criteria. Therefore, if therapists are differentially implementing therapy techniques to client clinical presentation, we would expect to detect that with our methodology.

This study was strengthened by its large and diverse sample of community mental health therapists drawn from a major metropolitan area. Several additional limitations exist. Therapists did not provide information about severity of client’s illness or comorbid conditions, although a recent study of children receiving services though the publicly funded mental health system in Philadelphia suggests that comorbidity was low, as reported in Medicaid claims (specifically, <5%; [48]). Information was obtained about one client on a therapist’s caseload, and treatment duration was not reported. It is not known whether treatment duration was associated with how many intervention techniques therapists used, and future studies would benefit from examining this empirically. Therapists chose which clients to report on. Therapists could have been motivated to report on a client whom they deemed to be a good candidate for EBP techniques; however, we observed a wide range of presenting disorders and ages, suggesting that this may not have been a significant limitation. Finally, data were lacking regarding the TPC-FR’s ability to predict therapist behavior or to distinguish between therapists who use varying levels of specific techniques.

Conclusions

Results suggest that both therapist and client factors predict self-reported use of therapy techniques. Most important, participating in an EBP initiative increased self-reported use of cognitive techniques, suggesting that a systemwide effort to implement EBPs can result in individual therapist report of use of evidence-based techniques. Therapists self-reported increased use of behavioral and family techniques with youths with externalizing disorders and less use of cognitive and psychodynamic techniques with young clients. Future studies would benefit from using observational methods to ascertain the veracity of these findings. In addition, studies examining therapists’ use of EBPs with clients with a range of presenting problems (14) and comorbidity are needed, as are those that examine the role of treatment length and sequencing of EBP techniques in community care.