The Influence of Team Functioning and Workload on Sustainability of Trauma-Focused Evidence-Based Psychotherapies

Posttraumatic stress disorder (PTSD) is a potentially disabling psychiatric disorder that disproportionately affects veterans (1,2). Fortunately, there are effective treatments (3–10). Treatments with the highest level of endorsement in clinical practice guidelines are trauma-focused evidence-based psychotherapies (TF-EBPs) (11,12). To implement two TF-EBPs, the U.S. Department of Veterans Affairs (VA) began nationwide competency-based training for cognitive processing therapy (CPT) (13) in 2006 and for prolonged exposure (PE) (14) in 2007 (15). VA policy requires that all VA patients with PTSD have access to one of these TF-EBPs (16).

Over the past decade, research has evaluated VA’s implementation of TF-EBPs in terms of provider and patient perceptions of TF-EBP acceptability, barriers and facilitators of provider uptake (17), and reach to patients with PTSD (18,19). Reach is a measure of the population impact of an intervention or program. It has been defined as the percentage and representativeness of individuals within a defined population who receive an intervention or program (20). Sayer et al. (18) found that outpatient PTSD teams with high levels of reach for TF-EBPs were organized differently than those with low reach. High-reach teams had a defined sense of purpose, or mission, to deliver TF-EBPs, internal and external leadership support for this mission, team members engaged in this mission, TF-EBP operational processes, and access to services outside the team for patients who do not want a TF-EBP or who have other treatment needs.

Another metric of implementation success is sustainability (21), which is defined as the extent to which a newly implemented treatment is maintained or institutionalized within a service (21,22). Although over a decade has passed since implementation of the TF-EBP training initiatives, very few studies have examined the sustainability of these psychotherapies (17). This omission is consistent with the larger dissemination and implementation literature, in which sustainability is understudied relative to initial adoption (23,24). In fact, sustainability is discussed in conceptual papers more often than it has been empirically investigated (21). Yet policy makers and program managers need to plan for sustainability so that the resources used in implementation are not wasted (25). Researchers have recommended that studies of program outcomes also identify factors that account for sustainability, given that such information can be used to guide effective implementation strategies and policy (26). In addition, data on the association among implementation outcomes are needed to determine whether improvements in various outcome domains are positively related with each other (for example, whether greater reach affects the extent to which a practice becomes a cultural norm, which promotes sustainability). Data can also show whether various outcome domains are negatively related (for example, whether a short-term increase in reach burdens the system and undercuts its sustainability) (21).

This study is guided by the British National Health System’s sustainability model, which uses a quantitative measure to estimate the likelihood of sustainability of a practice change. This model consists of 10 factors grouped into three larger domains: process (for example, credibility and adaptability of the change), staff (for example, leadership and staff support for the change), and organization (for example, infrastructure for sustaining the change and fit with the organization’s aims and culture) (22,27). The sustainability model was designed to help teams identify local contextual challenges to sustainability (27).

Is also important to determine whether there are additional factors that help ensure that a new practice or program will continue. Although the sustainability model assesses staff support for a practice change (22,27), it does not assess team functioning. Team functioning is the representation of how well team members work together and includes interpersonal cohesion and activities related to the work of the team (28). Prior research indicates that dimensions of team functioning affect program implementation and sustainability within health care settings (29–34). In the VA, TF-EBPs are primarily delivered by providers working in specialized outpatient PTSD teams, known as PTSD clinical teams (15,19). Providers on these teams may have varying levels of success in working together to maintain the infrastructure and care processes needed for TF-EBP delivery. Thus high levels of team functioning may translate into greater TF-EBP sustainability. Mental health teams with greater cohesion may be more willing or able to support team members who are implementing TF-EBPs. A positive association between team functioning and sustainability could represent an important managerial consideration.

Another potentially important factor missing from the sustainability model is workload. Workload has been found to affect the provision of evidence-based care in both general medical and mental health settings (35–37). CPT and PE are time-intensive (eight to 12 weekly sessions), manualized treatments (13,14). Adding weekly sessions of a TF-EBP, which are 60 or 90 minutes in length, may be challenging if providers already have a high caseload. In fact, providers reported that lack of time is a barrier to use of these treatments (32,38,39). Thus it would be important to determine whether greater workload diminishes TF-EBP sustainability.

This study examined TF-EBP sustainability in VA outpatient PTSD clinics 10 years after initial implementation. We hypothesized that there would be a positive association between team functioning and TF-EBP sustainability and a negative association between workload and TF-EBP sustainability. To inform future research, we conducted exploratory analysis to examine the association between TF-EBP reach and sustainability.

Methods

The VA Central Institutional Review Board approved this research. An alteration of informed consent (to allow for online consent) and waiver of documentation of informed consent were obtained for the anonymous staff survey. A waiver of informed consent and a HIPAA waiver were obtained for use of the administrative data.

Data Source

This study is based on VA administrative data and the PTSD team staff surveys collected as part of a mixed-methods study on reach of TF-EBPs. For the parent study, we used purposive sampling to select 10 PTSD teams from nine diverse facilities, with oversampling of teams that had high reach of TF-EBPs. Results from site visit interviews are presented elsewhere (18).

Procedures

To select sites for the parent study, we extracted administrative and chart note data for fiscal year (FY) 2015 (October 2014–September 2015) for all patients with PTSD who received psychotherapy from outpatient PTSD teams at the nine facilities. We identified staff assigned to the selected teams through a point of contact on each team. We used e-mail to recruit clinical staff and peer support specialists assigned to each team to complete an anonymous staff survey. Survey recruitment methods involved an introductory e-mail followed one week later by an e-mail with a link to the online consent and survey and then by three weekly reminder emails with the same link (40). At the time of the final reminder, we also mailed printed copies of the consent form and the survey and self-addressed stamped envelopes to our point of contact for distribution to the PTSD team clinicians. Surveys were completed between August 2014 and October 2015.

Dependent and Independent Variables

EBP sustainability.

Our outcome measure was the British National Health Service Sustainability Index. This 10-item measure assesses staff perceptions of factors that influence sustainability (22,27) and has been used to evaluate sustainability as an outcome (41–43). For each item, responders choose one of four weighted statements that they believe best describes each domain. Possible scores range from 0 to 100.9. Preliminary evidence suggests that a total score of 55 or higher means that an intervention or program will be sustained. A score of 45 or lower means that significant effort is required to address barriers to sustainability (26). The Sustainability Index also includes subscales reflecting the conceptual domains of process (N=4), staff (N=4), and organization (N=2). In the survey to staff, we assessed sustainability of CPT and PE separately (CPT, α=.84; PE, α=.73). However, because the Sustainability Index scores for CPE and PE were highly correlated (r=.95), we combined them into one score (α=.78).

Team functioning.

The staff survey included 53 items with 10 measures to assess team functioning domains. Twenty-one items drawn from the Survey of Organizational Attributes of Primary Care Practice assessed team communication, decision making, chaos and stress, and history of change (44,45). We used measures from the Learning Organization Survey to assess the extent to which the learning environment was supportive and leadership reinforced learning (46); Edmondson’s conceptualization of team behaviors to assess team learning behavior (47); organizational slack measures to assess organizational resources for staff, space, and time (48); and the Team Helping Measure to assess organizational citizenship behaviors (49). The response options range from 1 (disagree strongly) to 5 (agree strongly). Scores were computed as the means across completed items, and reliability was moderate to high for all scales (Cronbach’s α>.79) (50).

Given the large number of items and small sample size, we used principal-components analysis to determine whether these 10 measures could be combined to create a composite measure. The first component explained 90.1% of total variance (eigenvalue=5.73), demonstrating unidimensionality. Nine of the 10 team functioning measures collectively contributed to the total score, whereas one measure, history of change, did not (eigenvalue=.73, 12% of variance explained). Thus we calculated a composite team functioning measure on the basis of the first principal component by using nine scales, with higher scores indicating more positive team functioning (α=.94). History of change served as a separate independent variable. Kaiser’s overall measure of sampling adequacy was .90.

Team workload.

We determined the number of patients with PTSD seen by the outpatient clinical teams by using the stop codes that designate team workload. Workload was calculated by dividing the number of PTSD patients seen by team members for any treatment in fiscal year 2015 by the number of PTSD providers on the team.

TF-EBP reach.

In FY 2015, VA released chart note templates that allowed for identification of EBP type. Because the templates were new, we applied natural-language processing to identify additional TF-EBP cases from psychotherapy notes that were not derived from templates. Our natural-language processing system was based on the work of Shiner et al. (51) but was built by using hand-coded rules in the SAS Enterprise Guide, version 7.1. Performance for therapy note classification as measured by recall, precision, and F measure was strong (recall=.91 and .85, precision=.96 and .95, and F measure=.93 and .90, for CPT and PE, respectively). A patient who received at least one session of CPT, PE, or both in FY 2015 was designated as a TF-EBP recipient. TF-EBP reach was defined as the number of patients with PTSD who received a TF-EBP divided by the number of therapy patients with PTSD who received any form of psychotherapy on each team. Reach on each team ranged from 14% to 59% (mean±SD=36%±17%). Because we oversampled high-reach teams, the rates did not represent TF-EBP reach across the VA.

Covariates.

We modeled three classes of covariates, each representing patient, team, or facility characteristics. We extracted patient age, sex, race, and ICD-9 codes indicative of major psychiatric disorders (alcohol and drug abuse, anxiety, depression, and psychosis). We constructed team-level patient characteristics by calculating the proportion of patients who were male, the proportion who were Caucasian, the patients’ mean number of psychiatric comorbidities, and the patients’ age. The staff survey assessed team members’ gender, age, race-ethnicity, occupation, and VA tenure. For each participating facility, we extracted location, complexity level, and designation as urban or rural. VA facilities are classified into five complexity levels (1a, 1b, 1c, 2, and 3) on the basis of patient population, clinical services, educational and research mission, and administrative complexity. Rankings range from most (1a) to least complex (3).

Analyses

Twenty-two of 78 providers did not complete all Sustainability Index items. We used regression modeling to induce a monotone missingness pattern and the propensity method (52,53) to multiply impute (five times) the missing items prior to calculating Sustainability Index scores. All the analyses were conducted within each imputed data set separately; the reported numbers are the combined results. We used responses to the other survey measures (for example, team functioning, provider characteristics, and completed Sustainability Index items) to construct the propensity and regression imputation models.

Initially, we used Pearson correlations to examine the associations between independent variables and TF-EBP Sustainability Index total and subscale scores. To test our hypotheses, we then used multiple linear models with Sustainability Index total scores regressed on the independent variables (team functioning, workload, history of change, and reach) separately, each time accounting for the covariates. Because of the small number of teams, team-level predictors were used only for the means modeling. In a final multiple linear regression model, we included all explanatory variables and the covariates simultaneously.

To evaluate the robustness of the findings obtained from the combined analyses of multiple imputed data sets, a complete case reanalysis of our model was also performed. The results were virtually identical. Model assumptions were examined by using scatter diagram and residual plots. The normality assumption was not rejected. Gaussian multiple regression analyses were conducted by using SAS, version 9.4.

Results

Table 1 describes the participating sites, patients, and survey responders. A total of 140 PTSD team members received surveys, and 78 (56%) completed the team functioning survey. The response rate per team ranged from 33% to 75%. Compared with responders, nonresponders were just as likely to be psychologists or social workers. All but one participant spent at least 20% of his or her time providing direct patient care, and 78% (N=61) identified as clinical staff without team leadership responsibilities. Measures to assess team functioning and scores of responders are presented in Table 2.

Table 3 presents each team’s values for team functioning, workload, and reach, ranked by Sustainability Index score. Only one team (5a) had average Sustainability Index scores that fell below the preliminary cutoff of 55. Table 4 presents correlations between independent variables and Sustainability Index subscale and total scores. Higher scores on each Sustainability Index subscale were positively associated with better team functioning and reach and negatively associated with greater workload.

Multiple linear regression analysis in which we adjusted for covariates confirmed our hypotheses that a higher total score on the Sustainability Index was positively associated with better team functioning (B=9.18, SE=2.01, p<.001) and negatively associated with greater workload (B=–.26, SE=.12, p=.03). Exploratory analysis showed that after adjustment for the covariates, the association between reach and sustainability was not significant. These effects persisted when we included all three explanatory variables in one multiple linear regression analysis. That is, team functioning remained positively associated (B=9.16, SE=1.97, p<.001) and workload remained negatively associated with scores on the Sustainability Index (B=–.28, SE=–.12, p=.02), with no association found for reach.

Discussion

Although a decade has passed since the VA began implementation of TF-EBPs, this is one of the only studies to examine the sustainability of these psychotherapies (17). We found that mean TF-EBP Sustainability Index scores exceeded the threshold recommended by the British health care system, perhaps reflecting institutionalization of TF-EBPs through VA policy and training (15,16). Nonetheless, there was variability in sustainability of TF-EBPs, with some teams demonstrating room for improvement. We know from site visit interviews that the team with the lowest sustainability score was experiencing a leadership change and was worried about losing infrastructure and support for TF-EBPs. Thus the team’s relatively low sustainability scores reflected its concerns about the program’s viability during the site visits.

Even after adjustment for covariates, workload and team functioning were associated with TF-EBP sustainability. The finding that teams with better functioning had higher sustainability scores expands prior research suggesting that higher levels of team functioning are an important precursor for high-quality care (54). Higher functioning teams may have more frequent discussions about monitoring and meeting team goals, better allocation of job duties across work roles, better workload sharing practices, greater collaboration in solving problems and coping with challenges, and more effective team leaders (31), which may have facilitated sustainability.

The finding that higher workload was associated with lower sustainability converges with prior research suggesting that a high workload may interfere with delivery of evidence-based mental health care (35–37), including TF-EBP delivery (32,38,39). It appears that more research is warranted to determine whether there is a change point after which greater workload threatens the sustainability of EBPs more substantially than would be predicted by a linear relationship. In classical statistical approaches, the change point is examined by approximating the function between two variables with a polynomial and constructing a confidence interval around the maximizer (55). A larger sample than the one available here would be needed to fully examine this issue.

Research examining the relationships among implementation outcomes is needed to inform theory of implementation and the design of implementation trials (21). Although TF-EBP reach was correlated with sustainability, it did not predict sustainability over and above the patient- and facility-level covariates. This finding suggests that interventions to promote TF-EBP sustainability may differ from interventions to improve reach. In qualitative interviews related to this study, we found that some sites with low reach of TF-EBPs expressed strong support for continuing to use these treatments, but with only a small portion of their patients. We examined TF-EBP reach and sustainability over the same 12-month period; a longitudinal study with a greater number of clinical teams and patients and multiple measurement points would be needed to evaluate the possibility of temporal associations between reach and sustainability. Evaluating temporal associations would lead to a better understanding of whether changes in one measure precede changes in the other or whether a synergistic effect exists.

Methodologically sound research on sustainability of evidence-based treatments is rare (23,24). A strength of this investigation is our quantification of TF-EBP reach by using chart note data rather than clinician self-report. The response rate to the survey (56%) was better than what is often achieved in surveys of providers (32,43), but our sample was relatively small because of the limited number of clinical staff assigned to each team. This limited our ability to examine more complex relationships among independent variables. Because our survey was anonymous, we had limited information on nonresponders. We did not measure team mission or purpose, a factor that was associated with TF-EBP reach in prior research (18) and that may also be associated with sustainability. Last, we did not assess fidelity, another important implementation outcome (21), and thus we cannot address questions about the quality of treatment delivery or the relationship between TF-EBP fidelity and sustainability.

Conclusions

Much of VA’s TF-EBP implementation efforts has focused on training individual clinicians to ensure that they have the skills to deliver CPT and PE (15). Although training is crucial, attention must also be paid to the context in which clinicians work (18,19,32). We are aware of implementation strategies that have included leadership training (56,57), but less attention has been paid to how to develop high-functioning teams (58). Similarly, there is a dearth of guidance regarding workload levels that optimize TF-EBP implementation and sustainability. The contribution of team functioning and workload to the sustainability of evidence-based mental health care warrants further study.