Sustainment of Integrated Care in Addiction Treatment Settings: Primary Outcomes From a Cluster-Randomized Controlled Trial

Implementing evidence-based practices (EBPs) is an investment requiring time, effort, and expenditures. These investments are worthwhile insofar as the implemented EBPs are sustained. Sustainment, or the state of continued evidence-based intervention, service, or program delivery, is the final stage of the implementation process and is ongoing (1–3). It is a stage during which most organizations fail to continue to deliver an EBP as originally planned or intended (4, 5). Therefore, one could argue that the true measure of a successful implementation process and the effectiveness of any implementation endeavor lies in the sustainment phase. Maintenance or sustainment of outcomes of interest can serve to demonstrate and validate the value of investments and expenditures, ultimately translating to better care for patients. In this report, we present the sustainment outcomes of a study designed to evaluate the effectiveness of the Network for the Improvement of Addiction Treatment (NIATx)—a multifaceted implementation strategy—to implement integrated mental health and addiction treatment services for persons with such comorbidity (6, 7; Chokron Garneau H, Assefa MT, Jo B, et al., unpublished manuscript, 2020).

In 2018, in the United States, an estimated 9.5 million adults had co-occurring disorders (8). This is a significant public health concern, given the associated societal burden (9). Often under- or misdiagnosed due to their inherent complexity, co-occurring disorders are also often not treated effectively (10). Integrated care, in which treatment for mental health and substance use problems is delivered conjointly, at the same time, and by the same provider or providers, is the most effective form of treatment, yet it is rarely provided (11–17). Data from the 2018 National Survey on Drug Use and Health confirm that less than 8% of individuals with co-occurring disorders reported receiving integrated services (8).

Although persistent and significant efforts have been made to integrate substance use and mental health treatment services, barriers to delivery persist. Deterrents, such as financial cost, administrative obstacles, or provider resistance obstructing integrated care delivery, are situated at multiple contextual levels: system, organizational, and individual (18–20). It is thus imperative to use a multilevel, multifaceted approach, such as NIATx, that targets barriers at every level to enable and sustain delivery of integrated care. Additionally, because NIATx is effective for making practice change in behavioral health settings, promotes sustainment of change, and combines process improvement tools and techniques with quality improvement interventions, we hypothesized it to be potentially effective to implement and sustain integrated services (6, 21–28).

Outcomes from our longitudinal study confirmed NIATx as an effective implementation approach to conjointly deliver mental health and substance use treatment (7; Chokron Garneau H, Assefa MT, Jo B, et al., unpublished manuscript, 2020). Previous results indicate that most participating agencies, regardless of study arm, successfully transitioned from offering addiction-only treatment services at baseline to providing integrated services by the end of the implementation period (7; Chokron Garneau H, Assefa MT, Jo B, et al., unpublished manuscript, 2020). Furthermore, we demonstrated that organizations’ capability to provide integrated treatment services varied by level of NIATx adherence (7; Chokron Garneau H, Assefa MT, Jo B, et al., unpublished manuscript, 2020).

Here, we present sustainment outcomes associated with implementation of integrated care for co-occurring mental and substance use disorders by using NIATx. Sustainment was considered when improved capability for integrated services was maintained up to 2 years beyond the active implementation phase.

Methods

Design

This cluster-randomized, waitlist control group trial was designed to evaluate the effectiveness of NIATx strategies for implementing and sustaining integrated services for co-occurring disorders. Recruitment began in 2016, and the sustainment period of interest concluded in 2019. Following baseline assessments, organizations enrolled in the study were randomly assigned to one of two cohorts: NIATx1 (N=25) or NIATx2 (N=24). NIATx1 received active NIATx for the first 12 months (year 1, 2017), followed by 24 months of sustainment (year 2, 2018; and year 3, 2019). NIATx2 organizations were delayed in starting NIATx for 12 months (year 1), followed by active NIATx for 12 months (year 2) and sustainment for 12 months (year 3) (a figure in an online supplement to this article depicts the design). In this article, we report on results from the sustainment period. Given the study design, the length of the sustainment period varied between groups (2 years for NIATx1 and 1 year for NIATx2), thus providing the opportunity to examine whether gains in integrated service capability were differentially durable. The study protocol has been previously described in depth (6, 7).

Participants

To participate, organizations had to be state licensed to provide addiction treatment services at outpatient, intensive outpatient, or residential levels of care; be tax exempt; have government status or meet a 50% minimum of public funding; and have no previous NIATx training or participation in a NIATx project. Participating organizations were recruited via letter by state behavioral health authorities. Forty-nine of the initially enrolled 53 (92%) community-based addiction treatment programs from the State of Washington were randomly assigned at baseline to either the NIATx1 (N=25) or NIATx2 (N=24) study arms.

Measures

Dual Diagnosis Capability in Addiction Treatment Index.

The Dual Diagnosis Capability in Addiction Treatment (DDCAT) Index is a quantitative organizational measure used to evaluate the current capability of addiction treatment programs to deliver integrated treatment services for persons with co-occurring disorders (29). It comprises 35 items across seven dimensions, including program structure; program milieu; clinical process, assessment; clinical process, treatment; continuity of care; staffing; and training. The 35 items are rated on a 5-point Likert scale, which includes anchor scores of 1, addiction-only services (AOS); 3, dual diagnosis capable (DDC); and 5, dual diagnosis enhanced (DDE). Ratings of items within a dimension are averaged to generate a subscale score. A DDCAT total score is derived by averaging the 35 items.

The DDCAT also produces a categorization of the organization’s integrated service capacity by using the American Society of Addiction Medicine taxonomy (30, 31). If less than 80% of scores are rated 3 or higher, an organization is categorized as AOS. If at least 80% of scores are 3 or higher, an organization is categorized as DDC, which is the adequate and acceptable standard for all specialty addiction treatment organizations (30, 32). If at least 80% of scores are rated 5, then an organization is categorized as DDE. The reliability and validity of the DDCAT have been demonstrated in numerous psychometric studies, and public and private health care systems have adopted it as a measure of behavioral health service integration (29, 32–35). The DDCAT Index (version 4.1) and DDCAT Toolkit (version 4.0) are in the public domain, free and available from the authors (31).

Evaluators, blind to the study arm, were rigorously trained in data collection, specifically pertaining to the DDCAT for this study. Evaluators conducted site visits in pairs to the addiction treatment organizations every 12 months for data collection. Data were collected from key informant interviews with staff and patients, site observations, and chart or program manual review over a 2- to 3-hour period. The information collected was then synthesized and summarized across all data sources by the evaluators to complete the DDCAT Index measure.

NIATx Stages of Implementation Completion.

The Stages of Implementation Completion (SIC) was adapted to track each organization’s participation and adherence to the NIATx implementation process (36, 37). Similar to the original SIC, the NIATx-SIC measure captures the proportion of completed activities, the duration of activities, and the total time frame from first to last activity (7, 36–38). Organizations with values above the average across all three components are deemed to have full NIATx adherence. NIATx activities completed during the preimplementation phase and active implementation phase were recorded by our research team and entered into an online data collection and reporting system.

Ethics

The institutional review boards at Stanford University, the University of Wisconsin–Madison, and the Washington State Health Care Authority reviewed the study and determined it to be exempt.

Statistical Analysis

McNemar’s test was used to compare the proportion of agencies categorized as AOS versus DDC and DDE between baseline and study end. Longitudinal outcomes were analyzed both with and without consideration for NIATx adherence. Standard linear mixed-effects modeling was employed to estimate changes from baseline to end of sustainment in the organization’s capability for providing dual diagnosis in addiction treatment (39, 40). For all model estimation, we used maximum likelihood embedded in the Mplus program (41). Specifically, we used a random-intercept piecewise model, assuming three segments of linear change over time, from baseline to year 1, year 1 to year 2, and year 2 to year 3.

Per the intention-to-treat principle, all randomized agencies were included in the analyses as long as their data were available from at least one assessment. Unavailable data resulting from attrition or missed assessments were treated as missing-at-random, conditional-on-available (observed) information, a standard strategy of handling missing data in mixed-effects modeling. In the per-protocol analysis, we examined the two randomized arms after excluding organizations that were not fully adherent to NIATx, as determined by the NIATx-SIC. In both the intent-to-treat and per-protocol analyses, Cohen’s d, a measure of effect size, was calculated based on observed standard deviation pooled across the two conditions at each time point.

Results

Baseline Characteristics

Forty-nine of the 53 organizations enrolled in the study were randomly assigned to either NIATx1 (N=25) or NIATx2 (N=24). By study end, 40 organizations (82%) remained enrolled. A majority of organizations that discontinued participation did so because their facility closed or was sold (see online supplement for an extended CONSORT diagram [42]).

Baseline characteristics of organizations have been extensively presented in previous reports (7; Chokron Garneau H, Assefa MT, Jo B, et al., unpublished manuscript, 2020). Participants were community addiction treatment organizations located in Washington State. Participating organizations were primarily from the public sector and located in regions with shortages in health care services (43). Organizations in the NIATx1 group had more annual admissions on average (487.9±777.6), compared with the NIATx2 group (276.5±243.1). NIATx2 programs had longer lengths of stay on average (137.1±109.6), compared with NIATx1 programs (113.2±109.4).

Primary Outcome: Changes in Dual Diagnosis Capability

Across both study arms, dual diagnosis capability improved and was maintained throughout the study period. Both groups started with a substantial number of organizations categorized as AOS (78%, N=31). At study end, in a complete reversal, the preponderance of organizations (78%) was at least DDC across both groups (p<0.05). Specifically, 35% (N=7) of NIATx1 organizations were at least DDC at baseline, compared with 80% (N=16) at year 3 (p<0.001). For NIATx2, 10% (N=2) were DDC or DDE at baseline, compared with 75% (N=15) at year 3 (p<0.001). Although most gains were made between baseline and the end of the active implementation period, both cohorts continued to experience growth in DDC organizations between the end of the intervention and the end of the first sustainment period (NIATx1, 50% [N=10] to 65% [N=13]; NIATx2, 60% [N=12] to 70% [N=14]). NIATx1 saw further accrual in DDC organizations in the second sustainment year (from 65% [N=13] to 75% [N=15] in year 3).

Evidence for Sustained EBP

Intent to treat.

Intent-to-treat analyses, including all randomly assigned agencies (N=49), were conducted by using mixed-effects modeling (Table 1; see also online supplement). No significant changes occurred in the NIATx1 group in its first year after active implementation. During the second sustainment year, the NIATx1 group showed significant improvement in the DDCAT subscale metrics clinical process, assessment; clinical process, treatment; and continuity of care. No significant changes occurred for the NIATx2 group in the sustainment year. Effect sizes for changes from baseline to study end ranged from small to large (Cohen’s d=0.25 to 0.88). The only significant between-group difference in terms of the change from baseline to study end was for the training subscale: NIATx2 outperformed NIATx1 (Cohen’s d=0.88, p=0.02).

Per protocol.

Twenty-three agencies were fully adherent to NIATx per their SIC scores (NIATx1, N=11; NIATx2, N=12) and were included in per-protocol analyses (Table 2; see also online supplement). Neither NIATx1 or NIATx2 agencies showed any significant changes in the first year after active implementation. No significant changes were noted for the NIATx1 group in its second sustainment year. However, the NIATx1 group slightly improved on the subscale for clinical process, assessment, whereas NIATx2 slightly deteriorated on that subscale. Effect sizes for changes from baseline to study end were small and ranged from 0.04 to 0.34. There were no significant between-group differences at study end in terms of change from baseline.

Discussion

Gains in capability for integrated treatment services were sustained after the active implementation year. Both groups achieved successful implementation and sustained the implementation to a similar degree regardless of the sustainment year. At baseline, 78% of participating organizations were categorized as AOS. By the end of the trial, the ratio was reversed: 78% of participating agencies across both groups were at least DDC. These 31 agencies significantly improved and maintained care and services for their patients with dual diagnoses of mental and substance use disorders. Attainment of DDC or DDE status mainly was achieved during the active implementation phase, and these changes were sustained until study end. Additionally, NIATx1 organizations continued to improve on DDCAT subscales of clinical process, assessment; clinical process, treatment; and continuity of care throughout the sustainment period.

This study had some limitations. There was a volunteer bias for organizations that agreed to participate in the study, those that completed the implementation strategy, and those that maintained participation through study end. Also, baseline differences existed between cohorts. Although statistical analyses accounted for these differences, it would have been preferable to balance groups on key variables. Further, an additional measure of integrated treatment delivery would validate our results and add richness to the interpretation of findings, including patient-level or service-level outcomes. Nonetheless, previous studies with the DDCAT Index have found this measure to have good validity and to be associated with integrated service delivery and receipt and with patient outcomes (6, 7, 29, 34, 35, 44, 45).

These results build on, solidify, and extend our previous findings. We previously established the importance of using a proven manualized but adaptive strategy, such as NIATx, to enable change, as well as the importance of documenting participation during the active implementation stage and of measuring adherence to participation when examining outcomes (7; Chokron Garneau H, Assefa MT, Jo B, et al., unpublished manuscript, 2020). Here, we have demonstrated that increased capability to deliver integrated treatment services for individuals with co-occurring mental and substance use disorders was sustained. This finding is significant because individuals with comorbid disorders in Washington State now have increased access to integrated care, the gold standard for co-occurring disorders.

Results from both intent-to-treat and per-protocol analyses converged, and minor differences in sustainment outcomes emerged between both sets of analyses. Thus our results indicated sustained changes in behavioral health settings when these settings were evaluated under both actual and ideal circumstances (46).

It is, however, challenging to determine what accounted for the lack of difference between both sets of analyses. Following our year 1 and year 2 results, we would expect adherent organizations to outperform nonadherent organizations during the sustainment period. We suggest that this lack of differences was attributable to enhanced monitoring and feedback. Performance data were compiled and provided to participants for their agencies, along with data from all participating agencies combined. All organizations received this enhanced monitoring and feedback from baseline to study end, regardless of adherence and participation status. Our study design did not account for enhanced monitoring and feedback, and thus its role cannot be statistically confirmed. However, the results from the intent-to-treat analyses in sustainment make a convincing argument that enhanced monitoring and feedback is a viable and effective implementation strategy.

Conclusions

This article presents results from a study of the sustainment of integrated services after active implementation support was withdrawn. We observed wide variation in adherence to the study implementation strategy (NIATx) yet good overall sustainment of the target outcome. To further confirm these findings and advance the field of addiction treatment health services dissemination and implementation science, a better understanding of what and how an organization’s contextual factors may predict, moderate, and mediate sustainment outcomes is needed (47). This need is further strengthened by findings from Swain et al. (48), who reported differences in sustainability of mental health services for people with mental illness on the basis of agency characteristics, such as financing, staffing, training, fidelity, and agency leadership. Future studies will also be needed to determine whether NIATx is effective over time, compared with groups that did not receive the intervention.

In implementation practice and research, it is typical to offer a fixed protocol for expected strategy participation. A one-size-fits-all implementation strategy is, however, neither effective or efficient. It creates a high demand and burden on organizations to engage in all activities, whether they need to or not, to increase access to an EBP. Much like a stepped-care approach in a health care situation, a similar approach to implementation might apply whereby low-intensity strategies, such as monitoring and feedback, might be tried first and then evaluated for desired outcomes. If not effective, then increasingly resource-intensive strategies might be deployed. This stepped measurement–based implementation approach lends itself to an adaptive trial design and may inform future implementation endeavors with greater specificity and cost-efficiency. In addition, it would, decrease program burden and make better use of limited resources to support program implementation. Intensive supports that would have been spent on programs needing only a light touch could be redirected toward reaching other programs.