Fidelity to Recovery-Oriented ACT Practices and Consumer Outcomes

Since its inception in the 1970s, assertive community treatment (ACT) has grown from a pilot project at a state psychiatric hospital in Wisconsin (1) to one of the most studied and widely disseminated evidence-based practices for persons with severe mental illness (2,3). Among its core features, ACT is characterized by a multidisciplinary team-based approach with a small (1:10) staff-to-consumer ratio, 24/7 staff availability, and assertive outreach (4).

ACT has been adapted to serve urban and rural populations (5–8), homeless persons (9), and persons with severe mental illness who are involved with the criminal justice system (10,11). Evidence-based practices, such as integrated dual-disorders treatment (12,13), illness self-management (14), and supported employment (15) have successfully been added to ACT. These adaptations have led some observers to conclude that ACT is best viewed as a platform for service delivery and integration of other evidence-based practices (16).

The ethos and best practices of community mental health care have changed dramatically over the last 30 years, from a time when clinical staff essentially defined goals and created a high degree of daily structure for consumers with severe mental illness to today’s more person-centered and recovery-oriented approaches (17,18). In brief, recovery-oriented services support the fundamental components of recovery, which include self-direction, individualized and person-centered approaches, empowerment, holistic views, nonlinearity of service provision, a strengths-based approach, peer support, respect, responsibility, and hope (19,20). Some observers have long considered ACT to be inherently coercive and paternalistic (21–25). However, ACT can be adapted to align with recovery principles and practices (26). There is a new generation of ACT studies under way that focuses on the processes and outcomes of recovery-oriented ACT teams (27,28).

Over the past decade, the Dartmouth Assertive Community Treatment Scale (DACTS) (29) has become the gold standard for assessing the fidelity of ACT teams to the original clinical model. The research rationale underlying fidelity assessments is that in establishing high fidelity there is a high likelihood that the ACT team will produce results similar to those found in randomized clinical trials of ACT. Fidelity assessment also serves as a quality improvement mechanism to assist teams in improving performance over time. Although early efforts to establish a connection between fidelity and improved outcomes led to mixed results (25,30,31), recent meta-analyses that use aggregated data across dozens of clinical trials have confirmed that high fidelity, especially in regard to the organizational components of ACT, is associated with reduced psychiatric hospitalizations (32,33).

However, the DACTS has been criticized for focusing too much on the structure of ACT and not enough on care processes, especially those embracing recovery-oriented principles (34,35). Recently, the Tool for Measurement of Assertive Community Treatment (TMACT) has been developed as a recovery-informed update of the DACTS (35). The TMACT was created to monitor the implementation of Washington State’s recovery-oriented ACT initiative (PACT), which deployed ten recovery-oriented ACT teams statewide in 2007 (36). The PACT teams were developed to reduce the utilization of Washington’s two state psychiatric hospitals, along with other inpatient and costly services. Each team received in-depth training about the core elements of ACT, as well as training on integrating ACT with recovery-oriented principles (35,36). Training on recovery approaches was provided by national experts and was focused on strengths-based assessment, person-centered planning processes, and promotion of a culture of recovery within each PACT team.

Our earlier research in Washington State found that PACT participation led to reductions in the use and costs of state psychiatric hospitals (37), especially in the first six months after PACT enrollment and for consumers who were high users of hospitals before PACT enrollment (38). As part of that research, we had the opportunity to examine the relationship between fidelity as measured by the TMACT and consumer service use outcomes. We addressed the following specific research question in the study reported here: Did participation in a higher-fidelity PACT team lead to lower use of state psychiatric hospitals, local hospital psychiatric units, and local crisis stabilization units and to fewer arrests?

Methods

Design

We used a longitudinal study design and statewide administrative data to examine the relationship between fidelity to recovery-oriented ACT as measured by the TMACT and outcomes among 631 PACT consumers over a three-year study period (2007–2010). The study sample included PACT participants with no history of state hospitalization (N=181) and those with prior state hospital use (N=450). The latter group was a sample in our previous studies (37,38).

The research was conducted with the approval of the Institutional Review Board at the Washington State Department of Social and Health Services and at the University of North Carolina, Chapel Hill.

Data

We accessed administrative data from an information system maintained by Washington State’s Research and Data Analysis Division, Department of Social and Health Services, which links consumer demographic, diagnostic, and service utilization records from Medicaid, behavioral health, criminal justice, and other state data systems in a longitudinal, person-specific file (39). Data were collapsed to the person-month level, with each observation reflecting the use of services during that month (37). TMACT fidelity data were obtained from the Washington Institute for Mental Health Research and Training at the University of Washington, the organization responsible for initial training and continuing consultation to the PACT teams (35).

Measures

Fidelity.

The TMACT, an enhanced fidelity tool, was developed and pilot-tested to better assess critical ACT structures and processes (35). It consists of a 47-item inventory measuring team organization and infrastructure, staffing and roles, core practices, person-centered assessments, and recovery orientation. Each item is scored on a 5-point behaviorally anchored scale from 1, low, to 5, high; an overall team score is obtained by averaging the scores on each item. Two trained, independent reviewers conducted TMACT fidelity assessments for each PACT team during a site visit. Data sources for the fidelity review included team self-report data; observations of daily team meetings, treatment planning meetings, and staff provision of services; reviews of randomly selected charts; and interviews with staff and consumers. TMACT fidelity scores were available for each of the ten teams at baseline (within four to six months after start-up) and at six, 12, and 18 months later.

Outcomes.

Outcome measures included use of state psychiatric hospitals, local hospital psychiatric units, emergency departments, and local crisis stabilization units that functioned as evaluation and treatment alternatives to state hospitals, as well as arrests for gross misdemeanor and felony crimes (37). Costs of state hospital use are reported in 2010 dollars, having been adjusted by the Gross Domestic Product deflator.

Data analysis

All data were pooled across the ten PACT teams and collapsed to the person-month level, with each observation reflecting the use of services during that month. Our previous analyses (37) indicated that although there was some evidence of clustering at the team level, the explained variance based on the large sample of person-months was extremely small and essentially ignorable for these analyses. Specifically, we computed the intraclass correlation coefficient (ICC=.01121, 95% confidence interval=.00003–.02239) to assess clustering at the team level. Although statistically significant at p<.05, the ICC accounted for a miniscule portion of explained variance (R²=.01) on the basis of a very large sample (N=13,343 person months). These results indicated that clustering at the individual level was the appropriate model for these analyses and that clustering at the team level was so negligible it could be ignored in these analyses. The generalized estimating equation (GEE) models that we used, as described below, account for clustering at the individual level.

We used GEE models of the person-month data to examine the relationship between TMACT scores and service use outcomes. In our analyses, TMACT scores were interpolated monthly between assessment points. For the months before the baseline TMACT assessment, the baseline values were carried backwards, and for the months after the last assessment TMACT values were carried forward through the end of the study period. TMACT scores were appended to monthly ACT participant data by team and calendar month of assessment.

Continuous variables were analyzed with GEE models with a log link and a gamma distribution, binary variables were analyzed with GEE models with a logit link and a binomial distribution, and count variables were analyzed with GEE models with a log link and a Poisson distribution. Models controlled for race, ethnicity, gender, age (by using a quadratic term), and diagnosis. We used indicators of monthly time trends to allow for nonlinear effects when possible, but some of the models with less frequent outcomes used a linear time trend to facilitate model convergence. Most models were run with an autoregressive correlation structure, but exchangeable structures were required to achieve convergence in some of the models. Average marginal effects of the TMACT were estimated from GEE output.

Because our earlier research demonstrated that PACT effects differed by participants’ level of state psychiatric hospital use before PACT enrollment (37), we examined the difference in the marginal effect of fidelity scores for persons with any history of state psychiatric hospitalization from January 2001 through PACT enrollment (N=450). We did the same for PACT participants with high levels of state hospital use before PACT enrollment (defined as ≥96 days of hospital use in the two years before enrollment; N=263). This criterion for high use was adopted from a recent Cochrane Review of intensive case management for people with severe mental illness (32,33). We created indicators of whether study participants had each of these two characteristics (any use and high use). We reran the GEE models described above with these binary indicators included in separate models and report the marginal effects from these models for the defined subpopulations.

Results

Baseline TMACT scores ranged from 3.3 to 4.5 with an average of 3.9 on a 5-point scale, with higher scores indicating greater fidelity to recovery-oriented ACT (Table 1). Generally, teams showed increases in TMACT scores over the six-, 12-, and 18-month intervals; however, these increases were not statistically significant. There was an .18-point increase in the average score across the ten teams from baseline to six months, an .08-point increase from six to 12 months, and a .05-point increase in the average score across the ten teams from 12 to 18 months. TMACT scores at 18 months ranged from 3.52 to 4.57 with an average of 4.2.

The three comparison samples were fairly homogeneous with regard to demographic and diagnostic characteristics (Table 2). That is, the three samples were fairly homogenous because they consisted of one sample and two subsamples that were not independent. Participants were predominantly white and male, with an average age of 40, and many had multiple comorbidities. Nearly all had a diagnosis of schizophrenia, more than half also had a diagnosis of affective disorder, and about half also had a substance use disorder. By definition, members of the sample who were high users of the state hospital at baseline all had at least 96 days of state hospitalization in the two years before PACT enrollment, as did about half of the state hospital users and less than half of the PACT full sample.

The results from the GEE models are shown in Table 3. For the most part, the overall pattern of results suggests that higher TMACT scores were associated with reduced days of use of state hospitals, local inpatient services, and crisis stabilization units but not with the probability of use of any of these institutional settings. For state hospital days, we found that higher TMACT scores were associated with fewer hospital days per month (p<.01). These results were consistent across samples, although the marginal effect of higher TMACT scores on hospital days was somewhat smaller and not significant among high users of the state hospital at baseline. In contrast, the marginal effect of higher TMACT scores on reductions in local inpatient days achieved statistical significance only in the high user sample (p<.01). Also, higher TMACT scores were consistently associated with fewer crisis stabilization unit days across the three samples (p<.01). None of the results for the probability and use of emergency departments or for arrests were statistically significant.

Discussion

How well did TMACT scores predict consumer service use outcomes? There was an inverse relationship between TMACT scores and days in state hospitals and local crisis stabilization units. That is, higher TMACT scores were associated with reduced state hospital stays and reduced days in acute crisis units. However, TMACT scores were not associated with a lower probability of admission to state or local hospitals or emergency departments. There also was no association between higher TMACT scores and arrests. Accordingly, the overall pattern of results is consistent with our earlier findings (37) that the PACT initiative in Washington State affected the intensity of use (days of use) of state psychiatric hospitals but not the probability of use.

The results of this exploratory study are promising, but this study is only a first step toward assessing the psychometric properties of the TMACT. The results presented here are encouraging in that higher scores on the TMACT were associated with reductions in state hospital utilization, and these findings mirror results from previous studies of the DACTS. However, TMACT scores were not related to state or local hospital admissions. A likely explanation is that TMACT fidelity scores varied little among teams. That is, all teams appeared to begin with high fidelity to the ACT model and to recovery-oriented principles at baseline, which was most likely attributable to the intensive training each team received on these issues. Also, there was little variation in TMACT scores over time. With two exceptions, fidelity scores for teams started high and remained high. Thus, with little variation in TMACT scores between and within teams, it is not unexpected that we found few relationships between TMACT scores and consumer outcomes.

Further research is needed to establish the reliability and validity of the TMACT, and this study is only part of a broader research agenda focused on the development and fidelity measurement of recovery-oriented ACT and assessment of outcomes. Here, we capitalized on an opportunity provided by the creation of ten recovery-oriented ACT teams in Washington State and the availability of administrative data to answer questions about the relationships between TMACT scores and a number of important outcomes. Studies that compare outcomes among teams that score high on the TMACT and teams that score low are needed to further the evidence for the TMACT as a viable fidelity measure of recovery-oriented ACT.

Nevertheless, this study adds to the growing literature on recovery-oriented ACT. More information is needed about recovery-oriented ACT, and a number of fundamental questions have yet to be answered. What are the critical ingredients of recovery-oriented ACT? What are the outcomes of recovery-oriented ACT? How is fidelity to recovery-oriented ACT measured? What is the relationship between fidelity to recovery-oriented ACT and outcomes for persons with severe mental illnesses? These questions must be answered to establish recovery-oriented ACT as an evidence-based adaptation of ACT, and the development and testing of a reliable and valid fidelity measure is critical to answering these questions.

The strengths of this study include the large sample of consumers who received recovery-oriented ACT, the breadth of outcomes available in the statewide administrative data, and the fact that this is the first study of its kind to assess the relationship between fidelity to recovery-oriented ACT and consumer outcomes. The limitations of the study include the lack of personal recovery outcomes, the use of administrative data, and unknown generalizability of results from one state in the Pacific Northwest. Person-centered outcomes such as personal recovery, self-direction, and independence that were not recorded in the administrative data used for this study need to be examined in future assessments of the TMACT.

Conclusions

The TMACT predicted a range of consumer service use outcomes for recovery-oriented ACT teams. Future research should examine teams with a wider range of scores and explore the extent to which fidelity as measured by the TMACT also predicts personal recovery and other person-centered consumer outcomes.