Fidelity to Assertive Community Treatment and Client Outcomes in the New Hampshire Dual Disorders Study

Assertive community treatment is the most widely tested model of community care for persons with severe mental illness (1,2). Yet until recently, the model has not been defined operationally in a manual, and its critical components have not been identified and evaluated. The recent appearance of two manuals on assertive community treatment (3,4) and of studies of its critical components (5), coupled with the development of a scale to evaluate critical components (6), have fundamentally changed the potential for research on assertive community treatment.

Together these research tools should permit model-guided implementation, comprehensive and consistent process analysis, and theory-driven outcome studies. For clinicians and program directors, clear guidelines and measurement tools make it easier to shape program performance in formative stages. For researchers, a fidelity measure enables not only studies of process but also studies of the relationships between model fidelity and outcomes and between program components and outcomes (7).

In a number of areas, studies have shown that program fidelity is associated with participant outcomes. For example, Blakely and associates (8) studied criminal justice and education projects and found that high-fidelity programs produced better outcomes in a variety of areas than low-fidelity programs. Similarly, McDonnell and colleagues (9) found that standardization of procedures in supported employment programs was associated with better employment outcomes.

More specific to assertive community treatment, McGrew and coworkers (5) found that programs with higher fidelity, defined in terms of identified critical ingredients, were more effective in reducing hospital use. Finally, in treating persons with dual disorders, Jerrell and Ridgely (10) reported that when either behavioral skills training or intensive case management programs included their respective core elements, participants had significantly higher psychosocial functioning and lower costs for services, compared with participants in programs that did not include all the core elements.

Despite findings that the quality of implementation is related to outcomes, debate continues about implementation of model programs. At the theoretical level, the debate involves arguments between supporters of the classic research, development, and diffusion model and those who advocate local adaptation (8). For example, within the community mental health field, assertive community treatment advocates have argued strongly for faithful implementation (3), while others have argued that model programs cannot be transferred to other sites without local adaptation (11,12).

Essential to resolving this debate is the reliable and valid measurement of program implementation and fidelity to the model. Measurement begins with a thorough listing of a program's critical components and their operational definitions, accompanied by suggested sources of data from which to rate a program's performance on each component. Teague and colleagues (6) have described the development of an assertive community treatment fidelity scale, which profiles a program across numerous critical components and thereby enables comparison with other programs, with the same program at other points in time, and with a priori fidelity criteria. Thus classifying an assertive community treatment program as having high fidelity or low fidelity can be either relative or absolute, depending on the goals and design of a given study.

The purpose of this study was to examine the relationship between fidelity of assertive community treatment programs and outcomes of participants in a three-year study of treatment for persons with dual disorders of severe mental illness and substance abuse or dependence. One key issue was how to determine whether a study participant was exposed to a high-fidelity or low-fidelity assertive community treatment program. In general, data from multiple sources can be used to classify programs or to quantify the involvement of each participant (13). For the study reported here, we classified seven assertive community treatment programs as either high or low fidelity, based on extensive quantitative and qualitative information gathered at the program and client levels and analyzed for several years before the analysis of outcomes (14).

In addition, we included only participants who had the opportunity for exposure to the programs over the three-year course of the study. That is, analysis of client outcomes relative to program fidelity used only participants who remained engaged in treatment, even if minimally, for at least one year. A previous report of data from this three-year study focused on the outcomes of the randomized clinical trials and used an intent-to-treat approach (15), which included patients who were assigned to the assertive community treatment programs but who were not actually exposed to this treatment because of attrition, refusal, or prolonged institutionalization. The approach used in the current report thus shifts the focus from effectiveness toward efficacy.

Methods

Overview of the main study

The New Hampshire dual disorders study was a three-year randomized clinical trial of the effectiveness of assertive community treatment versus standard case management for persons with both severe mental illness and a substance use disorder. Seven community mental health centers (CMHCs) throughout New Hampshire provided both programs, and 223 persons with dual disorders were assigned randomly, within CMHCs, to receive treatment.

Diagnostic interviews were conducted before randomization, and research interviews were administered at intake and every six months for three years (1989 to 1995). The interviews included assessment of psychiatric symptoms, quality of life, alcohol and drug use, residential history, involvement in the legal system, medical status, and service utilization. Additional data were collected from case managers, CMHC management information systems, Medicaid records, hospital and legal system records, and participants' families. The methods and outcomes of the clinical trial have been reported elsewhere in fuller detail (15,16).

Participants

Of the 240 eligible referrals to the New Hampshire dual disorders study, 223 completed intake assessments and entered the clinical trial. During the three years of the study, 20 of the original 223 participants (9 percent) were lost from the study due to refusal to participate, relocation without contact, or death. Of the 203 participants who completed the study, 30 had less than one year of exposure to treatment for a variety of reasons, including relocation, long-term institutionalization, transfer to another program, or treatment refusal. Thus 173 participants constituted the group exposed to treatment. Of these participants, 87 were exposed to assertive community treatment programs, and they form the study group for these analyses.

The study group contained 68 men (78 percent) and 85 Caucasians (98 percent). The mean± SD age of the group members was 35±8.1 years. Sixty participants were diagnosed as having schizophrenia or schizoaffective disorder (69 percent); the remainder had bipolar disorder. All participants had one or more substance use disorders; sixty-two (71 percent) had an alcohol use disorder, and 38 (44 percent) had a drug use disorder, most commonly involving cannabis or cocaine.

Outcome measures

The client interview included the Time-Line Follow-Back (17) to assess days of alcohol and drug use during the previous six months and a detailed chronological assessment of residential history, using a self-report follow-back calendar, which was supplemented by outpatient and hospital records (18). The interview also assessed objective and subjective quality of life using Lehman's Quality of Life Interview (QOLI) (19), current psychiatric symptoms using the 24-item Brief Psychiatric Rating Scale (BPRS) (20), and overall functional status using the Global Assessment Scale (GAS) (21).

Self-reports of substance use from persons with severe mental illness are suspect for a variety of reasons (22). Therefore, to obtain valid ratings of substance use, a team of researchers evaluated all available data for each participant— from client self-reports, clinician ratings, and urine drug screens. They used this information to establish consensus ratings on the Alcohol Use Scale (AUS), Drug Use Scale (DUS), and Substance Abuse Treatment Scale (SATS) at intake and every six months for three years (23).

The AUS and DUS are 5-point scales based on DSM-III-R criteria for rating substance abuse status during the previous six months. A rating of 1 indicates abstinence; 2, use without impairment; 3, abuse; 4, dependence; and 5, severe dependence. The SATS is an 8-point scale based on a motivational model of substance abuse recovery in a series of stages. A rating of 1-2 indicates the engagement stage; 3-4, persuasion; 5-6, active treatment; and 7-8, relapse prevention and recovering.

Assessment of model fidelity

Assertive community treatment programs for persons with dual disorders were implemented in seven CMHCs in New Hampshire through a grant from the Robert Wood Johnson Foundation in 1988 (24). Implementation criteria for the programs included nine essential components of assertive community treatment and four additional components that focused on dual disorders. The components are listed in Table 1. Ratings on these program components were made throughout the study period by research staff, using information drawn from interviews with clinical and administrative staff, activity logs kept by case managers, clinical records, and direct observation. The details of the fidelity scale and the rating process used in the New Hampshire dual disorders study are presented elsewhere (14).

Using factor analysis, ratings on the 13 components were found to coalesce into two higher-order factors, one capturing program components related to structure and community treatment (staff continuity, multidisciplinary staff, community locus, assertive engagement, continuous responsibility, dual disorders treatment groups, and dual disorders model) and one capturing program components related to the team mandate and the integration of services (team approach, small caseload, high intensity, collaboration with support system, individualized substance abuse treatment, and focus on dual disorders). Each factor contains general assertive community treatment components and specific dual disorders components.

Little variation was found among the ratings of the seven assertive community treatment programs on the second factor, but substantial variation on the first factor was noted. Thus, based on the composite scores on the first factor, four programs form the high-fidelity group; their scores on the 5-point scale were 4.9, 4.1, 4, and 3.9. Three programs form the low-fidelity group; their scores were 2.9, 2.8, and 2.2.

Data analysis

In the study group of 87 participants, 61 were exposed to high-fidelity programs, and 26 were exposed to low-fidelity programs. Thus the group sizes are relatively small, and the analyses lack statistical power, especially when they involve a subset of the study group (for example, drug users).

In response to this limitation, we report effect sizes, so that the difference between the high-fidelity and low-fidelity groups can be judged apart from statistical significance. As used here, effect sizes depict the mean difference between groups, in standard deviation units, and thus they express the magnitude of the group effect. The effect size has a positive sign when the group difference favors the high-fidelity assertive community treatment group and a negative sign when it favors the low-fidelity group. By convention, group-difference effect sizes of .20, .50, and .80 represent small, medium, and large effects, respectively (25,26).

We examined group equivalence at baseline, using t tests and chi square tests on a set of demographic, illness-related, and outcome variables. Differences in outcome between fidelity groups were based primarily on data gathered in the third year of the study. Thus the outcome variables were often formed by averaging scores, or cumulating totals, from the 30-month and the 36-month assessment points. Analysis of covariance was used to compare group means in the third year, which were adjusted for baseline levels.

Results

Attrition and baseline equivalence

At the beginning of the study, the low-fidelity programs had 37 participants and the high-fidelity programs had 72 participants. Due to study attrition and to lack of exposure to treatment, both groups lost 11 participants, reducing the final group sizes to 26 in the low-fidelity programs and 61 in the high-fidelity programs (30 percent attrition for low-fidelity programs and 15 percent for high-fidelity programs; χ²=3.17, df=1, p=.08).

We examined 21 baseline variables for differences between the high-fidelity and low-fidelity groups; data for participants lost to attrition were not included. Only one difference was significant (p<.05). Participants in the high-fidelity programs began the study at a somewhat higher mean stage of substance abuse treatment, although both groups were between the engagement and early persuasion stages. The absence of differences indicates that the two fidelity groups were equivalent at the beginning of the study.

Outcome differences

Table 2 presents the results of tests for differences between the two fidelity groups during the third year of the study. Primary outcomes were from the domains of substance use and housing, which includes hospitalization. Substance use and housing are the outcomes that were targeted most directly by the assertive community treatment programs and for which conventional criteria were used to judge statistical significance (p<.05). Secondary outcomes were from the domains of psychiatric symptoms, functional status, and quality of life, and they were judged as statistically significant after the analysis adjusted for multiple tests. These important outcomes would be expected to differ between assertive community treatment programs and standard case management. However, few differences in these outcomes were expected to be found between the high-fidelity and low-fidelity dual disorders assertive community treatment programs, because the outcome variables were not the explicit focus of treatment.

Participants from high-fidelity programs reported significantly fewer days of alcohol and drug use than those in low-fidelity programs. These differences are complemented by differences in scores on the AUS and DUS; on average the high-fidelity group was in the remission range on both scales, with scores less than 3, whereas the low-fidelity group was in the abuse range, with scores of 3 or higher. In addition, the mean Substance Abuse Treatment Scale score for the high-fidelity group was in the active treatment stage, whereas the low-fidelity group was closer to the persuasion stage. The effect sizes indicate that substantial differences existed between groups during the third year of the study for each outcome variable related to substance use, although small group sizes precluded highly significant statistical results for some variables.

We also examined remission from alcohol and drug use as an important and clinically meaningful outcome. An individual was considered in remission during the third year of the study if both the 30- and 36-month AUS scores were less than 3 and both DUS scores were less than 3. The remission rate in the high-fidelity programs during the third year was 43.3 percent, whereas it was 15.4 percent in the low-fidelity programs (χ²=6.24, df=1, p=.01).

As a final step, we defined a participant as being in stable remission if the AUS and DUS scores were less than 3 at a given assessment point and at all subsequent points. Figure 1 shows the longitudinal rates of stable remission for the two fidelity groups. The Fisher's exact test for differences in proportions approached significance at the six-month follow-up (p=.10) and was significant (p<.05) at each subsequent assessment point.

The findings were mixed in the housing domain. To reduce the number of zeros and enable the use of parametric statistical tests, the hospital data were cumulated over the entire three-year period. The mean number of hospital admissions during the three years of the study was significantly lower for patients in the high-fidelity programs, although no difference between the two groups was found in the mean number of days spent in the hospital. Regarding residential status more generally, the two groups did not differ in the number of residential moves or in community tenure in the third year.

No substantial differences between clients in the high-fidelity and low-fidelity programs were found for the secondary outcomes. The groups did not differ on general life satisfaction or on scales from the Quality of Life Interview measuring social and family contact and satisfaction with social and family relations. The two groups also did not differ in their satisfaction with substance abuse treatment, although a marginal difference was noted in their satisfaction with medical care overall— that is, physical health, emotional health, and substance use combined. Both groups showed improvement over time in symptom severity as measured by the BPRS total score and functional status as rated on the GAS, but no significant differences between groups were found at the endpoint of the study.

Discussion

The results of this study indicate that the fidelity of the implementation of the dual disorders assertive community treatment model is related to client outcomes. However, the outcome differences noted in this study were confined primarily to those domains that were most proximal to the treatment provided. That is, clients in high-fidelity programs had better substance use outcomes than those in low-fidelity programs. In addition, consistent with the general assertive community treatment model, high-fidelity programs had higher client retention rates, and they also reduced hospital admissions during the three-year study period, although no differences on other residential measures were found.

Other outcomes, more distal from dual disorders treatment, such as psychiatric symptoms, functional status, and quality of life, were not associated systematically with fidelity to the assertive community treatment model. Clients in both fidelity groups improved over time on many of these outcomes.

These gains may have resulted from nonspecific factors associated with engagement with the treatment system and with participation in assertive community treatment programs. As is typical of persons with dual disorders, numerous participants at the time of study had not been recently connected with the mental health treatment system, or that connection had been ineffective. Reaching out to them and providing competent and broad-based clinical services over a three-year period enabled gains in many areas and in diffuse ways. Some of these gains at the group level were probably due to regression to the mean, and some were likely due to engagement with the assertive community treatment programs.

The differential gains found for participants in this study emerged despite the relatively minor differences between high-fidelity and low-fidelity assertive community treatment programs. The implementation ratings that were used to classify the seven assertive community treatment programs revealed as much similarity as dissimilarity among them. All seven programs faithfully implemented many components of the assertive community treatment model, and it was only on one cluster of program components that they differed. Thus labels like "high fidelity" and "low fidelity" may lead to a distorted view of what actually happened in the programs and must be understood as encoding only relative differences.

Nevertheless, the cluster of implementation criteria on which the two groups differed contained components related to the adoption of the dual diagnosis model and the provision of dual disorders treatment groups, which may explain why outcomes in the substance use domain were affected most prominently. In addition, this cluster contained items related to the extent of community outreach and assertive engagement, which are factors that may be associated with the difference in rate of hospitalization and retention in treatment. It is uncertain whether the correspondence between the differences in program components and the differences in client outcomes is as tight as suggested, but the plausibility of this strong relationship strengthens the case for attention to model fidelity and specific program components.

The results of this study must be interpreted in light of its limitations. Participants were not assigned randomly to high-fidelity and low-fidelity assertive community treatment programs; this distinction among the programs arose during the study period. Although self-selection is not a threat to validity because the programs were dispersed geographically, it is still possible that important unobserved differences existed among clients at the seven sites. Our analysis of baseline characteristics indicated equivalence of the two groups, but this study is a quasi-experimental evaluation and must be interpreted as a correlational study.

In addition, the external validity of the study is constrained by its sample size and constitution. The study group was small, and it was composed primarily of white males with schizophrenia-spectrum disorders and alcohol abuse. Similar research from diverse settings is needed to understand the influence of study group characteristics on program fidelity effects.

Another limitation is the small number of assertive community treatment programs. Not enough programs were included to design the study as a multilevel evaluation that used hierarchical statistical models to examine variation at the client level and at the program level. To the extent that dependency exists within the data due to the clustering of participants within single programs, unknown bias may be present in our results.

An additional consequence of the small number of programs is an alternative explanation of the findings, which is related to the small number of case managers involved. If more competent clinicians are better able to grasp the importance of model fidelity and to embrace the assertive community treatment philosophy, then their programs would score more highly on fidelity measures and their clients may have better outcomes. Consequently, it is difficult with a small number of programs to distinguish between the effectiveness of individual clinicians and the effectiveness of the assertive community treatment model per se.

Overall, these findings speak loudly of an association between program fidelity and participant outcomes, but they are silent about causal mechanisms. As noted, differences between participants and clinicians across sites are possible explanations, but so are the many ways in which faithfully implemented assertive community treatment programs differ from less competent, or locally modified, versions of the assertive community treatment model.

Despite its shortcomings, this study provides evidence that more complete and more faithful implementation of the model components of assertive community treatment is associated with better client outcomes. The importance of this finding cannot be overstated in this era of widespread proliferation of assertive community treatment programs. As variations of the model arise in practice due to philosophical differences, treatment focus, or local conditions, the link between model fidelity and client outcomes must not get lost. That is, to the extent that program effectiveness depends on fidelity to a particular model, modifications of program components may jeopardize positive client outcomes.

Each of the low-fidelity programs in this study justified their deviations from the assertive community treatment model with arguments about necessary modifications to fit local circumstances. That the assertive community treatment programs at these sites were associated with poorer outcomes leads one to question the wisdom of their decisions. Until systematic research identifies the critical components of the assertive community treatment model, there is risk in disassembling the model and reconfiguring its components unless careful attention is paid to both process and outcome evaluation.

Acknowledgments

This research was supported by grants MH-00839, MH-46072, and MH-47567 from the National Institute of Mental Health and by grant AA-08341 from the National Institute on Alcohol Abuse and Alcoholism. The authors thank Robin Clark, Ph.D., Tim Ackerson, M.S.S.W., Keith Miles, M.P.A., Barbara Helmstetter, B.A., Lindy Fox, M.A., and Joan Packard, B.A, for their work on this project.

Dr. McHugo and Dr. Xie are research assistant professors of community and family medicine and Dr. Drake is professor of psychiatry at Dartmouth Medical School in Hanover, New Hampshire. Dr. Teague is associate professor of community mental health at Louis de la Parte Florida Mental Health Institute at the University of South Florida in Tampa. Send correspondence to Dr. McHugo at the New Hampshire-Dartmouth Psychiatric Research Center, 2 Whipple Place, Suite 202, Lebanon, New Hampshire 03766 (e-mail, [email protected]).

Enlarge table

Enlarge table