Effectiveness and Outcomes of Assisted Outpatient Treatment in New York State

Forty-two U.S. states have laws mandating outpatient commitment, a form of involuntary civil commitment for people with mental illness. It is critical to understand the consequences of outpatient commitment, both because it is so widespread and because the potential harms and benefits are so significant. These include the well-being and safety of people with serious mental illness and those around them, as well as restrictions on personal freedom.

Yet it is difficult to obtain valid data on the consequences of outpatient commitment. Researchers must work within existing systems of treatment and criminal justice. Ideal experimental designs are usually precluded by ethical problems presented by randomly assigning an individual with a high risk of violence to a comparison group. Consequently, most studies of outpatient commitment have been naturalistic or quasi-experimental and subject to bias from selection and confounding.

Two randomized controlled trials of outpatient commitment have been conducted: the New York City Involuntary Outpatient Commitment Pilot Program ( 1 ) and the Duke Mental Health Study ( 2 ). As noted by Hiday ( 3 ) and Swartz and Swanson ( 4 ), the first of these ( 1 ) suffered from low statistical power and uneven enforcement of outpatient commitment during the study's startup phase. The other study, the Duke Mental Health Study ( 2 ), was noted by Hiday ( 3 ) to be the methodologically strongest study to date. That study randomly assigned 331 individuals court ordered to outpatient commitment to the experimental group that remained under these orders or to the comparison group that was released from these orders. Both groups received enhanced services. Participants with a recent history of serious violence (N=67), who for ethical reasons could not be randomly assigned to treatment, were given the same treatment protocol as the experimental group and were followed as a separate group. Many of the significant associations reported from the study were based on duration of mandatory treatment or frequency of service use for all participants combined (experimental group, comparison group, and non-randomly assigned group). Results from these analyses do not confer the causal-inference benefits of an experimental design.

In 1999 New York State established Section 9.60 of the Mental Hygiene Law (Kendra's Law), under which court-ordered outpatient commitment, termed assisted outpatient treatment, can be mandated for certain individuals with mental illness and a history of multiple hospitalizations or violence toward self or others. The New York State Office of Mental Health funded researchers at Columbia University and the New York State Psychiatric Institute to evaluate the effect of Kendra's Law. The Community Outcomes of Assisted Outpatient Treatment study is a longitudinal study of 76 individuals assigned to assisted outpatient treatment in Queens and the Bronx and a comparison group of 108 outpatients who were not assigned to such treatment.

The Community Outcomes of Assisted Outpatient Treatment study is quasi-experimental. Fortunately, the statistical literature demonstrates that under certain conditions causal effects can be identified by controlling for important characteristics of the two groups related to both the likelihood of treatment assignment and the outcomes ( 5 ). We used such a propensity score analysis to achieve the strongest causal inference possible without a randomized experimental design. Given the relatively small number of previous studies and their methodological strengths and weaknesses, the results of the Community Outcomes of Assisted Outpatient Treatment study should be considered an important part of the evidence landscape concerning outpatient commitment.

We focused on two sets of outcomes underlying important arguments for and against outpatient commitment. Supporters of outpatient commitment argue that lack of insight into mental illness necessitates coercion and that outpatient commitment improves psychiatric outcomes, social functioning, and quality of life, thereby reducing problems such as homelessness, incarceration, violence, and suicide ( 6 , 7 ). To assess such intended, positive consequences, we compared an outpatient commitment group and a comparison group in terms of psychotic symptoms, violent behavior, suicide risk, illness-related social functioning, and quality of life.

In opposition to outpatient commitment, the National Mental Health Association has argued that "service can only be effective when the consumer embraces it" and that coercive treatment can have negative unintended consequences ( 8 ). To assess such consequences, we compared the two groups in terms of perceived coercion and stigma. We measured stigma in terms of feeling different from others, because we expected this aspect of stigma to be particularly affected by the formal labeling entailed in outpatient commitment.

Methods

Sample and procedure

We ascertained 184 participants aged 18 to 65 in treatment facilities in the Bronx and Queens. Seventy-six had been court ordered to assisted outpatient treatment; 108 (outpatient comparison group) had been recently discharged from a psychiatric hospital and were attending the same outpatient facilities as the assisted outpatient treatment group. Participants in the assisted outpatient treatment group had been under court order for at least a month before being interviewed for our study. All participants were screened by a psychiatrist or doctoral-level psychologist, unaffiliated with the study, for capacity to provide informed consent. After a complete description of the study was given to participants, written informed consent was obtained. Institutional review board approval was obtained from the New York State Psychiatric Institute, Bronx Psychiatric Center, Creedmoor Psychiatric Center, Bronx-Lebanon Medical Center, and New York State Office of Mental Health. Interviews were conducted by doctoral- and master's-level psychologists, licensed clinical social workers, and doctoral-level social scientists. Respondents were compensated after each interview ($50 for baseline, $25 for follow-ups, and a $50 bonus for completing all interviews).

Interview data were augmented with clinical charts and information from the New York State Office of Mental Health to verify assignment to assisted outpatient treatment.

Chart diagnoses were as follows: 64 persons (35%) had schizophrenia (assisted outpatient treatment group, 28, 37%; comparison group, 36, 33%), 57 (31%) had schizoaffective disorder (assisted outpatient treatment group, 30, 39%; comparison group 27, 25%), 12 (7%) had major depressive disorder (assisted outpatient treatment group, one, 1%; comparison group, 11, 10%), 34 (18%) had bipolar disorder (assisted outpatient treatment group, 11, 14%; comparison group, 23, 21%), and 17 (9%) had other psychiatric disorders (assisted outpatient treatment group, six, 8%; comparison group, 11, 10%). The mean±SD age was 37.0±11.0 years (35.8±10.2 years for the assisted outpatient treatment group versus 37.8±11.5 years for the comparison group). Most were male (N=109, 59%) (assisted outpatient treatment group, 51, 67%; comparison group, 58, 54%). Most were from racial or ethnic minority groups: 90 (49%) were black (assisted outpatient treatment group, 41, 54%; comparison group, 49, 45%) and 70 (38%) were Hispanic (assisted outpatient treatment group, 27, 36%; comparison group, 43, 40%), 14 (8%) were white (assisted outpatient treatment group, six, 8%; comparison group, eight, 7%), and ten (5%) were from other racial or ethnic groups (assisted outpatient treatment group, two, 3%; comparison group, eight, 7%). The majority of participants had never been married (N=130, 71%) (assisted outpatient treatment group, 59, 78%; comparison group, 71, 66%) and had completed high school (N=111, 60%) (assisted outpatient treatment group, 43, 57%; comparison group, 68, 63%).

We compared the assisted outpatient treatment sample in our study (not including the comparison group) to the population of individuals assigned to assisted outpatient treatment in the Bronx and Queens, using Office of Mental Health data. The study sample overrepresented blacks compared with the state-collected data for the Bronx (Community Outcomes of Assisted Outpatient Treatment data, N=21 of 41, 51%; Office of Mental Health data, 49%) and for Queens (Community Outcomes of Assisted Outpatient Treatment data; N=20 of 35, 57%; Office of Mental Health data, 38%). Whites were underrepresented in our study sample in both boroughs. This discrepancy probably reflects the nature of the clinics from which we recruited. With the exception of race, the study sample was very similar to that of the assisted outpatient treatment population. Age, sex, marital status, employment status, income source, and educational attainment corresponded quite closely to the population of persons assigned to assisted outpatient treatment in the Bronx and Queens. All of these variables were weighted via propensity scores and controlled for in the main analysis.

Interviews were conducted between January 2003 and January 2006. Interviews were attempted at baseline and at three, six, nine, and 12 months. Baseline and 12-month interviews averaged two to three hours, whereas the other interviews averaged 30 to 60 minutes. Follow-up rates were 84% (N=155), 66% (N=121), 62% (N=114), and 72% (N=133), respectively, for three, six, nine, and 12 months. We used baseline data to compare individuals we successfully interviewed at 12 months (N=133) with those we did not (N=51). Persons lost to follow-up were younger (p<.006), more likely to be male (p<.006), and more likely to have never married (p<.02), compared with those interviewed at 12 months. Race, diagnosis, educational level, and length of unemployment in the previous year were unrelated to successful follow-up.

Independent variable: assisted outpatient treatment. Legal criteria for assignment to assisted outpatient treatment include a judgment—based on a history of treatment noncompliance that has resulted in hospitalization, incarceration, or violent acts or threats—that the individual is unlikely to voluntarily adhere to treatment and islikely to benefit from mandated treatment. Individuals appear in court to receive assisted outpatient treatment assignment and are ordered by the court to participate in outpatient treatment. Once assigned to assisted outpatient treatment, individuals who are judged by a physician to be noncompliant with treatment as ordered, who do not respond toefforts to engage them in treatment, and who are judged by a physician to be in possible need of civil commitment can be removed from the community and retained in ahospital for up to 72 hours to determine whether they need inpatient care and treatment. In addition, individuals receiving assisted outpatient treatment have priority access to certain services, including intensive case management, housing, and outpatient clinical services. Thirty-nine percent (N=42) of individuals in the comparison group did not have a case manager.

Dependent variables. We used six multiple-item scales and one binary indicator of serious violence. Scale scores represent the mean of the component items included in each scale.

Psychotic symptoms in the previous three months were assessed with the Structured Clinical Interview for DSM ( 9 ). The five-item scale ( α =.62) assesses delusions of reference, persecution, control, grandiosity, and thought broadcasting.

Suicide risk was assessed with a six-item scale ( α =.83) that examines recent self-harm, depression, serious consideration of suicide, suicide plans, suicide attempts, and injuries from attempts requiring treatment ( 10 ). Possible scores range from 0, low risk, to 6, high risk. The baseline measure assessed suicide risk in the previous 12 months; follow-ups assessed the previous three months.

Serious violent behavior was measured with the MacArthur Community Violence Interview ( 11 ). A score of 1 was given if the participant reported having, in the previous three months, kicked, beaten, or choked anyone; hit anyone with a fist or beaten up anyone; tried to physically force anyone to have sex against his or her will; threatened anyone with a knife, gun, or other weapon; or fired a gun at someone or used a knife or a weapon on him or her. Otherwise, the score was 0.

Illness-related social functioning was measured with a seven-item scale ( α =.80) assessing recent difficulty in completing tasks, such as "make routine decisions," "control the symptoms of your illness," and "deal with day to day stresses" ( 12 ). Possible scores on the items range from 0, very difficult, to 1, somewhat difficult, to 2, not difficult. The baseline measure assessed the previous 12 months; follow-ups assessed the previous three months.

Quality of life was measured with a 14-item scale ( α =.91) that was similar to Lehman's ( 13 ) more comprehensive scale. Respondents rate how they currently feel (today) about domains such as living conditions, employment, social life, physical health, level of independence, and self-esteem. Possible ratings include 1, poor; 2, fair; 3, good; and 4, excellent.

Perceived coercion was measured with a five-item scale ( 14 , 15 ) ( α =.86) assessing whether respondents recently felt free to choose outpatient treatment—for example, "It was your idea to get mental health treatment." Possible scores include 0, strongly agree; 1, agree; 2, disagree; and 3, strongly disagree. The baseline measure assesses the previous 12 months; follow-ups assess the previous three months.

Stigma (feeling different) was assessed with a three-item scale ( α =.75) assessing feelings of being set apart or different from other people because of a psychiatric hospitalization—for example, "There is something about the experience of being in a mental hospital that sets you apart from others." Possible scores include 3, strongly agree; 2, agree; 1, disagree; and 0, strongly disagree. Time frame is not specified and is assumed to reflect current states.

Analysis

Propensity score matching. Because we cannot compare the outcomes of individuals assigned to assisted outpatient treatment to what their outcomes would have been without being assigned to such treatment, we compared two groups (one assigned to assisted outpatient treatment and a comparison group that was not) that were as similar as possible on pretreatment characteristics that predict both the probability of assignment to assisted outpatient treatment and subsequent outcomes. Inverse probability of treatment weighting using propensity scores reweights the comparison group so that, with respect to the values of the confounding covariates, it mirrors the assisted outpatient treatment group ( 16 ). Assuming the propensity score model is correct, properly adjusting for the propensity score is sufficient to adjust for all the covariates used to create it ( 17 ).

Each participant's propensity score is the propensity that he or she was assigned to assisted outpatient treatment, conditional on the confounding covariates measured before treatment assignment. Thus the propensity score approach we employed sought to balance groups on factors occurring before assignment and is not intended to and therefore does not adjust for factors that follow from or are a consequence of assignment. We estimated propensity scores using probit regression and used the estimated scores to implement inverse-probability-of-treatment weighted models. Individuals receiving assisted outpatient treatment were given a weight of 1; individuals not receiving assisted outpatient treatment were given a weight equal to (propensity score)/(1-propensity score). The final model was chosen on the basis of its effectiveness at yielding weighted groups whose means were most similar. A combination of propensity score weighting and regression is generally more effective at reducing bias than either method alone ( 5 ). Therefore, we ran a series of weighted regressions in which the outcome variable was regressed on the confounding covariates, interview time point, and the assisted outpatient treatment indicator, with the weights described above incorporated as probability weights.

Main analysis. We employed generalized estimating equations (GEE), which do not restrict analysis to cases with complete data at every time point and which address the problem of correlated error between nonindependent observations in panel data. Because suicide risk, serious violent behavior, illness-related social functioning, and coercion were assessed for the 12 months before the baseline interview, a period that begins before assisted outpatient treatment assignment, we included these variables only for the three-, six-, nine-, and 12-month follow-ups. Because measures of psychotic symptoms, quality of life, and stigma assessed circumstances in the immediate or recent past, we used all time points, including baseline, in analysis of these variables. Consequently, the number of cases and longitudinal assessments varies across the dependent variables.

Using GEE, we conducted logistic regression for serious violent behavior and regression analyses for the other outcomes assessing the effect of assisted outpatient treatment assignment on each of the outcome variables using propensity score weighting. The following covariates were used to construct the propensity score weights. They were also included in the regression equations as an adjustment for any residual differences in balancing: gender, race or ethnicity (black, Hispanic, white, or other), age, marital status, education, psychiatric diagnosis, interview time point (baseline and three-, six-, nine-, and 12-month follow-up), natural log of number of involuntary commitments to psychiatric inpatient treatment, self-reported history of arrest, official history of arrest for violent crime and for nonviolent crime, self-reported antisocial behavior before the age of 15, likelihood of treatment noncompliance as judged by the interviewer, chart-indicated dangerousness, and history of causing bodily harm. Table 1 shows the unweighted and weighted means for the comparison group and the means for the assisted outpatient treatment group for selected propensity score weighting variables. We report the effects of assisted outpatient treatment assignment with all covariates controlled. Additionally, because all patients in assisted outpatient treatment but not all patients in the comparison group were assigned a case manager, we conducted supplementary analyses that controlled for availability of a case manager.

Enlarge table

Results

As Table 2 shows, most of the intended consequences of assisted outpatient treatment were achieved at least to some degree in our sample. Although the assisted outpatient treatment group and the comparison group did not differ in the extent to which they experienced psychotic symptoms, the assisted outpatient treatment group was less likely to perpetrate serious violent behavior (p<.05), had a lower risk of suicide (p<.05), and had better illness-related social functioning (p<.05). The odds of perpetrating serious violence during the 12-month follow-up were over four times as great in the comparison group as in the assisted outpatient treatment group (odds ratio=4.3, 95% confidence interval=1.15–16.17). Predicted mean (created by inserting the sample means for all covariates in the GEE analyses) suicide risk for the assisted outpatient treatment group was .134, compared with .314 for the comparison group (possible scores range from 0 to 6, with higher scores indicating higher risk). This effect size is approximately one-fourth of a standard deviation. Predicted mean illness-related social functioning for the assisted outpatient treatment group was 1.76, compared with 1.60 for the comparison group (possible scores range from 0 to 2, with higher scores indicating better functioning). This effect size is over two-fifths of a standard deviation. Quality of life was higher in the assisted outpatient treatment group but not significantly so. Neither of the potential unintended consequences was evident: the assisted outpatient treatment group actually reported marginally less (p<.10) stigma and coercion in the follow-up period than did the comparison group.

Enlarge table

For the most part, covariates were related to the outcome variables as one might expect, thereby giving us some confidence that our measures worked as expected in our sample. For example, female gender was related to suicide ideation (p<.01), antisocial behavior before age 15 was related to serious violent behavior (p<.01), and a history of involuntary commitment was associated with perceived coercion (p<.01). Tables showing all associations between covariates and outcome variables can be obtained from the first author.

As described above, the propensity score approach is not intended to and therefore does not control for factors that occur after or as a consequence of treatment assignment. For this reason, we controlled for the availability of a case manager in supplementary analyses and found that in some instances the significant or marginally significant effect sizes for assisted outpatient treatment became slightly stronger or slightly weaker; presence of a case manager was not itself significantly associated with any outcome variable. We also repeated our analyses but excluded participants who did not have a case manager. Again some effect sizes changed slightly, but the pattern remained essentially the same.

Discussion

Like all forms of involuntary psychiatric treatment, outpatient commitment is a high-stakes issue, with questions of personal freedom, well-being, and safety looming large ( 18 ). Therefore, it is essential to strive to obtain high-quality, valid data on the consequences of outpatient commitment. This is difficult because research necessarily takes place within existing systems of psychiatric care and justice and because of ethical issues involved in randomly assigning individuals with a high risk of violence to a comparison group. Working within these constraints, we employed a quasi-experimental design combined with state-of-the-art analytic techniques that allow the strongest possible conclusions from our nonexperimental data.

We assessed the impact of a large outpatient commitment initiative affecting the entire state of New York. Our results are certainly not the final word on outpatient commitment. Other studies, particularly the Duke Mental Health Study ( 2 ), have also used strong designs and analytic techniques. Moreover, every context in which outpatient commitment is enacted differs, for example, in terms of population and specific laws and procedures followed. Therefore, data from multiple operationalizations of outpatient commitment must be taken together in forming an overall assessment of its consequences. Our results will be useful in this larger assessment.

We evaluated New York's assisted outpatient treatment program by comparing a sample of individuals assigned to assisted outpatient treatment with a group receiving outpatient treatment after hospitalization. We compared the groups in terms of several desired consequences of assisted outpatient treatment (fewer psychotic symptoms, lower suicide risk, and less serious violent behavior and better illness-related social functioning and quality of life) and two possible unintended negative consequences (perceived coercion and stigma) that have been prominent in the debate over outpatient commitment. Most of the desired outcomes were achieved to some extent—suicide risk and violent behavior were significantly lower in the assisted outpatient treatment group, and illness-related social functioning was significantly higher. Neither of the negative consequences was observed; in fact, individuals assigned to assisted outpatient treatment were marginally less likely to feel different from others and to feel coerced in the follow-up period.

Our results were based on self-report, so they are limited by biases that can affect self-report data. A major goal was to assess how people felt about assignment to assisted outpatient treatment, and it is difficult to imagine doing so without asking people to self-report their experience of such commitment. Still it would be optimal to also have non-self-report data on outcomes such as violence and arrest, and a forthcoming report will include such data.

Although statistically significant and large, the effect of assisted outpatient treatment assignment on serious violence perpetration should be interpreted cautiously. Reports of serious violence were rare, with only six events reported in the follow-up period, five in the comparison group and one in the assisted outpatient treatment group. Uncertainty about the magnitude of this effect is evident in the large confidence bounds around it. It should also be noted that any bias introduced by nonrandom assignment of individuals to assisted outpatient treatment and not corrected by propensity score weighting would be expected to produce more negative outcomes for the assisted outpatient treatment group. That is, if the comparison group differed in relevant ways from the assisted outpatient treatment group, the fear would be that members of the comparison group would be less ill and have better functioning than those assigned to assisted outpatient treatment. The fact that our generally positive findings for assisted outpatient treatment were in the opposite direction from this possible bias adds credibility to the findings.

Our findings largely overlap with those of previous studies that measured the same outcomes. For example, Greeman and McClellan ( 19 ) found outpatient commitment to be associated with a reduction in symptoms, although Steadman and colleagues ( 1 ) did not. The Duke Mental Health Study found that although assignment to the outpatient commitment group did not reduce violence, a combination of extended outpatient commitment and frequent service use was associated with less violence ( 20 ). Similarly, Van Putten and colleagues ( 21 ) found outpatient commitment to be associated with less violence. However, Hiday and Scheid-Cook ( 22 ) found no such association. Regarding self-rated quality of life, the Duke Mental Health Study found that assignment to the outpatient commitment group was not related to prospective quality of life, but length of time on outpatient commitment did predict improved quality of life ( 23 ). Steadman and colleagues ( 1 ) found no association between outpatient commitment and quality of life.

The outcome on which our results most differ from previous studies is perceived coercion. The Duke Mental Health Study found that individuals assigned to the experimental outpatient commitment group and those with longer periods of outpatient commitment reported greater coercion ( 14 ). McKenna and colleagues ( 24 ) also found greater perceived coercion among individuals under outpatient commitment. The reason for these discrepant findings is not clear. However, the importance of feeling coerced is underscored by other findings from the Community Outcomes of Assisted Outpatient Treatment study. Although perceived coercion was not positively related to assisted outpatient treatment assignment, it was significantly associated with other factors, such as number of involuntary hospitalizations, and perceived coercion had detrimental effects on perceived stigma, quality of life, and self-esteem ( 25 ). These findings indicate that perceived coercion has consequences, as suggested by Pollack ( 8 ), and the impact of outpatient commitment on perceived coercion deserves further study. Ethical issues surrounding actual coercion must also be kept in mind in evaluating outpatient commitment programs.

Conclusions

Outpatient commitment in New York State affects the lives of many people; therefore, we were reassured to find that assisted outpatient treatment did not produce some of the negative consequences feared. Rather, in conjunction with the treatment and other enhancements that accompany assisted outpatient treatment, people's lives seemed to be modestly improved along several dimensions by outpatient commitment as enacted in New York State under Kendra's Law. Finally, it is important that our findings be viewed through the lens of the overall effect of outpatient commitment and not the effects of legal coercion per se. Assisted outpatient treatment is a "package deal" that includes coerced treatment but also access to enhanced services. Although our analysis found no differences when we controlled for the presence of an intensive case manager, assisted outpatient treatment clients also received other enhanced services, such as priority for housing and vocational services. We cannot conclude which of these elements of the package deal contributed most to the generally positive outcomes for participants. We therefore caution against using our results to justify an expansion of coercion in psychiatric treatment.

Acknowledgments and disclosures

This research was funded by the New York State Office of Mental Health.

The authors report no competing interests.