Comparison of Outcomes of Acute Care in Short-Term Residential Treatment and Psychiatric Hospital Settings

Public and private behavioral health care systems are struggling with the pressure to contain or reduce costs while simultaneously maintaining or improving the quality of care. A logical outgrowth of this struggle is research on treatment outcomes that attempts to address fundamental questions about what a particular treatment is accomplishing and whether it is worth the cost (1,2,3). The measurement of outcomes becomes especially important as publicly funded systems implement various forms of managed care. In this context, it is important to document effective, yet less costly, alternative models for providing needed services (4,5).

The study reported here focused on five publicly funded short-term acute residential treatment facilities located in San Diego County, California (6). Operated by Community Research Foundation and funded by San Diego County Mental Health Services, these 11- to 14-bed facilities are designed to provide a less costly alternative to hospitalization for adults who need acute psychiatric care but do not require restraint or a locked facility. They offer brief, intensive, milieu-based treatment with a psychosocial rehabilitation orientation in a homelike neighborhood setting.

Several features are reported to distinguish the short-term acute residential treatment facilities in San Diego from similar programs in other locales. The San Diego programs admit clients who would otherwise be at high risk for acute psychiatric hospitalization. For example, clients who are admitted to the programs may be experiencing intense suicidality or psychosis. About half of the admissions have comorbid substance-related diagnoses. Medical complications are also frequently present. Most staff members have master's- or doctoral-level training, and psychiatric and nursing care is available on site. Further, the treatment is client centered, focusing on the individual needs of each client. The homelike setting makes possible the inclusion of household chores and meal preparation as components of the treatment experience. Although most activity would be characterized as milieu based, clients also have frequent personal contact with staff in daily individual and group sessions. Two staff members are present during overnight shifts. Daytime staffing typically includes at least one staff member for every three to four clients.

The comparison facilities consisted of two acute psychiatric hospital programs located in San Diego County. The programs were accredited by the Joint Commission on Accreditation of Healthcare Organizations. Identifying information related to the hospitals has been withheld due to agreements for anonymity with both hospital programs.

The main goals of the study were to compare the two types of programs on the following dimensions: the severity of clients' disturbance at the time of admission to the program, the outcome of treatment, the length of stay, the cost per treatment episode, and clients' satisfaction with services.

Methods

The study was observational and used a repeated-measures design. We compared characteristics of clients from the five residential programs and the two hospital programs at admission, discharge, and four months after discharge. The research protocol received the approval of the institutional review boards with jurisdiction over each program. Data from each type of program were combined, and the combined data for each type were compared.

To increase the study's validity, multiple standardized measures, each with subscales addressing various dimensions of symptoms and functioning, were used. The Brief Symptom Inventory (BSI) (7) was administered at admission only and the Client Satisfaction Questionnaire (CSQ-8) (8,9) at discharge only. The Behavior and Symptom Identification Scale (BASIS-32) (10) and the Medical Outcomes Study Short Form-36 (SF-36) (11) were administered at admission, discharge, and four-month follow-up. The self-report measures were supplemented by information from charts and computerized information systems, including reports of diagnoses and previous admissions.

Participants were eligible for the study if they spoke English, did not have a substance-related primary diagnosis, and signed a consent form. Adults consecutively admitted to each facility who met participant criteria were approached for participation in the study. Within 48 hours of admission and discharge, participants completed a battery of self-report measures. Follow-up interviews were conducted over the telephone by highly trained staff. The study period, including the follow-up interviews, spanned the period from August 1993 to December 1995.

The analysis used demographic and diagnostic data and mean subscale scores on the BSI, BASIS-32, SF-36, and CSQ-8. Demographic variables included age, employment status, ethnicity, gender, and marital status.

The primary analyses examined the severity of clients' disturbance at the time of admission; the outcome of treatment, defined as change between admission, discharge, and follow-up; the length of stay; the episode cost; and satisfaction with services received. As multiple statistical tests were being conducted, results were held to the more stringent alpha level of p<.01 for statistical significance. Because the primary diagnostic category appeared to have clinically relevant implications, data for study participants in specific diagnostic groups were analyzed and reported separately.

Results

Study participants

A total of 554 clients participated in the study at program admission and discharge, 368 from the short-term acute residential treatment facilities (53 percent of admissions during the study period) and 186 from the hospital-based psychiatric programs (20 percent of admissions during the study period). A higher proportion of clients from both groups—75 percent of the clients admitted to the residential programs and 42 percent of those admitted to the hospital programs—agreed to participate and signed a consent form at admission. However, not all of those clients were represented in data collected at discharge; 16 percent of the clients admitted to the residential programs and 14 percent of those admitted to the hospital programs were discharged without completing the discharge testing, and 6 percent of the residential clients and 8 percent of the hospital clients withdrew consent.

Follow-up interviews were originally planned for three months after discharge, but due to administrative delays and difficulty locating participants, the actual time elapsed from the day of discharge to the follow-up contact averaged four months (mean± SD=127.8±11.7 days). Follow-up interviews were successfully completed with 162 study participants from the residential programs (44 percent) and 89 participants from the hospital programs (48 percent).

Primary DSM-IV diagnoses were grouped into primary diagnostic categories including major depression, psychosis (including schizophrenia and schizoaffective and other psychotic disorders), bipolar disorder, other depression, and other diagnoses.

To reduce variability, the 36 participants with diagnoses in the "other" category—24 from the residential programs and 12 from the hospital programs—were excluded from the analysis. The analysis thus included data collected at admission and discharge for the remaining 344 study participants from the residential programs, 150 of whom (44 percent) participated in the follow-up interviews, and 174 participants from the hospital programs, 82 of whom (47 percent) participated in the follow-up interviews.

The representativeness of participants was explored in an analysis using data from administrative databases. Demographic and diagnostic data and length of stay of clients who participated and those who did not were compared. Findings suggested a bias against inclusion of some of the most seriously disturbed clients in both treatment settings.

In both settings, a lower proportion of clients with psychotic diagnoses and a higher proportion of clients with major depression were found among the study participants than among the nonparticipants. In the residential programs, 14 percent of the participants had a psychotic diagnosis, compared with 29 percent of the nonparticipants, and 55 percent of the participants had major depression, compared with 37 of the nonparticipants. In the hospital programs, 23 percent of the participants and 49 percent of the nonparticipants had a psychotic diagnosis, and 41 percent of the participants and 27 percent of the nonparticipants had major depression.

Interviews with clinical staff at the study sites corroborated the perception that severely decompensated, oppositional, or paranoid clients were less likely to participate. Although study participants were not fully representative of the populations treated in these programs, the biases operating for and against inclusion appeared to have been consistent across treatment settings and therefore did not preclude comparison.

As more than half of the participants in both study groups could not be located at follow-up, we compared outcome measures and demographic and diagnostic data of the participants we were able to contact with those of the participants who were not contacted. We found little evidence of a selection bias through attrition. The residential program clients who participated in the follow-up included proportionally more female clients than the group lost to follow-up (58.7 percent, compared with 40.7 percent, χ²=10.96, df=1, p<.01). However, this difference was not related to outcome. Among the clients in the hospital programs, we found no significant differences between the participants lost to follow-up and those we located.

Comparability of participants

The groups from the residential programs and the hospital programs were compared on demographic and diagnostic characteristics and mean scores on the depression, paranoia, psychosis, and global severity subscales of the BSI at the time of admission. The results are shown in Table 1.

We found significant differences between the two groups on diagnostic categories, gender, marital status, and employment. To assess clinical relevance, we examined the effect of these factors on the mean admission scores on the overall scale of the BASIS-32 for each of the groups. The BASIS-32 overall scale was selected because it is a composite scale that summarizes information from all five subscales of the BASIS-32. We found a significant effect for primary diagnostic category among the participant group from the residential programs (F=3.76, df=3,330, p=.01). No other significant effects were noted.

Severity of disturbance at admission

Analysis of variance (ANOVA) procedures were performed for each diagnostic group using mean scores at admission on all subscales of the BASIS-32 and the emotional well-being, emotional role functioning, and social functioning scales of the SF-36.

Although the mean scores of the clients with major depression and psychotic diagnoses indicated slightly more disturbance for participants in the residential programs, the differences between treatment settings were not statistically significant. Similarly, the clients with bipolar disorder in the residential programs showed slightly more severe symptoms on most measures at admission, but only the difference in scores on the social functioning scale was statistically significant (mean±SD=18.9±17.2 for the residential program clients, compared with 32.9±22.7 for the hospital program clients; F=7.44, df=1,59, p<.01). The magnitude-of-effect estimates, which assess the strength of an association as opposed to its statistical significance (12), supported reporting this difference as meaningful (eta²=.11). Mean scores for study participants with other depressive diagnoses were quite similar for the residential program and hospital groups.

Although significant differences were found between the two groups on some demographic and baseline measures, separating the results into diagnostic categories eliminated the differences for all but the scores on the social functioning subscale of the SF-36 for participants with bipolar disorder. These findings suggested that the residential program clients and hospital program clients in separate diagnostic categories were satisfactory comparison groups.

Treatment outcomeand stability

Outcome was assessed by repeated-measures ANOVA procedures using six subscales of the BASIS-32 and three subscales of the SF-36 for each diagnostic category. The BASIS-32 subscales included relation to self and others, daily living skills, depression-anxiety, impulsive-addictive behavior, psychosis, and the overall scale, which summarizes the other subscales. The emotional well-being, emotional role functioning, and social functioning subscales of the SF-36 were selected, as they are most closely related to mental health issues.

Among clients with major depression, follow-up data were available for 89 of the 199 study participants from the residential programs, or 45 percent, and for 48 of the 85 participants from the hospital programs, or 56 percent. Repeated-measures ANOVA showed significant differences between groups on the daily living skills subscale of the BASIS-32 (F=5.36, df=2,268, p<.01), but not on other BASIS-32 subscales or on the SF-36 subscales. Clients in the residential programs with major depression reported higher levels of distress at admission than those in the hospital programs (mean score of 2.6, compared with 2.4 on the daily living skills subscale of the BASIS-32) and slightly more improvement at discharge (mean score=1.6, compared with 1.8). At follow-up, clients with major depression who had been discharged from the residential programs reported a slight improvement, and clients from the hospital programs reported a slight worsening (mean score=1.5 for the residential programs, compared with 1.9 for the hospital programs). The magnitude-of-effect estimates for these comparisons was moderate (eta²=.08).

Fifty-four clients from the residential programs and 53 clients from the hospital programs had a psychotic diagnosis. Follow-up data were available for 24 of the residential clients, or 44 percent, and 20 of the hospital clients, or 38 percent. We found no significant differences between the groups.

There were 61 participants in the bipolar disorder group and 66 in the group with other depressive diagnoses. However, follow-up data were available for only seven subjects in each of these categories, limiting our analysis to a comparison of mean admission and discharge scores. No significant differences in these scores between clients from the two types of treatment settings were found in either diagnostic category.

The effects of attrition on outcome were examined by repeated-measures ANOVA procedures comparing residential program clients and hospital clients in each diagnostic group at admission and discharge only. Excluding the follow-up data more than doubled the number of participants in the analysis. Participants from both types of settings and in all diagnostic groups demonstrated robust and significant improvements at discharge (p<.01) on all outcome measures except the emotional role functioning scale of the SF-36 for participants with major depression. However, no significant differences were found between the residential program clients and the hospital program clients.

Length of stay

Independent-samples t tests were used to compare the mean length of stay of clients from the two types of programs in each of the diagnostic groups. The difference between groups for clients with major depression was not significant. For clients with a psychotic diagnosis or with bipolar disorder, the mean length of stay for those treated in the residential programs was longer than for those treated in the hospital programs (12.7 days, compared with 10.1 days for the participants with psychotic disorders; t=1.84, df=105, p=.05; and 14.5 days, compared with 9.2 days for the participants with bipolar disorder; t=2.93, df=59, p<.01). For study participants with other depressive disorders, the difference in length of stay between the residential programs and the hospital programs was not statistically significant.

Cost of treatment episodes

The Medi-Cal (California Medicaid) reimbursement rate in San Diego County at the time of the study was $186 per person per day for short-term acute residential treatment programs and $395 per day for psychiatric hospital programs. Although not all study participants had their care funded in this manner, these standardized rates were used to approximate costs for comparison. Per-episode costs were calculated by multiplying the daily reimbursement rate for each program type by the actual length of stay for each study participant by diagnostic group.

The mean episode costs for the different diagnostic groups are presented in Table 2. The differences in program costs are even more striking because the residential program costs included costs for psychiatric, pharmacy, and laboratory services, while the hospital costs did not.

Client satisfaction

The CSQ-8, which was administered at discharge, comprises eight qualitative questions that probe areas related to the person's experience of receiving treatment. Mean scores of study participants in the residential programs and the hospital programs were compared using independent-samples t tests for each diagnostic group. There were no significant differences between the mean scores of the two groups in any diagnostic category except major depression.

Residential program clients with major depression reported significantly higher levels of satisfaction than hospital program clients with that diagnosis on two questions. On the question of how they would rate the quality of service received, residential clients with major depression had a mean score, on a scale from 1 to 4, of 3.6, compared with 3.4 for hospital clients with major depression (t=2.51, df=118, p=.01). On the question of whether the services helped them deal more effectively with their problems, residential clients with major depression had a mean score of 3.6, compared with 3.3 for hospital clients with major depression (t=2.85, df=119, p<.01). However, in both cases, the differences, although significant, were relatively small. Participants from both types of programs consistently reported mean levels of satisfaction that fell into the upper 25 percent of the response range.

Readmissions

We collected self-report information about readmissions to any hospital or residential program during the follow-up period. Twenty residential program clients, or 13 percent, and 19 hospital program clients, or 23 percent, reported at least one admission to a hospital during the follow-up period. One participant from the hospital programs, or 1 percent, and 21 participants from the residential programs, or 14 percent, reported readmissions to a residential program, and five participants reported readmissions to both a hospital program and a residential program during the follow-up period.

If we combine the reported number of readmissions to the residential programs and the hospital programs for a total number of acute care readmissions, 27 percent of the residential clients and 24 percent of the hospital clients reported acute care readmissions during the follow-up period, not a significant difference.

Discussion and conclusions

This study did not have an experimental design and thus involved no manipulation of study participants or experimental procedures. The study simply attempted to measure and document what occurred at admission, discharge, and four months after discharge in a naturalistic manner, using primarily self-report data. Each hospital and short-term acute residential treatment facility implemented a single-group repeated-measures design in which study participants served as their own control subjects. The results were then combined and compared.

Of the nine clinical outcome measures examined across four diagnostic categories, significant differences were found only on the daily living skills subscale of the BASIS-32 for the major depression group, with participants in the residential programs reporting slightly better outcomes. Perhaps a more important finding is the striking consistency of similarities in the results at admission, discharge, and follow-up between the residential clients and the hospital clients. The findings suggest that, compared with hospital programs, the residential treatment facilities admitted persons with similar levels of acute distress who demonstrated comparable improvements with treatment and similar short-term stability of treatment gains within the diagnostic groups considered. Because the cost of residential treatment is about half that of hospital treatment, the results of this study suggest that short-term acute residential treatment programs provide a cost-effective alternative to acute psychiatric hospitalization for many adult clients within these diagnostic groups who do not require restraint or a locked facility.

Limitations

Repeated-measures designs in which study participants serve as their own control subjects are cost-effective and relatively unobtrusive, but this approach imposes limitations on the conclusions that may be drawn. In addition, although it is important to ascertain the stability of treatment gains by assessing study participants at follow-up, their status at follow-up after receiving acute care services, especially after longer time periods such as four months, may not be attributable to factors connected with the treatment.

This study's generalizability was also limited by a lack of representativeness in the sample. However, the biases were consistent across treatment settings, therefore allowing comparisons between the residential programs and the hospital programs.

Future research

Defining and measuring the outcome of behavioral health care services, especially for persons with severe and persistent disorders, is a complex task (3). The authors intend to use the study reported here as a pilot study for a controlled randomized comparison. Despite researchers' efforts in this direction over the years (13,14,15,16,17,18), well-designed studies to provide empirical support for alternative forms of acute psychiatric treatment continue to be needed (4). In addition, distinctions need to be made among the range of programs calling themselves alternatives to acute psychiatric hospitalization and among the clientele they serve so that accurate comparisons of these programs can be made.

Another logical step is the analysis of differential outcomes (19,20), with the aim of determining which persons experiencing an acute psychiatric crisis are best treated in a hospital alternative and which are best treated in a hospital setting.

A last perplexing issue is the continuing slow pace with which alternatives to acute psychiatric hospitalization have been adopted. Various authors have commented on the failure of hospital alternatives to become more widely accepted (21,22,23). Mosher's query (22) of more than a decade ago— "Why has research failed to be translated into practice?"—continues to be relevant. In the late 1990s, fiscal pressures and an emphasis on verifiable treatment outcomes in the behavioral health care field are combining to bring another wave of attention to cost-effective treatment alternatives. Efforts need to be directed toward the adoption of valuable and cost-effective alternate treatments that have been shown to provide comparable or improved treatment outcomes.

Acknowledgments

The study was funded by San Diego County Mental Health Services, Community Research Foundation, and grants from the Vista Hill Foundation in San Diego.

Dr. Hawthorne is executive director of the Community Research Foundation, 444 Camino del Rio South, Suite 219, San Diego, California 92108 (e-mail, [email protected]). He is also associate clinical professor in the department of psychiatry at the University of California, San Diego. Dr. Green is research coordinator at Community Research Foundation and in private practice in San Diego. Dr. Lohr is chief of the psychiatry service at the Veterans Affairs Medical Center in San Diego. Dr. Hough is codirector for research on child and adolescent mental health services at Children's Hospital and Health Center in San Diego. Dr. Smith is chief of program review and development at San Diego County Mental Health Services. Earlier versions of this paper were presented at poster sessions of the annual meeting of the American Public Health Association held November 9-13, 1997, in Indianapolis, at the Institute on Psychiatric Services held October 18-22, 1996, in Chicago, and at the annual convention of the American Psychological Association held August 8-12, 1996, in Toronto, Ontario.

Enlarge table

Enlarge table