A Randomized Trial of Short-Term Acute Residential Treatment for Veterans

As health care costs continue to increase, the evaluation of alternative and innovative models of health care delivery has become increasingly important. Financial pressures are especially salient in the public mental health sector, which is characterized by both limited resources and a growing emphasis on evidence-based practices, creating an expectation that service providers will "do more with less" and at the same time document effectiveness. Typically, the most expensive service provided by mental health care systems is acute psychiatric hospitalization. The study reported here examined, in a randomized trial, a certified and accredited alternative to acute psychiatric hospitalization, the short-term acute residential treatment (START) model.

Since the 1950s, trends in the treatment of severe mental illness have moved away from reliance on long-term hospitalization toward ever-shorter hospital stays and community-based treatment. In addition to successful programs that have reduced the need for acute care, such as partial hospitalization and assertive case management, several innovative programs for treating patients who are experiencing an acute psychiatric crisis outside the hospital were developed during the 1970s and 1980s (1,2,3,4). These models are often referred to as "residential" but do not provide longer-term residential treatment. Rather, they provide acute-level care in non-hospital settings for patients whose psychiatric crises would otherwise require hospitalization. Despite well-constructed studies comparing hospital-based treatment with alternative models, it was apparent by the 1980s that these alternatives to the acute psychiatric hospital were not being widely adopted (5,6). After reviewing ten rigorous studies, Kiesler (7) concluded that research results showing that alternative care was more effective were largely ignored.

Various alternative models for providing 24-hour acute psychiatric care have continued to emerge, mostly in piecemeal fashion. Stroul (8,9) and Warner (10) compiled fairly comprehensive reviews of many of these programs. The START model was first described in 1989 (11). In 1999, Hawthorne and colleagues (12) reported the results of a nonrandomized study. In a repeated-measures design, 368 START patients and 189 hospital patients were assessed on multiple measures of symptoms and functioning at admission, discharge, and four-month follow-up. The findings suggested that START and hospital programs admitted persons who had similar levels of acute distress and demonstrated comparable levels of improvement by discharge and an equivalent degree of short-term stability of treatment gains. Initial-episode treatment costs were significantly lower for START participants across all diagnostic categories.

As was the case in the 1970s, several of the more recent models have been subjected to randomized trials. These models included an experimental day hospital and crisis respite program in Connecticut, designed specifically for the study, which demonstrated lower costs but no other significant differences compared with hospital treatment on measures of symptoms, functioning, quality of life, and satisfaction (13,14). Toprac and colleagues (15) compared hospital care with a complex set of eight different crisis alternative programs in Texas and reported lower cost for the alternative programs. These authors found no significant differences in symptoms, functioning, or satisfaction. Fenton and colleagues (16,17) compared a crisis residential program with a county hospital in Maryland and reported that in most cases the alternative provided "near-equivalent effectiveness for significantly less cost" (17).

Most previous studies compared one alternative program with a traditional hospital setting. Some alternatives were experimental and created for the duration of the study only. This study's alternative, START, has developed over 25 years and is now a fully accredited and certified network of six facilities. The study participants were veterans, an important subset of patients. As the largest mental health care provider in the United States, the Veterans Health Administration (VHA) has the scope and capability to promote and implement innovative programs on a widespread basis.

Our main goal was to compare hospital and START in terms of outcomes, satisfaction with services, and costs for the index episode within a randomized clinical trial. We hypothesized that participants who were treated in the START model would demonstrate greater improvement in symptoms and functioning, would report greater satisfaction with services received, and would have lower costs of care compared with those who were treated in the hospital.

Methods

This study was a randomized clinical trial of two well-established and accredited treatment programs for acute psychiatric care, START and the Veterans Affairs San Diego Healthcare System (VASDHS) inpatient psychiatry unit. The research protocol was approved by all relevant human subjects and research review committees. After initial randomization, participating veterans received treatment as usual; all follow-up outpatient care and any hospital or START facility readmissions were determined solely by the veteran and the treating clinician.

Recruitment of participants was conducted between February 2000 and April 2002.

Programs

The original START program was founded 25 years ago. There are currently six START programs, regionally located throughout the county and providing a total of 75 beds. These programs are funded primarily by Adult and Older Adult Mental Health Services of the Health and Human Services Agency of the County of San Diego to provide acute psychiatric hospital alternative services to Medicaid beneficiaries and indigent adults. On any given day, nearly half the adults who are voluntarily receiving publicly funded acute psychiatric care in San Diego County are being served in START facilities rather than hospitals. The programs also provide services on a contractual basis for the VASDHS and Kaiser Permanente.

All START programs are located in the community, most in large older homes that have been remodeled to accommodate 11 to 14 residents. The facilities are designed to provide acute psychiatric care for voluntary adults who do not require restraints or a locked facility. The programs are Medicaid certified by the California Department of Mental Health, licensed by the California Department of Social Services, and accredited by the Commission for Accreditation of Rehabilitation Facilities (CARF). The average length of stay is nine days.

Most patients who are admitted to START are experiencing acute suicidality or psychotic symptoms, and about half have concomitant substance-related diagnoses. Medical conditions are also frequently present, and nursing staff are on duty from 8:00 a.m. until midnight. However, only patients whose nonpsychiatric medical problems can be treated safely on an outpatient basis are admitted. Daytime staffing ratios are typically 1:3 or 1:4. Most staff members have master's- or doctoral-level training in psychology, social work, or nursing. Psychiatrists from the University of California, San Diego, provide services on-site three days a week and are on call 24 hours a day. All START patients are seen by a psychiatrist, and medications are prescribed and adjusted as needed during the stay. The program schedule includes two community meetings and two group sessions per day, individual counseling sessions, medication consultations, recreational activities, and patient participation, whenever possible, in chores and meal preparation. Except for laboratory work, if needed, all services are provided on-site. A core feature of START is adherence to the principles of psychosocial rehabilitation (18), including an emphasis on the patient's active involvement in treatment planning, goal setting, and planning for discharge as early as possible in the treatment process.

The VASDHS is considered to be one of the more modern and well-equipped veterans' hospitals in the country. Affiliated with the University of California, San Diego, School of Medicine, VASDHS includes an acute psychiatric hospital program with a total of 37 beds licensed by the California Department of Health Services and accredited by the Joint Commission on Accreditation of Healthcare Organizations. Staffing includes psychiatrists who are on-site seven days a week, nurses who are on duty 24 hours a day, and social workers and occupational therapists. Daytime staffing ratios are approximately 1:4 or 1:5. Treatment and discharge planning are under the direction of attending physicians. The program schedule includes a daily community meeting, recreation therapy and exercise groups, and other therapy-focused and educational groups. Veterans may be admitted on a voluntary or an involuntary basis, and seclusion and restraint are available if needed. In a recent nationwide satisfaction survey of the VA health care system, the VASDHS inpatient psychiatry unit received the highest quality-of-care scores in the United States.

Participants

Veterans were assessed in the VASDHS psychiatric emergency clinic by psychiatrists from the VA and UCSD. To be eligible for the study, veterans had to be judged to require an acute level of care, to be voluntary patients (that is, not involuntary or requiring restraint), and to be between the ages of 18 and 59 years. Voluntary admission status and the age limitation are licensing requirements for the START facilities. Veterans who had a diagnosis other than an affective or psychotic disorder—for example, a substance-related disorder only—were excluded, as were those with co-occurring medical conditions requiring hospital-level care. Participation in the study was also contingent on participants' signing an informed consent form after they had received a complete description of the study and on being admitted to the assigned facility.

Because the study involved veterans who were in need of acute care, randomization to treatment type occurred immediately after recruitment. The often emergent nature of these circumstances required a strict approach to participant recruitment. No veteran could be approached for participation in the study unless at least one bed was available at each of the two potential sites (hospital and START facility). All six START facilities participated in the study. If more than one START facility had a bed available, staff conducted a randomization procedure to select one facility.

A stratified block randomization (19) was used to maintain comparability between groups on the following characteristics that might affect outcome: age (18 to 38 years and 39 to 59 years), whether the presenting problem was psychotic or affective, and the presence or absence of substance abuse. The study's statistician generated in advance a separate randomly derived sequence of assignments for each of the eight possible combinations of the above stratification categories. After a veteran consented to participate, research staff determined the participant's status on the three stratification variables and then consulted the appropriate list of assignments. The complexity of this method precluded advance knowledge of randomization assignment by research staff.

Research staff who conducted structured interviews and administered instruments had completed master's-level or higher degrees in psychology or social work. All research staff had clinical experience working with patients with severe mental illness and in crisis. All staff underwent an intensive training module specifically on the study's instruments. Spearman rank correlations were computed on each rater and compared with a benchmark on the most complex of the instruments to score, the Structured Clinical Interview of the Positive and Negative Syndrome Scale. The mean correlation was .68, the median was .67, and the range was. 56 to .87. Item-by-item percentage of agreement on the PANSS, using the clinical trial standard of within two points (20), ranged from 70 percent to 97 percent, with a mean of 85 percent.

Instruments

The study participants were given an interview and a battery of instruments, usually within 24 hours of admission (mean hours from admission to testing, 18.6±24.8) and discharge (mean hours from discharge to testing, .2±89.3) from the index episode. Time flexibility and the interview format for assessments assisted in allowing the participation of the most severely disorganized or distressed participants. No significant differences were noted between hospital and START participants on hours to admission and discharge testing. The same instruments were then readministered at two-month follow-up interviews. Data on readmissions to the hospital or to a START facility during the study period were collected from VASDHS information systems and from San Diego County Adult and Older Adult Mental Health Services information systems.

When research is conducted among persons with severe mental disorders, it is recommended that multiple domains be assessed from various perspectives (21,22). Both clinician-rated and self-report standardized measures of symptoms, functioning, and satisfaction with services were included in this study, as were in-depth diagnostic interviews. We also collected information on participants' demographic and other characteristics, including age, gender, educational background, ethnicity, marital status, employment, living situation, and psychiatric and self-help treatment history.

The Structured Clinical Interview for the Positive and Negative Syndrome Scale (SCI-PANSS) (23,24) was used to assess symptoms and symptom severity at admission, discharge, and the two-month follow-up interview. The SCI-PANSS includes a 30-item rating scale, which yields subscales for positive and negative symptoms as well as for general psychopathology.

A Structured Clinical Interview for DSM-IV Disorders (SCID) (25) was conducted close to the time of discharge.

The alcohol and drug composite scales of the Addiction Severity Index (ASI) (26) were used to assess levels of substance abuse at discharge and at the two-month follow-up interview. These parts of the instrument asked participants for detailed information about their substance use and then required clinicians' judgment in producing ratings.

We used the mental component summary (MCS) scale of the VA Clinic Update version of the Short Form-36 Health Survey (SF-36V) (27,28) to assess emotional aspects of functioning at admission, discharge, and the two-month follow-up interview.

The Inpatient Version of the Perceptions of Care scale (POC) (29,30) was used to assess participants' satisfaction with different aspects of services received and was administered at discharge only. The measure contains 18 structured items that yield four subscales: information received, interpersonal aspects of care, continuity and coordination, and global evaluation.

Analysis

The SCI-PANSS and the SF-36V were used to ascertain whether participants who were treated in START programs would demonstrate greater improvement in symptoms and functioning. Because these measures were taken at multiple time points (admission, discharge, and two-month follow-up), repeated-measures analysis of variance (ANOVA) was used. Substance abuse was assessed with ASI scores collected at discharge and at the two-month follow-up interview, and these were also analyzed with use of repeated-measures ANOVA.

We also examined changes in homelessness and readmission data across the study period. Homelessness was categorized as a yes-or-no variable and was compared by using separate McNemar tests for changes between admission and discharge and separate Cochran's Q tests for changes across admission, discharge, and the two-month follow-up period for each group. Data on admissions to hospital or START facilities during the two-month follow-up period were collected from VASDHS information systems and from San Diego County Adult and Older Adult Mental Health Services information systems, respectively. Comparisons of the number of readmission episodes during the two-month follow-up period were examined by using Mann-Whitney U procedures, recommended for skewed distributions (31).

To test the hypothesis that START participants would report significantly greater satisfaction with services received as measured by the four subscales of the POC, Mann-Whitney U tests were used.

Initial-episode costs from the perspective of the health care purchaser were compared for persons who were initially assigned to the START and hospital groups. Costs were calculated by multiplying a daily rate by the length of stay. The daily rate used to estimate START costs ($251) includes indirect and all costs—for example, psychiatric, pharmacy, and laboratory costs—and was extracted from the Fiscal Year (FY) 2001-2002 Cost Report to the State of California. Estimates of the median VASDHS inpatient psychiatric facility cost per day (including indirect costs) for FY 2001-2002 ($864) were provided by the Health Economics Research Center (32,33). A log-linear model with a nonparametric retransformation factor (34) was used to compare initial-episode costs between the groups. Significance levels were calculated by using the bootstrap method with 1,000 replications. The bootstrap method is a resampling procedure that generates a nonparametric distribution of the variance (35). Stata software was used to compare costs; all other analyses used SPSS.

Results

During the participant recruitment period, when hospital and START beds were available, 278 veterans were assessed for participation in the study. Of these, 114 (41 percent) met criteria for study participation. Of the 99 participants who consented and received treatment, 47 received hospital treatment and 52 received START treatment for the initial episode. Mean length of stay did not differ significantly by group (10.6±6.8 days for the hospital group and 12.9±6.3 days for the START group).

During the two-month follow-up period, any subsequent readmissions or outpatient services were not rerandomized or restricted in any way by the study design. Researchers conducted two-month follow-up interviews with 80 participants (81 percent), 40 in each group. The 19 participants who were lost to follow-up and the 80 remaining participants were compared by using age, marital status, ethnicity, homelessness, the presence or absence of a substance-related diagnosis or a personality disorder, and discharge scores on the PANSS (total score), the SF-36V MCS, the POC, and the ASI drug and alcohol composite scales. Participants who were lost to follow-up were more likely than those who remained in the study to have a substance-related diagnosis (89 percent compared with 67 percent; χ²=4.2, df=1, p<.05). No other significant differences were noted between the groups. In addition, readmission data from databases were available for all 99 participants; when those lost to follow-up were compared with those interviewed at two months, no differences were found in readmissions to START facilities or to the hospital.

Participants were mostly men (96 participants, or 97 percent). Their ages ranged from 26 to 59 years, with a mean of 46±6.3 years. Additional demographic characteristics are listed in Table 1, and mean admission scores by group are listed in Table 2. To examine the comparability of the hospital and START groups, demographic and baseline dependent measures—including SF-36V, PANSS, diagnostic category, presence or absence of substance abuse, and homelessness—were examined. No significant differences were found between the groups.

To place study participants within the larger context of the veteran population receiving acute psychiatric care, we reviewed mean Global Assessment of Functioning (GAF) scores from study participants and from official VHA reports of veteran inpatients with psychotic and depressive disorders (36,37). Mean GAF scores included in VHA reports were based on each inpatient's lowest recorded GAF score for that year. Study participants' GAF scores were based on admission scores. Although these differences in data reporting precluded statistical comparisons, the mean GAF scores of study participants fell well below the mean GAF scores of veteran inpatients within these diagnostic categories (VA depression mean=38.0±14.0, study depression mean=32.4±7.4; VA psychosis mean=36.7±11.3, study psychosis mean=32.2±6.5).

Mean hospital and START PANSS total scores were compared at admission and discharge as well as across admission, discharge, and two-month follow-up. Both groups demonstrated robust within-group improvements between admission and discharge, with some additional improvements at the two-month follow-up (admission and discharge: F=93.8, df=1, 90, p<.001; admission, discharge, and two-month follow-up: F=51.0, df=2, 150, p<.001). However, no significant between-group differences were found in either group-by-time interaction terms or univariate comparisons of admission, discharge, and follow-up mean scores.

Results on the mental component summary of the SF-36V suggested that START participants had significantly greater group-by-time improvements than hospital participants between admission and discharge. By the two-month follow-up interviews, these differences were no longer significant (F=2.5, df=2, 138, p=.089).

Results on the alcohol composite scale and the drug composite scale of the ASI showed significant within-group improvements (reductions in use) between discharge and the two-month follow-up interview for both groups (alcohol composite scale, F=17.9, df=1, 76, p<.001; drug composite scale, F=5.3, df=1, 76, p=.025), but no between-group-by-time differences were found.

Significant reductions in homelessness were found between admission and discharge for the participants in the START group only (McNemar: START, p=.001, hospital, p=.09); in other words, more START participants reported that they were being discharged to living situations other than homelessness. However, by the two-month follow-up, both hospital and START groups showed significant reductions in homelessness (START Cochran's Q=14.0, df=2, p=.001; hospital Cochran's Q=13.6, df=2, p=.001).

There were no differences between the groups in the number of hospital or START readmissions or the number of inpatient days during the follow-up period.

Significantly greater satisfaction with services was reported by START participants on two of the four subscales of the POC: information received and global evaluation. However, because 23 percent of discharge testing was conducted by START staff, this finding should be interpreted with some caution, as discussed in more detail as a limitation in the Discussion section.

Mean costs for the index episode were found to be significantly lower for the START participants than for those who were treated in the hospital. Mean costs for the START participants were 65 percent less than for those who were treated in the hospital for the index episode.

Discussion

This study's hypothesis of superior outcomes in symptoms and functioning for participants receiving treatment through an alternative to the hospital was only partially supported. Both groups of veterans—those who were randomly assigned to the VA hospital inpatient unit and those who were assigned to the inpatient alternative START program—demonstrated significant improvements in symptoms, functioning, substance abuse, and homelessness when these variables were measured between admission, discharge, and two-month follow-up. On two measures of outcome—SF-36V mental component summary scores and reported homelessness—START participants fared better than hospital participants at the time of discharge. By the two-month follow-up, these between-group differences in outcome were no longer evident. Lower costs and greater patient satisfaction for START participants were found as hypothesized.

This study joins a long line of previous research, beginning in the 1970s, that has consistently demonstrated similar and sometimes superior outcomes from alternative acute psychiatric care compared with hospital-based treatment (1,2,35,6,7,8,9,1013,14,15,16,17). This was the first study of a state-licensed and nationally accredited residential alternative to hospitalization that demonstrated lower cost, greater satisfaction, and similar outcomes to those achieved under inpatient care.

Several recent research efforts on acute psychiatric care deserve more detailed comparison with our study's results. Although they differed in some important respects, START and the programs described by Fenton (16,17), Sledge (13,14), and Toprac (15) and their colleagues were all designed to treat adults who were identified as needing acute-level psychiatric care who did not require hospital-level detoxification or other serious medical care and who were being treated on a voluntary basis. Unlike START and the program studied by Fenton and colleagues (16,17), the acute care alternatives described by Sledge and colleagues (13,14) and by Toprac and colleagues (15) were both experimental programs designed for their respective studies. Length of stay varied widely, from an average three-day stay in the Texas alternative in the Toprac study to an 18.8-day average reported in the Sledge studies. Compared with all these programs, START provided the most comprehensive services on-site. The study reported by Sledge and colleagues by design made use of a day hospital for daytime care, and the programs reported by Fenton and Toprac used nonclinically trained staff with psychiatric care provided by the patient's outpatient psychiatrist.

A unanimous conclusion of these four hospital alternative studies was that costs were lower for the alternative program. For example, operating costs for the model of Sledge and colleagues (14) were 51 percent of inpatient hospital costs, whereas initial-episode costs were 44 percent lower in the Fenton study (17) and 65 percent lower in our study.

However, one way in which our study differed from the others was in the evidence to support greater patient satisfaction for participants treated at START facilities. The three other studies showed no differences in patient satisfaction, with one exception: of ten satisfaction ratings, participants in the Fenton study reported better food at the alternative program (16). Previous studies of START programs also reported higher patient satisfaction ratings for START facilities compared with hospitals: START programs were consistently rated higher than a group of nine psychiatric hospital programs in a 1998 San Diego County survey (38), and higher ratings were found for START participants with depression on two of eight satisfaction items in the earlier nonrandomized START study (12).

This study had several limitations that must be kept in mind when interpreting the results. Because participants were all voluntarily admitted veterans under the age of 60 and were overwhelmingly men, the results may not be generalizable to other groups, including involuntarily hospitalized patients, older veterans, women, older patients, and other nonveteran populations in need of acute psychiatric care.

In addition, the proportion of participants who had concomitant substance use disorders was higher than that reported in VA national data: 72 percent of participants also had diagnoses of substance-related disorders, compared with national estimates of 22 percent of veterans with depression (37) and 25 percent of patients with psychosis (36). However, this latter report discusses the issue of underestimating true prevalence and mentions that other studies have shown 50 to 75 percent of the general population of persons with serious mental illness to have a concomitant substance-related diagnosis (36). Diagnoses based on the SCID as used in our study were less likely to underreport substance-related diagnoses than chart or billing diagnoses.

As previously mentioned, this study's instruments were administered in interview format by trained research staff. Two staff members who were employed specifically for the research project conducted a majority of all interviews, including all admission PANSS ratings and all follow-up interviews, for both groups. However, because San Diego County is a large geographic region, 23 percent of discharge assessments (N=23) were conducted by staff at START facilities who underwent the same specialized training on the study's instruments as the two full-time research staff. This aspect of the process raised the question of whether differences existed between measures collected by research staff and START staff interviewers. Comparisons were therefore conducted on dependent and other measures collected at discharge by START staff and research interviewers. No differences were found on PANSS or SF-36V responses. However, POC scores on two of the four subscales were found to vary by interviewer type, in that participants who were interviewed by START staff had more favorable scores (Mann-Whitney U=426.5, Z=-3,39, p=.001). This finding raised the issue of a possible acquiescent bias on the satisfaction measure. Further analyses and review of how and when discharge interviews were conducted indicated that this finding was consistent with the practicalities of data collection during the study. START staff interviewers, on-site at the START programs, were most likely to conduct discharge interviews with START participants who had planned discharges after successful stays and who were more likely to be satisfied with their care. Research staff interviewed some START participants with planned discharges but were also responsible for tracking down and interviewing START participants who had left the program prematurely or against medical advice before a discharge assessment could be conducted on-site. Given these circumstances, it is not surprising that the 23 START participants who were assessed by START staff reported more favorable satisfaction scores than the 25 START participants who were assessed by research staff. Nonetheless, we cannot conclusively rule out an acquiescent bias, which should be kept in mind when interpreting these results.

Finally, alternative accounting methods were required to estimate the daily costs of the START and VASHDS programs. The cost assumptions and methods used in the START cost report may vary significantly from those in the report used to estimate VASDHS costs, and the use of these two accounting systems may have distorted the true differences in cost between these programs. Also, the limited perspective of our cost analysis does not consider other potential societal costs—for example, forensic and general medical costs—that may be differentially affected by admission to START versus VASDHS treatment. However, we note that the patients who were assigned to each program had similar psychiatric outcomes and that our estimate of VASDHS costs was similar to the daily cost of the local county psychiatric hospital.

Conclusions

In this randomized trial, START—an alternative to inpatient treatment—was less costly, yielded higher levels of satisfaction with services, and produced similar outcomes to those achieved with hospital treatment. The results of this study provide support for a certified and accredited hospital alternative treatment model that is suitable for adults who need voluntary acute psychiatric care and who meet age and medical criteria. These results add to the accumulated body of research supporting non-hospital-based acute psychiatric alternatives.

Acknowledgments

This study was funded by grant PCC-98051-2 from the Health Services Research and Development Service of the Veterans Health Administration. The authors also acknowledge Kathy Lyttle, Psy.D., and Len Lulow, M.S.W., for data collection and management as well as Nilah Archer, R.N., M.F.T., and Wayne Marshall, M.S.N., M.B.A., who were instrumental in the implementation of the study.

Dr. Hawthorne is affiliated with the department of psychiatry of the University of California, San Diego (UCSD) and with the Community Research Foundation in San Diego. Dr. Green is in private practice in San Diego and works on projects for San Diego County Mental Health Services. Dr. Gilmer is with the department of family and preventive medicine at UCSD. Dr. Garcia is director of Adult and Older Adult Systems of Care at San Diego County Mental Health Services. Dr. Hough is with the department of psychiatry of UCSD and the departments of psychiatry and family and community medicine at the University of New Mexico at Albuquerque. Dr. Hammond is with the Community Research Foundation. Dr. Lee is with the biostatistics department of the University of California, Los Angeles. Dr. Lohr is with the department of psychiatry of UCSD and with the Department of Veterans Affairs (VA) San Diego Healthcare System. Send correspondence to Dr. Lohr at VA San Diego Healthcare System, Department of Psychiatry, Mailcode 116A, 3550 La Jolla Village Drive, San Diego, California 92161 (e-mail, [email protected]).

Enlarge table

Enlarge table