A Randomized Study Comparing the Treatment Environment in Alternative and Hospital-Based Acute Psychiatric Care

As financial pressures in the public mental health sector continue to increase, the evaluation of innovative and lower-cost programs has become increasingly important. Because acute psychiatric hospitalization is typically the most expensive service provided by mental health care systems, alternative programs that demonstrate effectiveness may offer substantive benefits for patients as well as for health care systems. Since the 1970s studies have compared the outcomes of acute psychiatric care provided in alternative and hospital-based programs and have often found that alternative models offer equivalent or superior effectiveness at a lower cost than hospital care ( 1 , 2 , 3 , 4 , 5 , 6 , 7 , 8 , 9 ).

The idea that the treatment environment is an important aspect of care that influences the outcomes of patients with serious mental illness can be traced back to Pinel's successful unshackling of previously violent asylum patients in 18th century France and the Quakers' promotion of "moral treatment" for persons with mental illness ( 10 ). In the 1970s efforts began to systematically evaluate the environments in which psychiatric care was provided ( 10 , 11 , 12 ). One of the best known measures of treatment environment is the Ward Atmosphere Scale (WAS) ( 13 ). A substantial body of literature evaluating and comparing treatment environments has continued to grow over the past three decades ( 10 , 11 , 12 , 14 ). Although there are several studies of treatment environments that are alternatives to hospitals ( 7 , 15 , 16 , 17 ), the study presented here is the first randomized trial of a residential alternative to acute hospitalization comparing patient and staff ratings of both treatment environments on the same standardized measure. It is also the first randomized trial to compare patient and staff ratings of both treatment environments with national norms for hospitals.

Patients who participated in this study were all veterans and were randomly assigned to either a hospital or a well-established alternative program, short-term acute residential treatment (START). This study augments a previously published study comparing outcomes, satisfaction, and cost of acute psychiatric care at a hospital with those of START programs ( 1 ). The previous study by Hawthorne and colleagues ( 1 ), however, did not address the important issue of the treatment environment. Most studies of treatment environments that are alternatives to hospitalization have been based on observational data, and therefore patient characteristics and treatment preferences may have biased the findings.

The objective of the study presented here was to compare patient and staff perceptions of the treatment environment in a hospital with those of the treatment environment in START programs by using the WAS ( 13 ). The specific aims of this study were to compare perceptions of the treatment environment reported by participants randomly assigned to the hospital and to the START program, to compare perceptions of the treatment environment reported by hospital staff and START staff, and to compare congruence of staff and participant perceptions of the treatment environment at each of the treatment settings. To provide a broader context in which to view results, we also compared staff and participant ratings with published norms for hospitals.

Methods

Programs

Although the hospital and alternative programs in our study have been previously described ( 1 ), important aspects of the two treatment settings are summarized below.

START. The first START program was established in 1980 to provide an alternative to hospital-based acute psychiatric care ( 18 ). Currently, there are six START programs located throughout San Diego County providing a total of 75 beds. Treatment at START facilities is primarily funded by the County of San Diego, Health and Human Services Agency, Adult and Older Adult Mental Health Services, to provide services to Medicaid beneficiaries and indigent adults. On any given day almost half of the adults receiving publicly funded acute psychiatric care on a voluntary basis in San Diego are in START facilities rather than in hospitals. The START programs also treat Kaiser Permanente members and veterans from the Department of Veterans Affairs San Diego Healthcare System (VASDHS).

The START programs are located in residential neighborhoods in large older homes and accommodate 11 to 14 residents. The programs are Medicaid-certified by the State Department of Mental Health and are accredited by the Council for Accreditation of Rehabilitation Facilities. The average length of stay is nine days.

START programs provide acute psychiatric care for adults who do not require restraints or a locked facility. Patients admitted to START are often experiencing acute suicidal or psychotic symptoms, and in most cases they have a concomitant substance-related diagnosis. Nonpsychiatric medical conditions are also frequently present. Each START program has 24-hour staffing and a daytime staffing ratio of one staff member to every three patients. Clinical staffing at the START programs is quite different than at the Department of Veterans Affairs (VA) hospital examined in this study. The majority of hospital clinical staff consists of nursing staff providing 24-hour coverage. Clinical staff at START programs typically have a master's degree in psychology, social work, or other related field. START programs also employ nursing staff, who provide coverage 16 hours per day (8 a.m. to midnight), but nurses comprise a lower proportion of clinical staff than at the hospital.

Psychiatrists from the University of California, San Diego, provide services on site three days per week and are on call 24 hours a day. All patients are evaluated by a psychiatrist, and on the basis of shared decision making ( 19 ), medications are prescribed or adjusted as needed.

START offers intensive, milieu-based treatment with a client-centered psychosocial rehabilitation philosophy. Patients participate in daily individual, group, and community meetings. This creates a less institutional atmosphere and provides more continuity with daily life outside of the treatment setting. A core feature is the emphasis on individualized goals and the patient's active involvement in setting treatment goals, taking medication, and planning for discharge.

VA hospital. The VASDHS is considered to be one of the most modern and well-equipped VA hospitals in the country. Affiliated with the University of California, San Diego, School of Medicine, VASDHS includes an acute psychiatric hospital program with a total of 37 beds. Staffing for the psychiatric unit includes psychiatrists on site seven days a week, nurses on duty 24 hours a day, social workers, and occupational therapists. Typical staffing ratios are approximately one staff member to four or five patients. Treatment includes a daily community meeting, recreation therapy, exercise groups, and therapy-focused and educational groups. Attesting to the high quality of the facility, in a 2001 unpublished internal nationwide satisfaction survey of the VA health care system, the VASDHS inpatient psychiatry unit received the highest quality-of-care scores in the country.

Participants and data collection

The study protocol was approved by all relevant human subjects and research review committees. Recruitment took place between February 2000 and April 2002. Veterans eligible for inclusion in the study were required to be between 18 and 59 years old (a licensing requirement for START programs at that time). Patients with comorbid (nonpsychiatric) medical conditions requiring more than outpatient care were excluded from the study. Patients provided informed consent. All were veterans and were judged by psychiatrists at the psychiatric emergency clinic of the VA hospital to need admission to an acute level of care and were subsequently randomly assigned to either the inpatient psychiatry unit of the hospital or START.

Patients completed the WAS at or near the time of discharge. For patients who left the START or hospital program abruptly or against medical advice, research staff endeavored to conduct discharge interviews as soon as possible. Staff members from both programs, including all six START facilities, completed the WAS during the study period.

Instrument

The WAS is the most widely used instrument for measuring the social climate of inpatient psychiatric programs ( 20 ). The WAS is composed of 100 true-or-false items that yield ten subscales: involvement, support, spontaneity, autonomy, practical orientation, personal problems orientation, anger and aggression, order and organization, program clarity, and staff control. Possible scores on the subscales range from 0 to 10, with higher scores indicating a high level of the characteristic measured. Both staff and patient norms are available, which are based on a sample from 160 VA, state, and university hospitals.

Hypotheses

In this study, four groups completed the WAS: patients randomly assigned to the hospital, patients randomly assigned to START, hospital staff, and START staff. We compared these groups both across and within program types. Our primary hypothesis was that patients treated in the START model would rate the treatment environment more favorably than would those treated in the hospital. This hypothesis was based on previous findings from two published studies ( 1 , 2 ) and one local government report ( 21 ) in which patients expressed greater satisfaction with services at START than services at hospital-based programs.

We next compared staff ratings of the two types of programs. Because START programs utilize a client-centered treatment approach emphasizing patient involvement in treatment and discharge planning, we hypothesized that START staff members would rate their environments more favorably than hospital staff members, especially on the subscales of involvement, support, and practical orientation.

Many studies have found differences between staff and patient perceptions of treatment environments ( 10 , 14 , 22 , 23 ), with staff typically having more favorable perceptions than patients on most subscales. Staff and patient congruence on ratings can be an indicator of program cohesion; because of the START client-centered treatment philosophy, we hypothesized that ratings of participants and staff members in the START program would be more congruent than those of patients and staff members in the hospital. We also examined the WAS profile of each staff and patient group in relation to that of the other group and to national norms, expecting to find higher subscale scores among START staff and patients.

Analysis

Because some distributions were nonnormal, Mann-Whitney U tests were used to compare WAS subscale scores. To test the main hypothesis, we compared scores of patients in the hospital with those of patients in START. Next we compared perceptions of the treatment environment of hospital staff with those of START staff. Although hypotheses were directional, we used the more conservative two-tailed tests for the analyses because of the number of comparisons.

Finally, we examined the congruence in WAS scores between patients and staff in START and hospital programs separately in two ways. First, using standardized scores we compared staff and participants within their respective programs. To estimate congruence, we calculated the mean of differences between subscale scores of staff and participants at each program type and compared them by using a Mann-Whitney U test. In addition, we compared the WAS profiles of patients and staff at both types of program to published norms ( 13 ).

Results

Ninety-three patients, 45 randomly assigned to the hospital and 48 randomly assigned to START, completed the WAS at discharge. Fifteen of 50 hospital staff (30% response rate) and 75 of 112 START staff (67% response rate) completed the WAS. As described in detail in the previous publication ( 1 ), there were no significant differences between the two patient groups on any demographic, diagnostic, or baseline criteria. Eighty-seven patients (94%) were male (41 males, or 91%, in the hospital and 46 males, or 96%, in START). Mean±SD age for both participant groups was 46 years (45.9±6.7 years in the hospital and 46.0±6.1 in START). Demographic and diagnostic information is presented in Table 1 .

Enlarge table

When we compared WAS scores of the patients, we found that those in the START program rated the treatment environment more favorably than those in the hospital program on five of ten WAS subscales (involvement, p<.01; support, p<.01; autonomy, p=.03; order and organization, p=.01; and program clarity, p<.01) ( Table 2 ). Mean scores of patients in the hospital were not significantly higher than those of patients in START on any of the WAS subscales. When we compared the staff ratings, we found that START staff rated their treatment environment more favorably than hospital staff on five subscales (involvement, p<.01; practical orientation, p=.01; personal problem orientation, p=.02; order and organization, p=.05; and program clarity, p<.01) ( Table 3 ). The first two of these were specifically targeted in the second hypothesis. Compared with START staff, hospital staff had significantly higher scores on the anger and aggression subscale (p=.01). Comparisons on the other subscales were not significantly different.

Enlarge table

Enlarge table

Our third hypothesis, that START staff and patient ratings would be more congruent than those of hospital respondents, was partially supported. We found no difference when comparing the hospital and the START program on the mean of differences between staff and participant subscale scores. However, compared with published norms, there were three subscales on which both START patients and staff had significantly higher ratings than national norms (>1 standard deviation higher) (support, patient=1.76 SD, staff=1.44 SD; order and organization, patient=1.73 SD, staff=1.10 SD; and program clarity, patient=2.22 SD, staff=1.49 SD), whereas hospital staff and patients did not have significantly higher ratings than national norms on any of the subscales.

In comparing the WAS ratings for the hospital in this study to published norms ( 10 , 13 ) ( Figure 1 ), we found that the standardized subscale scores for hospital patients and staff were not significantly different from hospital norms on nine of ten subscales. Both hospital staff and patient ratings on the autonomy subscale were significantly lower than published norms (>1 standard deviation lower). START participants rated the START program more than 1 SD higher than the norm on five subscales (involvement, 1.63 SD; support, 1.76 SD; practical orientation, 1.20 SD; order and organization, 1.73 SD; and program clarity, 2.22 SD), whereas START staff rated the START program more than 1 standard deviation higher on four subscales (support, 1.44 SD; spontaneity, 1.29 SD; order and organization, 1.10 SD; and program clarity, 1.49 SD). Staff rated the START program 1.14 SD lower than the norm on staff control. Both staff (-1.52 SD) and participants (-1.10 SD) rated START significantly lower than hospital norms on the anger and aggression subscale.

Discussion

The main finding of this study was that both patients and staff members rated the treatment environment at START more favorably than the hospital on five of ten WAS subscales. The results regarding patients are strengthened because there was random assignment to treatment condition and because there were no significant differences in demographic or clinical characteristics between the two groups of patients, thereby reducing the likelihood that these findings were due to differences in patient characteristics or treatment preferences. To our knowledge this is the first study of a program that is an alternative to a hospital that reported both patient and staff perceptions of the treatment environment on a standardized measure that also randomly assigned patients to treatment condition. More favorable participant ratings of the START treatment environment are consistent with previous findings of more favorable client satisfaction for START than hospital treatment ( 1 , 2 ).

Our hypothesis of more favorable staff perceptions of the treatment environment at START was partially supported, in that START staff responses were more favorable than hospital staff responses on five subscales (involvement, practical orientation, personal problem orientation, order and organization, and program clarity), including two of the three subscales specifically targeted in our hypotheses. However, we found no differences between the ratings of hospital and START staff on four subscales, and hospital staff had significantly higher ratings than START staff on the anger and aggression subscale.

The anger and aggression subscale is considered somewhat problematic and has been the subject of considerable discussion in the literature ( 14 , 20 , 24 ). Moos ( 10 ) discusses how higher elevations of the anger and aggression subscale can often be favorable, because they indicate a level of comfort with the expression of such feelings and thoughts. However, higher scores on the anger and aggression subscale are not always viewed as positive, because they may be associated with angry acting-out behavior and may be stressful for more disturbed patients. Several studies have found that lower perceived levels of anger and aggression are beneficial for more severely impaired patients ( 24 , 25 , 26 ). However, hospital staff and participant ratings of anger and aggression in this study both fall within ±1 SD of hospital norms.

Over the years, various WAS researchers have investigated patient-environment match to determine characteristics of treatment settings considered optimal for different types of patients. In addition to the finding that lower levels of anger and aggression are beneficial to more impaired patients, other environmental factors found to be helpful for more disturbed patients in multiple studies are high levels of support, problem orientation, and order and organization ( 25 , 26 , 27 ). Both patient and staff WAS profiles for START more closely resemble recommended treatment environments for more disturbed patients, with higher than average scores on the support, problem orientation, order and organization subscales and lower ratings on the anger and aggression subscale ( Figure 1 ). This finding suggests that the START environment may be more beneficial to seriously disturbed patients in need of acute psychiatric care.

Participants were all voluntarily admitted veterans under age 60 and were overwhelmingly male. The results, therefore, may not generalize to other groups, including involuntarily hospitalized patients, older veterans, females, older patients, and to other nonveteran populations in need of acute psychiatric care. In addition, this study compared patients treated at START with those receiving care in one psychiatric hospital. The lower WAS response rate among hospital staff (N=15, 30% response rate for hospital staff, versus N=75, 67% response rate for START staff) leaves open the possibility that staff who responded were not fully representative of all hospital staff. The hospital required an additional approval of the institutional review board and the union before the WAS could be administered to hospital staff and also required that responses of hospital staff be anonymous, which precluded collection of information on staff respondents. The low response rate from hospital staff may have been due to this delay and our inability to identify and follow up with specific staff members who had not completed the WAS. Another possibility based on anecdotal reports is that the hospital's status as a well-known research and training institution may have inadvertently yielded a "research burnout" factor among its staff, resulting in a lower level of participation.

Furthermore, the inpatient setting also included patients who were not in the study and were involuntarily hospitalized, some for aggressive behavior. The START programs do not treat patients admitted involuntarily, and this difference in patient groups may partially explain the higher scores on the anger and aggression subscale reported by hospital participants and staff. Nonetheless, these findings, particularly when combined with previous findings of similar or better outcomes, greater satisfaction, and lower cost, seem to have important practice and policy implications. Finally, this study was based on self-report instruments. Despite these limitations, the findings were strengthened by the fact that both patient and staff perceptions were assessed using a standardized measure and that patients were randomly assigned to the two treatment options.

Conclusions

Patients randomly assigned to START and START staff reported more favorable perceptions of the treatment environment than did patients and staff at the hospital. This finding is consistent with previously published findings indicating that patients treated in START programs have greater levels of satisfaction than those receiving hospital-based acute psychiatric care. Because this study compared the START model with only one hospital, multisite research is needed to confirm and further develop these findings. The expanded use of hospital alternative programs, such as START, that demonstrate lower cost, equivalent or better outcomes, and more favorable ratings of satisfaction and perceived treatment environment may have enormous potential benefits for both patients and mental health care systems.

Acknowledgments and disclosures

This study was funded by grant PCC-98015-1 from the Health Services Research and Development Service of the Veterans Health Administration.

The authors report no competing interests.