Cost-Effectiveness of Two Vocational Rehabilitation Programs for Persons With Severe Mental Illness

The measurement of cost-effectiveness has become imperative in this era of scarce resources and evidence-based medicine. Research on different models of vocational rehabilitation for persons with psychiatric disabilities has generally focused on direct work-related outcomes. Recent research has shown some advantage of supported employment over traditional train-and-place models (1,2). However, as has been noted by other researchers (3,4), few cost-effectiveness studies of vocational rehabilitation programs have been undertaken.

A largely unsolved question raised in the study of mental health services and vocational rehabilitation is the relationship of improved vocational status to mental health treatment and cost. The "improved functioning" theory of cost offset posits that participation in supported employment improves clients' functioning, which results in less need for services and lower costs for mental health care. The "substitution theory" holds that when clients enroll in a supported employment program, they disenroll from another program. Costs thus remain the same or decline. However, it is also possible that employment and the associated stress could increase the cost of mental health care (5).

Previous work suggests some potential for cost-offset when supported employment programs are added to mental health services. Bond and colleagues (6) compared accelerated with gradual supported employment in a group of 74 clients randomly assigned to either condition; the accelerated condition cost 27 percent less because of reductions in mental health costs. The conversion of two rehabilitative day treatment programs to supported employment programs resulted in improved vocational outcomes with no increase in costs (7). Another randomized controlled comparison of a supported employment model and a more traditional train-and-place model found no net monetary benefits from the experimental supported employment program despite improved outcomes for participants (8).

Drake and associates (9) evaluated the effectiveness of two vocational programs for persons with severe mental illness, individual placement and support (IPS) and enhanced vocational rehabilitation (EVR). After 18 months, participants in the IPS program had spent significantly more hours and weeks in competitive employment than EVR participants. However, no differences in combined earnings were found because of the high rates of sheltered work among EVR participants.

This study addressed whether the advantage of IPS in terms of competitive employment comes at a greater cost. It extends the findings of previous reports (9) by evaluating the cost-effectiveness of individual placement and support and enhanced vocational rehabilitation.

Methods

Study group and design

All study participants met the enrollment criteria of severe mental illness, unemployment, willingness to give informed consent, and absence of memory impairment or medical illness that would have prevented them from being employed or participating in research interviews. Individuals were considered to have a severe mental disorder if they met criteria for schizophrenia, schizoaffective disorder, bipolar disorder, recurrent major depression, or borderline personality disorder and had two years of impaired role functioning. Participants were recruited at a mental health agency in southeast Washington, D.C., that serves persons who need intensive case management, usually because their psychiatric disorders are complicated by homelessness, a comorbid substance use disorder, or HIV infection.

During an 18-month period from January 1994 to July 1995, all agency clients were encouraged to attend informational groups, and interested attendees were screened for eligibility by the project director (9). Random assignment was stratified according to work history—more than one year of continuous employment in a previous job. A total of 76 participants were assigned to IPS and 76 to EVR. However, two of the IPS participants dropped out during the study. Nonparametric tests revealed a near-significant trend of less hospitalization for the EVR group in the period before the study. Additionally, we reviewed our data and excluded an "outlier" from the IPS group because IPS was not considered to be the cause of this person's costs or effects. He had not engaged in competitive work in the year before the study, was hospitalized for more than 300 days during the study period, and had virtually no participation in the IPS program.

Interventions

Individual placement and support was developed to provide supported employment services for people with severe mental disorders in community mental health centers (10). IPS integrates mental health and vocational services with the addition of an employment specialist to the multidisciplinary case management team. IPS employment specialists help clients search for jobs rapidly. After employment is secured, the specialists provide individualized, follow-along supports such as counseling, transportation, or intervening with an employer. Services are provided as needed and with no limits. Three employment specialists were hired to implement IPS. The IPS program was implemented according to a manual (11) and was monitored by the research team. Ratings indicated that the IPS program consistently scored within the "high fidelity" range on the IPS Fidelity Scale (12).

The comparison condition, EVR, consisted of vocational rehabilitation services provided by several well-established agencies recommended by the District of Columbia Rehabilitation Services Administration. The program was considered "enhanced" because an extra vocational rehabilitation counselor was placed in the Rehabilitation Services Administration office to ensure rapid and assertive linkage with service vendors to avert the problem of dropout during the referral process. The vocational rehabilitation counselor monitored participants monthly, and if clients were dissatisfied with the program to which they were assigned or dropped out of vocational services, the counselor attempted to link the participants with another agency.

It is important to note that all of the EVR agencies endorsed competitive employment as their goal but used stepwise approaches that involved prevocational experiences, primarily paid work adjustment training in sheltered settings.

Assessments

Participants received an extensive assessment as part of the parent study (9). Major assessments were conducted at baseline and at six-, 12-, and 18-month follow up points when research interviewers completed the Employment and Income Review (13). Employment specialists in both programs assessed employment weekly. Research interviewers asked participants about employment every two months.

The primary outcome, competitive employment, was defined as work in the competitive job market in integrated work settings at prevailing wages with supervision by personnel employed by the business (14). Competitive employment was chosen as the main outcome even though other vocational opportunities, such as sheltered work, were available, because there is evidence that people with psychiatric disorders prefer competitive employment (15,16). Community integration is another advantage of competitive work. Participants work alongside persons without disabilities and receive comparable pay. Competitive employment counters the pattern of segregation that often accompanies mental illness.

Costs

We adopted the cost perspective of a hypothetical single payer of all treatment costs (17). Like most care systems, the one we studied involves multiple public payers and agencies, but focusing on any one of those entities would limit the generalizability and relevance of the study for other systems of care. All costs were calculated in fiscal year 1995 dollars. Overall costs were measured as the sum of direct costs for mental health and vocational rehabilitation services for the entire 18-month study period.

Mental health costs had both inpatient and outpatient components. Outpatient mental health costs included costs for medication management, intensive case management, group therapy, and family therapy. Overall costs were calculated by multiplying the number of hours of service by the hourly unit cost estimates for each service type.

Because the same agency was responsible for outpatient services to both IPS and EVR participants, unit costs for mental health services were the same for both study conditions. The unit costs for outpatient services were calculated by using data abstracted from the agency's audited 1995 financial statements. Direct and overhead costs for medication management, intensive case management, and family therapy were divided by the estimated number of hours of service reported in daily service logs completed by all staff (Table 1).

Estimates of the costs for group therapy were derived from the unit cost for intensive case management—$102.87. Staff case managers conducted groups of eight patients for 1.5 hours. Thus the unit cost for group therapy was $19.29, or $102.87 multiplied by 1.5 hours and divided by eight.

The unit cost for inpatient mental health services was calculated by multiplying the number of hospital days for each participant by an average hospital cost per day derived from Medicare Cost Report data for two of the most heavily used hospitals. The average hospital cost per day—$477.89—was determined by dividing the total cost for psychiatric beds by the number of psychiatric bed days in fiscal year 1995.

Vocational rehabilitation costs were calculated differently for each program. For IPS participants, IPS workers completed service logs indicating the number of service hours each participant received. The number of hours per participant was multiplied by an hourly unit cost. The unit cost was computed by using the two-year ratio of overall costs for IPS vocational rehabilitation services reported in cost statements for fiscal years 1994 and 1995, or $209,000+$209,000= $418,000, divided by the number of hours spent with clients reported in service logs for both years, 3,240.3+2,677.1=5,917.4. A comparison of the resulting unit cost—$418,000 divided by 5,917.4= $70.64—with year-specific unit costs (that is, fiscal year 1994 versus fiscal year 1995) indicated only modest start-up costs. Although the study continued for part of 1996, the same level of IPS activity did not remain the same. Fiscal year 1996 data reflect a reduction in the level of services delivered to participants.

For EVR participants, vocational rehabilitation costs were the sum of three costs: the cost of the coordinator, the cost to the Rehabilitation Services Administration, and additional assessment costs routinely used by the Rehabilitation Services Administration, such as the costs of medical and psychological evaluations. The coordinator cost included both the coordinator and supervision. The coordinator had a senior supervisor from the Rehabilitation Services Administration and worked to match EVR participants with community vocational rehabilitation programs (vendors). We considered the annual salary, fringe benefits, and overhead for the coordinator and for one-seventh of the supervisor's time, because the supervisors typically worked with seven frontline workers. The estimated annual cost for the coordinator was $61,353. We assumed that over a three-year period, the cost for the coordinator was three times that amount, or $184,059.

The coordinator logged all time dedicated to each study participant. Using methods similar to those used to calculate IPS costs, we developed an hourly unit cost based on the coordinator's productivity in the first 18 months of the study to avoid an underestimate of the coordinator's productivity in the third year. During the first 18 months, EVR participants used 1,247 hours of vocational rehabilitation services that were provided by the coordinator. Thus we estimated that over the three years, the coordinator provided 2,494 hours of services (2×1,247). We then divided the three-year cost by the estimated three-year service use to obtain a unit cost for the coordinator of $73.80 ($184,059 divided by 2,494). Individual use of each type of service was multiplied by the unit cost to estimate the total costs for enhanced vocational rehabilitation. The additional start-up costs were modest, as they were for IPS.

For the second component of the costs for EVR, we used actual payments from the Rehabilitation Services Administration to individual vendors for services provided to the EVR group. The third EVR cost component included additional assessments such as medical and psychological evaluations arranged by the Rehabilitation Services Administration. These were assigned a nominal value of $50 per service, which reflected the administration's estimate of its costs for these services.

Effectiveness

Our analyses of effectiveness included the number of hours of competitive work, weeks of competitive work, and combined earnings—that is, earnings derived from competitive and noncompetitive employment.

Data analysis

We compared the costs and effects of IPS and EVR. Because of the skewness of the data and the small sample, nonparametric statistical tests—the Wilcoxon rank sum test—were used. Overall costs were further investigated with two multiple linear regression analyses. The more parsimonious regression model estimated the effect of vocational intervention on the natural logarithm of overall costs while adjusting for the prestudy hospitalization rate. We also estimated overall costs by using a richer model that adjusted for vocational intervention, age, gender, race, substance abuse, and a diagnosis of schizophrenia as well as for the prestudy hospitalization rate.

To estimate the cost-effectiveness of IPS compared with EVR, we constructed an incremental cost-effectiveness ratio (ICER). The ICER is the statistic of interest in cost-effectiveness analysis and has been used in economic evaluations of other mental health programs (17,18,19,20). The ICER was computed by dividing the difference in average cost for each program by the difference in average effectiveness. The ICER we used estimated the increased cost associated with gaining a one-unit improvement in effect over 18 months. Because we considered three effects—hours of competitive work, weeks of competitive work, and average combined earnings—we generated three ICER estimates. We used bootstrapping methods to create 5,000 ICER estimates that were plotted on cost-effectiveness planes (scatterplot graphs) (19,20,21). In this way, we were able to assess the degree of uncertainty associated with our estimates of the ICERs. Details of the bootstrapping process are available from the authors.

Results

Service costs

The average overall costs for IPS participants were 16 percent greater than those for EVR participants—$29,087 compared with $25,119 for 18 months. However, as shown in Table 1, no statistically significantly differences in mental health costs, vocational rehabilitation costs, or overall costs were found for IPS and EVR participants. In both cost regression models, the coefficient on the IPS variable was positive but not significant. The parsimonious cost regression model, which included treatment condition and prestudy hospitalization rate, had an adjusted R² of .067 (F=1.50, df=2,146, p=.23).

The results were similar in the larger regression model that adjusted for prestudy hospitalization rate, age, gender, race, substance abuse, and a diagnosis of schizophrenia. The coefficient for IPS was positive and not significant. The adjusted R² of .069 (F=2.56, df=7,141, p=.02) indicated that the richer model picked up only a modest amount of the variation in the natural logarithm of overall costs. Had we retained data for the excluded participant who was considered an outlier, costs would not have differed significantly, but IPS would have cost on average 20 percent more than EVR.

Outcomes

As shown in Table 2, the IPS participants spent a significantly greater number of hours and weeks in competitive employment than the EVR participants. However, no differences were found in the average combined earnings of the two groups. Although information about the time spent in noncompetitive employment was not available, the findings on combined earnings provide an indication of outcomes in the area of noncompetitive employment. Participants in the two programs did not differ in global functioning, symptoms, general quality of life, and self-esteem; however, IPS participants had more psychiatric hospitalizations during the study, a finding that was consistent with baseline differences. These group differences disappeared when the baseline value was entered as a fixed covariate rather than as part of the response vector (9).

Incremental cost-effectiveness ratios

The results of analyses conducted to examine the cost-effectiveness of IPS in providing an additional hour or week of competitive employment are shown in Table 3. The ICER estimates indicated that participation in the IPS program was associated with an additional week of work—that is, an additional hour at $13 per hour—at an additional cost of $283.

The 5,000 ICER estimates were plotted on scatterplot graphs (available from the authors). In the graph for providing an additional hour of employment, the vast majority of the points (91 percent) fell in the quadrant in which IPS both costs more and provides more competitive employment. A small proportion of the points (9 percent) fell in the quadrant in which IPS costs less and provides more. The scatterplot graphs of the bootstrap estimates of the ICER for an additional week of employment were quite similar.

As shown in Table 3, the ICER estimate for combined earnings from competitive and noncompetitive employment indicated that the IPS program cost more and provided lower average combined earnings. The scatterplot graph showed a high level of statistical uncertainty associated with the estimate of the ICER for providing an additional dollar of combined earnings. In the graph, the bootstrapped estimates are scattered on the cost-effectiveness plane such that any economic interpretation is possible: IPS costs more and provides less (indicated by 47 percent of the points); IPS costs more and provides more (44 percent); IPS costs less and provides less (4 percent); or IPS costs less and provides more (5 percent). The statistical cost-effectiveness analysis thus suggests that there is a great deal of statistical uncertainty for the ICER estimate when combined earnings are used as the outcome measure.

Discussion

This study investigated alternative methods of providing vocational rehabilitation for persons with severe mental illness. We first examined the overall costs of individual placement and support compared with those of enhanced vocational rehabilitation. Although the IPS program cost 16 percent more on average than the EVR program, comparison of the means and more complex regression methods did not reveal statistically significant cost differences between the programs.

It is important to note that the higher rates of hospitalization among IPS participants before they entered the study might have introduced a bias against IPS. The difference in overall costs was accounted for by the higher inpatient costs for IPS participants. About half of the overall costs of IPS (49 percent) were related to psychiatric hospitalization, compared with only 38 percent of the costs of EVR. When hospital days were removed from the overall cost calculation, IPS actually cost 4 percent less than EVR, although this difference was not statistically significant.

We calculated the correlation between competitive work and hospitalization to explore the possibility that competitive work led to higher costs because participants who engaged in competitive work required more hospitalization because of work-related stresses. The correlation was in fact negative and small (r=-.09). Our findings thus do not support the notion that participation in the IPS program is associated with more hospitalization. On the other hand, our study did not support the notion that IPS participation is associated with less need for outpatient or case management services, because no statistically significant cost offset or substitution of IPS services for case management services was found. However, costs for case management were approximately 17 percent greater for EVR participants than for IPS participants, although the difference was not significant.

Current research indicates that supported employment approaches, such as individual placement and support, do not produce undue stress or exacerbate the symptoms of illness (22). A recent study using data from the parent study (9) compared clients in four groups, which reflected the predominant work activity of group members over 18 months: competitive work, sheltered work, minimal work, and no work. Clients in the competitive work group—predominantly IPS participants—were more likely to show improvements in symptoms, quality of life, and self-esteem than clients in the combined minimal work and no work groups. Those who had engaged mostly in sheltered work—exclusively EVR participants—showed no such advantages (23). The differences in nonvocational outcomes and the advantage of competitive work were not apparent in the overall comparison between the IPS and EVR groups, because, as in most experimental trials of employment interventions, the effects of competitive employment are diluted; generally less than half of the study group are competitively employed at any given time. Therefore, it is plausible that the higher rate of hospitalization among IPS participants at baseline—and not IPS itself—explains the greater rate of hospitalization and the associated costs during the study.

The markedly different average cost of family therapy between IPS and EVR participants is worthy of comment. The cost for family therapy for the two groups was not significantly different and contributed less than 2 percent to total costs of mental health care. However, for IPS participants the cost for family therapy was about 100 times greater than for EVR participants. The explanation is that for three of the ten IPS participants who received family therapy, the cost was $2,000; the next lowest cost for family therapy was $686. When data for these three IPS participants were removed from the analysis, the average family therapy cost for an IPS participant (based on seven participants) was $18, compared with $3 for an EVR participant (based on four participants). This difference was not statistically significant. Thus the few subjects who received any family therapy and the minimal contribution to total costs of family therapy suggest that the use and cost of this service for participants in the IPS and EVR programs do not differ.

Having analyzed costs and outcomes separately, we next computed the ICER to link program costs with program benefits. This approach enhanced our analysis by providing an estimate of the trade-offs associated with dollars invested in IPS versus EVR. The ICERs for competitive work outcomes provided estimates of the additional cost associated with an additional hour or week of competitive work. The ICER estimates suggested that compared with EVR, IPS costs an additional $13 per additional hour of competitive work and an additional $283 per additional week of competitive work. Our scatterplots, which reflected uncertainty about the ICER estimate, suggested that it is statistically highly likely that IPS both costs more and produces more competitive employment. On the other hand, an ICER using combined earnings as the outcome precluded any definitive conclusion because of the high degree of variability.

This study had numerous limitations, including its use of direct mental health costs only. We were unable to obtain information about the costs of some mental health services such as the cost for use of emergency department services and the cost of receiving a physician's care in the hospital. We used different costing strategies for the IPS and EVR programs because of the nature of the services and the use of multiple vendors. Furthermore, the estimated trade-off of $283 for an additional week of competitive employment among IPS participants may overestimate IPS costs, because hospital costs may have been misattributed to IPS. Additional limitations include the relatively small sample and the large variation in costs.

In generalizing the results of this study to actual practice, the "enhanced" nature of the vocational rehabilitation condition must be taken into account. The enhancement was probably largely responsible for the retention of EVR clients in vocational services. The enhancement probably increased both the costs and the effects of the EVR program, which produced statistical uncertainty about the overall result of the cost comparisons reported here. For our study, we adopted the perspective of a single hypothetical payer for services. However, future analysis of the IPS approach might seek to gather cost data from a societal viewpoint. In addition, a longer follow-up period might give participants in the EVR program a chance to "catch up" if they made the transition from sheltered to competitive work after the 18-month study period.

Conclusions

This study underscored the importance of choice of outcomes in evaluations of employment programs. ICER estimates indicated that individual placement and support programs both cost more and produce more competitive employment. When total earnings were evaluated as the outcome, the degree of variability in the ICER estimates prevented us from drawing a firm conclusion. Consideration of the results for competitive employment will be contingent on society's valuation of competitive employment. Is the trade-off worth it? Future studies that focus on total earnings with a larger sample may illuminate the extent to which trade-offs are necessary to generate increased total earnings.

Acknowledgments

This work was supported by grant K20-MH-01250-01 from the National Institute of Mental Health (NIMH) to Dr. Dixon and by grants MH-51346 and MH-00839 from NIMH and grant HD5-SM-51346 from the Substance Abuse and Mental Health Services Administration to Dr. Drake. Dr. Hoch is a recipient of the Ontario Ministry of Health and Long-Term Care Career Scientist Award; he also received funding from the Natural Sciences and Engineering Research Council of Canada.

Dr. Dixon is associate professor in the department of psychiatry at the University of Maryland, 701 West Pratt Street, Room 476, Baltimore, Maryland 21201 (e-mail, [email protected]). She is also affiliated with the Veterans Affairs Capitol Health Care Network Mental Illness Research, Education, and Clinical Center. Dr. Hoch is affiliated with the departments of psychiatry, family medicine, and epidemiology and biostatistics of the University of Western Ontario, London, Ontario. Dr. Clark, Dr. McHugo, Dr. Drake, and Ms. Becker are affiliated with the Dartmouth Medical School in Lebanon, New Hampshire, and the New Hampshire-Dartmouth Psychiatric Research Center in Hanover, New Hampshire. Dr. Bebout is the clinical housing director at Community Connections in Washington, D.C. and adjunct assistant professor at Dartmouth Medical School.

Enlarge table

Enlarge table

Enlarge table