Effects of Job Development and Job Support on Competitive Employment of Persons With Severe Mental Illness

Although the principles, operating practices, and outcomes of employment interventions for persons with severe mental illness have been investigated (1,2), few studies have explored direct effects of specific program services on employment outcomes. Services are groups of activities that have one or more common goals. Job development is one example of a service. Principles for employment programs have been identified, and the evidence of their effectiveness has been reviewed (2,3,4). Principles focus on the manner in which services are provided. Stronger empirical evidence exists for some principles—a focus on competitive employment, eligibility based on client interest, and rapid job search—than others, such as integration of vocational and clinical services, attention to consumer preferences, time-unlimited and individualized support, and benefits counseling (2,3,4).

Several recent studies and reviews have highlighted the effectiveness of supported employment services for people with severe mental illness (1,2,5,6,7,8,9). Studies have shown that supported employment helps consumers with severe mental illness obtain competitive employment (2) and that higher fidelity to supported employment is positively correlated with competitive employment rates (10) and other employment outcomes (3,4). However, studies have shown that job tenure is often short (11). One formulation of supported employment is individual placement and support (12), which has received the most systematic research (2). However, the principles and operating practices of supported employment, like those of other employment interventions, are embodied in a number of different types of program and delivery mechanisms. Less is known about the effectiveness of specific services. Thus employment programs have been referred to as "black boxes" (13). The possibility remains that certain employment services work better than others, independent of program context.

Programs such as employment interventions, which provide many different vocational and nonvocational services, have "complex arrangements and soft boundaries" (14). One question about replicating socially complex interventions generally and evidence-based ones specifically that has yet to be answered for most interventions is whether all the services and complex arrangements currently included in the interventions are necessary to achieve desired outcomes. To better understand this issue, we examined data from a two-year, seven-state demonstration program, the Employment Intervention Demonstration Program (EIDP) (15), which was funded by the Substance Abuse and Mental Health Services Administration. Specifically, we examined types and amounts of services received in relation to the achievement and retention of competitive employment (termed "dismantling" research) (16). We implemented dismantling analyses to increase our understanding of factors that improve employment outcomes and to identify more cost-effective and less demanding approaches to achieving these outcomes.

The study reported here is unique in that it enabled us to look at specific services across multiple sites and program models. This report focuses on two employment services hypothesized by the EIDP investigators to be particularly effective in obtaining and retaining a competitive job—job development and on-site job support, respectively. We focused on competitive jobs because they best reflect recovery and community integration, two objectives of community care for persons with severe mental illness (17). Our study differs from other studies of the EIDP that have been reported in that sites focused on site-specific primary outcomes that have no relation to this meta-analytic study of job development and job support and their relationship to employment outcome (5,18,19). Reports from the EIDP Coordinating Center pool data for the sites and look at the data from a program level and not at the level of specific services (20,21,22).

The receipt of a service can be measured in terms of penetration (whether or not the service is received) and amount (how much of the service is received). We focused on service penetration, because we had no theoretical or empirical basis for distinguishing amounts of service that were too little, appropriate, or too much.

We hypothesized that participants who received job development would be more likely to acquire competitive jobs than those who did not receive it. Another hypothesis was that participants who received job development sooner would likely be more prepared for work and more likely to acquire competitive jobs than those who received it later. With respect to job support, we hypothesized that participants who received job support would work more months and hours than those who did not receive job support.

Methods

Data used in the analysis came from seven sites of the EIDP study. The sites were in Arizona, Connecticut, Maine, Maryland, Massachusetts, South Carolina, and Texas. The sites, conditions, and methods are described elsewhere (15). Individuals were included in the study if they were 18 years or older at enrollment, were willing and able to provide informed consent, had a DSM-IV diagnosis of mental illness, and were unemployed.

Procedures

Interview assessments with EIDP participants elicited information about demographic characteristics, previous employment, current income, clinical indicators, and other relevant information at the time of study enrollment (baseline) and at six-month follow-up intervals for 24 months. Some data were collected on a weekly basis, including vocational data on the nature of the participant's job (among those employed), job title, wages earned, hours worked, benefits received, and level of workplace integration. The sites also collected data on the types of vocational and clinical services received by EIDP participants. The EIDP common protocol and data collection methods are described elsewhere (23). Sites received approval of human subjects' protection and confidentiality safeguards required for recruitment, informed consent, and data management procedures from their organizations' institutional review boards. Recruitment of study participants took place between February 1996 and May 2000, and all participants received monetary compensation for each interview.

Analysis sample

A total of 1,340 persons from the seven sites were included in this analysis. Persons doing paid work at baseline (N=28) and those with baseline but no follow-up employment data (N=98) were excluded.

Measures

Employment outcome variables. Job development was defined as direct or indirect contact with potential employers or networking with individuals or organizations that had job information. The outcome variable for the job acquisition analyses was whether or not the individual obtained competitive employment. In this study, the criteria for competitive employment included four components: pay at minimum wage or higher, a job located in a mainstream integrated setting, a job that was not set aside for mental health consumers, and a job that is held independently (that is, not controlled by a service agency). The first two criteria match the Department of Labor's definition of competitive employment (24), and the second two were developed to differentiate competitive employment from transitional employment placements or enclave jobs. Under this definition, transitional employment placements, such as those provided through International Center for Clubhouse Development (ICCD) clubhouses, were not categorized as competitive employment. The final report for the Massachusetts site of the EIDP provides for further details of the site-specific analysis that included such transitional placements (25). For each month in the study, EIDP participants were categorized as having obtained a competitive job (coded as 1) or not (coded as 0). Jobs were excluded if they were temporary and lasted less than five days. Participants were dropped from the analysis for the study months after they obtained a competitive job.

Job support was defined as on-site counseling, support, and problem solving. Three outcome variables were selected for the job retention analyses. First, total duration of the first competitive job in months was calculated by counting the number of days from the start date to the end date and dividing by 30.4 (the average number of days in a month). Second, total hours worked in the first competitive job was calculated. Third, whether the participant retained the job for an entire month was documented. For each month, participants were categorized as being still employed at the end of the month (coded as 1) or not (coded as 0). A total of 503 jobs were included in the analyses. This sample excluded jobs that began less than nine months before data tracking ended (N=72) to maximize the number of jobs analyzed but to exclude arbitrarily truncated jobs.

Vocational services variables. In addition to job development and job support, eight other types of vocational services were tracked: vocational assessment and evaluation; client-specific collaboration with an employer; vocational support groups; collaboration with family or friends; vocational treatment planning and career development; off-site skills training or education; off-site vocational counseling; and transportation. Definitions for these other services are available elsewhere (26).

Demographic and clinical variables. Because participants were not randomly assigned to services, we needed to explore the possibility of selection bias and potential confounders. Demographic and clinical variables explored to control for selection bias were gender, race, previous work experience, a diagnosis of schizophrenia, a diagnosis of a substance use disorder, education, marital status, children living in the same household, Supplemental Security Income (SSI) and Social Security Disability Income (SSDI) status, work motivation (measured by a scale created by the EIDP steering committee), Positive and Negative Syndrome Scales (PANSS) scores (27) (a measure of psychiatric symptoms), and hours in nonvocational services. These demographic and clinical variables were included on the basis of a review of the literature by EIDP investigators to identify characteristics of participants that might be associated with vocational outcomes. Other potential covariates were excluded from the EIDP study because of their high correlations with other covariates or lack of variance across the sample. To identify potentially biasing demographic and clinical variables, we conducted regression analyses to determine which variables were associated with both service group membership and outcomes.

We postulated two potential confounders of service effects. One was the receipt of vocational services in addition to the services studied (multiple vocational services). A second was the degree to which programs integrated vocational and clinical services. The EIDP Level of Integration Scale, which ranged from 0 to 100, was used, and programs that scored above 50 were categorized as having integrated services; those that scored below 50 were categorized as having nonintegrated services. Further information about the EIDP Level of Integration Scale is available in another report of the EIDP project (15). Programs that were more integrated were similar to supported employment programs. We implemented analyses to test whether these variables altered findings for job development and job support.

Data analyses

Random-effects meta-analyses were fit to the data over multiple sites. Random-effects models consider within- and between-site variability and adjust the effect sizes to account for sample size and site variability. This method makes detecting statistical significance harder if there are between-site variations. The meta-analysis of multisite trials is well established in clinical research (28,29) and adds clarity to other analyses of the EIDP data (5,18,19,20,21,22). The analyses provide a synthesis of the results over all sites controlling for and clearly showing the effects of site and selectively testing the importance of differences in participant mix to give overall effect sizes for the entire study. Cochran's Q statistic was used to examine variability between sites; a significant Q rejects the null hypothesis of homogeneity and indicates that variability among the effect sizes is greater than what would be expected from subject-level sampling error alone (30). All our analyses showed consistency between sites.

Effect sizes for the job acquisition and job retention variables were calculated by using Comprehensive Meta-Analysis (31) statistical software. All hypotheses were two-tailed, and the standard p value of p<.05 was used for rejection of the null hypothesis. Because participants were not randomly assigned to services, we constructed multiple groups to test the possibility that selection and contextual effects were responsible for our findings. Thus we refer to our dismantling data analytic approach as "constructed comparisons" meta-analysis. In this method successive groups are constructed to investigate potentially confounding variables. However, there is no way to completely rule out the possibility of selection bias or potential confounders that are attributable to unobserved characteristics.

Relative probability of acquisition and retention of a job. To examine effects of job development and job support on acquisition and retention of competitive employment, the meta-analyses we conducted compared the relative probability of job acquisition or retention given the participant's previous receipt of job development or job support. The relative probability of obtaining or retaining a job was defined as the ratio of the percentage of participants who received a service and who had a desired outcome to those who did not receive the service but who had a desired outcome.

Methods for analyzing data for job development and job support are described separately below.

Job development and overall effects. To rule out "reverse causality"—for example, participants who already had jobs received job development services for career advancement—we carried out analyses in which job acquisition was analyzed as a function of previous receipt of job development. We postulated that the effects of job development on job acquisition would be seen in the month that the service was received or the month immediately afterward, which allowed us to analyze data for individual months and for multimonth periods. Because individuals who did not acquire a job would appear more than once in multimonth analyses, we computed weighted monthly averages for persons who obtained competitive employment and who did and did not receive job development for multiple months. The inputs into the meta-analyses were monthly average employment rates for participants who received and did not receive job development in that month or the previous month (weighted by total N for the month) and average monthly number of participants who did and did not receive job development.

Job development and effects of time. Because one of our hypotheses about the effects of job development involved time, time periods were constructed on the basis of the number of months a participant was in the program. The months were divided into three groups: months 1 to 6, months 7 to 12, and months 13 to 24. Time periods were chosen in order to have enough participants in each period. Separate meta-analyses examined outcomes for each period and across all months.

Job development and effects of multiple services. To examine the potentially confounding effects of multiple vocational services, a meta-analysis was conducted to examine whether effects found for job development were maintained when participants received multiple services.

Other unplanned analyses were conducted to compare the relative probability of obtaining a job while examining the effect of other specific vocational services and job development on job acquisition. Separate analyses compared participants who received only job development services in a given month with those who received only the comparison vocational service.

Job development and effects of program context. To examine the potentially confounding effects of integrated services, meta-analyses were conducted to determine whether the effects found for job development were maintained when participants who received job development in the context of integrated services were compared with those who received integrated services but not job development. Analyses were conducted to investigate the role of job development in programs that were not integrated.

Job support and overall effects. The overall effects of job support were examined by meta-analyses over sites. The inputs into the meta-analyses for total duration of employment and total hours were the overall mean and the number of participants who did or did not receive job support during their first competitive job. The inputs for the job retention variable were the rates of participants who were still employed at the end of the month for those who did or did not receive job support in that month or in the previous month (weighted by total number of participants for the month) and the average monthly number of participants who received and did not receive job support.

To again rule out reverse causality—the possibility that people with longer-lasting jobs would have a higher probability of receiving unneeded job support, which would result in an association between the two variables—we carried out analyses in which job retention was analyzed as a function of previous receipt of job support. We postulated that the effects of job support on job retention would be seen in the month the support was received or the month afterward, which again allowed us to analyze data for individual months and for multimonth periods. Because retention of a job would appear more than once in multimonth analyses, we computed weighted monthly averages for participants who retained competitive employment and did or did not receive job support for multiple months.

Additional analyses to explore potential selection or confounder variables were not conducted because no significant difference was found in the relative probability of retaining competitive employment after the receipt of job support.

Job support and effects of program context. The effects of program context could not be tested in the job support analyses, because the number of people who received job support in programs with nonintegrated services was low. Of the 103 jobs held by participants in programs with nonintegrated services, job support was provided for only 15.

Results

Baseline characteristics

The mean±SD age of the 1,340 participants in the study was 38.4±9.3 years, and 716 participants (53 percent) were men. Less than half the participants were white (653 participants, or 49 percent). A total of 408 participants (30 percent) were black, 205 (15 percent) were Hispanic, 35 (3 percent) were American Indian or Alaskan Native, ten (.7 percent) were Asian, and 27 (2 percent) were classified as other. A total of 482 participants (36 percent) had less than a high school degree, and 468 (35 percent) had no work experience in the five years before they entered the EIDP program. A total of 1,196 participants (89 percent) were not married, and 1,013 (76 percent) had no children under the age of 18 years living with them. A total of 465 (35 percent) received only SSI, 330 (25 percent) received only SSDI , and 167 (13 percent) received both SSI and SSDI.

All participants received a DSM-IV axis I primary diagnosis. The most prevalent diagnosis was schizophrenia (429 participants, or 32 percent), followed by major depression (265 participants, or 20 percent), schizoaffective disorder (247 participants, or 18 percent), bipolar disorder (213 participants, or 17 percent), psychotic disorder (44 participants, or 3 percent), and other (95 participants, or 7 percent). More than a third of the participants (498 participants, or 37 percent) had a primary or secondary diagnosis of a substance use disorder. The mean PANSS scores were 14.4±5.1 on the positive subscale, 16.3±5.5 on the negative subscale, and 34.3±8.8 on the general subscale (31).

Employment services received

Job development was the service most used by EIDP participants (840 participants, or 58 percent). On-site job support was one of the least used employment services (423 participants, or 29 percent). More than half of the participants (812 participants, or 56 percent) received vocational assessment and evaluation and off-site job skills training and education (747 participants, or 52 percent). Half of the participants (726 participants, or 50 percent) received vocational treatment planning or career development. Fewer than half of the participants (707 participants, or 49 percent) received off-site vocational counseling, collaboration with family and friends (558 participants, or 39 percent), collaboration with employers (463 participants, or 32 percent), and vocational support groups (439 participants, or 30 percent). Transportation was the employment service least used (410 participants, or 28 percent).

Job development

Competitive jobs obtained. Of the 1,340 participants, 575 (43 percent) obtained one or more competitive jobs. A total of 362 of the 575 participants (63 percent) obtained only one competitive job.

Effects of job development on job acquisition.Table 1 shows the relative probability of obtaining competitive employment in an average month on the basis of receipt of job development. The table presents the average number of participants who received job development per month at each site, the average weighted monthly percentage of those who received job development and obtained a job, the average number of participants who did not receive job development per month at each site, the average weighted monthly percentage of those who did not receive job development and obtained a job, the random effect for each individual site, and the overall effect for all sites when the random-effects model was used.

As shown in Table 1, the relative probability of obtaining competitive employment was significantly greater for participants who received job development. This finding was consistent within and between sites, and the overall effect across sites was also significant. The effect was consistent at all sites and statistically significant at two sites. This pattern of findings shows the benefit of multisite studies when samples at individual sites are small. Individuals who received job development were almost five times more likely to obtain competitive employment than individuals who did not receive job development.

Effect of time period.Table 2 shows that effect sizes were similar for different time periods.

Effects of demographic and clinical variables. Having previous employment was the only variable significantly associated with ever obtaining a competitive job and ever receiving job development (β=1.79, p<.001 and β=1.34, p=.02, respectively). Thus we implemented subgroup analyses only for this variable. Results in Table 2 show that regardless of work experience, receipt of job development increased the likelihood of obtaining competitive employment and that individuals with no previous work experience had virtually no chance of acquiring a competitive job without job development.

Effects of multiple services.Table 2 also shows that the effects found for job development were maintained for participants who received multiple services but were not statistically significant.

Results not shown here indicated that participants who received only job development had a greater likelihood of obtaining competitive employment than those who received only a comparison vocational service, although none of these differences were statistically significant. It is possible that the lack of significant findings is attributable to the small numbers of participants who received only one service.

Effects of program context.Table 2 also shows that job development helped participants obtain competitive employment in both integrated and nonintegrated services. The effect was stronger in nonintegrated services than in integrated services, because there was such a small likelihood of getting a competitive job without job development when services were not integrated. In addition, provision of job development is rare in nonintegrated services.

Job support

Effects of job support on job retention. Data presented in Table 3 show a significant association between number of months in the first competitive job and receipt of job support. The association with total hours worked approached significance. However, these analyses did not take into account whether the job support was delivered before job retention, leaving open the possibility that job retention allowed persons to receive job support rather than vice-versa.

As shown in Table 4, no significant difference was found in the relative probability of retaining versus losing competitive employment for the entire month after the receipt of job support. Additional analyses not shown indicated that a significant overlap existed between the participants who received job support and those who received vocational counseling.

Discussion and conclusions

Job acquisition and retention are important outcomes to consumers (1) and to policy makers (32). If resources are limited, programs might decide to focus more on job development than on job support. These nonexperimental data do not establish such modifications as an evidence-based course of action, and the resulting services would not be supported employment. However, these data do underscore and support the importance of job development for job acquisition.

On average, receipt of job support was positively correlated with the number of months and hours worked in the first competitive job. However, the analyses that took the time of intervention into account did not rule out the possibility of reverse causality. The weaker results for job support go along with the weaker effects of employment interventions on job retention compared with job acquisition that have been found in other studies (11). We need to experiment with services specifically focused on this admittedly difficult outcome and to design studies appropriate for this task. We knew when individuals needed job development, because they had no job. We had no equivalent measure of a need for job support. Thus we were unable to determine the effects of job support only on persons judged to need this service.

It is important to point out that although the analyses show that job development was effective for the acquisition of competitive employment for 57 percent of persons who received the service, 43 percent of EIDP participants (N=318) who received job development services were unable to obtain competitive employment. How these persons should be served is another important question to be addressed.

It is important to note other limitations to this study. Participants were randomly assigned to programs but not to services. Although we examined the possible effects of selection bias and potential confounders, the possibility of bias and confounders remains. Ideally, randomized studies or statistical modeling should be conducted to confirm the findings presented here. Also, more studies are needed to establish the generalizability of our findings to different subgroups of people with severe mental illness and currently available vocational rehabilitation programs. In addition, the analyses were limited to services received during the first competitive job. It is possible that services received during other competitive jobs may have different effects on job acquisition and job retention. Indeed, in the EIDP study, second and third competitive jobs lasted longer on average than first competitive jobs.

It is also important to point out that the service data were collected differently at the sites. Some sites collected the data for the study by using the definitions provided by the Coordinating Center; most others had to cross-reference service definitions used in their systems to those provided. These differences in data collection may well have affected the results. Future studies of obtaining competitive employment could focus on the day from study entry to the day of beginning the first competitive job. Future studies could also conduct higher-order Markov analysis that would allow determination of whether the accumulated amount of job services is related to the probability of a person's converting from never being employed to obtaining a job for the period that begins at the start of the study and ends at job acquisition.

This study only begins to unpack the black box of vocational rehabilitation programs. The effects of patterns of services received (as suggested by cluster analysis) on job acquisition and retention should be explored. Our analysis of the confounding effects of multiple services left this door ajar. Future studies could group job support and vocational counseling into a composite category. In addition, when the effects of job support were examined, one site had negative results compared with the other. This and other site variation might be explored to gain a deeper understanding of effective services.

Finally, given the importance of return-to-work initiatives for persons with severe mental illness, this study provides support for prioritizing supported employment and other programs that emphasize individual job development.

Acknowledgments

The research was funded by cooperative agreement SM51820 from the Center for Mental Health Services of the Substance Abuse and Mental Health Services Administration.

Dr. Leff, Dr. Camacho-Gonsalves, and Ms. Raab are affiliated with the Human Services Research Institute, 2269 Massachusetts Avenue, Cambridge, Massachusetts 02140 (e-mail, [email protected]). Dr. Cook is with the mental health science research programs department at the University of Illinois at Chicago. Dr. Gold is with the department of psychiatry at the Medical University of South Carolina in Charleston. When this work was done, Dr. Toprac was affiliated with the Texas Department of Mental Health and Mental Retardation in Austin. She is currently a private consultant. Dr. Blyler is with the Center for Mental Health Services in Rockville, Maryland. Dr. Goldberg is with the department of psychiatry at the University of Maryland in Baltimore. Dr. McFarlane is with the Center for Psychiatric Research of the Maine Medical Center in Portland. Dr. Shafer is with the community rehabilitation program at the University of Arizona Health Sciences Center in Tucson. Dr. Allen is with Babson College in Babson Park, Massachusetts.

Enlarge table

Enlarge table

Enlarge table

Enlarge table