Implementation of Supported Employment for Homeless Veterans With Psychiatric or Addiction Disorders: Two-Year Outcomes

There has been growing interest in helping people with mental illness participate in competitive employment by using approaches such as individual placement and support (IPS) ( 1 , 2 , 3 ). IPS emphasizes rapid job placement, a focus on competitive jobs, ongoing support without a time limit, client choice of jobs, integration of vocational support and clinical care, and openness to all who want to work, regardless of clinical status or past work experience ( 4 ).

Experimental studies have robustly demonstrated the effectiveness of IPS for people with severe mental illness ( 2 , 4 ), and several studies of real-world implementation have shown significant cross-sectional correlations between employment success and fidelity to the IPS model ( 5 , 6 , 7 , 8 , 9 ). Experiences with IPS programs have stimulated broad efforts to disseminate evidence-based practices ( 10 , 11 ).

Although multiple dissemination efforts are under way, analyses on the benefits of efforts to disseminate IPS have been on cross-sectional and site-level data, rather than on longitudinal information on individual clients. To implement IPS in health care systems with limited previous experience, sustained and individualized training programs may be needed to realize potential client benefits. Few studies have demonstrated that IPS, or other evidence-based practices for that matter, can both be implemented in a system that lacks prior experience with the model and yield outcomes for comparable cohorts that are superior, over several years, to those of a comparison group for which this intervention was not available.

In 1987 the Department of Veterans Affairs (VA) implemented a series of national programs for homeless veterans diagnosed predominantly as having substance abuse disorders, many of whom also have psychiatric disorders ( 12 ). Although these programs did not include a vocational component, VA's compensated work therapy (CWT) program has long provided veterans with an opportunity to work in sheltered workshops or in transitional employment programs either on VA grounds or with community employers ( 13 , 14 ). In contrast to IPS, CWT offers a limited choice of jobs; jobs are "owned" by the program, not by the veteran; and until recently, supported employment could not be provided because it did not fall with VA's statutory definition of medical care.

In 2000 a demonstration was implemented at nine VA programs for homeless veterans. Funds were allocated to hire and train an employment specialist who would work within each program to provide IPS services to this population, which, unlike IPS demonstrations in other organizations, included substantial numbers of clients with primary addictive disorders. Implementation was evaluated both with client-level service delivery and with program-level fidelity measures. Individual client outcomes were assessed with a pre-post nonequivalent control group design. A cohort of 30 veterans newly entering each program who expressed an interest in work was targeted for recruitment at each site before the IPS employment specialist was hired and trained. These clients (phase 1 cohort, N=308) were followed and interviewed quarterly for two years. After the employment specialist was hired and IPS training was completed, a second cohort of veterans was recruited, offered IPS, and followed quarterly for two years (phase 2 cohort, N=321).

This article describes the training approach and presents data on the implementation of IPS. It also compares employment and other outcomes between the pre-IPS and IPS cohorts (phase 1 and phase 2). Because group assignment was not conducted at random, we first compared the groups at the time of program entry on a broad range of baseline measures and used multivariate analysis to adjust outcomes for characteristics that were significantly different between the two cohorts at baseline. The principal hypothesis was that after the analyses adjusted for baseline differences, the cohort that was offered IPS services would have superior outcomes for days of competitive employment, compared with the comparison cohort.

Methods

Program design and training

Each of the nine VA medical centers hired an employment specialist to implement the IPS model of supported employment within a specialized program for homeless veterans. A training program was developed in accordance with research showing that active approaches to shaping provider behavior—such as social modeling, individual case reviews, feedback with timely performance data, and demonstrations of top management support—are more likely to shape behavior than lectures by experts and other passive approaches ( 15 , 16 ).

Training began with a one-day, on-site, face-to-face orientation for both employment specialists and other VA mental health staff that was led by a social worker with experience training IPS specialists and conducting IPS fidelity ratings. For the duration of the project regular teleconferences continued providing weekly individual case reviews for the first three months, monthly individual case review conferences, and monthly conference calls with all nine employment specialists jointly. A review of ratings of model fidelity at six, 12, and 18 months after program implementation was also performed with a modified version of the IPS fidelity scale ( 5 , 6 ). Training was provided by a social worker with extensive experience disseminating the IPS model in other research projects ( 17 , 18 ). He was assisted by a VA occupational therapist with experience in supported employment. Monthly conference calls were also held with administrators responsible for program implementation at each site.

Ongoing training and review of progress was thus provided throughout the life of the program at three levels: the employment specialists, VA clinical staff working with IPS clients, and facility administrators. Training costs averaged $6,033 per site.

Sample

A total of 308 veterans were recruited during phase 1, starting in January 2001, when IPS was still unavailable anywhere in the VA. An additional 321 veterans were recruited in phase 2, starting in July 2001, after the IPS employment specialist had been hired and trained, and were invited to participate in IPS for at least two years. Follow-up data collection continued through March 2005.

Veterans were eligible for either phase if they were homeless and not receiving VA health services, expressed interest in seeking competitive employment, and were diagnosed as having a psychiatric or substance abuse problem. Participants were considered to be eligible under the homeless criteria if they had slept in a shelter or on the street in the previous 90 days. Interest in competitive employment was assessed by asking prospective participants, "Are you interested in working for pay in the community—somewhere other than at the VA?" Individuals were invited to participate if they responded "yes" to this question and were judged to have a genuine interest in work.

Sources of data

Data were collected by independent research assistants. Efforts were made to collect outcome data on all veterans, even if they discontinued participation in IPS.

Clinical assessments. Patients gave written informed consent to participate in the project and for their employment specialist to communicate with their employers. They were paid $10 for their time for each interview. Institutional review board approval was obtained at the authors' parent institution and at each of the nine participating VA facilities.

Employment specialist activity logs. Employment specialists used a weekly log to document contacts with clients at the VA and in the community, contacts with VA mental health staff, and client employment status. Annualized per-client costs for employment specialist services averaged $2,063 (site range of $1,400 to $2,700).

Fidelity ratings. IPS fidelity ratings were conducted three times, at six, 12, and 18 months, after program implementation by using a shortened version of the IPS Fidelity Rating Scale ( 5 , 6 ), which excluded three staffing items (staffing and caseloads were documented at the individual client level and were constant across sites) and rated several items on 3-point scales rather than on 5-point scales. Fidelity assessments were conducted by an IPS trainer-rater with extensive experience using the IPS Fidelity Rating Scale. The assessments were based on telephone interviews with employment specialists and administrators, and they used information from ongoing teleconference supervision, data logs, and a review of clinical progress notes and treatment plans for five current cases selected at random. Feedback on the fidelity scores with a written explanation of the scores for each item was integrated into the ongoing training.

Client-level measures

Employment status. Employment was assessed with three measures representing the number of days in the past 30 of competitive employment, noncompetitive employment (for example, transitional employment in VA's CWT program), casual or volunteer work, and any type of employment (that is, the sum of the previous three measures). The proportion of clients who indicated working one or more days in competitive employment was also calculated along with the proportion of clients who worked at a noncompetitive job or in any job. Employment data were based on client interviews.

Health status. Diagnoses were based on clinical assessments by homeless outreach staff. Subjective distress was measured with 30 items from the 90-item Symptom Checklist-Revised ( 19 ). Alcohol and illicit drug problems were assessed by using composite indices from the Addiction Severity Index (ASI) ( 20 ). The Medical Outcomes Study 12-item Short Form (SF-12) was used to assess mental and physical health status ( 21 ).

Community adjustment and housing status. Housing status was measured by questions concerning sleeping arrangements during the previous 90 days in each of 12 types of places and used to calculate days of independent housing (whether in their own place or with others).

Attitudes toward rehabilitation and employment. Client interest in both conventional vocational rehabilitation and IPS was assessed with brief vignettes describing each type of service and questions asking how appealing the client found each approach and how willing he or she would be to participate in it. Ratings ranged from 0, no interest, to 100, maximum interest.

Broader attitudes toward work were assessed by using a 21-item measure used in the Social Security Administration's Project NetWork in 1991 ( 22 ). A factor analysis of these 21 items (varimax rotation) produced five component measures reflecting attitudes toward work that can be summarized as "I can't work," "I want to work," "Work helps me cope with problems," "I don't like the jobs I can get," and "Others expect me to work" ( 23 ).

Analyses

Participants in phase 1 were compared with participants in phase 2 on baseline characteristics by using chi square tests and t tests. All available participants were included in all analyses whether or not they were currently participating in IPS. Mixed models were then used to compare outcomes between the phases over the two-year follow-up period, controlling for baseline differences, using the MIXED procedure of SAS, with the alpha set at <.05.

Results

Baseline characteristics

Tables 1 and 2 show the baseline sociodemographic and clinical characteristics. The sample was similar to other populations of homeless veterans ( 12 ) but was less likely than participants in IPS research studies to be diagnosed as having severe mental illness and more likely to be diagnosed as having substance use disorders ( 2 , 3 , 4 ).

Enlarge table

Comparison of participants in the two phases on baseline measures showed that participants in phase 2 had fewer lifetime years of homelessness, were more likely to have worked in the previous three years (but not in the previous 30 days), and had fewer psychiatric symptoms and better physical health. However, the groups did not differ in major psychiatric diagnoses or substance abuse problems ( Tables 1 and 2 ).

Enlarge table

Compared with clients in the phase 1 group, participants in the phase 2 group reported less interest in vocational rehabilitation and more interest in supported employment. Also, participants in the phase 2 group scored lower on the "I can't work" attitudinal factor but not on other attitudinal measures. These significant differences suggested a greater disposition to work among veterans in phase 2 and were included as covariates in subsequent analyses.

Follow-up rates

Overall follow-up rates across all time points averaged 70% (1,730 of 2,464 possible interviews) for the phase 1 cohort and 72% (1,858 of 2,568 possible interviews) for the phase 2 cohort ( Table 2 ). Examination of more than 90 baseline characteristics showed none that were significantly associated (p<.01) with higher follow-up rates for veterans in either phase.

IPS retention

Data from client interviews show that substantial but declining proportions of veterans reported involvement in IPS. Altogether, 82% (N=263) of all veterans in the phase 2 group reported contact with the employment specialist during the first three months of the program, and of these 68% (N=218) were still participating at six months, 56% (N=180) were still participating at nine months, 53% (N=170) were still participating at 12 months, 41% (N=132) were still participating at 15 months, and 28% (N=90) were still participating at 24 months. The median participation rate, as reported by the clients at follow-up interviews, was thus 13 months among those with any reported participation in IPS.

Data from employment specialist administrative records show substantially higher rates of involvement, with 94% (N=302) of clients in the phase 2 group having documented contact with an employment specialist, of whom 90% (N=289) were still involved at six months, 77% (N=247) were still involved at 12 months, and 49% (N=157) were still involved at 24 months, with a median participation rate of just under 24 months. Thus administrative records suggest substantially greater involvement than self-report data from follow-up interviews, although both sources suggest more prolonged involvement than in real-world "high-fidelity" programs ( 24 ).

One percent (N=2) of veterans in the phase 1 group reported any contact with an employment specialist.

Face-to-face contacts between employment specialists and veterans were more intensive during the first three months, averaging 86 minutes per week and then plateauing at an average of 34 minutes per week from six months to 24 months.

During the first three months of participation, when many veterans were in residential treatment facilities, only 27% of contacts took place in community settings. The proportion of community contacts rose to 35% at six months and averaged 46% thereafter, demonstrating substantial emphasis on community-based services delivery, one of the hallmarks of IPS.

Employment specialists also documented contacts with mental health clinicians. All nine employment specialists had at least one such contact during the first three months, declining to 72% from nine to 12 months and 49% from 21 to 24 months. They also documented face-to-face contacts with veterans' employers, ranging from 66% of all veterans involved in the first three months, 33% from nine to 12 months, and 23% from 21 to 24 months.

Fidelity ratings. Fidelity ratings ( Table 3 ) were used to classify sites as having no fidelity (8% of all 27 ratings over all three assessment periods), low fidelity (23% of ratings), acceptable fidelity (35% of ratings), or excellent fidelity (35% of ratings). During the final two rating periods only one site (22%) had "no" or "low" fidelity, while the rest (78%) were "acceptable" or "excellent."

Enlarge table

Outcomes. In mixed models without baseline covariates that used all completed follow-up interviews, days of employment were substantially greater for the IPS group (least-square means=8.8 days for the IPS group compared with 5.6 days for the control group, a 57% difference). However in mixed models that also controlled for significant baseline characteristics that were different between the two phases, veterans in phase 2 had only 15% more days per month of competitive employment on average (least-square means [that is, adjusted for baseline differences]=8.4 days compared with 7.3 days; F=14.0, df=1 and 496, p<.001) along with 32% significantly fewer days of noncompetitive employment (for example, CWT) (least-square means=1.5 days compared with 2.2 days; F=8.6, df=1 and 583, p=.004) ( Figure 1 ). Veterans in the phase 2 group had 5% more days per month of any kind of employment, which was not significantly greater than the number of days for the phase 2 group (least-square means=11.6 days compared with 11.0 days; F=2.8, df=1 and 548, p=.09).

Site-level results show small differences in days of competitive employment favoring IPS at eight of nine sites ( Table 3 ). An exploratory analysis of bivariate correlations (Pearson's r) of fidelity ratings and employment outcomes across all sites revealed small correlations (that is, less than .20). However, the site with the lowest fidelity rating (site "I" in Table 3 ) showed a relatively high level of competitive employment in phase 2 and of improvement from phase 1 to phase 2. Because the number of sites in the sample was small, we reexamined these exploratory relationships with this site excluded. We found a substantial association between fidelity ratings and days of competitive employment in phase 2 (r<.6) and between fidelity ratings and improvement in days of competitive employment (r<.4), although the sample was too small to allow meaningful significance testing.

Paralleling the findings for days employed each month, the proportion of clients with any competitive employment at follow-up interviews was greater for the phase 2 group (44%, compared with 40%; F=15.5, df=1 and 469, p<.001), as was the proportion with any employment (65% compared with 63%; F=3.3, df=1 and 557, p<.04). Smaller percentages of veterans in the phase 2 group participated in noncompetitive employment (10% compared with 14%; F=9.9, df=1 and 578, p=.002). There were no significant between-group differences in hourly wage ($8.52 in the phase 2 group compared with $8.10 in the phase 1 group) or monthly earnings ($1,242 in the phase 2 group compared with $1,150 in the phase 1 group). Average annualized employment income among all participants was $1,299 greater for those in the phase 2 group ($8,889 compared with $7,590; F=4.5, df=1 and 596, p=.01).

Although at follow-up veterans in phase 2 had a greater average number of days housed than veterans in phase 1 (least-square means=34.1 days compared with 29.8 days; F=4.2, df=1 and 644, p=.04), there were no significant differences in other clinical outcomes on any other measure (mental health status, psychiatric symptoms, substance abuse, general health, and social support).

Relationship to transitional employment. Because the VA is a system in which transitional employment was previously well established, we compared employment sequences of transitional employment and competitive employment in phase 1 and phase 2 ( Figure 2 ). According to these unadjusted figures, 23% (N=71) of veterans in the phase 1 group never worked, compared with only 12% (N=39) in the phase 2 group. It is notable that there were similar proportions who were involved in transitional employment only, in transitional employment followed by competitive employment, and competitive employment followed by transitional employment. The greatest difference between phase 1 and phase 2 was in the proportion who worked in competitive employment only (102 clients, or 33%, in phase 1, compared with 177 clients, or 55%, in phase 2). Thus, even when IPS was available, although most veterans went directly into competitive employment, some veterans entered transitional employment before or after entering competitive employment.

Analysis of vocational outcomes by diagnostic subgroups showed that among clients with diagnoses of substance use disorders without psychiatric comorbidity employment gains for persons in the IPS group were significantly greater than those in the control group (least-square means=9.6 days per month for the phase 2 group compared with 8.5 days per month for the phase 1 group), a 14% difference favoring IPS (F=6.8, df=1 and 435, p=.01; N=194). Among patients with diagnoses of serious mental illness, either with or without substance abuse, benefits for IPS in days of competitive employment were slightly smaller, with a statistically significant increase of 9% in days of competitive work per month (least-square means=7.4 days per month for the phase 2 group compared with 6.8 days for the phase 1 group; F=5.0, df=1 and 688, p=.025, N=194).

Discussion

This VA study suggests that IPS can be implemented in an organization with no previous experience with this model through a modestly intensive, but sustained, training effort guided by an outside expert in the implementation and fidelity assessment of IPS. By using an array of measures available at the time that the study was initiated and by using judgment of an experienced IPS rater-trainer, the model appears to have been successfully implemented at most sites. Increases in days of competitive employment were observed at eight of nine sites and were statistically significant overall. However, these gains were substantially smaller than those reported in other studies ( 2 , 3 ). In Bond's ( 2 ) IPS review, almost 2.9 times as many clients in the IPS groups were employed, as compared with those in the control groups (56% compared with 19%), whereas in a study on the Employment Intervention Demonstration Program ( 3 ), 1.6 times as many clients in the supported employment group (experimental model) were employed as clients in the control group (55% compared to 34%). These are much larger effect sizes than the 15% gain in days competitively employed found in this study. Perhaps more intensive on-site training would have led to greater gains in competitive employment.

A major difference between this VA IPS demonstration and initiatives in other systems is that veterans served in both phases of this initiative were more likely to have substance abuse problems than severe mental illness and had higher rates of past, although not recent, employment. It is possible that these differences in the target population reduced the potential gains for veterans in the phase 2 group. However, further analysis of diagnostic clients with diagnoses of substance use disorders without psychiatric comorbidity showed significant employment gains for persons in the IPS group, compared with those in the control group. These gains were similar in magnitude to those observed for clients with serious mental illness. Diagnostic differences thus do not seem to explain our relatively small IPS effects.

It is also notable that this program was implemented in a system with a strong tradition of transitional employment assistance through VA's CWT program and that opportunities for transitional employment may have competed with supported employment. One site (site "I" in Table 3 ) in particular had very close ties with local employers and thus had positive employment outcomes despite having a low IPS fidelity score. Some veterans worked in transitional employment before entering competitive employment, and some went back to transitional employment after their first competitive job. It is not clear whether the combination of opportunities for supported and transitional employment offers useful flexibility and broadening of client options or results in the lowering of expectations for competitive employment.

It should also be noted that unlike other IPS research studies, there was only one IPS employment specialist assigned to each VA homeless program, rather than several, and most veterans were not treated by a structurally unified treatment team. Consistent with other studies ( 5 ), however, moderate correlations were observed between the modified IPS Fidelity Rating Scale and employment outcomes after one anomalous site was excluded.

Several methodological limitations require comment. First, veterans recruited for IPS were significantly different from veterans in the comparison group at the time of program entry, especially in that they had a better long-term work history (even though they did not differ in their recent employment), lower levels of psychiatric symptoms, and a less negative attitude toward work on one of five measures. The baseline covariate with the greatest effect on the adjusted results was the client's history of regular employment in the previous three years ( Table 1 ). As a result, we cannot be sure that raw outcome results are entirely attributable to the IPS intervention, because differences in the populations at baseline may also have contributed to differences in study outcomes at follow-up. To minimize the influence of such selection biases, we used multiple regression analysis to adjust our assessment of outcomes for baseline differences, as noted above. The risk of selection bias may be an unavoidable hazard of studying outcomes in real-world projects to change systems, because random assignment is often not practical when the primary goal of a project is dissemination.

Differences between veterans in the phase 1 and phase 2 groups are also of substantive interest because they suggest that, despite the IPS principle of openness to all who want to work regardless of clinical status or past work experience, some selectivity appears to have been at work, because veterans with greater employment potential were recruited for phase 2. It is difficult, under any circumstances, to objectively measure interest in employment, and it would not be surprising if the employment specialists, new as they were to this kind of work, were more encouraging to veterans whom they thought would have greater vocational success. It is also possible that their awareness of ongoing monitoring of client outcomes led them to select veterans whom they thought would be more successful workers. Selection of clients according to any assessment of their interest in working is inconsistent with the IPS model.

A second major limitation was that only 71% of scheduled follow-up assessments were completed. This is not surprising in a two-year follow-up of clients who were recently homeless at the time of program entry. It is somewhat reassuring that follow-up rates were not different between participants in the two phases of the study and were not associated with any baseline characteristics.

Third, we used a shortened version of the IPS Fidelity Rating Scale. However, the rater who developed the brief version has extensive experience with the full scale, and the spread of scores was substantial, with a range of 10 to 22 out of a maximum of 25. After the exclusion of one site with anomalous findings in this small sample, the fidelity scale showed nonsignificant but moderate correlations both with days of competitive employment in IPS and with change in competitive employment from phase 1 to phase 2. However, our failure to use what has become the gold-standard rating scale for evaluating fidelity to IPS remains an important limitation of the study. It is further notable that although rates of continuation in IPS approach 100% in highly controlled research studies, in a study of 12 "high-fidelity" supported employment programs, as demonstrated by scores of 70 to 75 on the IPS Fidelity Rating Scale, Latimer and colleagues ( 24 ) found that service episodes ranged from an average 103 to 254 days (3.4 to 8.5 months) across sites, substantially less than the 13-month median reported here. Data on coordination with other VA professionals, on coordination with employers, and on community-based contacts and sustained involvement are consistent with the IPS model.

Because the study focused on a real-world implementation, data were not collected on clients who were judged to be ineligible or who refused to participate, and quantitative data on delivery of the training intervention were not obtained.

Conclusions

This dissemination effort appears to be one of the first to evaluate both site-level model fidelity data and two-year longitudinal outcome data that included information from an untreated comparison group. It provides evidence that a sustained training program based on teleconferencing can be used to implement IPS with apparently good model fidelity at most, if not all, sites in a system that has no past experience with the approach. Although significant improvement was observed in rates of competitive employment as well as in independent housing, 20% of sites did not achieve acceptable model fidelity, and the degree of improvement in employment was smaller than that found in more conventional research studies. More intensive, on-site training and performance monitoring may be needed to optimize dissemination of IPS and other evidence-based practices.

Acknowledgments and disclosures

The authors thank Gay Koerber in the VA Central Office for providing overall leadership of this project with assistance from Theresa Hayes. Rich Toscano, M.S.W., and Mary Sperrazza, O.T., provided IPS training, and Mr. Toscano conducted the fidelity ratings. Sandra Resnick, Ph.D., provided helpful comments. The authors also thank the site coordinators of the Therapeutic Employment and Support program: Kathleen Scott, Gary Warner, Steve Mortimer, Steve Gilligan, Greta Mankins, Desi Jackson, George Castillo, Vince Clancy, Jim McGuire, Quita McBride, Gary Glacken, Wendy Hellickson, and Laurie Harkness. Michele D'Amico coordinated data management, and computer programming support was provided by Alex Ackles and Ilan Harpaz-Rotem, Ph.D.

The authors report no competing interests.