Outcomes-Based Funding for Vocational Services and Employment of People With Mental Health Conditions

National policy embraces a commitment to the full participation in society of people with mental health conditions (1). Integral to achieving this goal is that people with mental health conditions have the opportunity to work. Currently, however, despite government policy, improved treatments, and the successful employment experiences of many, people with mental health conditions continue to be underrepresented in the labor force (2).

Policy makers, in search of ways to promote employment, have turned to funding strategies. One strategy is outcomes funding, which pays providers for milestones that mark consumers' sustaining work in integrated competitive employment rather than paying providers for services independent of the consumer's work status (3,4,5). However, documentation is lacking about the effectiveness of performance-based contracting (6,7). The purpose of this paper is to describe the employment experience among people with psychiatric disabilities who received services through one such outcome-based funding program, New York State's performance-based contracting demonstration. Our study investigated two questions: What was the likelihood that participants in the performance-based contracting demonstration secured a placement and retained work? And what factors were associated with securing a placement, time to secure a placement, and job retention? The study used a nonexperimental, qualitative approach to explore in depth the employment outcomes of people with mental health conditions who receive vocational services through a program based on outcomes funding.

Methods

Sample

Seven nonprofit social service agencies across New York State responded to a Request for Proposals and were purposively selected by the state to participate in the study. The agencies were chosen to represent different regions, to serve diverse populations, and to represent rural, suburban, and urban areas. The agencies also differed in size, from 40 employees to more than 2,000 employees. All offered vocational rehabilitation but varied in other services, such as housing, mental health services, children's services, and welfare-to-work programs.

Performance-based contracting program structure

Performance-based contracting milestones were developed as six incremental, reimbursable steps toward the goal of sustained integrated competitive employment. Each milestone was reimbursed at a different rate, with the later milestones (sustaining employment for three, six, and nine months) weighted more heavily than the earlier milestones (life skills assessment, initial placement, and job skill acquisition) to provide further incentive to help consumers retain employment. To avoid the risk of selection by agencies of consumers who were easier to serve, a risk of outcomes-based funding, New York developed and funded a system in which agencies were paid 20 percent more for consumers who were deemed more difficult to serve because they lacked work experience (that is, had no competitive employment in the past 60 months) or their conditions were considered to be more severe (that is, three or more psychiatric hospitalizations or 12 or more weeks in the hospital over the past two years) or any combinations thereof.

Agencies had to meet the service requirements at each milestone to be reimbursed, and the state verified service delivery through an audit of agency records (8). For example, requirements for the initial placement phase included developing a career plan, teaching job search skills, making employer contacts, providing workplace accommodation counseling and negotiation, identifying natural and job supports, and providing job training.

New York State's Office of Vocational and Educational Services for Individuals With Disabilities (VESID) partnered with the state in the provision of performance-based contracting services. VESID funded work-related training, reimbursed agencies for completing life skills assessments, and offered expedited VESID enrollment to eligible consumers who were served in the performance-based contracting demonstration.

Procedures

As an evaluation of a service program, this study was designated as exempt by Columbia University's institutional review board in accordance with federal guidelines. Staff assigned to the demonstration attended a training session to introduce them to the milestone structure and the evaluation procedures. At the time of consumer enrollment, vocational counselors collected demographic data and began a record of services. Counselors updated the record every time that they met with the consumer, at the completion of each milestone, and when a consumer was placed in employment. Counselors obtained informed consent, and they surveyed consumers about their perceptions and knowledge of issues important to working with a mental health condition at enrollment and at the end of the demonstration, the end of the last milestone, or when the consumer left the program, whichever came first. Consumers were compensated $20 for each completed survey. Intake was continuous throughout the demonstration period, from December 2000 to December 2002.

Semistructured interviews were completed with agency directors, supervisory staff, and line staff who were responsible for performance-based contracting before the implementation of the demonstration and at the end of the demonstration. Staff were asked to describe their role in providing performance-based contracting services and how they perceived performance-based contracting changed the vocational unit's operations, services, and policies. Results from these interviews are reported elsewhere (8).

Variables

Outcomes of interest were placement, defined as working for at least three days in the initial competitive job secured after program enrollment; time to placement, defined as the number of weeks between enrollment and the first placement; and retention, defined as the number of weeks employed starting with the first day on a job. The retention measure was cumulative, so that if a consumer left one job but started another, all weeks worked in any job counted toward the weeks of retention.

Six categories of descriptive variables were examined. The first included current employment information for each job obtained during the demonstration, including title, industry, employment sector, first and last dates of employment, number of hours worked per week, hourly wage, whether the consumer's mental health condition was disclosed to the supervisor, and when relevant, reasons why a consumer left the job. The second category was employment history, including dates and number of previous jobs, the amount of time since the most recent employment, and number of below-average work skills. The third category included program variables, such as VESID status, whether the criteria indicating the harder-to-serve designation were met, whether job coaching was provided, number of weeks to complete milestones, date that the client left the program (if applicable), and reasons that the consumer chose to leave. The fourth category was the intensity of service, as measured by the average number of provider hours spent with consumers per week. The fifth category was the effectiveness of the services offered in the life skills assessment and initial placement phases. Effectiveness was measured by the difference in scores calculated from the pre- and post-consumer survey responses to questions that asked about consumers' knowledge or perceptions of issues targeted by these first two milestones, including rights under the Americans With Disabilities Act, the impact of work on benefits, the risks and benefits of disclosure, nonvocational issues that interfere with work, job search skills, need for workplace accommodation, and need for social support while finding or keeping a job. The last category included demographic and disability information, such as including age, gender, race or ethnicity, benefits status, marital status, living situation, DSM-IV axis I and II diagnoses obtained from consumers' case records, and number of nonvocational barriers to employment, as indicated by substance abuse, physical health problems, criminal justice record, and no high school diploma or equivalent.

Data analysis

Data were examined for participants who were still active in the performance-based contracting demonstration at the end of the study period. To respond to the study question about the likelihood that participants in the performance-based contracting demonstration secured and retained work, the likelihood of placement was calculated as the proportion of consumers enrolled in the program who were placed in jobs. The likelihood that participants in the demonstration retained their jobs was assessed by using life tables. Life tables estimated the probability of leaving a placement at different points in time, taking into account data for individuals who were placed but did not complete a milestone because they were not in the job for a long enough time and not because they lost their jobs (censored cases).

Logistic regression was used to respond to the study question about what factors were related to initial placement. Factors significantly related (p<.05) to placement (0, not placed, and 1, placed in employment) were entered as a block into the equation. Stepwise regression was used to determine the demographic, disability, employment, and program factors associated with the time to secure the initial placement. Cox regression analysis (9) was used to explore the factors associated with job retention. This analysis showed the effect of independent variables on the time it takes for the job loss to occur (hazard rate), and it takes into account censored cases. Covariates selected for entry into the model were theoretically relevant and significantly correlated with time on the job. All variables were entered in one block. The effect of agencies was controlled for by entering agencies as dummy variables in all analyses. SPSS 12.0 was used to perform all analyses.

Results

Overall, 310 consumers enrolled in the program. By the end of the demonstration, 171 consumers were active in the program. Most of the consumers who transferred out left before they achieved their first placement. This pattern may have been due, in part, to the agencies' recruitment strategies. In an effort to include those who were harder to serve, some were enrolled who were not ready or interested in competitive work.

The mean±SD age of the participants was 41±9 years. Most were male (97 participants, or 57 percent), and 73 (43 percent) were from a racial or ethnic minority group. A majority was single (112 participants, or 66 percent) or separated or divorced (40 participants, or 23 percent). Few were married (18 participants, or 11 percent).

Consumers tended to live in supported housing (59 participants, or 35 percent), with relatives (38 participants, or 22 percent), or alone (41 participants, or 24 percent). Most frequently consumers received federal income assistance through Supplemental Security Income (56 participants, or 33 percent), Social Security Disability Insurance (48 participants, or 28 percent), or both (30 participants, or 18 percent). These programs provide benefits to persons who are evaluated as being unable to secure substantial gainful employment at any job in the economy. Thirty-nine participants (23 percent) did not receive a high school degree or equivalency, 48 (28 percent) received a high school diploma, and 81 (47 percent) completed some or received a degree for post-high school education.

Most commonly consumers' diagnoses were schizophrenia (53 participants, or 31 percent), bipolar disorder (38 participants, or 22 percent), or other mood disorders (40 participants, or 23 percent). The remainder had a diagnosis of schizoaffective disorder (24 participants, or 14 percent) or anxiety and other disorders, such as posttraumatic stress disorder or anorexia (16 participants, or 9 percent). Consumers were given a diagnosis an average of 12±10.2 years before enrolling in the performance-based contracting demonstration.

Likelihood of job placement and retention

Among the 171 active participants, 108 (63 percent) were placed in jobs. Most of these individuals were placed in one job (70 participants, or 65 percent); however, 38 (35 percent) were placed more than once, so our study examined 159 placements. Most of the 159 placements were permanent (127 placements, or 80 percent), in the for-profit sector (130 placements, or 82 percent), in retail (54 placements, or 34 percent), and in food services (38 placements, or 24 percent). The most frequent positions were administrative (38 placements, or 24 percent) or sales (38 placements, or 24 percent). Consumers were working a mean of 22±10.2 hours per week and earned $7.25± $3.00 per hour.

At the end of the demonstration, 79 (73 percent) of those placed were working, and among these clients, 43 (54 percent) sustained work for at least six months and into the last milestone or ongoing support. Table 1 shows that there seemed to be a tendency for those who left their jobs to do so during the job skills acquisition phase (corresponding to the first month on the job). Of the 15 participants whose highest milestone reached by the end of the demonstration was job skills acquisition, eight (53 percent) left their placement. Although consumers left their employment throughout the demonstration, 31 (94 percent) of those who were employed for more than nine months were still working by the end of the demonstration.

Table 2 shows the probability of sustaining work for the 108 consumers who were placed in jobs and remained active in the demonstration over the 104-week demonstration period. Overall, it showed that the probability of sustaining work through the nine months of employment was .69. The likelihood of leaving a job (hazard rate) and the number of consumers who terminated employment were higher during the first month of employment and around the sixth month. Among the 29 consumers who left a job placement, the most frequent reasons were problems on the job, including being fired or quitting for poor work performance or for not liking the work (13 participants, or 45 percent); leaving because the job was a temporary assignment (nine participants, or 31 percent); and leaving because of an exacerbation of the mental health condition (seven participants, or 24 percent).

Factors associated with placement, time to placement, and retention

Among all the variables, only the program variables were significantly related to securing an initial placement. Results of the logistic regression showed that the number of provider hours stands out as most important. The odds of an initial placement increased 6.35-fold for every additional hour that providers spent assisting consumers. Providers averaged half an hour per week in the initial placement phase for those who were not placed and 1.67 hours per week for those who were placed.

For this sample, receiving services through VESID was also significantly related to placement. The odds of placement were 3.81 times greater for VESID recipients than those without VESID status. Among the 171 participants, 87 (51 percent) were VESID recipients. Among these, 72 (83 percent) were placed. Among those without VESID status, 37 (44 percent) were placed.

The median time to first placement was 13.6 weeks and ranged from less than a week to 72 weeks. Variables significantly correlated (p≤.05) with time to placement were entered into a stepwise regression. As shown in Table 3, the analysis showed that time to placement was significantly related to the number of provider hours and whether the consumer met criteria for being harder to serve. Together they explained 19 percent of the variance. As expected, more provider hours resulted in faster placements, and consumers who were considered harder to serve took longer to place. Among the consumers enrolled in the performance-based contracting demonstration, 99 (58 percent) met the criteria for being harder to serve, and 55 (51 percent) of those who secured jobs met these criteria. The other disability, employment, and program factors did not make a further significant contribution to explaining the variance in time to placement and thus were excluded.

Results of the Cox regression analysis showed that the model was a good fit with the data, as indicated by a significant overall chi square value (χ²=25.54, df=12, p<.05). As shown in Table 4, the risk ratios (Exp[B]) indicated that as hourly wage increased, the likelihood of sustaining work increased. Conversely, among participants with temporary jobs, the likelihood of job loss increased significantly. By definition job retention is less likely in a temporary job than in a permanent job. However, these data suggest that the temporary jobs were also short term. Also, the likelihood of job loss increased the longer consumers spent in the initial placement phase and, unexpectedly, the more hours per week that providers spent with consumers after placement.

Discussion

The evaluation of the performance-based contracting demonstration describes consumers' employment experience in an outcomes-based funded program. It was limited in its generalizability by the nonexperimental approach, including the purposive selection of agencies, and the small sample. However, an in-depth study carried out over several years enables an understanding of the factors related to successful employment outcomes for people with mental health conditions. Results show that the likelihood of placement and retention matched or exceeded performance outcomes of best practices cited in the literature (10,11,12,13,14) and suggested that outcomes-based funding contributes to placement and retention in several ways.

First, as indicated by the significant inverse relationship between number of provider hours and time to initial placement, performance-based contracting may encourage providers to secure funding in the shortest time by increasing the amount of time spent with consumers each week. The fact that increased provider hours after placement was associated with a greater likelihood of job loss may seem counterintuitive. This finding suggests that consumers who were struggling in their jobs and eventually left their placements received more provider attention. It was expected that consumers who met the criteria for being harder to serve would take longer to place. However, what is important in this context is that no significant difference was found in the likelihood of getting or keeping a job between those who met the criteria for being harder to serve and those who did not. It may be that more provider hours had a positive impact on employment for all consumers, including those considered more difficult to serve.

These results have implications for program staffing. They suggest that caseloads need to accommodate the greater demand on providers' time during placement and when consumers experience problems on the job, underscoring the fact that providers need to be able to intensify their service throughout the employment process, not just at the start. Furthermore, because consumers deemed harder to serve took more provider time, it is important to take into account the proportion of the caseload who meets the criteria for this group.

Second, the performance-based contracting payment structure, which reimbursed more heavily the longer the consumer was employed, may have provided incentive to place consumers in good jobs that they will want to keep. Results from this demonstration showed that higher hourly wages, a possible indicator of a good job, was positively associated with retention.

Third, consistent with the findings of Gowdy and colleagues (15)—who reported that when eligibility for state vocational rehabilitation services occurs in a timely manner, consumers are more likely to get jobs—findings from this study showed a positive association between VESID status and placement. Performance-based contracting may be effective in linking consumers in a timely manner to the additional source of support that VESID represents. In so doing, the likelihood of securing employment is enhanced.

The study also suggests that there are potential pitfalls with services provided through an outcomes-based program. Performance-based contracting may detract from the desired outcomes to the extent that providers feel pressured to place consumers in jobs at the expense of job fit. Some consumers left jobs because they did not like them, and although temporary jobs may be a good employment option for some, the finding that temporary jobs were associated with poor retention may suggest that some providers placed consumers in jobs to gain reimbursement for early milestones at the expense of longer-term placements.

Data also suggest that job loss tends to occur during the first month and at about six months of employment. Leaving a job was often due to ineffective workplace accommodation, as suggested by those who were hired, and therefore qualified, for the job but were fired or quit because of poor performance. This finding suggests that agencies need to assess whether the workplace support that they provide, or for which they arrange, adequately promotes successful outcomes.

Conclusions

Outcomes-based funding has the potential to promote integrated competitive employment for people with mental health conditions. This study provided evidence that the rates of placement and retention for people who were served through a performance-based program are comparable to those in other best-practice vocational programs. Outcomes-based funding may lead to temporary placements to gain reimbursement at early stages at the expense of longer-term jobs. However, the preliminary data suggest that performance-based contracting promotes successful outcomes by encouraging providers to work more intensively with consumers early in the placement process, find jobs that consumers want to keep, and coordinate more effectively with the state rehabilitation agency.

Acknowledgment

The research for this article was funded in part by the New York State Office of Mental Health.

Dr. Gates and Dr. Akabas are affiliated with Columbia University School of Social Work, Workplace Center, 1255 Amsterdam Avenue, New York, New York 10027 (e-mail, [email protected]). Ms. Klein is with the department of education and family support at Abt Associates in Boston. Dr. Myers, Ms. Schawager, and Ms. Kaelin-Kee are with the New York State Office of Mental Health in Albany.

Enlarge table

Enlarge table

Enlarge table

Enlarge table