Fidelity of Supported Employment Programs and Employment Outcomes

Employment is a primary goal for the majority of people with severe mental illness. Recent research shows that supported employment is a more effective approach to vocational rehabilitation for this population than traditional methods (1). However, supported employment is currently implemented in a variety of ways. Bond (2) and Cook and Razzano (3), among others, have identified several aspects of vocational services that are associated with good employment outcomes. These factors have been incorporated into a fidelity scale for supported employment (4).

Despite the movement toward evidence-based medicine, the use of fidelity measures—that is, the accurate representation of critical program components—is somewhat controversial in the mental health field. On one hand, proponents of evidence-based treatment argue that faithful adherence to guidelines is a necessary condition for good outcomes. On the other hand, some mental health advocates have argued that we do not know what works, that good treatment cannot be measured, that offering a variety of models is necessary, or that local solutions are more effective than model programs (5,6).

Program administrators and state mental health and vocational rehabilitation agencies need information about evidence-based practices in order to make decisions about staff management, funding, and policy issues and to promote the value of employment outcomes (7). Research spanning a number of mental health programs suggests that fidelity does predict outcomes (8). However, there has been little research on fidelity of supported employment.

The purpose of this study was to examine fidelity and specific components of supported employment in relation to competitive employment in a state mental health system. A statewide assessment was undertaken as part of a long-term collaborative effort between the Vermont Division of Vocational Rehabilitation and the Vermont Department of Developmental and Mental Health Services to improve employment outcomes for persons with severe psychiatric disabilities.

Supported employment has been an integral part of Vermont's mental health programs for more than a decade, although the various programs have used different implementation approaches, including clubhouses, work crews, and individual placement. The supported employment programs enjoy a strong collaboration with the state's Division of Vocational Rehabilitation and mental health agencies, and vocational rehabilitation funding is blended at the local program level.

Methods

The sample for this study comprised 2,639 individuals aged 18 to 64 years who had been diagnosed as having severe and persistent mental illness and who were enrolled in community rehabilitation and treatment programs at ten mental health centers in Vermont between October 1 and December 31, 1999. The largest of the programs had 533 participants, and the smallest had 91.

The case manager or another staff member completed an employment survey for each eligible client that included questions about competitive employment, average number of hours worked, and number of weeks worked in the last quarter of 1999. This data collection method has been validated by the Mental Health Statistics Improvement Project of the Center for Mental Health Services.

Researchers also visited the ten mental health centers to learn how employment services were provided. For each center, the researchers completed the Individual Placement and Support Fidelity Scale (4). This 15-item scale measures components of supported employment in three categories: staffing, organization, and services. Each component is rated on a 5-point anchored dimension. A score of 1 signifies "not implemented," and a score of 5 signifies "fully implemented"; thus, the lowest possible score is 15, and 75 is a perfect score. The scale has good interrater reliability and internal consistency, and it clearly differentiates supported employment programs from other types of vocational rehabilitation programs (4). We used Pearson's correlation to compare fidelity scores with employment rates.

Results

The quarterly rate of competitive employment among the ten community mental health centers ranged from 14.4 percent to 26.4 percent (mean±SD= 21.5±3.6 percent). The total scores on the Individual Placement and Support Fidelity Scale ranged from 45 to 68 (mean=61.3± 7). The mean total score on the scale was significantly correlated with the mean competitive employment level (r=.76; p≤.01).

Sufficient variance in scores was found for seven of the 15 program components to warrant further examination. However, because of the small number of programs in the sample and the inherently lower reliability of individual component scores, the analyses for these components must be considered exploratory. As shown in Table 1, significant correlations with competitive employment were found for two components: community-based services and full-time employment specialists. Moderate correlations were found for most of the other components, although these correlations were not statistically significant.

Discussion

This study provides preliminary validation of an overall supported employment fidelity scale. Several aspects of supported employment have been validated individually in previous studies, but this is the first study that we know of that shows evidence for a supported employment scale as a whole.

Despite its small sample size, the study has several strengths. It was conducted in a routine mental health system with naturalistic variation in a state system that is beginning to emphasize employment outcomes. The mental health centers were comparable in terms of funding, regulations, and state contracts, but their supported employment programs varied in several respects. Additionally, the methods of assessing employment outcomes and supported employment components have been validated in previous research.

This study points out the importance of several aspects of the supported employment approach. The two individual components that were the most strongly correlated with better employment outcomes were provision of community-based services and use of employment specialists. Provision of community-based services means that engagement, job finding, and follow-along supports are provided in community settings, away from the mental health center. The advantage of the community-based approach is that it reduces the need for people to transfer skills from one setting to another; it facilitates a more naturalistic identification of clients' interests, strengths, and natural supports, which may not have been clear in an office setting; and it builds relationships with employers directly, thereby increasing the effectiveness of job searches.

The correlation between the level of competitive employment and the presence of employment specialists reflects the importance of having staff members who provide only vocational services. Vocational workers who are assigned case management or day program responsibilities are distracted from the employment goal when they must handle other pressing issues, such as crises, illness exacerbations, housing problems, or other emergencies.

Other program components with strong or moderate correlations with competitive employment included a zero-exclusion policy, team-based integration, a vocational unit, and work-based assessment. The zero-exclusion policy means that individuals are not screened for work readiness; everyone is encouraged to consider work opportunities and is supported in these efforts. Team-based integration indicates that the employment specialists are part of the treatment team, so that mental health treatment and rehabilitation are coordinated.

Vocational unit refers to how employment specialists form a group for back-up and support. During weekly supervised group meetings, they discuss clients and share job development leads. Work-based assessment reflects a deemphasis of prevocational assessment; instead, vocational assessment is based on work experiences in competitive jobs. Additional components may also be critical, but they could not be examined here because of lack of variance.

This pilot study suggests that the supported employment components it measured account for about 50 percent of the variance in competitive employment outcomes. Other factors, such as the relationship with local offices of the Vermont Division of Vocational Rehabilitation (7), the proportion of the budget devoted to supported employment services (9), local employment conditions, staffing capacity of employment specialists, and demographic characteristics, could account for additional variance.

Most of the programs scored in the midrange of the supported employment scale, indicating that there is clearly room for improvement. Studies to determine whether improvements result in better employment outcomes are planned.

Conclusions

This study provides preliminary evidence that faithful implementation of key aspects of a supported employment program is associated with competitive employment outcomes. Providing supported employment services in the community and using full-time employment specialists appear to be critical components of such programs. We recommend that this study be replicated in other systems and in longitudinal studies to confirm our findings.

Acknowledgments

Work on this paper was supported by a grant from the Social Security Administration and the Substance Abuse and Mental Health Services Administration and grant MH-56147 from the National Institute of Mental Health. The authors thank Laura Flint and Boyd Tracy, M.Ed., for their contributions.

Ms. Becker and Dr. Drake are with the New Hampshire-Dartmouth Psychiatric Research Center, 2 Whipple Place, Lebanon, New Hampshire 03766 (e-mail, [email protected]). Mr. Smith and Mr. Tremblay are affiliated with the Vermont Division of Vocational Rehabilitation in Waterbury. Ms. Tanzman is affiliated with the Vermont Department of Developmental and Mental Health Services in Waterbury.

Enlarge table