Contents and Intensity of Services in Low- and High-Fidelity Programs for Supported Employment: Results of a Longitudinal Survey

Employment for people with severe mental illness is a global issue. Previous studies of supported employment, particularly those concerning the individual placement and support (IPS) model, have developed international evidence of its effect on vocational outcomes (1–6). In this context, research interest has gradually shifted from investigating the effectiveness of supported employment to examining implementation and dissemination issues in real-world settings through evaluations of the service structure and service processes (7, 8).

In terms of evaluation of service structure, several studies have conducted fidelity scale development research to measure quality of services in each agency at the structure level (e.g., on the 25-item IPS [IPS-25]) (9–11). The Japanese version of the individualized supported employment fidelity scale (JiSEF) was also developed through modification of IPS-25 to create an evidence-based supported employment program fidelity scale that fits Japanese social and labor service systems (11). Previous fidelity development studies compared fidelity scores with vocational outcomes or self-assessment scores (9–14) but rarely focused on the association with the service process. With regard to service processes, past studies have addressed process evaluations by assessing the contents and intensity of supported employment programs, but the results have been inconsistent.

For example, U.S. studies of process evaluation on supported employment services have reported that clients contact IPS program 2–3 times per month (15) or receive vocational services for approximately 2–3 hours per month (16) and mental health services for approximately 4–19 hours per month (15, 16). A Japanese pilot study also reported that the mean service hours for vocational and mental health services in supported employment programs was approximately 11 hours (17). Although greater service intensity was likely associated with finding competitive employment, the relationship between service intensity and work tenure varied between the studies (15, 18–20). One reason for these heterogeneous findings regarding the service process may be differences among supported employment programs in quality of service structure. For example, programs with high-quality service structure, as indicated by a high score on the fidelity scale, may provide more services and, consequently, better vocational outcomes compared with those with a low fidelity score (15). However, few studies have compared the amount of services provided by programs on the basis of their fidelity to the IPS model.

Implementation and service processes also vary between countries in accordance with the relevant policies. For example, Japan has unique laws for mental health services, including employment services. These laws do not focus on individual services for people with severe mental illness. Instead, group services are offered in agency facilities regardless of disability type (6). Therefore, employment service providers often prioritize the provision of group services. Moreover, the Japanese system does not include community mental health centers that combine mental health services and employment services (6). Instead, the majority of employment services are provided by community social service agencies, which serve people with various social needs and types of disability (11). These agencies generally do not care only for clients with severe mental illness, such as schizophrenia; however, clients with a severe mental illness may exhibit severe symptoms and low cognitive function that require frequent contact with supported employment programs (21, 22). In fact, a diagnosis of schizophrenia itself has a negative effect on employment compared with other diagnoses (23–25). The unique Japanese mental health system may affect the service process, and service contents and intensity may differ from those of U.S. programs.

To address the evidence gap, we conducted fidelity research and a prospective study to assess the service provision hours and service contents of supported employment programs among clients with schizophrenia in a Japanese setting. This study aimed to examine the association between the intensity and quality of an agency’s supported employment services, as indicated by fidelity score, and to further validate the JiSEF. In particular, we investigated service contents and compared service intensity among low- and high-quality supported employment programs before clients found a job and during their employment.

Methods

Overall Design

We conducted a prospective longitudinal study with a 12-month follow-up in a routine setting. Through the scale development research for the JiSEF, we identified 25 agencies that at least partially provided supported employment programs for people with mental illness (11). Of these, 13 programs participated in the study (the flow diagram is available in Figure 1 of the online supplement to this article). Five programs were located in or near the Tokyo area, and eight programs were located in other areas. Four programs were part of medical institutions such as psychiatric day care and outpatient services, and nine were in community social service agencies. The mean±SD number of employment specialists and their caseloads were 1.9±1.6 and 25.2±24.0, respectively.

Using the JiSEF, we assessed the program fidelity of each program between August 2014 and February 2015. The JiSEF, which has 25 items (possible scoring range 25–125), was developed on the basis of the IPS-25 (10) and was modified to fit Japanese settings. For example, because the Japanese mental health care system does not have community mental health centers (6), the relevant items in JiSEF were modified from IPS-25 (see Sasaki et al. [11] and its supplementary material). The JiSEF’s convergent validity and interrater reliability have previously been confirmed (11). In addition, the JiSEF has a cutoff point (90/91), which can be used to screen supported employment programs that have high employment rates (26).

In this study, the fidelity research was conducted by researchers who received 1 day of training. We found good interrater reliability with an intraclass correlation coefficient (ICC) of 0.93. We compared the data for service provision and vocational outcomes between the low-fidelity group (mean±SD score for low-quality supported employment programs, 80.1±6.5; N=7) and the high-fidelity group (high-quality supported employment programs, 100.8±5.3; N=6).

Participants

We recruited participants at each agency between December 1, 2014, and November 30, 2015. The eligibility criteria were as follows: clients who were receiving employment services for the first time in each agency; had been diagnosed as having schizophrenia on the basis of the ICD-10; and were between ages 20 and 60 years. Written informed consent was obtained from all participants. This study was approved by the Research Ethics Committee at the National Centre of Neurology and Psychiatry (A2014–078).

Measures

Service provision data.

Because past studies have measured vocational and mental health service data separately (15, 16), the process data service inventory sheets for employment specialists and case managers used in this study were independently developed. In a Japanese pilot research project to collect process data and categorize service types, the initial inventory sheets were created in collaboration with supported employment staff members (17). Before this study was conducted, we revised the inventory sheets through an outreach service process study to clearly identify the delivery methods of the content of each service (27). The inventory sheet for employment specialists had nine codes for individual services (e.g., assessment, job counseling, job development) and one code for group services. The inventory sheet for case managers had 10 codes for individual services (e.g., assessment, individual life counseling, and individual life skill training) and one code for group services (the inventory sheet codes is available in Table 1 of the online supplement to this article).

On both inventory sheets, all individual service codes were divided into three subcodes: outreach services, agency-based individual services, and telephone and e-mail services. Outreach services means services provided outside of the agency or its offices. Agency-based individual services means individual services (rather than group services) provided by the agency staff members in facilities. The agency-based group services code was divided into four subcodes: work skills, cognitive rehabilitation, illness management, and recreation and physical activities.

During the 12-month follow-up period, employment specialists and case managers completed an inventory sheet each time they provided a participant with services. They selected the appropriate code to reflect the service provided on that day and then wrote the minutes of their service provision in each code. The study collaborator affiliated with each supported employment program regularly checked clients’ service records for omissions and incomplete data.

Vocational outcomes.

Competitive employment status data for participants during the 12-month follow-up were collected from their service records and based on their employment contract. Employment was operationally defined as working at least 1 day a month at minimum wage or more as determined by Japanese law during the follow-up period. We calculated length of time to find first job, work tenure during the study period, hours worked per week, and work income. In addition, we collected information about whether the participants maintained their employment for 1 year after starting their job, which was beyond the 12-month follow-up period. Data collection was completed in December 2017.

Background variables.

As a baseline assessment, we obtained sociodemographic variables and information from each program’s client records about participants’ employment experience for the 6 months before the study. In addition, we assessed the participants’ function using the Global Assessment of Functioning (GAF) (28). The participants also completed a scale to assess the motivation for competitive employment of people with mental illness (motivation scale) (29). The scale development study for the motivation scale confirmed its factor validity and reliability, including high internal consistency (Cronbach’s α=0.91) and high test-retest reliability (ICC=0.83) (29).

Data Analysis

We compared sample characteristics at baseline and vocational outcomes between the low- and high-fidelity groups. In terms of the service provision data, we first presented descriptive statistics for number of service uses per month during the 12-month follow-up. We calculated the number of service uses by determining the number of days on which the participants received services from either employment specialists or case managers as indicated on the inventory sheets.

Second, we computed the mean duration of services (minutes and hours per month) for each code on the basis of the inventory sheet data. The codes were classified into four categories: outreach services, agency-based services, telephone and e-mail services, and agency-based group services. We then summarized the mean hours in each category of service by month for each group. Third, we computed the mean hours per month in each category of service before the participants found a job by dividing the number of hours per service by the number of months until participants started a new job. If a participant did not find competitive employment during the follow-up period, the service hours were divided by 12 (months). Similarly, for participants who worked during the study period, we calculated the mean hours per service per month between the job start and the end of 12-month follow-up (or job termination).

On the basis of our primary interest, we implemented the generalized linear model (GLM) with the robust standard error adjusted within each program (Stata syntax, vce[cluster] option) to compare the mean hours provided in the categories of outreach services, agency-based individual services, and agency-based group services. The dependent variable was service hours, combining employment specialists and case managers, because the integration of vocational and mental health services and an interdisciplinary team approach is an essential factor of effective supported employment services (a detailed description is available in the online supplement) (30, 31). The independent variable was group (low- or high-fidelity program). We simultaneously adjusted for employment status during the 12-month study period as a covariate when analyzing the hours of service provided before the participants found a job. We also performed a sensitivity analysis to adjust for the other potential covariates, including sex, age, educational qualifications, past work experience, GAF score, and motivation scale score (32–34). In addition, we conducted supplemental analyses to examine the association between hours of service provided and vocational outcomes using a univariate logit model and Spearman’s correlation. Statistical significance was set at the 5% level. All analyses were conducted by using Stata version 15.

Results

Sample Characteristics and Vocational Outcomes

During the recruitment period, 93 clients met the inclusion criteria, and 51 were included in the analyses (see flow diagram available in the online supplement). No significant between-group differences were found in sample characteristics and scale scores at baseline (Table 1). As shown in Table 2, participants in the high-fidelity group were significantly more likely than those in the low-fidelity group not only to find jobs (68% versus 38%, p=0.032) but also to retain the initial job after 1 year (59.1.% versus 20.7%); they also had a longer mean work tenure (166.2 versus 56.4 days, p=0.002).

Descriptive Statistics for Service Provision

We found no difference in mean service uses per month between the low-fidelity group (6.8±4.8) and the high-fidelity group (6.9±3.8) during the 12-month follow-up. However, the mean hours of total services per month were greater in the low-fidelity group (18.6±17.1) than in the high-fidelity group (10.9±7.7) (see Table 2 in the online supplement). Overall, services provided to the high-fidelity group included a balance among outreach services (29%), agency-based individual services (36%), and agency-based group services (32%). By contrast, a large proportion of services provided to the low-fidelity group was agency-based group services (83%). For both groups, about 70% of the total hours of service were provided in the first 6 months (see Figures 2 and 3 and Table 3 in the online supplement). With regard to the service provision codes for the employment specialists, the high-fidelity group expended more effort on outreach-based job development (136.8 minutes per month), whereas the employment specialists in the low-fidelity group spent more time on group services for work skills (349.6 minutes per month) (Table 3). For case managers in both groups, minutes of service provided were highest for recreation and physical activities (low-fidelity group, 127.7 minutes per month; high-fidelity group, 95.3 minutes per month). The case managers in the high-fidelity group also provided more individual employment services than did those in the low-fidelity group (Table 4).

Comparison of Service Intensity Between Groups

Table 5 presents the GLM results. Before the participants found jobs, the high-fidelity group was provided significantly more hours of outreach services (B=7.2, 95% confidence interval [CI]=0.2 to 14.1, p=0.043) and agency-based individual services (B=5.7, 95% CI=2.6 to 8.7, p<0.001). Conversely, providers in the low-fidelity group spent more effort on agency-based group services (B=−13.4, 95% CI=−22.3 to −4.5, p=0.003). However, we found no significant between-group differences in service hours for clients who found employment (see Table 4 in the online supplement), but clients in the low-fidelity group tended to spend more hours in agency-based group services (B=−5.6, 95% CI=−11.8 to 0.6, p=0.076). Sensitivity analyses obtained the same significant trends as the unadjusted models. In addition, supplemental analyses showed that the total hours of service provided to participants before they obtained a job were significantly associated with whether the participants found a job (B=0.1, 95% CI=0.02 to 0.11, p=0.007), but the total hours of service provided during employment were not correlated with work tenure.

Discussion

In this study, we compared the contents and intensity of services provided in low- and high-fidelity supported employment programs for clients diagnosed as having schizophrenia. The findings suggest that high-quality programs, as identified by a high fidelity score, have better vocational outcomes and provide more hours of individual services than low-quality programs.

Supported employment programs with a high fidelity score expended more effort on outreach services and agency-based individual services, particularly job development services, and less effort on group services. In supported employment programs, particularly those using the IPS model, staff members focus on individuals’ preference for job seeking (35). In addition, employment specialists are expected reach out to the community to address job development (18, 36). The high-fidelity group may intentionally increase individual services rather than group services when implementing supported employment services corresponding to individual needs

Both groups provided group services in addition to individual services. The low-fidelity group offered more total service hours per month during the 12-month follow-up than the high-fidelity group, although there was an almost equal number of service uses (days) between the groups. We attribute this seemingly paradoxical finding to the large amount of time spent per day on group services in the low-fidelity group. However, even the high-fidelity group expended about 30% of their efforts on group services. Providing group services even in a high-fidelity supported employment agency appears to be affected by the Japanese mental health care system, which mainly supports group services rather than individual services; thus, this finding may reflect the Japanese context.

Our study is consistent with the findings of McGuire et al. (15), who reported that clients received intensive services in the first 6 months of high-fidelity IPS programs. Their results were similar to those of this study. Because the principles of the IPS model of supported employment include rapid job searches (35), IPS programs appear to provide intensive services during the initial phase of services, regardless of country. In particular, this study’s high-fidelity group received more outreach and individual services before the participants obtained jobs. Considering an association with outcomes, as past studies noted (18, 19, 37), a potential reason that the high-fidelity group had better outcomes is that the participants received more intensive services prior to finding a job.

Contrary to the intensity of services before clients obtained employment, we did not find significant between-groups differences in service intensity during clients’ employment period. Previous studies have demonstrated that the need for follow-up services varies among individuals, particularly those diagnosed as having schizophrenia, as a result of symptom severity or cognitive impairment (15, 21, 22, 38, 39). Because we assessed neither clients’ symptoms nor their cognitive function, this is a limitation of the study. However, the amount of follow-up services may also be influenced by individual preferences and the functional aspect. For example, if a client prefers no follow-up services, service hours are unlikely to increase. Moreover, staff members generally expend effort to solve problems when a client embraces a particular need (e.g., a relationship problem with a colleague) to keep their job rather than when a client’s work is stable. In other words, service quality at the structure level as rated by fidelity may simply not be associated with follow-up service hours. Clarifying the relationship of quality of service structure, amount of service hours, and individual needs and conditions remains an issue for future research.

We recognize that this study had several other limitations. First, collection of service data throughout the study was not perfect. On occasion, staff members did not properly document their service provision on the inventory sheet. However, the mean total service hours per month for the high-fidelity group (10.9±7.7 hours) was similar to that in a Japanese pilot study (11.2±9.4 hours) (17). Second, the sample size was relatively small, although we recruited participants from 13 programs. Hence, the study may have had inadequate statistical power. Third, about 70% of the participating agencies were community social service agencies that did not focus exclusively on people with severe mental illness. Fourth, the quality ratings were performed by using JiSEF. Although this study partly replicated previous findings, its generalizability appears to be limited. Future studies could use the IPS-25 to compare service processes in standard settings in other countries.

Conclusions

This study validated the JiSEF and demonstrates that high-fidelity programs produce better vocational outcomes and provide more individual services before client employment than do low-fidelity programs. Future research should examine fidelity scores (service quality at the structure level), amount of service hours, and individual needs and conditions.