Job Offers to Individuals With Severe Mental Illness After Participation in Virtual Reality Job Interview Training

Individuals with severe mental illnesses, including bipolar and major depressive disorders and schizophrenia, and military veterans with posttraumatic stress disorder (PTSD) experience disabling symptoms that hinder their vocational outcomes (1,2). Although approximately 60% of individuals receiving mental health services want to work and may actively seek employment (3,4), they have low employment rates. Specifically, only 22% to 38% of individuals with severe mental illness (including veterans with PTSD) are competitively employed (5–7).

One factor that may contribute to a low employment rate is the social deficits experienced by individuals with severe mental illness that may interfere with successfully navigating the job interview (8–12). Thus these individuals may have difficulty translating their life experiences or military training into work-related skills within a community setting. Therefore, we developed a virtual reality training program to help facilitate learning of job interviewing skills (13–15). Virtual reality job interview training (VR-JIT) provides trainees with the opportunity to engage in repeated interviewing trials with a virtual human resources representative and learn from didactic online materials.

Two randomized controlled trials (RCTs) assessed the efficacy of VR-JIT, compared with a waitlist group, in cohorts of individuals with severe mental illness (13) and veterans with PTSD (14). Approximately 37% of participants were currently or previously enrolled in vocational services but not in supported employment. The studies were administered by the same research team and used identical procedures. The participants were informed that they could complete up to ten hours of training across five sessions. Each simulated interview trial lasted 20 to 30 minutes, and trainees completed approximately 15 trials. The training followed a hierarchical learning framework in which trainees advanced from easy to medium to hard levels on the basis of their performance, and the results suggested that VR-JIT enhanced interviewing skills (13,14). However, the utility of the training would be further validated if use of VR-JIT was associated with receipt of job offers after completion of real-life interviews.

In this study, six-month follow-up data were collected from participants who completed the efficacy studies. First, we hypothesized that the increased interviewing self-confidence observed among trainees during the efficacy study would be maintained at six-month follow-up. Second, we hypothesized that trainees would have greater odds of receiving a job offer compared with waitlist comparison participants. Third, we hypothesized that a larger number of completed VR-JIT trials, a greater VR-JIT performance slope, and increased self-confidence would predict greater odds of receiving a job offer. Fourth, we hypothesized that a larger number of completed VR-JIT trials, a greater VR-JIT performance slope, and increased self-confidence would predict fewer weeks spent searching for employment before receipt of a job offer. Finally, we hypothesized that after six months, trainees would report that VR-JIT helped them find employment. We evaluated neurocognition and recent employment history as covariates in the analyses because they are known predictors of vocational outcomes in this population (16–18).

Methods

Participants

Participants included 70 individuals with severe mental illness or U.S. military veterans who completed prior efficacy studies of VR-JIT (13,14). For inclusion in the efficacy studies, nonveteran participants were required to have a diagnosis of bipolar disorder, major depressive disorder, schizophrenia, or schizoaffective disorder; veterans were eligible for the study if they had a diagnosis of PTSD and a comorbid mood or psychotic disorder. Details about participant recruitment were reported previously (13,14). Bachelor’s or doctoral-level research staff confirmed diagnoses with the Mini-International Neuropsychiatric Interview (19).

The following inclusion criteria were employed: 18–65 years old, minimum sixth-grade reading level as demonstrated on the Wide Range Achievement Test–IV (20), willingness to be video recorded, unemployed or underemployed, and actively seeking employment. Exclusion criteria included having a medical illness that significantly compromised cognition (for example, traumatic brain injury), an uncorrected vision or hearing problem, or current substance abuse or dependence. The Northwestern University Institutional Review Board approved the study protocol, and all participants provided informed consent.

Participants from both efficacy studies were recontacted after six months and asked to complete a follow-up survey. Seventy participants completed the original efficacy studies. [A figure in an online supplement to this article displays the flow of participants from the efficacy studies to the follow-up study.] All comparison participants were informed that they could return to use VR-JIT after the study and were reminded of this option during their final visit. Six waitlist comparison participants completed VR-JIT after the efficacy study and transitioned to the VR-JIT group for six-month follow-up. However, one was lost to attrition. The final sample included 39 VR-JIT participants and 12 comparison participants who chose not to use VR-JIT.

Intervention

VR-JIT is a software application developed by SIMmersion LLC (www.simmersion.com). VR-JIT includes educational content about finding employment, an interactive role-play simulator, and integrated feedback. VR-JIT provides individuals who have a range of disabilities the opportunity to repeatedly practice interviews in a risk-free environment. During each virtual interview, “Molly Porter,” a human resources manager at a large department store, asks questions about skills and experiences. Molly selects questions from 1,200 options to tailor each virtual interview on the basis of customizable information (for example, military history and gaps in employment), skill level, and responses. This variation allows trainees to practice until they master the skills at three difficulty levels and gain confidence to successfully interview for employment. Images of Molly and the VR-JIT interface can be found at the company’s Web site (www.jobinterviewtraining.net). Additional methodological details have been reported elsewhere (13,14).

Study Procedures

Research staff contacted participants weekly beginning six months after they completed the efficacy studies and asked them to complete the follow-up survey by phone and e-mail. If participants were unreachable, a recruitment letter was sent to their last known address. The efficacy studies and six-month follow-up occurred between January 2013 and January 2014.

Baseline Study Measures

Participant characteristics.

The participants’ demographic characteristics and vocational history were obtained via self-report.

Cognitive measures.

The Repeatable Battery for the Assessment of Neuropsychological Status (RBANS) (21) was administered to assess neurocognition. The total index score of the RBANS reflects immediate memory, visuospatial capacity, language, attention, and delayed memory. We assessed emotion recognition as a measure of basic social cognition using the Bell-Lysaker Emotion Recognition Task (22). We used an emotional perspective–taking task as a measure of advanced social cognition (23). Accuracy ratings for each task were generated by using the number of correct responses.

Process measures.

Participants’ VR-JIT performance scores and the number of VR-JIT trials completed were recorded. The software scored each simulated interview from 0 to 100 by using an algorithm based on the appropriateness of trainee responses throughout the interview in the following domains: conveying oneself as a hard worker, sounding easy to work with, sharing information in a positive way, sounding honest, sounding interested in the position, acting professionally, displaying good negotiation skills, and establishing overall rapport with the interviewer. We evaluated VR-JIT performance score improvement across trials as a process measure by computing a linear regression “performance score slope” for each participant on the basis of the regression of their performance scores on the log of trial number.

Six-Month Follow-Up Measures

The survey included a series of questions that asked participants to reflect on the prior six months and to report the total number of weeks they searched for employment, whether or not they completed a job interview (0, no; 1, yes) (and if so, how many), and whether they received (and accepted) a job offer (0, no; 1, yes).

The survey included the same self-confidence items that were collected during the efficacy studies (13,14). Participants rated their self-confidence at performing interviews by using a 7-point Likert scale to answer nine questions, with higher scores reflecting more positive views (for example, “How comfortable are you going on a job interview?”). Item ratings ranged from 1, extremely uncomfortable or unskilled, to 7, extremely comfortable or skilled. Total scores at the end of the two-week follow-up for the efficacy study and six-month follow-up were computed. Across all participants, the internal consistencies were strong at two-week follow-up (α=.91) and six-month follow-up (α=.86).

The six-month survey also assessed whether trainees felt that VR-JIT helped them improve their interview skills, prepared them for interviews, helped them attain employment, and gave them more confidence to interview and whether using the training again would help them prepare for future interviews. Item ratings ranged from 1, strongly disagree, to 5, strongly agree.

Data Analysis

Chi square tests and analyses of variance (ANOVAs) were conducted for participant baseline characteristics to examine between-group differences. To evaluate whether improved self-confidence observed during the efficacy trial was maintained at six months, we conducted a paired-sample t test between the two-week and six-month follow-up scores.

We conducted a stepwise logistic regression with job offer as the dependent variable to evaluate whether VR-JIT trainees had greater odds of receiving a job offer. Step 1 included neurocognition, the number of months since prior employment, and PTSD diagnosis, and VR-JIT group status was added in step 2.

Among VR-JIT trainees, we conducted a stepwise logistic regression with job offer as the dependent variable to evaluate whether VR-JIT process measures and self-confidence contributed to higher odds of receiving a job offer. Step 1 included neurocognition, the number of months since prior employment, and PTSD diagnosis. Step 2 included the number of completed VR-JIT trials, VR-JIT performance slope, and self-confidence at six-month follow-up. For both logistic regressions, odds ratios (ORs) and 95% confidence intervals were generated for each step. Nagelkerke R² was computed to determine the proportion of explained variance.

Among VR-JIT trainees, we conducted a hierarchical linear regression, with the number of weeks searching for employment as the dependent variable to evaluate whether VR-JIT process measures and self-confidence were associated with fewer weeks searching for employment before receipt of a job offer. Step 1 included neurocognition, the number of months since prior employment, and PTSD diagnosis. Step 2 included the number of completed VR-JIT trials, VR-JIT performance slope, and self-confidence at six-month follow-up.

Finally, we used descriptive statistics to evaluate whether participants found VR-JIT to be helpful to them.

Results

Participant Characteristics

Baseline characteristics of the VR-JIT and comparison groups did not differ with respect to their demographic characteristics and clinical, cognitive, and vocational histories (Table 1). A greater proportion of participants with PTSD were observed in the comparison group compared with the VR-JIT group, but the difference did not attain significance. To conservatively analyze the data, we evaluated PTSD as a fixed-effect covariate in the ANOVAs and regressions.

Vocational Outcomes at Six Months

We found that a similar proportion of VR-JIT and comparison participants completed job interviews (before and after adjustment for a PTSD diagnosis) (Table 2). Although 51% of VR-JIT trainees obtained a job offer, compared with 25% of comparison participants, this difference attained significance only after adjustment for PTSD diagnosis (p=.04). The groups did not differ with respect to the adjusted and unadjusted proportion of participants who accepted job offers, the number of weeks they searched for employment, or the mean number of job interviews completed (Table 2). A PTSD diagnosis was significantly associated with whether an interview was completed (p<.05); but this diagnosis was not associated with any other between-group differences.

Self-Confidence at Six Months Among VR-JIT Participants

For VR-JIT trainees, mean scores for self-confidence in interviewing skills did not differ between the postintervention (51.9±7.9) and the six-month follow-up (52.1±7.6) assessments. This analysis excluded the five waitlist participants who subsequently completed VR-JIT.

Odds of Receiving a Job Offer

Step 1 of the logistic regression included neurocognition, months since prior employment, and PTSD diagnosis and explained 40% of the variance in receipt of a job offer. Step 2 added VR-JIT participation into the model, which explained 52% of the variance in receipt of a job offer (Table 3). Overall, step 2 had 77% accuracy in terms of predicting receipt of a job offer, and the omnibus test of model coefficients was significant (χ²=24.90, df=4, p<.001). Moreover, the odds of receiving a job offer were higher for VR-JIT trainees compared with comparison participants (OR=9.64). For each month since prior employment, participants had reduced odds of receiving a job offer (OR=.96). Neither PTSD nor neurocognition was significantly associated with receipt of a job offer.

VR-JIT Trainees and Job Offers

Step 1 of the logistic regression included neurocognition, months since prior employment, and PTSD diagnosis and explained 59% of the variance in receipt of a job offer (Table 4). Step 2 added total completed trials, performance slope, and self-confidence at follow-up, which explained 68% of the variance in receipt of a job offer. Overall, step 2 had 80% accuracy in terms of predicting receipt of a job offer, and the omnibus test of model coefficients was significant (χ²=27.79, df=6, p<.001). For each completed VR-JIT trial (15.1±5.5 total trials completed), the odds of receiving a job offer were greater (OR=1.41). VR-JIT trainees had lower odds of receiving a job offer for each month since prior employment (OR=.93). Neurocognition, PTSD diagnosis, performance slope, and self-confidence were not significant predictors.

VR-JIT Training and Searching for Employment

Step 1 of the linear regression included neurocognition, months since prior employment, and PTSD diagnosis, but it did not explain significant variance in the number of weeks spent searching for employment before receipt of a job offer (Table 5). In step 2, the addition of total completed trials, performance slope, and self-confidence at follow-up explained 53% of the variance in the number of weeks searching for employment (F=3.37, df=3 and 13, p=.05). A greater number of completed VR-JIT trials was associated with fewer weeks searching for employment (B=−3.79, SE=1.37, p=.02). A larger slope of VR-JIT performance scores, neurocognition, months since prior employment, PTSD diagnosis, and self-confidence were not significantly associated with number of weeks searching for employment (Table 5).

VR-JIT Helpfulness

Approximately 90% of VR-JIT trainees agreed or strongly agreed that the training increased their confidence to go on interviews, helped improve their interview skills, and better prepared them for interviews; 90% also agreed or strongly agreed that they would like to use VR-JIT again to enhance their interviewing skills. Sixty-nine percent agreed or strongly agreed that VR-JIT helped them get a job (data not shown).

Discussion

Prior studies assessing the efficacy of VR-JIT during lab-based RCTs found that trainees improved their interviewing skills after simulating job interviews with a virtual character (13,14). This study found that trainees were more likely than comparison participants to receive a job offer after interviewing for a position. Moreover, this study observed a dosing effect: the odds of getting a job offer increased by 1.4 times for every completed virtual interview. A similar dosing effect was found for time spent searching for a job prior to obtaining an offer. Finally, trainees demonstrated sustained improvement in their self-confidence between the RCT and six-month follow-up. In addition, most trainees reported that VR-JIT helped them prepare for interviews and obtain a job offer.

There are several directions in which to continue the evaluation of VR-JIT. Only 37% of participants were receiving vocational services at entry into the efficacy studies and were not previously enrolled in supported employment. Therefore, it would be important for future research to evaluate the impact of VR-JIT on larger study samples with and without access to vocational services. This is an important question because not all individuals with severe mental illness have access to evidence-based vocational services. Also, a recent report suggests that research is needed to study technological advances in standardized vocational services (24). Thus an equally important area for future research would be to evaluate whether VR-JIT can enhance vocational outcomes for individuals who are using evidence-based vocational services (for example, Individual Placement and Support) (25,26) and to evaluate strategies to implement VR-JIT within a large community mental health service provider.

Some limitations to the study must be considered. The sample was relatively small, and not all participants were retained at six months: VR-JIT participants, 71% (N=34 of 48); comparison participants, 55% (N=12 of 22); and comparison participants who transitioned to the VR-JIT group, 22% (N=5 of 22). We did not record data on the types of jobs obtained and the pay received. Moreover, we did not directly assess participants’ motivation to find a job. Thus the transfer of five participants from the waitlist to the training group could reflect lower motivation among participants who did not use VR-JIT. However, we observed that comparison participants did not differ from VR-JIT trainees with respect to the proportion that completed a job interview, the number of interviews completed, and the number of weeks searching for a job during the six-month follow-up period. Finally, we did not assess the types of vocational services received during the six-month follow-up.

The study had notable strengths to help overcome the limitations. First, during the efficacy studies, the participants were randomly assigned to VR-JIT or a waitlist group. This approach enabled us to follow up on a larger sample of trainees who used VR-JIT after the efficacy study. In an effort to address any bias associated with this approach, we conducted a post hoc analysis that excluded these five waitlist participants. The results showed VR-JIT continued to be associated with greater odds of receiving a job offer. [A table in the online supplement presents results of this analysis.] The analysis of baseline characteristics indicated that the two groups completing the six-month follow-up did not differ with respect to prior vocational training and cognition, and the primary analyses were conducted while controlling for known predictors of vocational outcomes (16–18). We evaluated VR-JIT among individuals who were motivated to actively seek employment, which is the target group most likely to use the intervention. The training appeared to be effective across diagnoses, but larger and more focused community samples are needed to examine whether the training has diagnostic specificity.

Conclusions

VR-JIT trainees were more likely than comparison participants to receive job offers by six-month follow-up. Moreover, greater odds of receiving a job offer were associated with the amount of training, and the amount of training and performance scores were associated with how quickly trainees received a job offer. Thus VR-JIT is a promising intervention, and future studies can evaluate whether this training enhances vocational outcomes for individuals with severe mental illness with and without access to evidence-based services.