Impact of the COVID-19 Pandemic on a Statewide Individual Placement and Support Employment Initiative

The individual placement and support (IPS) approach to supported employment for adults diagnosed as having serious mental illness is the gold standard evidence-based practice for helping people find and keep meaningful jobs (1, 2). Twenty-eight randomized controlled trials of IPS have found the approach to be efficacious (https://ipsworks.org/index.php/evidence-for-ips), and Modini et al. (3) found that those receiving IPS were 2.4 times more likely to be employed than were participants in the control group.

Implementation of IPS began in New York State over a decade ago (4), and the approach has achieved outcomes consistent with national benchmarks. In 2020, the Center for Practice Innovations at Columbia Psychiatry (CPI) worked with 89 community psychiatric rehabilitation programs and state facility clinics to implement IPS. Both settings serve adults diagnosed as having serious mental illness, are recovery oriented, and view employment as part of their mission. Community psychiatric rehabilitation programs employ multidisciplinary teams and emphasize group interventions as well as individual meetings, whereas the clinics emphasize individual appointments. Before the COVID-19 pandemic, CPI provided a comprehensive set of supports and resources to participating IPS implementation sites: learning collaborative experiences where sites learned from experts as well as one another; online training modules; training and consultation provided onsite; regional face-to-face training meetings; webinars and other online experiences; an employment resource book useful to recipients of services, employment staff, and other treatment team members; and an extensive online resource library (4).

The pandemic has resulted in many changes in how services are being delivered by behavioral health providers (5–8). Understandably, IPS service provision has been affected as well, as identified by the IPS Works Learning Community (https://ipsworks.org/index.php/2020/04/21/providing-ips-services-during-the-covid-19-pandemic), who note that IPS services are often being provided remotely, addressing pandemic-related financial changes and adjusting to the rapidly shifting labor market.

This column focuses on the impact of the COVID-19 pandemic on the implementation of IPS services across New York State. It describes changes in the training and implementation support provided by CPI, the model as delivered by the sites, self-reported IPS fidelity, and in employment outcomes.

Changes in CPI’s Training and Implementation Support Provided

Although several previously existing supports and resources have not been affected by the pandemic, those involving face-to-face interaction have been adjusted. The start of the pandemic resulted in a new focus for CPI’s longstanding webinar series, with sites sharing innovations, accomplishments, and challenges in adapting to the pandemic. Trainees supported one another through brainstorming exercises and generously sharing lessons learned. The pandemic has brought new challenges concerning how to address finances and benefits, and some discussions focused on stimulus payments, unemployment benefits, and other entitlements. One topic receiving considerable attention was the ambivalence and concern expressed by some service users concerning the health risks related to community exposure to COVID-19 while working. We introduced a shared–decision-making perspective in which employment staff helped service users to think through their options and personal decisions.

Site-specific training and consultations continue, albeit remotely through the use of online platforms, with 62 remote site visits from April through December 2020. Training of newly hired staff has also shifted to remote platforms, with training focused on modeling and role-plays of remote job development skills and on reviewing the IPS fidelity scale.

In summary, training and implementation supports have changed considerably with the arrival of the pandemic; learning collaborative meetings have placed considerable emphasis on solving the challenges and sharing the opportunities raised by new conditions, and site visits and staff training have become virtual. In a survey of learners conducted in December 2020 asking about satisfaction with the various training events and supports provided, 18% (N=30) rated CPI’s efforts as “extremely helpful”; 39% (N=64) “reasonably helpful”; 28% (N=46) “somewhat helpful”; 10% (N=16) “slightly helpful”; and 5% (N=8) “not helpful.” Mean satisfaction ratings of specific training events and supports provided ranged from 2.63 to 3.80 on a 5-point scale; these ratings were consistent with ratings provided in 2019 by clinic staff, which ranged from 2.60 to 3.64.

Changes in What the Sites Are Doing

In May and June of 2020, an online survey was sent to all 89 participating sites asking about the impact of the pandemic on IPS implementation. Eighty-eight sites responded, with 64 (73%) continuing to provide IPS services. A closer look indicated that 96% (N=50) of the community psychiatric rehabilitation programs and 39% (N=14) of the state facility clinics were providing these services. For state clinic sites that suspended IPS, staff were redeployed to inpatient and residential sites. By November 2020, 70 sites reported providing IPS services, and by February 2021, that number had increased to 84. Open-ended responses to several survey questions were provided. (The questions and a representative sample of responses are presented in section 1 of the online supplement.) Themes included remote provision of services, increased need for benefits counseling, technology challenges, and fear of COVID-19 exposure.

Employment staff noted that several new jobs had emerged because of the pandemic and were being accepted by IPS recipients. Some were new kinds of positions, and others reflected an increased availability of positions that had already existed. Opportunities included jobs in warehouses; in grocery stores, including shopping for and delivering to people at home; in janitorial and cleaning services; in telemarketing; as remote COVID-19 tracers; and as home health aides. These positions are now routinely considered by IPS recipients, and those positions that continue postpandemic will be options for future IPS recipients.

In summary, although many IPS implementation sites continued to provide services, several (mostly state facility clinics) did not, largely because of staff redeployment to other roles necessitated by the pandemic. Over time, increasing numbers of sites have renewed their IPS efforts. Many of the sites continuing to provide IPS creatively shifted to remote delivery of services, working through challenges as they emerged. Some service users receiving IPS have struggled, especially with the required technology, whereas others appear to be thriving as the pandemic continues. For some IPS recipients, training in use of online technology may contribute to marketability, now more than ever. Interestingly, employment staff recommend retaining some of the innovations that have been developed, noting the benefits of remote service delivery for some service users.

Fidelity

CPI’s IPS initiative has utilized self-assessed fidelity ratings that provide a snapshot of statewide IPS implementation efforts, identify systemic issues, and guide and tailor our training and implementation support efforts. They also are used by implementation sites to identify targets for continuous quality improvement efforts. Margolies et al. (9) compared self-assessments and independent IPS fidelity assessments at 11 sites and found that the self-assessments may be valid for examining a program’s overall functioning. Margolies et al. (10) found that self-assessed higher fidelity implementation of IPS was associated with higher employment rates for service users, which were sustained over time.

Fidelity self-assessments were completed in October and November 2020 by using the IPS Supported Employment Fidelity Scale (11). In addition, we added seven items to reflect COVID-19 conditions (e.g., remote provision of services at many sites), which are adjusted versions of seven original items (see section 2 of the online supplement). These items allowed us to understand the impact of the pandemic on traditional fidelity and also capture new ways in which sites are providing services during the pandemic. However, use of the scale with adjusted items has not been empirically validated, so the insights gathered are tentative at best. Scores on the IPS Supported Employment Fidelity Scale range from 25 to 125, with higher scores indicating greater fidelity.

As noted in Table 1, 70 sites providing IPS at that time submitted fidelity self-assessments: 46 community psychiatric rehabilitation programs and 24 state facility clinics. A paired t test of fidelity data from all sites in 2020 (N=70) showed a significant mean difference between the standard and the adjusted fidelity scores, with the latter being 4 points higher (p<0.001). There was no significant difference in mean fidelity scores (standard and adjusted) between community psychiatric rehabilitation programs and state facility clinics.

From 2018 through 2020, 52 sites completed the standard fidelity scale all 3 years (28 community psychiatric rehabilitation programs, 24 clinics). Repeated measures analysis of variance (ANOVA) indicated a significant decrease in mean fidelity score among this subset of sites from 2018 (97.92) to 2020 (89.74), with a mean difference of 8 points (p<0.001). Similarly, mean fidelity score decreased from 2019 (95.83) to 2020 by 6 points (p=0.001). There was no significant difference in mean fidelity scores for all sites combined from 2018 to 2019.

In summary, the pandemic has had a clear and significant impact on self-assessed implementation fidelity, as measured by the standard IPS Supported Employment Fidelity Scale. This impact was not seen with the adjusted scale, which considers (and credits) the remote provision of IPS services.

Employment Outcomes

CPI’s IPS initiative routinely achieves employment outcomes on par with national benchmarks (4). For example, in 2019, the mean monthly employment rate of individuals receiving IPS varied from 44.8% to 52.1%, with no monthly rate below 44.8% for any month in 2019. Averaged across the 12 months, the 2019 mean monthly employment rate was 49.1%.

Examination of the mean monthly employment rates for those receiving IPS services in 2019 and 2020 indicated a clear drop in April 2020, and then a quick rebound in the following months (see section 3 of the online supplement). With a focus on employment outcomes from March through May 2020, 37 sites had all 3 months of performance indicator data. A repeated measures ANOVA of this data set (N=37) showed a significant change in mean employment rate from March (49.9%) to April (38.9%), with a mean decrease of 10 percentage points (p=0.01), and from April to May (43.7%), with a mean increase of 4 percentage points (p=0.03). In summary, outcomes were clearly affected soon after the start of the pandemic in April. They rebounded quickly in May 2020 and have maintained prepandemic levels since.

Discussion and Conclusions

Several lessons can be learned from our experience providing IPS services during this period of the COVID-19 pandemic. First, IPS is clearly durable and adaptable. Many implementation sites were able to pivot to remote service delivery relatively quickly, continuing to provide valued support through these difficult times. Despite a modest impact on fidelity as measured by the traditional prepandemic fidelity scale, the negative impact on employment outcomes was short-lived, with employment outcomes of those receiving IPS rebounding very quickly to prepandemic levels. Second, IPS outcomes rebounded even though so many services (e.g., job development) that were traditionally provided in person and in the community were being provided remotely. It appears as if, at least during the pandemic, remote services have helped to accomplish valued outcomes. It remains to be seen whether this employment trend will continue after the pandemic. Third, intermediary organizations, such as CPI, are capable of adapting their training and supports for the realities of the pandemic. They are able to pivot to new methods, new emphases, and new areas of focus to meet emerging needs and conditions.

It is important to recognize the limitations to our findings. CPI has been supporting programs across New York State with IPS implementation for over a decade. CPI’s experience in providing IPS training and support, and sites’ experiences with IPS implementation, provided a strong foundation upon which to make needed adaptations during the pandemic. These efforts may not be so easy or so successful for intermediary organizations without such extensive experience. The same may be true of implementation sites whose ability to pivot to innovative service delivery was made easier by a history of success prepandemic. Another limitation was CPI’s experience, prepandemic, with providing hybrid training and support both onsite or face-to-face regionally and online through training modules and webinars. This experience with remote training and support provision over the past decade resulted in a relatively easy transition to fully remote operations. This might have been a much more difficult, and perhaps less successful, transition without such extensive previous experience. As we look toward the future, a very important question will be which innovations in training and implementation support and in service provision should be maintained, if funded, in the postpandemic service delivery environment.