Implementing Evidence-Based Interventions to Improve Vocational Recovery in Early Psychosis: A Quality-Improvement Report

A first episode of psychosis (FEP) usually occurs in adolescence or early adulthood (1)—a period that is crucial to developing identity and independence, building social relationships, finishing education, and gaining first experiences in employment. A psychotic episode can disrupt these important processes, which cannot always be sufficiently repaired (2). Young persons’ social and vocational recovery directly after their first psychotic episode has a strong predictive value for long-term recovery. Because it can also considerably reduce their symptoms (3), early support in social and vocational recovery is an essential element of treatment.

In a preparatory study preceding our project, Hendriksen-Favier and colleagues (4) asked adolescents and young adults who had experienced a first psychotic episode what type of care they valued most in programs for early psychosis. These young people reported first and foremost a need for help in their education and employment. They also expressed a need for more support with reduced cognitive abilities, such as memory and planning, which made them insecure and hindered them in work and education. Finally, to take control of their own recovery process, they wanted to hear positive stories of their peers’ recovery and to be involved in selecting interventions that closely matched their personal needs and objectives (4).

On the basis of these findings, we sought a suitable means of implementing three interventions targeting vocational recovery for individuals who had recently experienced FEP. The first intervention was intended to improve participation in education and employment. For this, we introduced individual placement and support (IPS), an evidence-based program that helps patients find and maintain regular employment or education (5–8). The second intervention, intended to reinforce adolescents’ cognitive skills, is a computerized program for cognitive remediation (9, 10). When combined with vocational rehabilitation, cognitive remediation can improve cognitive performance and real-life functioning in the workplace (11, 12). The third intervention, intended to strengthen adolescents’ self-management, is a computerized method to facilitate shared decision making. Shared decision making has shown positive effects on psychological well-being (13, 14) and treatment adherence (15, 16). Such general effects may also boost the effects of the two other interventions specified above.

Because there is insufficient knowledge about effective strategies for implementing evidence-based interventions in the care of individuals with severe mental illness (17–19), the question was how these interventions could best be implemented. In this study, we launched and evaluated a quality improvement collaborative (QIC) within early intervention services (EIS) for psychosis and flexible assertive community treatment teams in the Netherlands. A QIC is a structured improvement approach that organizes multiple sites to collaborate for 12 to 24 months to significantly improve a specific area of care. QICs have shown promising results in improving processes of care, with less robust findings for impacts on patient-level outcomes (20, 21).

In this article, we describe the methods used in the QIC and present results of the QIC implementation in terms of uptake, availability, and fidelity (i.e., the extent to which delivery of an intervention adheres to the standards of a protocol or program model) (22, 23). We also describe barriers and facilitators encountered during the QIC implementation. More specifically, our study aimed to examine whether implementation of these three interventions with good fidelity, uptake, and availability was possible over a period of 24 months with teams delivering early psychosis services. We also examined whether certain components of the interventions were implemented with more fidelity than others and which local barriers, facilitators, and implementation strategies had influenced implementation outcomes.

Methods

Known as the Early Psychosis QIC, the collaborative was carried out during 2014–2016. The implementation activities were accompanied by a randomized controlled trial (RCT) evaluating the beneficial effect of cognitive remediation as an add-on to IPS in EIS (a diagram and timeline are included in an online supplement to this article). Results of the RCT will be presented elsewhere, including patient outcomes on participation in employment and education (van Duin D, de Winter L, Kroon H, et al., submitted manuscript, 2020). Although formal consent is not required for the type of study reported here, which focused on the quality of care, written informed consent was obtained from all patients participating in the RCT and fidelity assessments. (The RCT and implementation study were reviewed and approved by the Amsterdam “Vrije Universiteit Medisch Centrum” institutional review board, 2014.355/NL50176.029.14.)

Implementation Approach

For the model for change, we used the Breakthrough Series, a type of QIC that has been applied in many countries for various clinical problems, mostly in general medical care (24). Generally, this implementation approach has five central features: a focus on a specific aim, related to gaps between best and current practice; participation of multidisciplinary teams from multiple sites; a national advisory board with clinical experts; a model for improvement (i.e., setting targets, collecting data, and continuous feedback loops); and a quality improvement process with a series of structured activities in a given time frame (25). The specific mix of structured activities that was offered to the participating teams during the 2-year period of the Early Psychosis QIC is listed in Box 1. These implementation strategies had three levels of focus: professionals, patients, and organization.

Participating Teams

Teams were selected after a general call to all the teams treating individuals with early psychosis that had participated in the study of Hendriksen-Favier and colleagues (4) and to teams participating in the Dutch Early Psychosis Network. Early psychosis in this project was defined to include the prodromal phase of psychosis, in which subclinical symptoms are present; FEP; and the 5-year period following FEP. Under our selection criteria, we sought multidisciplinary teams that had at least 25 patients with early psychosis in their caseload, were motivated to implement at least two of the three interventions proposed, had active leadership support, and could designate time for a local team coordinator.

Interventions

IPS

Because many first episodes occur at an age when patients have not yet finished their education, our IPS program was enhanced with a specific focus on education. For this purpose, the training of IPS coaches was extended with a module on obtaining and maintaining education and, in collaboration with an international group of experts, the IPS Fidelity Scale was extended with items for the educational component of IPS (26).

Cognitive remediation

Computerized Interactive Remediation of Cognition Training for Schizophrenia (CIRCuiTS) (see online supplement) was implemented, a Web-based program for cognitive remediation that helps users practice cognitive skills (drill) and learn new strategies (strategy) (27). During individual sessions, trained therapists supported the processes of motivating participants, enhancing metacognition, and generalizing acquired skills to daily life. Cognitive goals could focus on functioning in employment and education or on other aspects of real-life functioning.

Shared decision making

To facilitate the process of shared decision making, a computerized program called “Deciding Together” was implemented as a tool for patients to prepare for and to evaluate decisions made in consultation with their practitioner. This program (see online supplement) is an adaptation of an American example developed by Deegan (28) that was positively evaluated by mental health care consumers. In our study, patients using the program were supported during individual sessions by a peer expert from the local mental health organization. The focus of shared decision making could concern vocational rehabilitation issues or other aspects of care.

Measures

Fidelity

Fidelity rates for each intervention were obtained by rating adherence to the principles specified in the evidence-based practice models. Fidelity to the IPS model was assessed at the program level by using the 25-item version of the IPS Fidelity Scale, which has a good internal consistency of 0.88 and a fair predictive validity of 0.34 (29). Fidelity to the shared decision–making model was assessed at the intervention level by using the therapist and patient versions of the nine-item Shared Decision Making Questionnaire (30). Because the patient and therapist cannot accurately observe and recall the overall process of shared decision making, this scale requires individual therapists and patients to reflect on a recent therapy session in which a treatment decision needed to be made. The scale has a fair face validity and a good internal consistency of 0.88 in the therapist version (31) and 0.94 in the patient version (32). On the basis of the therapist manual written by the program developers, a 16-item fidelity scale for cognitive remediation was developed for this study to measure an initial “proxy” for fidelity at the intervention level. A proxy is used when direct measurement of the exact value is not possible, and calculable values are thus used instead.

For each intervention, fidelity was assessed by two external assessors at month 10 and month 22 of the QIC (6 and 18 months after the start of enrolling patients in the interventions). At site visits, both assessors rated fidelity independently and discussed discrepancies to arrive at consensus ratings. The design of the fidelity visits was similar for shared decision making and cognitive remediation, each of which involved half-day visits to interview therapists and patients working with the intervention. Additional information on cognitive remediation was gathered from the content management system of the digital program (CIRCuiTS). The IPS assessment was more extensive, involving a 1-day visit to gather information from various sources (see online supplement). For shared decision making, individual fidelity scores were calculated for each person interviewed. For IPS and cognitive remediation, one fidelity score was calculated for all team therapists and patients being interviewed.

Secondary outcomes

Information on barriers and facilitators was obtained at the start and end of the implementation period in a questionnaire for clinicians, matching the categories of the Consolidated Framework for Implementation Research (33). In this questionnaire, clinicians also rated their satisfaction with the various implementation strategies used, and they described the availability and uptake of interventions.

Descriptive Analysis

Fidelity outcomes for each intervention were established by calculating the total mean, standard deviation, and range of all fidelity items for all participating teams and persons. For shared decision making, mean scores were established separately for patients and therapists. On the basis of prior studies, a cut-off score of ≥4 on the 5-point IPS Fidelity Scale was defined as good fidelity, ≥3 as moderate fidelity, and <3 as low fidelity (34). For cognitive remediation and shared decision making, mean scores were interpreted according to the normed proportions of the IPS Fidelity Scale. To facilitate this interpretation, all item scores were also standardized to the 1–5 scale range of the IPS Fidelity Scale.

Results

Participants in this project were 14 multidisciplinary outpatient teams treating patients with early psychosis at nine mental health organizations. Each organization appointed one or two local project coordinators (N=11). Eighty-five professionals participated in the QIC, including psychiatrists, nurses, psychologists, vocational therapists, and peer support specialists.

According to their own priorities, teams selected at least two of the three interventions to be implemented. In total, 11 teams selected three interventions. Altogether, 13 teams selected IPS plus cognitive remediation (Table 1). Selection of this combined intervention automatically included participation in the accompanying trial, which was confirmed through a written statement signed by the local management. During the initial 6 months, designated practitioners received training in their chosen interventions, with two half-day sessions for shared decision making, four half-day sessions for cognitive remediation, and nine half-day sessions for IPS. In addition, to learn the key principles of each of the chosen interventions, all team members received basic training. In total, 332 patients were included in the project, with a mean of 37 patients at each organization (range, 10–98). Regarding the uptake of interventions, results showed that of the total group of 332 patients, 252 (76%) participated in the IPS program, 52 (16%) in cognitive remediation, and 39 (12%) in shared decision making (see diagram in online supplement).

Fidelity of Interventions

Table 1 presents the baseline availability of interventions and the mean total fidelity scores for each intervention at month 10 and month 22 of the QIC. At month 22, teams reached moderate-to-high mean fidelity scores for each of the two or three interventions they selected, with a mean score of 3.2 on a 4-point scale for cognitive remediation, 3.7 on a 5-point scale for IPS, and 4.9 on a 6-point scale for shared decision making.

Fidelity of Components

Item scores on the fidelity scales were analyzed to establish whether certain components of the interventions had been implemented with more fidelity than others. (Mean item scores for the three interventions at months 10 and 22 of the QIC are presented in the online supplement.)

For IPS, components that were implemented with high fidelity included integration within mental health care through team assignment (mean score IPS-Em=4.8 on a 5-point scale) and zero exclusion criteria (mean score IPS-Em=4.4/IPS-Ed=4.4). For cognitive remediation, components implemented with high fidelity included motivating patients (mean score=3.7 on a 4-point scale) and supporting patients with cognitive strategies (mean score=3.7). For shared decision making, therapists at month 10 reported a higher fidelity than did patients. Between the two assessment points, most therapist and patient scores converged.

Barriers, Facilitators, and Strategies

Table 2 presents barriers and facilitators that were encountered during implementation of the three interventions and the implementation strategies used to address them. The barriers and facilitators identified concerned the interventions, the individuals (professionals), the inner setting (organization), the outer setting (society and patients’ needs), and the process. Most implementation strategies were included in the original mix of project activities. Some of the strategies were added in response to specific barriers.

Table 3 presents elements of the QIC that were rated as the most and least useful by means of a questionnaire with a 1–5 scale. Strategies perceived as most useful were the increased sense of urgency that occurred via presentation of a central action plan, interactive training sessions, strong local leadership, and monitoring and feedback on fidelity. Most teams (83%, N=10 of 12 teams) reported that implementation of the interventions had resulted in better treatment. Two-thirds (67%, N=8 of 12 teams) reported that they had accomplished a real breakthrough during the QIC, mainly by implementing IPS.

Discussion

Fourteen teams that participated in the Early Psychosis QIC for 24 months managed to attain moderate-to-high fidelity scores on two or three of the evidence-based interventions that they chose to implement. Although it is promising that implementation with moderate-to-high fidelity was possible within 2 years by using a Breakthrough QIC, the degree of implementation was lower than that reported by some other implementation studies in this field (34–36). For example, McHugo and colleagues (34) attained an overall high-fidelity rate of 55% for the implementation of five evidence-based interventions in mental health care, including a high-fidelity rate of 89% for IPS.

The teams in our study reported several general barriers to implementation that might have contributed to the lower fidelity rates. First, they reported that an excessive burden was imposed on routine daily practice by the simultaneous implementation of three new interventions, combined with participation in fidelity assessments and in an RCT. In the project of McHugo and colleagues (34), each site had selected one or two interventions for implementation, which might in retrospect be established as a maximum if implementation is to be given sufficient priority and focus. Second, our implementation project took place shortly after the 2008 economic crisis, and thus the financial context was unstable. A third factor that may have contributed to the poorer fidelity results is the intensity of the implementation model we used. Overall evidence indicates that modest implementation efforts result in modest fidelity outcomes, which in turn result in modest improvements in patient outcomes (37). The highly intensive model of McHugo and colleagues (34), which consisted of a comprehensive implementation toolkit and up to 2 years of monthly face-to-face expert consultant contact, might have contributed to a higher degree of implementation fidelity, compared with our findings of moderate-to-high fidelity in the intensive Breakthrough QIC, which included interactive training, two national conferences, and several team visits.

Implementing IPS

Results showed a large increase in the availability of IPS. This intervention was available in three of the participating teams at baseline and in all 14 teams at the end of the QIC. In addition, a high uptake was seen; 252 of the 332 patients in the QIC were engaged in IPS (76%). However, results showed a moderate (albeit a high-moderate) mean fidelity score for IPS of 3.7 on a 5-point scale at month 22. Implementing IPS with high adherence to the principles in the model can be challenging and time consuming, because it requires changes at the level of professionals and teams, as well as at the organizational level and even at the intersectoral level (8, 38). As found in earlier studies, our study found multiple barriers on all three of these levels. At the micro level of individuals and small groups, the therapists reported fear that competitive work might be an overambitious goal for patients and that it would have a negative impact on them (8, 39, 40). At the meso level of institutions, we saw a lack of leadership and commitment of upper management to create the required financial and practical conditions (such as low caseloads, an exclusive focus on IPS for IPS workers, and organization-wide IPS implementation) (17, 39). Lastly, at the macro level of society, we encountered a national employment policy that contradicted the IPS scheme (e.g., financial disincentives to employment because of the “benefit trap” and a risk for the employer because of a strict policy related to firing staff) and the requirement for funding from multisector finance systems (38, 39, 41).

Notably, the fidelity item scores for IPS-education were similar to the fidelity scores for IPS-employment. This is remarkable, because a specific focus on the educational part of IPS has not yet been fully developed. In fact, this QIC was one of the first initiatives undertaken with such a focus. The project showed that use of the IPS-education scale was feasible. However, despite the relatively high scores on fidelity, IPS workers reported that it was harder to attain educational goals, compared with employment goals; that it was more difficult to finance educational support; and that their team needed at least one educational specialist (26). For this reason, future research is needed on the content and predictive validity of the IPS-education items, and the practice of IPS-education will have to undergo substantial development.

Implementing Cognitive Remediation

The availability of cognitive remediation also increased substantially over the course of the study. Some type of cognitive remediation was available in three of the teams at baseline and in 13 teams at the end point. This increase in availability was partly attributable to the fact that the license for the digital program (CIRCuiTS) was provided during the QIC. The uptake was low, with 52 (16%) of the 332 patients in the QIC engaged in cognitive remediation. This limited uptake was not surprising, because the program for cognitive remediation was introduced in the context of a randomized trial. Up until month 20 of the QIC, a team could offer cognitive remediation to patients only if they matched inclusion criteria and were enrolled in the trial (see timeline in online supplement). Of the 64 patients included in both the QIC and the trial, 29 were randomly assigned to the experimental condition (IPS plus cognitive remediation) and 35 were randomly assigned to the active control condition (IPS plus computer games) (see diagram in online supplement). Therefore, the uptake of cognitive remediation was hindered by these inclusion criteria and by the chance of random assignment to the active control condition. Teams reported problems with the cognitive remediation software, but they also reported that patients’ cognitive skills could be improved easily. Teams achieved a mean proxy score indicating good fidelity—3.2 points on a 4-point scale. The good score on fidelity may be due to the fact that cognitive remediation can be implemented on an individual professional level.

Implementing Shared Decision Making

The Deciding Together program was implemented to facilitate the process of shared decision making. The uptake of Deciding Together was low, with only 39 (12%) of the 332 patients in the QIC being engaged in this specific program. This low uptake appears to be correlated with poor usability and acceptability of the Deciding Together program. Teams reported that it had an unattractive interface and experienced multiple technical start-up problems related to matches with available Web browsers, log-in problems, inefficient reports produced by the application, incorrect translations, and problems with the switch to a new version of the program. These factors also resulted in a low availability; the Deciding Together program was available in six teams at the end point. However, on the basis of a modest number of fidelity scores, a mean score indicating good fidelity—4.9 on a 6-point scale—to general principles of shared decision making was achieved at the assessment at month 22. Thus therapists in teams that used the program appeared to conduct shared decision making with good adherence to the specific steps in the model.

Fidelity results for shared decision making indicated a gap between therapist and patient scores at month 10, but scores converged at month 22. Such a patient-therapist gap has previously been reported for shared decision making, indicating that professionals overestimate the extent of patient involvement, compared with the experience of patients (42, 43), and that the actual occurrence of shared decision making is limited when examined with an observer-based assessment (44, 45). Although professionals’ positive attitude toward shared decision making (46, 47) seemed to be a facilitator of implementation, two important barriers appeared to exist: insufficient knowledge of the exact definition and specific steps of shared decision making, together with a lack of the skills needed to perform these steps in daily practice. The converging fidelity scores in our study can thus be interpreted as progress, suggesting that therapists and patients had greater agreement on the definition of shared decision making, with therapists becoming more aware that there was room for improvement and patients experiencing increased sharing of decisions.

Apparently, this progress in fidelity to the shared decision–making model was accomplished despite the shortcomings of the Deciding Together program. Even though the uptake of this program was low because multiple teams did not use the program or stopped using it during the QIC, teams showed converging patient and therapist fidelity scores. Two other factors appear to have contributed to this progress: participation in the training on principles of shared decision making, including theory and role-plays concerning specific steps in shared decision making, as well as feedback from the fidelity scale assessing adherence to the specific steps in shared decision making.

Differences Between Components

Consistent with prior research (48), IPS scores indicated that within the 24 months of the QIC, sufficient implementation of any components of IPS that required structural changes at the team level was possible and attainable through a change in team policy or administrative mandate. For example, organizations could easily use team assignments to switch to integrating vocational rehabilitation and mental health care and to apply zero exclusion criteria. On the other hand, fidelity to the model was more challenging for components that required organizationwide commitment and changes. For example, it was highly challenging to organize IPS coaches from multiple teams into a single vocational unit and for executives to be supportive of IPS. In the category of “fast movers” were components of IPS and cognitive remediation in which individual professionals were able to increase fidelity scores by acquiring new knowledge and skills—such as by focusing more on competitive jobs for IPS and by teaching patients to apply cognitive strategies for cognitive remediation.

Strengths and Limitations

Our study had several limitations regarding the fidelity measures. Findings must be viewed as descriptive because of the lack of comparison sites, the low number of fidelity assessments (divided unevenly between interventions), and the risk of confounding (for example, by scales and raters). With regard to cognitive remediation, no specific validated fidelity scale was available for programs that use components such as massed practice, cognitive strategies, metacognition, and transfer to daily life—that is, components matching the cognitive remediation program that was implemented.

The method used to implement the three interventions had several strengths and limitations. Use of a multifaceted model was a strength, including learning within a collaborative, interactive training, and feedback on fidelity outcomes. However, conducting a randomized controlled trial of effectiveness that used an active control group within an implementation collaborative was more problematic than we had anticipated. Therefore, we recommend that future projects that combine these two ambitions consider research designs that interfere less with implementation goals. We should have taken more time and effort in the preparatory phases to address technical and usability problems in the software packages; inclusion of a thorough testing procedure would have helped avoid needless barriers to implementation. Given the general problems with engaging young adults in therapy after a first psychotic episode, the combination of IPS and cognitive remediation may have been too intensive for some patients. To address this issue, the goals of the separate practices could have been more strongly integrated, and a more critical estimation of the required intensity of the programs might have been helpful. Finally, the goal of implementing three interventions in the same period might have caused an initiative overload, making it too ambitious.

Conclusions

Use of a Breakthrough QIC for 24 months to implement three interventions aimed at improving vocational recovery in teams delivering services for early psychosis yielded mixed results in terms of uptake and availability and moderate-to-high results in terms of fidelity. Important barriers were the heavy burden exerted on daily practice by the simultaneous implementation of three interventions, the presence of a control condition for cognitive remediation, technical problems with digital applications for shared decision making and cognitive remediation, acceptability issues with the shared decision–making application and the need for a paradigm shift, lack of commitment from upper management, and a multisector finance system for IPS. A multifaceted implementation model (at least as intensive as the Breakthrough QIC) and a concentrated testing phase for computerized applications appear needed for implementation of these types of interventions in this setting, preferably with a maximum of two medium-intensive interventions implemented simultaneously.