Use of Learning Collaboratives by the Center for Practice Innovations to Bring IPS to Scale in New York State

The Center for Practice Innovations (CPI) is funded by the New York State Office of Mental Health (OMH) to bring best practices to mental health programs across the state (1). Large systems face the challenge of helping practitioners in diverse programs adopt evidence-based practices. This challenge includes reaching all programs, regardless of whether they would be naturally inclined to participate. Because of the large number of programs and the large geographic area, New York State (NYS) cannot afford traditional approaches to dissemination, such as use of consultant trainers for individual agency training.

To encourage community programs to adopt best practices, OMH developed a program model in 2004 that set a clear expectation in regard to the implementation of recovery-oriented evidence-based practices. Through funding policies, OMH provided incentives for adoption of these practices. These Personalized Recovery Oriented Services (PROS) programs are designed for adults living in the community who have a diagnosis of serious mental illness. The programs help them identify and achieve personally meaningful recovery goals. The number of PROS programs has increased significantly: in early 2014, 86 programs were serving 10,600 individuals.

OMH leadership identified supported employment as a key service in PROS programs. Rates of competitive employment of mental health consumers have historically been very low. PROS programs across NYS routinely report competitive employment rates of less than 10% among program participants, similar to rates in other states (2,3).

IPS and LCs

Individual placement and support (IPS) has been recognized as the evidence-based approach to supported employment for more than a decade (2,4). A study that focused on 127 sites involved in the Johnson & Johnson–Dartmouth Community Mental Health Program found that the mean quarterly employment rate for individuals receiving IPS services over eight years was 41% (5,6). Employment was defined as working in a competitive job for one day or more in the quarter. IPS principles include zero exclusion, integration of employment and mental health services, competitive employment as the goal, and rapid job search. Key services include developing a meaningful employment plan that reflects the consumer’s wishes, as well as job development and job supports (7).

Learning collaboratives (LCs) have been used in health care and are increasingly used in behavioral health care. LCs were initially patterned on the Institute for Healthcare Improvement Breakthrough Series (8), and variations have been developed (9). Fundamental elements include a number of organizations working together, using quality improvement methods to close the gap between potential and actual performance, learning from experts as well as from one another, and using data to track performance (8). Reviews have reported promising results in regard to the impact of LCs, but they are cautious in reaching firm conclusions because of difficulties comparing studies and other methodological issues (9–11).

The NYS initiative builds on the LC model designed by the team that developed IPS with the goals of guiding and supporting IPS dissemination in 12 states and the District of Columbia (12). The Johnson & Johnson–Dartmouth national LC used a two-tiered approach. IPS researchers used LCs to work with mental health and vocational rehabilitation leaders at the state level, and these leaders in turn worked with programs in their states. The LC approach has helped states sustain and expand IPS, improving quality and achieving good outcomes. The NYS LC initiative works directly with large numbers of programs.

CPI’s Dissemination of IPS

In 2013, CPI worked with 47 of 77 PROS programs across NYS that were clustered into four regional LCs. In 2014, the number of participating programs increased to 59 of 86, clustered into seven regional LCs. OMH strongly encouraged participation, but no fiscal or other incentives were provided. Supported employment is funded through a combination of Medicaid and state dollars. CPI staff (3.2 full-time-equivalent [FTE] trainers-consultants) helped these programs by leading LC meetings, providing guidance concerning the development of quality improvement plans, collecting and analyzing data submitted by the programs, leading informational webinars and conference calls, and providing onsite program-specific training and consultation. The LCs provide both face-to-face and online training and support activities. Programs agree to attend LC meetings, conduct periodic fidelity self-assessments, submit monthly performance indicator data, develop and use IPS quality improvement plans, and share experiences with one another.

LCs offer the following training and support activities: two online training modules, one providing an overview of IPS and the other focusing on the practitioner skill of job development; statewide webinar and regional online meetings focusing on important topics (for example, IPS fidelity and supervision) and processes (for example, using data to drive continuous quality improvement efforts) (in 2013, one webinar and 24 online meetings were conducted, and from January to May 2014, four webinars and eight online meetings were conducted); an IPS library, available through CPI’s learning management system, that provides archived webinars, presentations from past training events, and tools to help IPS implementation; regional face-to-face meetings where programs learn from each other (eight meetings in 2013 and 15 from January to May 2014); individualized consultations, both at the program site and by telephone that focus on addressing implementation challenges and enhancing practitioner competencies (83 site visits in 2013 and 45 from January to May 2014); and special interest conference calls focusing on issues identified by participating programs (for example, implementation in rural settings) (five calls from January to May 2014).

It is important to note that between LC meetings, participants engage in follow-up work, including completion of online training; submission of performance indicator data, including caseload size and in-person employer contacts and employment outcomes; and development and use of quality improvement plans based on program-specific data.

Data collected in 2013 produced several findings. In 2013, a total of 49 (64%) of 77 eligible programs joined the initiative, and two dropped out before the end of the year. Thus over 60% of eligible programs chose to join and remain with the initiative throughout the year, and 96% of programs that enrolled remained actively involved throughout 2013. By early 2014, 59 (69%) of 86 eligible programs had joined. Improved IPS fidelity has been noted. For the 47 programs that began in 2013 and remained in the LC, 46 had submitted self-reported fidelity data, using the Supported Employment Fidelity Scale (13) adapted for NYS PROS programs. Fidelity increased significantly from January to September 2013 (from 91.3±16.7 to 96.1±14.7 out of a possible score of 125; p<.05). (The adapted scale is available online at adobeformscentral.com/?f=zO40AePRbfe73yRJIGhRZg.)

In addition, the programs have achieved national IPS benchmarks. In November 2013, CPI revised the LC performance indicators, including definitions of key concepts, to be more consistent with indicators collected in national studies (indicators are available at adobeformscentral.com/?f=DlAWcIQ1sQVHnkdHaENlfw). For 41 of the 47 programs that used the revised indicators to provide data for November 2013, we found that IPS caseload per FTE staff (calculated by dividing the total caseload across the 41 programs by total FTE staff at those programs) achieved expected national standards: 19.6 consumers receiving IPS services per one FTE. The statewide average rate of employment of consumers on the IPS caseload (calculated at the program level and then averaged across the state) is within the range reported nationally for effective IPS programs: 48% over the past month and 44% over the past three months worked at least one day of competitive employment. There were 1,045 individuals statewide on the past-month caseload and 1,596 different individuals on the caseload over the past three months. These data do not represent a fixed cohort because the IPS caseload fluctuates monthly.

Data collected in 2013 indicated one serious limitation and some continuing challenges. Programs reported a mean of 1.3 FTE employment staff per PROS program (median=1.0, mode=1.0). The overall program census per program ranged from 41 to 373. The limited availability of employment staff suggests that a very small number of program participants can receive IPS services. This has a dramatic impact on the absolute numbers of program participants who will find competitive employment. Across all PROS programs, the percentage of individuals enrolled on the last day of the year who were competitively employed increased from 8.6% in 2012 to 9.5% in 2013. Although the trend is in a positive direction, the low number of employment staff severely limits the achievement of better employment outcomes. In addition, difficulties with specific aspects of fidelity have been noted. The September 2013 fidelity self-assessment identified, on average across programs, difficulties with staff assignments, including the number of staff assigned to employment duties; the percentage of time spent in the community providing IPS services, including job development; and expectations in regard to staff completion of tasks not related to employment.

Promoting Sustainability and High Fidelity

CPI and OMH are examining ways in which OMH can strengthen or clarify policies, expectations, monitoring systems, and incentives for achieving better employment outcomes and increasing program involvement in CPI’s IPS initiative. OMH has looked for opportunities to communicate a clear expectation that supported employment services are a crucial component of PROS programs. This communication occurs in meetings and conference calls with program leaders. In addition, OMH issued a guidance document that details the manner in which IPS services can and should be incorporated into PROS program design. OMH monitors program involvement in the IPS initiative through detailed monthly reports provided by CPI. OMH monitoring of implementation and employment outcomes occurs in several ways: through fidelity and performance indicator data provided by programs to CPI, through data provided by programs directly to OMH, and through program licensing visits, which now focus more fully on IPS activities and employment outcomes.

OMH is also committed to aligning incentives with desired outcomes. It provides PROS programs with state funds to assist with IPS implementation. OMH is examining ways in which these funds can be made contingent on involvement with CPI’s IPS initiative, IPS fidelity, and employment outcomes. In addition, through the national Ticket to Work program, programs can now receive funds from the federal government that are contingent on employment outcomes. OMH has designed an administrative system that will facilitate programs’ access to this important incentive.

Creative Planning for the Future

CPI and OMH are working with program leaders to find creative ways to increase staff involvement in employment activities. Because low current overall employment staffing levels are likely to continue, employment must become a significant focus of other (nonemployment) staff members. CPI plans to build on the IPS foundation now in place and to look at creative adaptations and facilitators that incorporate IPS principles into employment practices designed specifically for the PROS environment. One such facilitator is the Employment Resource Book (14), which was developed by CPI and funded in part by the Employment Development Initiative of the Substance Abuse and Mental Health Services Administration and by the National Association of State Mental Health Program Directors. Informed by IPS principles, the book is designed for use by consumers on their own as well as with employment staff members, other practitioners, and peer specialists. The book’s 32 topic areas and ten appendices cover three critical time periods: before the job search, during the job search, and after getting a job. Information, personalized activities, and next steps in each topic area are described. We hope that this resource will inspire and empower consumers to pursue employment and that it will increase staff involvement with employment activities.

Conclusions

Reports by participating programs suggest that a state mental health authority (OMH) working closely with a training center (CPI) can indeed “move the bar” toward dissemination of IPS across a large state, resulting in improved fidelity and outcomes that are consistent with national benchmarks. The self-reported nature of these data, however, limits our ability to reach firm conclusions. Structural and fiscal barriers have resulted in difficulties with implementation for some programs (such as the small number of staff assigned to employment duties and the nature of their assignments). We have learned that planning efforts must go well beyond training to include a focus on these barriers.