State Mental Health Policy: Scaling Up Evidence-Based Behavioral Health Care Practices in New York State

In November 2007, the New York State Office of Mental Health (OMH) and the Department of Psychiatry, Columbia University, established the Center for Practice Innovations (CPI) at Columbia Psychiatry and New York State Psychiatric Institute to promote the widespread use of evidence-based practices throughout New York State. CPI uses innovative approaches to build stakeholder collaborations, develop and maintain practitioners’ expertise, and build agency infrastructures that support implementing and sustaining evidence-based practices. CPI works with OMH to identify and involve consumer, family, provider, and scientific-academic organizations as partners in supporting the goals of OMH and CPI. CPI receives the bulk of its funding from OMH through the Research Foundation for Mental Hygiene. Therefore, CPI has always enjoyed strong support from OMH leadership.

This column describes the activities of CPI, including the use of learning collaboratives and a learning management system, as well as processes undertaken by CPI to ensure continuous quality improvement.

CPI’s activities and learning collaboratives

CPI has five core initiatives that provide training in five areas: treatment of co-occurring mental and substance use disorders (called Focus on Integrated Treatment or FIT), assertive community treatment (ACT), supported employment and education via Individual Placement and Support (IPS), wellness self-management (WSM) (an adaptation of Illness Management and Recovery), and treatment of first-episode psychosis (called OnTrackNY). CPI plays secondary roles in other state training initiatives, such as increasing the use of clozapine, suicide prevention, and tobacco dependence treatment (as part of the FIT initiative). CPI also convenes panels of experts to review research evidence and provide practical recommendations around practices that OMH is considering adopting or changing (1).

CPI’s initial charge was to provide training for the New York State behavioral health care workforce. Given the size and geographical dispersion of this workforce, CPI turned to distance learning technologies. Distance technologies may offer cost-effective alternatives to typical training methods, and some evidence suggests that such technologies are at least as effective as face-to-face training (2). CPI has collaborated with key stakeholders and content experts to create more than 40 award-winning online modules to provide training for each initiative. Industry awards from 2010 to 2013 include Brandon Hall, Horizon Interactive, Silver Omni Intermedia, , and Communicator. CPI also earned a 2010 Science to Service Award from the Substance Abuse and Mental Health Services Administration for developing practical resources, tools, training materials, and evaluation methods that have and continue to successfully assist mental health agencies to adopt and sustain WSM (3).

As of February 28, 2014, a total of 14,376 participants had completed 160,650 modules. Programs participating in CPI initiatives are geographically dispersed through New York State, reaching 61 of the 62 counties (the remaining county, contained fully within a state park, is the least populous county in the state, indeed in the eastern half of the United States). The modules use personal recovery stories, clinical vignettes, interactive exercises, frequent knowledge checks, and expert panel presentations to engage the learner. Each core initiative now provides electronic learning modules as part of its training and implementation curriculum. In addition to online training, CPI conducts an array of regional and on-site agency face-to-face training.

Although the initial charge to CPI was to provide training, CPI has increasingly moved toward providing supports for programs to implement evidence-based treatments. CPI offers participating programs online resources and other supports to help programs build capacity to implement these practices. Implementation supports include interactive webinars, an online resource library with practical tools, consultations (both in person and by telephone), and monthly conference calls for participants in specific initiatives during which program staff share successes and receive expert consultation from peers and implementation experts on their implementation challenges. Consultations may follow difficulties with licensing reviews or staff turnover or are provided upon request for technical assistance.

Recently these components have been combined into a training approach that blends online supports with face-to-face meetings organized around learning collaboratives. Learning collaboratives (4) have been used as a vehicle for disseminating evidence-based practices in health care, behavioral health care, and other areas (5,6). To join a learning collaborative, participating programs commit to forming implementation workgroups whose role is to develop an implementation plan and oversee work toward the goals of that plan, to participate in learning collaborative meetings, and to collect and submit performance indicator data. Some learning collaboratives meet exclusively online, whereas others offer a blended approach (both face-to-face and online meetings). Feedback from participants in learning collaboratives has been overwhelmingly positive. Participants have noted how the experience of learning from experts and one another has helped them to effectively implement particular practices. Positive improvements in performance indicator data support this assertion.

CPI’s learning management system

Creating CPI’s online modules and resources required the creation of an online learning platform, a learning management system (LMS), that facilitates access to online training, event registration, and resource libraries for each initiative. For each user, the LMS automatically tracks what modules are viewed, registration for and attendance at training events (for example, webinars and face-to-face trainings), and what modules are completed. CPI, via its LMS, also collects several levels of evaluation from learners: satisfaction with and usefulness of training, knowledge mastery, self-efficacy, and reports of practice change as a result of the training. In addition, the LMS provides program managers with the ability to assign training and with real-time reports of staff members’ status regarding required training.

CPI has been using this LMS for nearly five years to deliver and track training and implementation supports across its initiatives. CPI staff extract data from the LMS in a variety of formats that upload easily into statistical programs to be merged with outside data sources, such as performance indicator data and other program-level information collected externally. Because the LMS tracks individuals’ progress, CPI staff can summarize training data in a number of ways (for example, by program, by agency, by region, by job title, and by program type).

Quality improvement and incentives

In April 2012, CPI conducted a self-evaluation of activities following guidance from the implementation literature. By this point, CPI had created a solid LMS infrastructure and a significant online course library on which to build additional tools and supports to enhance its work. For example, CPI began to collect additional program information (self-reported fidelity and performance indicator data) using online tools outside the LMS. CPI merges this information with LMS data to inform a quality improvement process to fine-tune each of its offerings. CPI has engaged in this process transparently with programs to demonstrate the positive impact of their feedback and to model a quality improvement process.

For example, CPI’s IPS supported employment initiative works with four dozen community mental health programs to use fidelity self-assessments, learning collaborative performance indicators, and data routinely reported to the OMH in a process of continuous quality improvement. These data allow programs to identify challenges in IPS implementation and work with CPI staff to address these challenges. These same data, in aggregate, also allow OMH to identify systems issues needing attention. For example, many programs report difficulty in assigning employment staff to IPS duties off site, a critically important aspect of this approach.

Similarly, the ACT initiative provides technical assistance and consultations to individual ACT teams. Consultations and technical assistance are initiated and guided by data routinely reported by teams to OMH. Using a data-driven and outcome-based approach to assist ACT teams in improving outcomes has allowed for more intensive and purposeful work, both at the team and at the individual ACT provider levels.

In addition to internal adjustments, CPI has increased its attention to the outer setting (7), particularly with respect to external policies and incentives, including policy and regulations, external mandates, recommendations and guidelines, pay for performance, and public or benchmark reporting (8). This has led to additional regulations and mandates, updated recommendations and guidelines, fiscal and other incentives, and the use of data to help benchmark programs and encourage continued improvement. For example, beginning in January 2013, outpatient mental health clinics became eligible to receive an increased reimbursement rate for participation in a quality improvement project. As part of the project, clinics were required to complete a set of related CPI training modules. In March 2013, OMH also revised its guidance documents for clinics so that participation in FIT is necessary if a clinic wishes to be considered “exemplary” for several standards of care. Given these changes, clinic participation in FIT has increased and continues to rise—for example, from 49% (211 of 427 adult mental health clinics) in December 2012 to 68% (338 of 499 adult mental health clinics) in February 2014.

Separate from these program-specific interventions, beginning in August 2012, OMH and the New York State Office of Alcoholism and Substance Abuse Services collaborated with CPI to offer a training certificate, signed by commissioners from both agencies, to practitioners who complete all of the basic, intermediate, and advanced competency modules of integrated treatment for co-occurring disorders, including three devoted to tobacco dependence treatment. In the eight months before the certificate was offered, total module completions averaged 1,641 per month; in the eight months after introduction of the certificate, completion rates averaged 6,344 per month (a nearly fourfold increase). As of February 28, 2014, a total of 1,282 individuals had earned this training certificate.

Conclusions

CPI has evolved from a center providing high-quality training to a center that not only provides this training but also supports programs to implement evidence-based treatments with fidelity. As CPI’s reputation has grown, it is increasingly receiving requests from a variety of community stakeholders, from individuals to managed care plans, asking which provider agencies have staff who have completed various CPI trainings. In addition, CPI is also exploring how to respond to numerous requests from providers outside New York State who are interested in taking advantage of its training and supports.