0
Get Alert
Please Wait... Processing your request... Please Wait.
You must sign in to sign-up for alerts.

Please confirm that your email address is correct, so you can successfully receive this alert.

1
Open Forum   |    
Scaling Up the Dissemination of Evidence-Based Mental Health Practice to Large Systems and Long-Term Time Frames
Sandra G. Resnick, Ph.D.; Robert A. Rosenheck, M.D.
Psychiatric Services 2009; doi: 10.1176/appi.ps.60.5.682
View Author and Article Information

Dr. Resnick is with the Northeast Program Evaluation Center (182), Department of Veterans Affairs Connecticut Healthcare System, 950 Campbell Ave., West Haven, CT 06516 (e-mail: sandy.resnick@yale.edu). Dr. Rosenheck is with the Department of Psychiatry, Yale University School of Medicine, New Haven, Connecticut.

Abstract

This Open Forum raises issues related to large-scale dissemination of sustainable evidence-based practices. Current dissemination efforts have been time limited and primarily conducted at volunteer sites with the skills of external expert trainers. The authors describe an effort to implement supported employment at 166 veterans programs in what is hoped to be a permanent addition to mental health services offered by the Department of Veterans Affairs. A two-stage process is described for developing a cadre of internal trainers who can realize the goals of this large-scale dissemination effort. Such strategies appear necessary to fully realize the broad national changes envisioned in the New Freedom Commission report. (Psychiatric Services 60:682–685, 2009)

Abstract Teaser
Figures in this Article

In recent years there have been several pioneering efforts to disseminate evidence-based practices beyond the research context and into real-world practice. These initiatives have often taken place at multiple sites within multiple mental health systems (1,2,3). However, most have been time limited, lasting no more than a few years; were spearheaded by experts external to the parent organization; and used a modest number of volunteer sites, often selected for the availability of skilled clinicians or strong leadership committed to implementing the practice. If, as the President's New Freedom Commission report urges (4), these practices are to be made universally available, methods must be developed for implementation and monitoring on a much larger scale, by sustaining ongoing intervention and including sites led by those who want to implement the practice as well as those who are skeptical. Such implementation requires a triple focus: persuading the leadership of reluctant agencies to implement the practice, achieving high levels of initial implementation, and developing procedures for sustaining the practices over time in the face of inevitable staff turnover and resulting skill loss (5).

In this Open Forum, we share our experiences from a four-year process of implementing supported employment at 166 Department of Veterans Affairs (VA) Medical Centers. This implementation is intended to be permanent and ongoing. Nationwide there are a limited number of trainers who are expert in teaching clinicians to provide supported employment, and we believe an initiative of this size and for permanent duration requires the development of self-sustaining internal expertise. We first identify the scope of the challenge and then outline how we are meeting this challenge through a cascaded training strategy created for the purpose of developing a cadre of supported employment trainers who are skilled in providing technical assistance and performing fidelity assessment within the VA.

In the first phase of the implementation, the sole national supported employment educator (supported by a national team of administrators and evaluators) focused on 21 "mentor-trainer" sites (6); in the second phase the focus shifted to the process of supporting the 21 mentor-trainer sites as they worked to disseminate the program to the remaining 145 sites throughout the country. The unique challenge of this project was to institutionalize a sustained process of training trainers to maintain program fidelity for the foreseeable future. Time-limited external demonstrations conducted thus far have not been designed to meet this challenge.

The supported employment implementation and dissemination initiative began in late 2004, after the allocation of $16 million in recurring funds through the VA Strategic Mental Health Plan for the implementation of the individual placement and support model of supported employment (7) within VA's existing Compensated Work Therapy program (6,8). The training program began with a series of national conferences and conference calls in which the principles of supported employment were reviewed and the rationale and structure of the dissemination plan were outlined. Written materials on supported employment were also distributed to each site.

At the heart of the dissemination effort was the assumption that periodic, on-site, face-to-face performance review with frank and specific feedback at multiple organizational levels is essential to effective dissemination. As mentioned above, the plan relied upon a two-stage cascading training strategy, in which 21 sites, one in each VA geographic region (the Veterans Integrated Service Networks, or VISNs), were selected as the mentor-trainer sites that would be the focus of the initial training effort. Once trained, mentors would themselves be responsible, in the second phase, for training personnel in other programs within their VISN. Conference calls, e-mail groups, and performance data were used to foster continuous communication among all involved. The mentor-trainer programs typically have three or four supported employment specialists and a supervisor, with one of these employment specialists designated as the mentor trainer. The 145 other programs are divided among 21 VISNs with mentor-trainer sites responsible for between four and ten programs in their region (6).

A central approach to fostering successful dissemination has been the use of fidelity measures that provide a methodical accounting of adherence to the principles and specific activities that define each practice (9). Fidelity scales were originally developed for use in single-site or small multiple-site clinical trials to demonstrate that the intervention being tested was being implemented as intended by its originators. More recently they have also been adapted for use as a tool to provide systematic, concrete feedback to practitioners on program areas that need adjustment in order to reach a high level of adherence to the practice. Documentation of the process and outcome of treatment at the level of each client, collected through an electronic data collection system designed specifically for this purpose, has been a central informational foundation for the national dissemination of supported employment in the VA (10). Regularly distributed reports summarizing outcome data allow each site team to compare its data with VISN and national averages (11).

For the first two to three years of the dissemination, each of the 21 sites received a series of visits every six months from an expert trainer in supported employment who communicated the goals of the program to facility leadership, provided didactic training on supported employment to the entire mental health service, and provided ongoing technical assistance for the supported employment team. At each visit, the trainer conducted a fidelity assessment and used the fidelity scale results as the foundation for providing feedback to the supported employment team as well as to the higher administrative levels of each facility. In addition, twice-monthly national conference calls and e-mail groups provided a forum for ongoing dialogue, case reviews, and exchanges of experiences. During this phase, the primary emphasis was on fostering support from facility leaders and guiding the mentor-trainer sites to reach full implementation of supported employment, although all 166 sites participated in most activities.

Once this goal was largely achieved, the energies of the expert trainer and national program staff shifted to the second, more challenging phase: the dissemination of supported employment to the 145 nonmentor sites and development of self-sustaining institutional structures. In this phase, guidance, instruction, and support were provided to the 21 mentor trainers in their efforts to offer training and technical assistance to the programs under their purview, with the eventual goal of developing the skills to provide fidelity assessments, with constructive feedback, to all sites. A more elaborate dissemination framework was developed to provide structure for this effort and to articulate the principles and values that shape supported employment mentorship.

The first goal in phase 2 was simply to provide a structural framework for the interaction of mentor trainers with supported employment participants at the 145 other VA sites. There were five structural elements in this phase.

First, a national conference was held on phase 2 of the implementation. Participants from the mentor-trainer sites were introduced to the goals and the proposed structures of this phase. The conference also established a sentinel group that was charged with undertaking this effort.

Second, mentor trainers were advised to establish a monthly conference call involving all sites in their VISN and to make two to four face-to-face site visits per year.

Third, every six months, corresponding to the mentor-trainer site's fidelity assessment, the national trainer accompanied the mentor trainer to a different site in that mentor trainer's region to perform a fidelity assessment. The mentor trainer conducted as much of the assessment as she or he was able, with the national trainer providing training and backup during the visit. This allowed the mentor trainer to receive important hands-on training in fidelity assessment and in delivery of feedback.

Fourth, mentor trainers were encouraged to meet regularly with overall VISN mental health leaders in their area and with medical center directors at each supported employment site to garner top-level support for their work with each site.

Fifth, an e-mail group and periodic conference call for mentor trainers was established. This provided mentor trainers and their supervisors with ongoing guidance and support from national supported employment program leadership, as well as opportunities for sharing experiences and discussing challenges. The conference calls were also intended to provide opportunities for skill building, problem solving, and fostering a learning community among the mentor-trainer sites.

In addition to establishing a structural framework for this large-scale mentorship effort, a training initiative in the art of supported employment mentorship was also initiated. Although it is common to utilize subject matter experts as trainers, the skill set necessary to master a skill is not likely to be the same skill set needed to teach the skills to someone else (12). There is little consensus in the literature as to what makes a good trainer and even less guidance on how to teach someone to be an effective trainer (13). Some characteristics hypothesized to be related to being an effective trainer include being approachable, supportive, willing to answer questions (14), and willing and able to provide helpful feedback. Model trainers also are respectful of trainees and encourage their self-reflection (15).

There is a similar paucity of information about the key components of effective technical assistance that lead to high-fidelity implementation (16). To address these subtler issues, plans were developed for each mentor trainer to participate in intensive two-day training with a small number of their peers at a centrally located site. No more than five mentor trainers and approximately three or four faculty participated in each workshop. Interactive training was intended to develop interviewing techniques, assessment of specific fidelity scale items, methods of teaching the philosophy and nuances of supported employment, and approaches to giving both positive and negative feedback to encourage change.

A more specific goal of this effort was to develop the mentor trainer's skills in the use of the Supported Employment Fidelity Scale for both assessment and providing feedback to guide program development. Frequent fidelity ratings are viewed as an invaluable training tool, a way to document performance for national VA administrators, and a process for supporting sustainability (5,6).

Fidelity scales were originally created as a means of systematizing expert knowledge, allowing individuals already expert in supported employment to document their fidelity to the supported employment model, often with minimal additional instruction or support. Most if not all supported employment fidelity assessments described in the research literature have been performed by a relatively small number of experts who have been hand-picked for the purpose because they already have demonstrated expertise in assessing and fostering fidelity through supportive feedback. Intensive training methods have not yet been needed, and thus they have not been fully developed or tested (17). Although manuals and instructional videos are available for training people in supported employment and use of the Supported Employment Fidelity Scale (18) (dms.dartmouth.edu/prc/employment), handing out such materials is insufficient for teaching clinicians who have limited teaching and training experience in conducting constructive and accurate fidelity assessments and providing useful feedback. Assessors need to have specialized skills, such as how to ask open, nonleading questions; how to understand and identify common implementation challenges; how to examine data critically; how to interact effectively with all levels of the organization, including leadership; and how to conduct a site visit in a supportive and nonjudgmental manner. If interventions such as supported employment are ever to become widely available, train-the-trainer procedures such as those we have outlined will need to be deployed on a large scale.

Although there have been challenges, we consider this to be a successful implementation. As of this writing, all 21 sites have received five fidelity visits, and all but one have received six over the course of the three-year implementation period. Nine (42.9%) sites are at full implementation (>65 on the Supported Employment Fidelity Scale), and the mean±SD fidelity score at the end of three years is 64.9±2.94. Mentor trainers from all 21 sites are conducting VISN-wide conference calls and conducting site visits based on the needs of the VISN, and they are receiving ongoing training on fidelity assessment by the national consultant.

Our experience with this project thus far suggests that successful implementation is facilitated by supportive leadership at the local level, familiarity of mentoring staff with rehabilitation values, and possession of both skill and experience in leadership and teaching roles. A major challenge for the mentors has been dealing supportively with skeptical administrators and clinicians who have not "bought in" to the supported employment model and genuinely do not feel a need to try a new approach. Although the implementation of supported employment is a national priority, mentors derive limited direct authority from their selection for this task and must reach out to construct alliances with local mental health and medical center leaders in support of the effort. This demands skills in the art of persuasion and alliance building as much as or more than it requires knowledge of and experience in supported employment. Staff turnover in this effort is perhaps the most serious impediment because it sometimes results in a nearly complete loss of acquired skills. At all levels, interactive communication—be it in person or via conference calls and e-mail groups—is the key vehicle for promoting the effort, because formal written materials have little impact. We have been continuously challenged to clarify goals of the program and to provide concrete guidance and support to mentors with a wide range of skills and relevant past experience.

Although great progress has been made in the dissemination of evidence-based practices in recent years, if we are ever to reach the ambitious goal of rebuilding the mental health system as outlined in the New Freedom Commission report (4), such efforts will eventually need to be carried out on a far larger scale than has been described in the literature thus far and with the explicit goal of establishing self-sustaining internal institutions. We have shown previously that by using one national trainer and a strong support team it has been possible to achieve nearly acceptable fidelity levels at almost two dozen sites (6). It will take years to collect and analyze the data generated through this effort. However, we believe that if we are to bring about the wholesale broad national change envisioned in the New Freedom Commission report, we must now begin the important dialogue on how best to scale up these efforts. We hope that this Open Forum will stimulate this discussion.

This material was based on work supported by grant MNT 05-098 from the Department of Veterans Affairs Health Services Research and Development.

The authors report no competing interests.

Drake RE, Becker DR, Goldman HH, et al: The Johnson & Johnson-Dartmouth community mental health program: disseminating evidence-based practice. Psychiatric Services 57:302–304, 2006
 
McHugo GJ, Drake RE, Whitley R, et al: Fidelity outcomes in the National Implementing Evidence-Based Practices Project. Psychiatric Services 58:1279–1284, 2007
 
Rosenheck RA, Mares AS: Implementation of supported employment for homeless veterans with psychiatric or addiction disorders: two-year outcomes. Psychiatric Services 58:325–333, 2007
 
Achieving the Promise: Transforming Mental Health Care in America. Pub no SMA-03-3832. Rockville, Md, Department of Health and Human Services, President's New Freedom Commission on Mental Health, 2003
 
Rosenheck RA: Stages in the implementation of innovative clinical programs in complex organizations. Journal of Nervous and Mental Disease 189:812–821, 2001
 
Resnick SG, Rosenheck RA: Dissemination of supported employment in VHA. Journal of Rehabilitation Research and Development 44:867–877, 2007
 
Becker DR, Drake RE: A Working Life for People With Severe Mental Illness. New York, Oxford University Press, 2003
 
Mental Health Strategic Plan Workgroup: A Comprehensive VHA Strategic Plan for Mental Health Services. Washington, DC, Veterans Health Administration, 2005
 
Bond GR, Evans LJ, Salyers MP, et al: Measurement of fidelity in psychiatric rehabilitation. Mental Health Services Research 2:75–87, 2000
 
Resnick SG, Kaczynski R, Medak S, et al: Tenth Progress Report on the Compensated Work Therapy (CWT)/Veterans Industries (VI) Program: Fiscal Year 2005. West Haven, Conn, Northeast Program Evaluation Center, 2007
 
Resnick SG, Kaczynski R, Medak S, et al: Tenth Progress Report on the Compensated Work Therapy (CWT) Program: Fiscal Year 2006. West Haven, Conn, Northeast Program Evaluation Center, 2007
 
Ricks JM, Williams JA, Weeks WA: Sales trainer roles, competencies, skills and behaviors: a case study. Industrial Marketing Management 37:593–609, 2008
 
Latham GP, Heslin PA: Training the trainee as well as the trainer: lessons to be learned from clinical psychology. Canadian Psychology 44:218–231, 2003
 
Farrant P, Cohen SN, Burge SM: Attributes of an effective trainer: implications of the views of UK dermatology trainees. British Association of Dermatologists 158:544–548, 2008
 
Boendermaker PM, Conradi MH, Schuling J, et al: Core characteristics of the competent general practice trainer: a Delphi study. Advances in Health Sciences Education 8:111–116, 2003
 
Salyers MP, McKasson M, Bond GR, et al: The role of technical assistance centers in implementing evidence-based practices: lessons learned. American Journal of Psychiatric Rehabilitation 10:85–101, 2007
 
Bond GR, McHugo GJ, Becker DR, et al: Fidelity of supported employment: lessons learned from the National Evidence-Based Practice Project. Psychiatric Rehabilitation Journal 31:300–305, 2008
 
Becker DR, Lynde D, Swanson SJ: Strategies for state-wide implementation of supported employment: the Johnson & Johnson-Dartmouth Community Mental Health Program. Psychiatric Rehabilitation Journal 31:296–299, 2008
 
+

References

Drake RE, Becker DR, Goldman HH, et al: The Johnson & Johnson-Dartmouth community mental health program: disseminating evidence-based practice. Psychiatric Services 57:302–304, 2006
 
McHugo GJ, Drake RE, Whitley R, et al: Fidelity outcomes in the National Implementing Evidence-Based Practices Project. Psychiatric Services 58:1279–1284, 2007
 
Rosenheck RA, Mares AS: Implementation of supported employment for homeless veterans with psychiatric or addiction disorders: two-year outcomes. Psychiatric Services 58:325–333, 2007
 
Achieving the Promise: Transforming Mental Health Care in America. Pub no SMA-03-3832. Rockville, Md, Department of Health and Human Services, President's New Freedom Commission on Mental Health, 2003
 
Rosenheck RA: Stages in the implementation of innovative clinical programs in complex organizations. Journal of Nervous and Mental Disease 189:812–821, 2001
 
Resnick SG, Rosenheck RA: Dissemination of supported employment in VHA. Journal of Rehabilitation Research and Development 44:867–877, 2007
 
Becker DR, Drake RE: A Working Life for People With Severe Mental Illness. New York, Oxford University Press, 2003
 
Mental Health Strategic Plan Workgroup: A Comprehensive VHA Strategic Plan for Mental Health Services. Washington, DC, Veterans Health Administration, 2005
 
Bond GR, Evans LJ, Salyers MP, et al: Measurement of fidelity in psychiatric rehabilitation. Mental Health Services Research 2:75–87, 2000
 
Resnick SG, Kaczynski R, Medak S, et al: Tenth Progress Report on the Compensated Work Therapy (CWT)/Veterans Industries (VI) Program: Fiscal Year 2005. West Haven, Conn, Northeast Program Evaluation Center, 2007
 
Resnick SG, Kaczynski R, Medak S, et al: Tenth Progress Report on the Compensated Work Therapy (CWT) Program: Fiscal Year 2006. West Haven, Conn, Northeast Program Evaluation Center, 2007
 
Ricks JM, Williams JA, Weeks WA: Sales trainer roles, competencies, skills and behaviors: a case study. Industrial Marketing Management 37:593–609, 2008
 
Latham GP, Heslin PA: Training the trainee as well as the trainer: lessons to be learned from clinical psychology. Canadian Psychology 44:218–231, 2003
 
Farrant P, Cohen SN, Burge SM: Attributes of an effective trainer: implications of the views of UK dermatology trainees. British Association of Dermatologists 158:544–548, 2008
 
Boendermaker PM, Conradi MH, Schuling J, et al: Core characteristics of the competent general practice trainer: a Delphi study. Advances in Health Sciences Education 8:111–116, 2003
 
Salyers MP, McKasson M, Bond GR, et al: The role of technical assistance centers in implementing evidence-based practices: lessons learned. American Journal of Psychiatric Rehabilitation 10:85–101, 2007
 
Bond GR, McHugo GJ, Becker DR, et al: Fidelity of supported employment: lessons learned from the National Evidence-Based Practice Project. Psychiatric Rehabilitation Journal 31:300–305, 2008
 
Becker DR, Lynde D, Swanson SJ: Strategies for state-wide implementation of supported employment: the Johnson & Johnson-Dartmouth Community Mental Health Program. Psychiatric Rehabilitation Journal 31:296–299, 2008
 
+
+

CME Activity

There is currently no quiz available for this resource. Please click here to go to the CME page to find another.
Submit a Comments
Please read the other comments before you post yours. Contributors must reveal any conflict of interest.
Comments are moderated and will appear on the site at the discertion of APA editorial staff.

* = Required Field
(if multiple authors, separate names by comma)
Example: John Doe



Web of Science® Times Cited: 6

Related Content
Articles
Books
Gabbard's Treatments of Psychiatric Disorders, 4th Edition > Chapter 22.  >
Gabbard's Treatments of Psychiatric Disorders, 4th Edition > Chapter 22.  >
Gabbard's Treatments of Psychiatric Disorders, 4th Edition > Chapter 22.  >
Dulcan's Textbook of Child and Adolescent Psychiatry > Chapter 30.  >
Dulcan's Textbook of Child and Adolescent Psychiatry > Chapter 61.  >
Topic Collections
Psychiatric News