Practices that have been demonstrated to be effective by clinical services research could improve the lives of many people if they were widely adopted in routine health care settings. The desire to promote evidence-based practice has led to a proliferation of practice guidelines throughout medicine (1,2). As Rogers noted (3), government and professional organizations formulate guidelines "to help translate the scientific literature into concise statements intended to change practice." However, studies of the impact of practice guidelines suggest that publication and distribution of guidelines is not enough to change the practice of clinicians (2,4,5,6).
In the mental health field, several groups have published recommendations, guidelines, consensus statements, and treatment algorithms that are relevant to the care of adults with severe mental illnesses (7,8,9,10,11,12,13,14,15). Although healthy debates continue about the types and levels of research evidence, the efforts of these groups highlight the fact that much is known about how to help persons living with severe mental illnesses. For example, research strongly supports the use of specific medications prescribed in specific ways as well as the use of psychosocial interventions such as supported employment, various approaches to illness self-management, family psychoeducation, case management based on the principles of assertive community treatment, and substance abuse treatment that is integrated with mental health treatment (16).
More significant than disagreements about levels of evidence is the fact that the practices supported by research are not widely offered in routine practice settings. In the most extensive demonstration of this problem, the Schizophrenia Patient Outcome Research Team (PORT) showed that patients with a diagnosis of schizophrenia in two state mental health systems were highly unlikely to receive effective services (17). Even simple medication practices meet standards of effectiveness about half or less than half of the time. A minority of patients—often as few as 10 percent—receive psychosocial interventions supported by effectiveness research.
Evidence from other sources supports the PORT findings. For example, Anderson and Adams (18) noted that family psychoeducation is rarely available in routine practice settings, and Tashjian and associates (19) reported that fewer than 5 percent of persons with severe mental illnesses receive supported employment services.
Thus a critical challenge for the mental health field is to facilitate the widespread adoption of research-based practices in routine mental health care settings so that persons with severe mental illnesses can benefit from services that have been shown to work (20). In this paper we review the essential ingredients of an implementation plan for evidence-based practice.
This paper is part of a series of reports that stem from the Implementing Evidence-Based Practices for Severe Mental Illness Project, which is sponsored by the Robert Wood Johnson Foundation, the Center for Mental Health Services of the Substance Abuse and Mental Health Services Administration, the National Alliance for the Mentally Ill (NAMI), and state and local mental health organizations in New Hampshire, Maryland, and Ohio. The project aims to develop implementation toolkits for effective practices. Subsequent papers will describe the project in more detail and address the issue of what constitutes evidence.
To inform the design of the implementation toolkits, we assessed five complementary sources of knowledge: the literature on efforts to change practice in health care; the discussions the authors had with executive administrators from provider organizations in New Hampshire, Maryland, and Ohio during several days of meetings on implementing evidence-based practice; the findings of focus groups with frontline clinicians who serve people with severe mental illnesses in New Hampshire and Baltimore; the advocacy experiences of NAMI; and the observations of services researchers who have implemented a range of mental health services in demonstration progiams.
Implementing a practice involves promoting change in the behavior of mental health care providers. Theorists suggest that behavior changes when intention to change is combined with the necessary skill and the absence of environmental constraint (21). Green's model (22,23) for promoting change prescribes three elements: predisposing or disseminating strategies, such as educational events or written material; enabling methods, such as practice guidelines and decision support; and reinforcing strategies, such as practice feedback mechanisms.
Research shows that education alone does not strongly influence the practice behaviors of health care providers (24,25,26). Additional efforts, such as increasing consumer demand for services, changing financial incentives and penalties, using administrative rules and regulations, and providing clinicians with ongoing supervision and feedback on practices, are also necessary (24,27,28). The more elements of the system of care that can be marshaled to support change and reduce resistance, the more likely practice improvements will occur. In other words, intensity of effort appears directly related to success in studies of practice change (29,30).
Furthermore, complicated changes, such as modifying the practice of a whole clinical team, require a greater intensity of effort than is needed to affect a relatively simple change, such as shifting a single prescription pattern (3,29). Guidelines are not self-implementing and must be contextualized to the actual processes of care. Sustained change requires a restructuring of the flow of the daily work so that routine procedures make it natural for the clinician to give care in the new way (31).
Katon and colleagues (32), for example, were able to improve primary care treatment of depression to conform to treatment guidelines developed by the Agency for Health Care Policy and Research by using an intensive, multifaceted program that included extensive physician education and restructuring of services. They found, however, that the improved care did not generalize to other depressed clinic patients seen outside the restructured intervention program, even when the patients were seen by the same physicians, and did not endure after the processes of care reverted to their previous form when the year-long intervention was complete (25).
Research also indicates that when mental health programs attempt to implement evidence-based practices, the quality of the implementation strongly influences outcomes (33,34,35,36). In other words, if two programs offer a practice of care that is known to be effective, the program that has the higher fidelity to the defined practice tends to produce superior clinical results. This finding suggests that efforts to promote evidence-based practice must include fidelity measures and self-correcting feedback mechanisms.
Implementation efforts are most effective when they address the specific needs, values, and concerns of the persons whose behavior the implementation aims to change (37,38). Specifically, administrative elements of an implementation plan must be tailored to mental health administrators, clinical training elements to clinicians, and consumer and family education to those groups.
Mental health administrators influence the implementation of evidence-based practice by setting priorities for care and organizing the operational details of practice. During meetings with the researchers in New Hampshire and Baltimore, administrators from organizations that provide services to adults with severe mental illnesses strongly urged the development of organized resources to help programs offer evidence-based practices. Having access to clearly defined practices that are known to work and that are packaged for implementation would help administrators in their efforts to meet their responsibility to the communities they serve. High turnover of clinical staff can easily overwhelm the training resources of any one agency and underscores the need for focused, effective training materials that can be used repeatedly.
Administrators indicate that financial incentives and administrative rules and regulations must be aligned to support the implementation of evidence-based practices. The overall financing structure must be relatively stable and support the effort. To sustain evidence-based practice over time, ongoing funding must be competitive with the potential income from alternative clinical staff activities. Additional financial concerns of administrators are the cost of initial training and the cost of collecting clinical outcome data and measuring fidelity over time. Training must be efficient because the cost of training includes both the direct costs and, in fee-for-service settings, the opportunity costs of the training time. The processes for tracking clinical outcomes and practice fidelity need to be streamlined because data collection takes time and energy away from direct clinical service.
To understand the needs of clinicians who directly provide services to adults with severe mental illnesses, we conducted a series of seven focus groups with case managers, psychiatrists, vocational specialists, and community support program therapists from numerous practice sites in New Hampshire and Baltimore. The purpose was to ascertain what factors motivate them to change, how they learn a new practice, and what they perceive as barriers to change.
Clinicians indicated that, first, they must be convinced that the practice is worth learning; second, they need to learn a practice through observation, training, and reading; and, third, they benefit from efforts to reinforce the practice over time, such as regular supervision and feedback on activities.
Clinicians are generally not eager to change and must be convinced to adopt a new practice. In addition to research support, they are influenced by compelling vignettes, impressions of the practice seen in action, and a practice ideology, or theory, that resonates with their values and experiences as providers. Practices that can be learned and put into action quickly are more appealing than those that require intensive learning or dramatic system change.
Clinicians are particularly motivated to learn a new practice if they believe it will help them in a clinical area where they currently feel ineffective or if they feel there will be broad consumer demand for the practice. An opportunity to discuss rationale, theory, and concerns is helpful to clinicians as they evaluate a proposed practice.
Once convinced to adopt a practice, clinicians need practical instruction. At this stage, most clinicians indicate that seeing the practice in action has more of an impact than reading about it. They endorse traveling to a training site where the practice can be observed, being visited by a skilled trainer who can demonstrate the practice, or watching instructional videotapes. Written materials complement and expand understanding gained from observing the practice. Clinicians prefer practical workbooks with clear and simple aims, principles, and examples, rather than books. The workbook should provide a clear structure of practice with examples and challenging questions. Ideally the workbook should lay out the practice in brief, easy-to-read modules allowing a clinician to read a section and immediately try the intervention.
Clinicians cite the importance of supports to consolidate and reinforce new practices. Practice-specific supervision of their clinical work is essential. Supervision helps clinicians translate the theory into daily action. Supervision can be done by an expert at a distance over the phone or, preferably, by a skilled local program leader who can motivate others to learn. Strategically placed posters listing key practice principles or relevant practice slogans support the supervision and reinforce learning through repetition. Follow-up trainings and feedback on practice are also helpful. Clinicians suggest that internet resources, advanced seminars, and additional readings may all be helpful for people who have the basics and want to extend their understanding and skill.
Advocacy can help systems move to research-based practice. For example, the National Alliance for the Mentally Ill (NAMI) has had success promoting the Program for Assertive Community Treatment (PACT). By focusing on the replication of PACT as a national priority, packaging the practice for implementation, engaging the media, coordinating NAMI state organization efforts, and communicating progress, NAMI has created a grassroots demand for PACT. Active NAMI PACT steering committees are working with providers to establish PACT in 19 states.
From the PACT dissemination effort, NAMI has learned the importance of designing communication to match the needs of the participants and reinforcing learning through repetition. NAMI created printed material to capture the attention of policy makers and administrators at the state and local levels, built information for family members into family education programs, and developed a detailed implementation manual for providers. NAMI promotes PACT at every opportunity, including media events, annual NAMI conventions, state conferences, and on the Web.
Observations of researchers
The final source of knowledge about practice implementation is the combined experience of the authors from implementing and studying medication algorithms and psychosocial interventions in routine practice settings. Our experiences make it clear that a core set of steps is required to successfully establish and maintain a desired practice. These include clearly voiced administrative support for change before training; initial clinical training using didactic methods, observation of practice, and written materials; ongoing weekly supervision by an expert, based on written principles and practices; follow-up visits by a program expert with feedback on implementation; and feedback on services and outcomes.
Model programs often dissipate after demonstration or research projects end. Practices that persist have funding and organizational structures to support the practice. These programs typically have an influential clinical or operational leader who ensures the continuity of the structures and training that maintain the practice (39,40,41,42,43).
On the basis of the knowledge reviewed above, the team for the Implementing Evidence-Based Practices for Severe Mental Illness Project proposes a model for effecting change. The team will develop integrated written material, Web-based resources, training experiences, and consultation opportunities, packaged as implementation toolkits. The toolkits will be designed to promote the consistent delivery of effective services and will be developed for practices for which the evidence base is extensive and the consensus supporting the practice relatively high. The impact of the toolkits on practice will be carefully studied to learn more about how to translate research-based knowledge into practice in the care of adults with severe mental illnesses.
Clearly, practice implementation can easily fail. To succeed, the system of care must have adequate resources and be reasonably organized, and the efforts of multiple stakeholders must be aligned to support the practices. Stakeholders who play essential roles include mental health authorities (funders), who create administrative rules and financial incentives; administrators of mental health organizations, who set priorities and organize care; mental health clinicians, who provide direct care; and service consumers and their families, who create demand for and receive mental health care. The toolkits will address all of these stakeholders.
The first challenge is to predispose stakeholders to work on restructuring services. For each practice, materials will include a brief video that describes the evidence-based practice, reviews the scientific support, and provides testimony from clients and clinicians who have participated in the intervention. The video will link the research evidence for the practice with the personal experiences of people who have benefited from receiving the intervention, thereby making the practice come to life for administrators, clinicians, and consumers and their families.
Written material for each practice will also include a list of commonly asked questions and answers, a paper summarizing the scope and limits of the evidence supporting the practice, and a general paper articulating the rationale for adopting practices that are strongly supported by research. In addition to the materials, introductory training and consensus-building group facilitation will be offered to actively engage stakeholders in the change process.
Second, once the stakeholders are predisposed to change, the toolkits will enable them to make the desired changes. For administrators of mental health authorities, implementation toolkits will include a brief document specifying the funding structures and administrative rules that create effective incentives, administrative consultation, and links to other administrators who have found mechanisms to promote specific evidence-based practices. To support change efforts by administrators of provider organizations, implementation toolkits will include general recommendations for promoting change in health care settings; a brief document that lays out recommended practice processes, such as staffing, training, meeting structure, supervision, and fidelity monitoring; and administrative consultation to help organizations overcome obstacles to establishing the practice.
Implementation toolkits for clinicians will recommend initial training, including job shadowing at a site where the practice is well established. The primary ongoing training tool will be a practical workbook that articulates the aims of the practice and the practice principles and provides clear examples. Additional supports for clinicians will include extended practice-specific supervision by trainers to help clinical supervisors learn to apply practice principles to their specific clinical dilemmas, posters articulating the practice principles, and a Web site for the practice to link providers with other relevant research, training opportunities, and materials. For consumers and their families, a booklet will be developed that describes the practice, indicates reasonable expectations about services and outcomes, and refers people to organizations that advocate for practices, such as NAMI.
Third, once an evidence-based practice has been established, organized practice feedback will help stakeholders maintain and extend the gains. To provide system feedback, implementation toolkits will include fidelity scales for program self-assessment, simple outcome measures to track the effects of the practice, follow-up trainings either at the practice site or a training site, and a recommended process for review and revision of the implemented practice to further adapt it to site-specific circumstances and to continue to engage stakeholders in implementation.
Extensive empirical research demonstrates that several pharmacological and psychosocial interventions are effective in improving the lives of persons with severe mental illnesses. Despite this knowledge, there has been widespread failure to implement evidence-based practices in routine mental health settings. In this report, we have recommended ingredients of an implementation plan for evidence-based practices based on the perspectives of the research literature, administrators of provider organizations, clinicians, family advocates, and services researchers. The effectiveness of the proposed plan will be studied and the findings shared with the field.
This article was supported by contract 280-00-8049 from the Center for Mental Health Services, a grant from the Robert Wood Johnson Foundation, and grants MH-00839 and MH-56147 from the National Institute of Mental Health. The authors thank Paul B. Batalden, M.D., for his comments on an earlier draft of this article.
Editor's Note: This article by Dr. Torrey and his colleagues initiates a series of papers on evidence-based practices that is planned for monthly publication in 2001. These papers will focus on mental health practices for which there is substantial evidence of effectiveness and should therefore be routinely offered in clinical settings. Other articles in the series will address evidence-based practices for supported employment, for patients with dual diagnoses, for case management, for children's services, and for family psychoeducation, among other topics. Robert E. Drake, M.D., Ph.D. (firstname.lastname@example.org), and Howard H. Goldman, M.D., Ph.D. (email@example.com), are the series editors. Please contact them about submitting papers.
Dr. Torrey is associate professor of psychiatry, Dr. Drake is professor of psychiatry, and Dr. Clark is associate professor of community and family medicine and psychiatry at Dartmouth Medical School in Hanover, New Hampshire. Dr. Dixon is associate professor of psychiatry at the University of Maryland in Baltimore. Dr. Burns is professor of medical psychology at Duke University Medical Center in Durham, North Carolina. Ms. Flynn is former executive director, National Alliance for the Mentally Ill in Arlington, Virginia. Dr. Rush is professor of psychiatry at the University of Texas Southwestern Medical Center at Dallas. Dr. Klatzker is chief executive officer of the Riverbend Community Mental Health Center in Concord, New Hampshire. Address correspondence to Dr. Torrey at the New Hampshire-Dartmouth Psychiatric Research Center, 2 Whipple Place, Lebanon, New Hampshire 03301 (e-mail, firstname.lastname@example.org).