Policy Implications for Implementing Evidence-Based Practices

The U.S. Surgeon General's 1999 report on mental health (1) alerted the public, mental health advocates, and policy makers to the disparity between the opportunities for improving treatment and services and the reality of everyday practice. Services and programs based on scientific advances in treatment and services are not routinely available to meet the needs of individuals who have mental illness. The report identified courses of action and called on the field to "ensure the supply of mental health services and providers" and "ensure delivery of state-of-the-art treatments."

Throughout 2001, each issue of Psychiatric Services has focused attention on this public health problem and has offered a range of responses to the Surgeon General's call to action. Various articles have reviewed individual evidence-based practices for adults and children. They have described efforts to implement these practices, highlighting facilitators and barriers, including rules, regulations, and mental health financing policies. In this article we synthesize that material, focusing on the role of policy makers in the process of implementing evidence-based practices, particularly in the public sector.

Returning to a focus on policy and administrative practices brings us full circle in the process of reforming mental health services. In the earliest stages of the community mental health and community support reforms, emphasis was placed on organizational and financing solutions to the problems of individuals with mental illness, particularly those with severe and persistent mental disorders (2,3,4). Treatment technology was comparatively weak, and the reforms focused on the locus of treatment in the community and on the system of care (4).

Evaluations of several national service demonstrations have indicated that although system reforms occurred, the direct impact of system changes on individuals was limited (4,5,6). When system interventions alone proved necessary but insufficient for improving the lives of persons with mental illness, attention shifted to the content and quality of services. Research identified both the potential benefits of services and treatments and the deficiencies in usual care (1,7).

Both policies and administrative practices have been identified as specific barriers to the implementation of evidence-based services; policies have also been identified as facilitators. Policies create incentives and disincentives that shape the mental health service system. A major challenge is to identify policy interventions that facilitate implementation of evidence- based practices but also minimize barriers to implementation. This article is addressed to policy makers and to those who advise them and who would influence their rules and regulations—namely, the rest of us.

Quality and accountability

Quality and accountability have become the watchwords of health and mental health services (8). Implementing evidence-based practices has become a means to achieving both ends. In this context "quality" means positive outcomes obtained by using cost-effective services, and "accountability" means documentation of adherence to evidence-based practice.

Michael Hogan, commissioner of mental health in Ohio, refers to a triangular relationship among these three service system elements: quality improvement, accountability through performance measurement, and evidence-based practices. He describes this relationship as central to providing effective mental health services (personal communication, Hogan M, 2001). Implementing evidence-based practices is a quality-improvement process that provides accountability through the monitoring of the fidelity of practices to models that have been demonstrated by research to be effective.

Using this framework, policy makers can approach their funders with greater confidence. They can argue for resources to implement evidence-based practices with greater assurance of accountability and value for money. Monitoring for adherence to evidence-based practices is possible through the use of fidelity measures. Programs that are faithful to the evidence-based models produce good outcomes in general, but not necessarily for all individuals or for all circumstances. Achieving consistently positive outcomes is at the heart of the definition of an evidence-based practice.

With common agreement about the validity and appropriateness of these positive outcomes as policy goals, the quality of mental health services can be continually improved. Measures of fidelity, like other process measures, are a means to an end, not an end in themselves. It is critical that fidelity to a particular model or practice not be regulated in a way that prevents client choice, clinical judgment, or continuing change as new evidence emerges. Yet fidelity should be a goal to which systems and practitioners aspire, with the assumption that the greater the fidelity, the greater the likelihood of good outcomes.

Unfortunately, although the Surgeon General concluded that a range of efficacious treatments exists for almost every mental disorder, for many clinical conditions there is no evidence to support particular treatments or services. For example, although effective treatments are available for schizophrenia and bipolar disorders, many patients with these disorders have complications and comorbid disorders that have not been considered in studies of treatment effectiveness. In many cases, the existing evidence comes from clinical trials that may not be generalizable without adaptation to typical treatment settings—for example, the trials may have been conducted by clinicians with specific levels of training or with homogeneous patient groups.

For some problems with the greatest salience—such as youth suicide, posttraumatic stress disorder, and borderline personality disorder—there is not yet a satisfactory research base to guide policy and practice with clarity, although the evidence base for each of these problems is growing. Rosenberg and colleagues (9) have suggested that while we wait for definitive answers to emerge, policy makers hold off on endorsing specific models and instead support studies of comparative effectiveness.

Not every problem has an evidence-based solution, and not every evidence-based practice that works for a majority of persons who have similar symptoms, history, and needs will work for all such individuals. There continues to be much room for clinical judgment, client choice, and development of innovative treatments and services. However, evidence-based practices do exist for certain clinical conditions, as documented in the pages of this journal throughout the past year. Yet too often these practices are not implemented, even when their benefits are well understood; when clients, clinicians, and policy makers agree on desired outcomes; or when models exist of successful implementation.

States are moving forward in their implementation of evidence-based practices with varying levels of commitment and success. Many are struggling with the implementation of evidence-based practices that have existed for more than a decade and that have been proven effective in a variety of settings. Even when states have had the political and administrative will—or at least the stated interest—to implement evidence-based practices, they have not always done so by using mechanisms that ensure adherence to fidelity. And even when evidence-based services have been implemented with fidelity, systems have had to address questions of how these fit with each other and with services that lack a strong evidence base.

Many factors contribute to these implementation problems, including lack of a long-term vision for the service system, lack of agreement on desired outcomes, lack of penalties for practices that are not evidence based, short-term horizons for policy planning, political mandates or competing public-sector priorities, resource limitations, and uncertainty associated with change and untoward events. In such a context, administrative practice and the policy infrastructure are of paramount importance.

Overcoming systemic barriers

Although the focus has shifted from organization and financing to the content and quality of services, policy makers cannot ignore the systemic barriers to implementing evidence-based practices. Each of the articles on evidence-based practices has identified barriers related to organizational policy and financing policy, and some have identified strategies for overcoming those barriers and creating appropriate incentives to support implementation. We use the eight courses of action outlined by the Surgeon General to organize this section.

Continue to build the science base

As we have noted, there are limitations in the treatment-effectiveness research base that defines the evidence-based practices. More research is needed to determine whether these practices are effective in all ethnic subpopulations, among persons who have multiple disorders, and in all practice settings—for example, rural as opposed to urban settings. In addition, more research is needed on nontraditional approaches that give clients more control of their own recovery or that utilize professionals trained in nontraditional methods.

Furthermore, although thousands of studies have been conducted on dissemination of innovation and implementation of health and mental health services, there is virtually no definitive evidence to guide implementation of specific evidence-based practices. However, some experts, such as Argyris (10), warn that the results of experimental studies that involve human interaction may not generalize to any great degree to typical treatment circumstances, because the complexity of social systems cannot be captured in controlled experiments. There is uncomfortable irony in moving forward to implement evidence-based practices in the absence of an evidence base to guide implementation practice.

Torrey and colleagues (7) reviewed some of the literature on dissemination and implementation but uncovered more about what we do not know than about what we do know. The literature is better at telling us what does not work and what not to do than it is at guiding our work. We intend to study the earliest experiences with evidence-based practices to identify significant barriers and successful strategies to inform future implementation efforts.

Overcome stigma

Few of the authors in the evidence-based practices series in Psychiatric Services identified stigma as a special barrier to implementing evidence-based practices. However, a special section of four research articles in this issue of the journal examines stigma as a barrier to recovery. It is possible that the pervasive stigma associated with mental illness and its treatment has resulted in discriminatory financing policies. As a result of stigma, individuals who are in need are unwilling to seek care. They experience forms of discrimination that can exacerbate their illness if they do seek treatment. In addition, stigma often produces service delivery systems that view mental health treatment as less valuable or necessary than general health care.

For example, all too often Medicaid does not cover the evidence-based practices or covers them in a way that precludes faithful implementation of the model. With lack of fidelity comes the risk of losing the positive outcomes documented in the research. Furthermore, there is growing evidence that budgets for public mental health systems are eroding (11). All the authors in the evidence-based practices series identified financing policies as barriers to implementing evidence-based practices.

Improve public awareness of effective treatments

All the articles began with a careful description of evidence-based practices. It cannot be assumed that all readers of Psychiatric Services are familiar with all the evidence-based practices, let alone understand all the barriers to and facilitators of implementation. Although awareness alone is not sufficient for implementation, it is certainly a necessary first step. Consumers and family members can affect the demand for evidence-based services if they are aware of the benefits associated with these services (12). Evidence from general medical care supports the effectiveness of raising awareness (13). Providers—both clinicians and administrators—must understand the new practices and their utility before they can be expected to adopt them. The same, of course, is true for policy makers.

Ensure the supply of mental health services and providers

Ensuring the supply of mental health services and providers, along with the next course of action—ensuring the delivery of state-of-the-art treatments —is at the heart of the matter. Policy makers have a responsibility to ensure that individual clinicians and service providers are available in their mental health systems. This responsibility involves making a commitment to recruiting individuals who have the necessary skills to deliver evidence-based services, creating incentives to attract these individuals to practice in their systems, and training, supervising, and supporting the work of providers of evidence-based services.

Retaining skilled providers and minimizing job burnout are critical to maintaining a workforce that is capable of supplying evidence-based services. According to the Surgeon General and the authors of the articles in the Psychiatric Services evidence-based practices series, there is a shortage of trained personnel who are able to provide the evidence-based services described. The erosion of the resources of state mental health programs undermines the ability of mental health agencies to attract and retain competent clinicians. It will be necessary to develop mechanisms for retraining the current workforce and to influence the training of new professionals and paraprofessionals.

The Evidence-Based Practices Project, which is described in more detail below, is designed to increase the number of individuals and clinical service teams who are able to practice in a manner that is supported by research findings. Some practices require that consumer-providers and family members receive special training. All need informed and engaged individuals at all levels of service provision—consumers, family members, clinicians, program administrators, and policy makers.

Without these informed and committed administrators and policy makers, no amount of literature or evidence will matter, and no amount of accountability through measurement of fidelity will increase public commitment to seeking or funding mental health care. Fidelity will give way to whatever clinicians can get paid for, and accountability will give way to whatever questions funders want answered. Program administrators need assistance in understanding the need, making the case, and sustaining the effort to lead systems either to promote evidence-based practices or, at least, to get out of the way.

Ensure delivery of state-of-the-art treatments

Each of the authors of the papers on evidence-based practices reinforced the need for leadership in implementing state-of-the-art practices. The authors also pointed out that ensuring delivery is not a trivial matter. Evidence-based practices must be a priority for care. Architects of the mental health system must organize services with quality improvement in mind. Regulations often impede the implementation of evidence-based practices. It is not possible to deliver state-of-the-art treatments if, for example, newer antipsychotic medications are not on the formulary of a program, or if an insurer does not cover family interventions.

Regulations may create unanticipated barriers. For example, supported employment may not be an approved service for Medicaid reimbursement. Most states cannot afford to offer evidence-based services without Medicaid coverage; often, a majority of individuals in public-sector programs are not eligible for Medicaid. Organizational and financial barriers to integrated treatment have been identified for supported employment (between vocational rehabilitation and mental health agencies) and for integrated treatment of co-occurring substance abuse and severe mental illness (between separate substance abuse and mental health service authorities). This is a special problem in which federal mental health and substance abuse block grant funds cannot be mingled to provide integrated care. Overcoming these agencies' divisions is often an important first step in the effort to provide better-integrated services. On the other hand, some of these services, such as assertive community treatment, are designed to provide the services themselves instead of relying on a fragmented service system.

Tailor treatment to age, sex, race, and culture

Although the research base is not sufficient to support all the evidence-based practices with each of the sociodemographic groups encountered in practice, it is always important to be culturally sensitive and respectful of diversity when designing and delivering services. It is also important to realize that, for the most part, when research on evidence-based practices has been conducted in ethnic subpopulations, the outcomes have been good. As emphasized by the Surgeon General, tailoring treatment will be of special importance in situations in which "culture counts" in specific ways (14).

For example, family interventions must take into account the cultural meanings of family and respect the differences in meaning associated with age, sex, and stage of the life cycle. Language-appropriate services are critical to successful outreach and for encouraging members of linguistic minorities to use evidence-based services. Medications should be used appropriately, with an awareness of ethnopsychopharmacologic variations in physiology and in attitudes and behaviors associated with drug taking. In addition to being faithful to program models, evidence-based services must reach out and include everyone in a community who might need or benefit from the services.

Facilitate entry into treatment

In most cases, people cannot benefit from evidence-based treatments if they do not seek help. Occasionally treatment is provided under a court order, but in general the goal is to have consumers receive services on a voluntary basis. Evidence-based services must be available and accessible, and, as noted above, they should be inviting. The Surgeon General expressed the belief and the hope that evidence-based practices will reduce the need for coercion in mental health services. He encouraged multiple "portals of entry" to services by creating incentives for many service providers to receive referrals and accept all individuals seeking services (1,14).

Subsequently, individuals can be matched with appropriate evidence-based services that are provided by specially trained clinicians, teams, and programs within the service system. Not every service provider will offer all the evidence-based services, but every clinician and provider organization should offer choices of some of the evidence-based services that are delivered in their organization or elsewhere in the system. There should be no "wrong door" for services. Awareness of evidence-based practices and of where such services can be received is essential information for the contemporary mental health service system.

Reduce financial barriers to treatment

No single policy issue received more attention from the authors of the papers in the evidence-based practices series than the adequacy of financing. Realistically, a service is not available if a person with a mental illness cannot afford to use it or a program cannot afford to provide it for the price offered by payers. It is a simple truism that a service system runs on its financing policies. If evidence-based practices are not covered services, or if the fees paid are below the cost of providing them, they will not be used.

Until very recently, Medicaid policy almost uniformly discouraged assertive community treatment. Federal block grant regulations have complicated the funding of integrated services for individuals who have co-occurring disorders. Payment for multifamily groups is not always covered or reimbursed adequately. The same may be true for various components of self-managed care. Newer medications may not be on the formulary of a pharmacy benefit plan, or copayments may discourage the use of newer agents. Supported employment may not be reimbursed at a rate that compares favorably with the rate that could be obtained through a sheltered workshop.

These are recurrent issues in every discussion of barriers to implementing evidence-based practices. The remedy is self-evident—remove unreasonable financial barriers. However, these policies are often out of the decision-making purview of the mental health authority. Working on these policies with other agencies has become the standard approach for supporting the implementation of evidence-based practices. Resources must support the transition to evidence-based practices in agencies that have historically been involved in older practices. It is difficult to be motivated to learn a new practice if the old practice generated the agency's revenues. Policy makers and administrators need the tools to shift funding in a logical and incremental manner from old ways of practice to new ways. They also need the resources—both human and financial—to provide technical assistance or quality oversight to ensure that funds are being spent in new ways rather than in old ways that have new names. Funds are needed to offset the opportunity costs associated with learning a new practice.

By and large, the move to evidence-based practices will not be accompanied by a permanent increase in resources. Many successful implementations have occurred when agencies have switched from an older practice, such as brokering case management or rehabilitation-oriented day treatment, to a new practice, such as assertive community treatment or supported employment. These agencies benefit from additional one-time-only resources to support the transition to evidence-based practices.

Implementation might be enhanced by better planning among the agencies responsible for financing care—federal, state, and local authorities—to develop the necessary incentives for implementing and sustaining evidence-based practices. To provide adequate financing, planners also need accurate information about the costs of providing evidence-based services. As with other aspects of the research, cost data from experimental studies often are not generalizable to usual care settings. Cutting across all these courses of action is the need for informed leadership from mental health policy makers and administrators—and increasingly from other sectors, such as Medicaid, the criminal justice system, vocational rehabilitation services, and the education system.

Most authors of the papers in the series indicated the need for a dedicated individual and for infrastructure to support the implementation of evidence-based practices. Infrastructure with continuity of leadership in implementation is important because of the frequent turnover of state mental health program directors. This type of infrastructure is also important in efforts to move from research or pilot projects to systemwide implementation. What may be conceptualized by a clinical or policy leader in an administrative office and supported in the throes of change may become compromised when multiple practitioners or providers or multiple locations are involved.

Infrastructure is needed not only to provide assistance for both leaders and implementers to sustain changing practices but also to change again as new evidence emerges. This capacity for managing change is often not present in public-sector settings that are buffeted by the political or public priority of the day. The necessary research and resources—for training, ongoing support, and travel—to move from a pilot project to full-scale implementation are needed if evidence-based practices are to be implemented broadly and sustained over time with at least a modicum of fidelity.

Infrastructure to support systemic change

Without a template to guide them, various mental health authorities have developed similar infrastructure to support systemic change toward evidence-based practice and quality improvement. Leadership is critical for sorting through all the treatment recommendations and guidelines that are being promoted by various organizations and for developing an evidence standard for assessing practice. Agencies such as the Substance Abuse and Mental Health Services Administration, the National Institute of Mental Health (NIMH), and the Agency for Healthcare Research and Quality as well as foundations such as the Robert Wood Johnson Foundation and the MacArthur Foundation have supported these efforts. Organizations such as the National Association of State Mental Health Program Directors (NASMHPD) and its research institute and the National Alliance for the Mentally Ill (NAMI) have established initiatives and partnerships to promote evidence-based practices.

The Evidence-Based Practices Project began in New Hampshire, Maryland, and Ohio and has spread to several other states. Each of these three states developed its own center for implementing evidence-based practices, taking advantage of local opportunities and preferences. Each state has created its own model and priority practices for implementation. The project has stimulated some cross-fertilization, so the centers share many of the same functions, but the differences are illustrative and might encourage other states to develop similar centers of their own. Each center is sponsored at least in part by the state mental health authority.

Each center views its mission as supporting the implementation of evidence-based practices, which involves training, supervision, ongoing clinical and administrative support in the new practice, and structural support with regulations and financing technical assistance. Each of the centers sponsors needs assessment activities, training events, and various services that support the implementation of evidence-based practices. The centers work with all the stakeholders—the state and local mental health authorities, program administrators, clinicians and other providers, and consumers and their families.

In New Hampshire, the West Institute for Implementing Evidence-Based Practices is a partnership between the state and a private family foundation. The institute grew out of the well-established public-academic linkage between the state mental health authority and Dartmouth Medical School. It is affiliated with the New Hampshire-Dartmouth Psychiatric Research Center, where several of the evidence-based practices were developed and evaluated. The Evidence-Based Practices Project is run out of the West Institute and the New Hampshire-Dartmouth Psychiatric Research Center. The centralized model in New Hampshire is well suited to a small state with a single academic center.

In Maryland, the Center for Implementing Evidence-Based Practices is a newly established center within the Maryland Mental Health Service Improvement Collaborative. Sponsored by the state mental health authority, the center is an outgrowth of the original collaborative that has been devoted to providing training and conference opportunities for service providers in Maryland. Like the New Hampshire center, the Maryland center is a key element of one of the oldest public-academic liaisons in the country, between the department of psychiatry at the University of Maryland and the Mental Hygiene Administration. The specific link is with the Center for Mental Health Services Research, which together with the Johns Hopkins University includes the NIMH-funded research center that conducted the Schizophrenia Patient Outcomes Research Team (PORT) study.

The PORT study was one of the first to identify and explore the major problem of the disparity between research and practice (1,15). No private funds have yet been obtained to support the center, but a network grant from the MacArthur Foundation to the university may fund pilot research on implementing evidence-based practices in these centers.

In contrast with the centralized model used by New Hampshire and Maryland—both comparatively small states—Ohio uses a decentralized approach. The coordinating centers of excellence are a series of centers—currently eight, but ten are planned —decentralized throughout Ohio. Most are linked to a research-oriented institution, either a university or a private-sector entity, that specializes in one area of evidence-based practice. In Ohio, where there are multiple small research centers and where local mental health authorities are largely autonomous and statutorily responsible for mental health services, the decentralized and specialized approach makes the most sense. Some of Ohio's coordinating centers of excellence focus on practices for which there is substantial research evidence; others focus on important areas such as school-based mental health services that cannot wait for an evidence base to accumulate before some guidance is provided to local mental health authorities.

In Texas, statewide implementation of evidence-based practices has occurred through collaboration with academic centers and stakeholder groups, who advocate for resources, as well as through contractual requirements, including financial sanctions, with local mental health authorities. These collaborations have also resulted in major research initiatives related to the implementation of evidence-based practices.

The NASMHPD and the NASMHPD Research Institute, using a grant from NIMH to advance their research on evidence-based practices, are coordinating these research efforts at the level of the state mental health authority. The NASMHPD Research Institute has created a Center on Evidence-Based Practices, Performance Measurement, and Quality Improvement to support state efforts to implement evidence-based practices and to monitor the quality and impact of the services being provided.

The functions of the center are to identify, share, and promote knowledge about evidence-based practices, performance measurement, and quality improvement; conduct research and develop knowledge; provide technical assistance; and coordinate activities across organizational entities and levels of government. Several other states are involved in the project and have their own approaches to infrastructure development. Private entities, such as the nonprofit Institute of the Technical Assistance Collaborative, are emerging to provide technical assistance related to infrastructure and policies to support evidence-based practices.

Conclusions

The time has come to add to the body of knowledge about implementing evidence-based practices at different levels, including knowledge about policy, program priorities, clinician practice, consumer adherence, and family member support. However, implementation at the policy level is both primary and paramount. The national initiative embodied by the project is one of the most important innovations on the mental health horizon. It will serve as the testing ground for what can be learned about bridging the gap between science and service.

This important initiative will not go far if it is not supported by mental health policies—at state and federal levels—that create the organizational and financial incentives to implement evidence-based practices. In addition, it will be a time-limited activity if it does not also yield lessons about how to adapt to new evidence and ongoing systemic changes. Organizations must be flexible and must be able to learn and adapt.

The promise of decades of research must be realized in practice. The Surgeon General simultaneously identified the promise and documented the shortcomings. His report outlines courses of action for policy makers that should guide us away from service disparities and that support the implementation of evidence-based practices. We have the opportunity to combine quality improvement with accountability through performance measurement and the implementation of effective new services and treatments.

Acknowledgments

Support was provided by grants R24-MH-53148 and P50-MH-43703 from the National Institute of Mental Health and by the Center for Mental Health Services, the Robert Wood Johnson Foundation, and the John D. and Catherine T. MacArthur Foundation.

Dr. Goldman is affiliated with the department of psychiatry at the University of Maryland School of Medicine, 685 Baltimore Street, MSTF Building, Room 300, Baltimore, Maryland 21201 (e-mail, [email protected]). Dr. Ganju is with the National Association of State Mental Health Program Directors Research Institute in Alexandria, Virginia. Dr. Drake is with Dartmouth Medical School and the New Hampshire-Dartmouth Psychiatric Research Center in Lebanon, New Hampshire. Mr. Gorman is with the West Institute for Implementing Evidence-Based Practices in Lebanon, New Hampshire. Dr. Hogan is commissioner of mental health in Ohio. Ms. Hyde is with the Technical Assistance Collaborative office in Santa Fe, New Mexico. Mr. Morgan is director of the Maryland Mental Hygiene Administration in Baltimore.