Implementing Evidence-Based Practices in Routine Mental Health Service Settings

An important focus of Psychiatric Services in 2001 is on the implementation of evidence-based interventions in mental health care. In last month's issue (1), the journal initiated a series of papers on implementing evidence-based practices for the care of persons with severe mental illnesses in routine mental health service settings. Papers in this series will describe the conceptual framework of a national demonstration project, the Evidence-Based Practices Project, which is sponsored by the Robert Wood Johnson Foundation, the Center for Mental Health Services of the Substance Abuse and Mental Health Services Administration, the National Alliance for the Mentally Ill, and several mental health research centers, state mental health authorities, and local mental health programs in New Hampshire, Maryland, Ohio, North Carolina, and Texas.

The goal of the project is to develop standardized guidelines and training materials, in the form of toolkits, and to demonstrate that the toolkits can be used to facilitate the faithful implementation of evidence-based practices and to improve client outcomes in routine mental health service settings.

In this paper we present the concept of evidence-based practices. We describe the background of the movement toward such practices and define evidence-based practices and related concepts. We also discuss common concerns about the use of evidence-based practices in routine mental health service settings.

The rationale for evidence-based practices

Psychiatric Services' decision to dedicate 2001 to evidence-based practices rests on a series of research findings and philosophical commitments. First, a great deal is now known about efficacious and effective mental health interventions, which we refer to here as evidence-based practices. For example, numerous recent reviews of the research evidence identify a core set of interventions that help persons with severe mental illness attain better outcomes in terms of symptoms, functional status, and quality of life (2,3,4,5,6). The core set includes medications prescribed within specific parameters, training in illness self-management, assertive community treatment, family psychoeducation, supported employment, and integrated treatment for co-occurring substance use disorders (7). In upcoming issues, experts will review research evidence on outcomes, present current knowledge about barriers and strategies related to implementation, and discuss implications.

A second reason for the journal's focus on evidence-based practices is that despite extensive evidence and agreement on effective mental health practices for persons with severe mental illness, research also shows that routine mental health programs do not provide evidence-based practices to the great majority of clients with these illnesses (8). This finding was a major conclusion of the mental health report of the Surgeon General (6). In the most extensive demonstration of this problem, the Schizophrenia Patient Outcome Research Team showed that in two state mental health systems, clients with a diagnosis of schizophrenia were highly unlikely to receive effective services (9). For example, antipsychotic medications were often prescribed at dosages outside the effective range. A minority of clients—often as few as 10 percent—received evidence-based psychosocial services, such as family interventions. Findings from other sources suggest a dearth of evidence-based practices in routine mental health settings (10).

Third, research indicates that offering a service that resembles an evidence-based practice is not sufficient; adherence to specific programmatic standards, often referred to as fidelity of implementation, is necessary to produce expected outcomes (11,12,13). In other words, if two programs offer a practice of care that is known to be effective, the program with higher fidelity to the defined practice model tends to produce superior outcomes. This critical finding, which contradicts the conventional wisdom that model programs do not transfer and need to be modified extensively to fit local circumstances, suggests that implementation guidelines and toolkits should begin to incorporate manuals and fidelity measures (14).

Fourth, mental health services for persons with severe mental illness should reflect the goals of consumers. People with severe mental illness, like people with other long-term illnesses, want to pursue normal, functional, satisfying lives to the greatest extent possible (15,16). Mental health services therefore should not focus exclusively on traditional outcomes of treatment compliance and prevention of relapses and rehospitalizations. The new paradigm emphasizes helping people attain outcomes such as independence, employment, satisfying relationships, and good quality of life.

Fifth, given that mental health resources are limited, persons with severe mental illness have a right to have access to interventions that are known to be effective and that are delivered in a manner faithful to or consistent with current understandings of the interventions' active ingredients. In other words, to the extent that evidence-based practices exist, they should be the bedrock, the minimum of acceptable offerings, in all mental health settings that provide services for persons with severe mental illness. Additional services may enhance the service offering and may prove to be effective over time, but the basic offering of evidence-based practices should not be displaced by interventions of unknown or lesser effectiveness. Researchers, state mental health directors, consumers, and families support this commitment and have endorsed the urgent need for dissemination and implementation of evidence-based practices (17).

Finally, evidence-based practices do not provide the answers for all persons with mental illness, all outcomes, or all settings. Below we discuss several caveats as well as the judicious and informed use of evidence-based practices. Accurate information about effective practices, including a clear understanding of the limitations of current research and the need for further evidence in many areas, can improve services and outcomes. However, to produce positive changes, the research evidence on effective services and the implementation procedures must be available to all stakeholders in the service system (1).

What are evidence-based practices?

Evidence-based practices are interventions for which there is consistent scientific evidence showing that they improve client outcomes. For example, research shows that using antipsychotic medications within specific dosage ranges and providing education and skill training for family caregivers over several months prevents or delays relapses of schizophrenia (4). The requirements for scientific evidence used by different groups sometimes vary, but in general the highest standard is several randomized clinical trials comparing the practice to alternative practices or to no intervention. When the separate trials are considered together, such as through a meta-analysis, the evidence supports the superiority of the evidence-based practice over the alternatives, including no intervention.

In some situations quasi-experimental studies, with comparison groups that are not assigned by randomization, constitute the best available evidence and consistently support a specific evidence-based practice. Rather than rigid decision rules, which inevitably are more appropriate for some types of interventions than others, we encourage panels of research scientists to review the available controlled studies and to make explicit their criteria for inclusion, the studies reviewed, the review procedures, and the conclusions so that others can examine the same evidence and reasoning.

Open clinical trials, which lack independent comparison groups, provide a significantly lower level of research evidence and are generally not considered to provide sufficiently strong scientific evidence. The lowest level of evidence, which should not be considered research evidence, consists of clinical observations collected as expert opinion. Best practices based on clinical opinions or open clinical trials do not constitute evidence-based practices, because they are not research based, are fraught with potential for error, and are often contradicted by later findings of controlled research.

Some groups, such as the Agency for Healthcare Research and Quality, have identified levels of scientific evidence, which they use to score evidence-based practices. The various practice guidelines developed in the 1990s by the agency—then known as the Agency for Health Care Policy and Research (18)—exemplify this approach by using three levels of evidence: level A refers to good research-based evidence, with some expert opinion. Level B indicates fair research-based evidence, with substantial expert opinion, to support the recommendation. Level C denotes a recommendation based primarily on expert opinion, with minimal research-based evidence.

Although the term evidence-based practice is sometimes used to refer to guidelines that are not based on research, true evidence-based practices are by definition grounded in consistent research evidence that is sufficiently specific to permit the assessment of the quality of the practices rendered as well as the outcomes. For example, supported employment is sufficiently well defined and standardized so that its fidelity can be measured and it can be differentiated from other approaches to vocational rehabilitation (14).

Other guidelines, such as those developed by the American Psychiatric Association (19), are based on a mixture of randomized clinical trials and expert consensus and often lack sufficient specificity to provide tools for judging the quality of interventions. There are also consensus guidelines, such as those developed by the triuniversity consortium (20), that systematically define desired practices through an iterative consensus process among experts. The advantage of the expert consensus approach is that it permits development of guidelines about practices for which no systematic research evidence exists. The major disadvantage is that the expert opinions may reflect current biases in the field rather than demonstrated effectiveness. At the far end of the continuum are guidelines that are written primarily to serve some administrative function and that are supported neither by research evidence nor by expert consensus.

Treatment decisions are typically complex, involving multiple decision steps that depend on the patient's response to treatment at each step. This complexity has led to the development of treatment algorithms that map out a series of decision points based on responses to the previous steps. For example, an algorithm for pharmacologic treatment of a person experiencing acute symptoms of schizophrenia may begin with initiation of an antipsychotic medication within a specific dosage range. If the person fails to respond adequately to the treatment, the algorithm may specify a second option—for example, switching to another class of antipsychotic agent. Failure to respond to the second treatment would lead to a third decision point—for example, initiation of clozapine treatment. Such algorithms may be particularly useful in guiding clinicians through a complex series of treatment decisions.

As with guidelines, algorithms may have varying levels of scientific evidence to support them. A current challenge in algorithm development is that the scientific evidence supporting the successive steps quickly becomes quite thin. Hence even the most evidence-based algorithms typically begin with steps supported by multiple clinical trials and evolve into steps defined through expert consensus. An excellent example of treatment algorithms has been developed in the Texas Medication Algorithm Project (5).

Questions about the useof evidence-based practices

A number of questions arise in relation to the recent emphasis on evidence-based practices. For example, what is the role of ethical values in shaping practices? What should be done when there are different levels of evidence or changes in evidence? What are the limits of evidence? What should be done in clinical situations for which there is no scientific evidence? We briefly address each of these points below and will do so in greater detail in upcoming issues of the journal.

Values

Mental health services appropriately incorporate humanistic values, ethical principles, and legal standards. For example, addressing adult clients as adults, interacting benignly and respectfully with families, and showing sensitivity with respect to age, sex, race, and cultural background are core values of the health care system (21). Research is not required to support these standards, and nothing about evidence-based practices contravenes the importance of such standards. On the contrary, this discussion assumes that evidence-based practices must incorporate consensual values, ethical principles, and legal standards.

Nature of the evidence

As noted, research evidence is often complicated, with inconsistent results and differences in the design, quality, and number of studies of any single intervention. Moreover, the evidence evolves rapidly. Thus the need for scientific review is critical and ongoing. For policy purposes, however, the transition from existing practices to evidence-based practices is more clear-cut. Existing practices typically rely on tradition, convenience, clinicians' preferences, political correctness, marketing, and clinical wisdom—none of which is consistently related to improving outcomes. Historically, clinical practice in mental health is often completely unhelpful and sometimes tragically harmful, as in the case of psychosurgery.

The crux of the matter is that precisely because evidence-based practices are grounded in the qualifications imposed by current science, they are standardized, replicable, and effective. A switch to research-based interventions with known effectiveness can dramatically improve outcomes in large practice systems—for example, the overall rate of employment of persons with severe mental illness (22).

Limitations of the evidence

Research evidence on interventions is often quite specific with respect to population, outcome, and context. For example, research clearly demonstrates that assertive community treatment effectively reduces hospital use for clients with schizophrenia who are unstable, homeless, treatment resistant, or hospital prone, especially in settings in which the alternative services are hospital based or clinic based (23,24,25). But the evidence is less clear in several areas. Does assertive community treatment improve functional outcomes such as employment? Should clients who are more stable be given assertive community treatment? What about different ethnocultural groups? What about clients with other diagnoses, such as borderline personality disorder, posttraumatic stress disorder, or substance use disorders? Is assertive community treatment needed in settings in which the usual services have incorporated its basic features such as outreach?

The central strategy in defining evidence-based practices is to be straightforward about the limits of the evidence. The provision of evidence-based practices even under circumscribed conditions would be an improvement and would move stakeholders toward awareness of the potential of other evidence-based practices. For example, research indicates that assertive community treatment does not consistently improve vocational outcomes and that supported employment must be a well-integrated component of the intervention to achieve high rates of competitive employment (25). Thus providing assertive community treatment should increase pressure to implement supported employment.

Extensions of the evidence should be careful, logical, and able to be evaluated. Because supported employment improves employment outcomes for white, African-American, and Hispanic-American clients, offering it to other cultural minorities would be a logical decision, unless there are obvious cultural differences in the meaning of work. On the other hand, logic dictates that family psychoeducation may have different meanings and effects in different cultures, and some negative research findings among Hispanic-American families (26) reinforce caution about extending the research on family psychoeducation to other groups (27).

In some situations, essentially no research exists on treatment outcomes, such as treating hepatitis C infection or the sequelae of trauma among people with severe mental illness or evaluating the effects of self-help services. The emphasis on evidence-based practices should create pressure to develop and test these interventions to fill the need for informed rather than expedient public policy. The corpus of evidence-based practices is not static, and outcomes that are valued by consumers and families should influence which interventions are developed and studied.

Conclusions

For each area of evidence-based practices, implementation in routine mental health practice settings is complex and difficult (1). Issues of organizational structure and commitment, resource development, and clarity of roles and responsibilities must be addressed before training can be effective (28). Service boundaries are often involved as well. For example, supported employment involves the interface between clinical and rehabilitative services, and dual diagnosis services highlight differences between the mental health and substance abuse treatment systems. Upcoming articles in Psychiatric Services will address implementation barriers and strategies as well as the specific evidence, including boundaries and extensions, for each of several evidence-based practices. Emphasis on evidence-based practices also has implications for public policy, education, research, medical information systems, managed care, liability, and many other topics.

Dr. Drake and Dr. Mueser are professors of psychiatry and Dr. Torrey is associate professor of psychiatry at the New Hampshire-Dartmouth Psychiatric Research Center, 2 Whipple Place, Lebanon, New Hampshire 03766 ( [email protected] ). Dr. Leff is assistant professor of psychiatry at Harvard Medical School and at the Cambridge Hospital in Cambridge, Massachusetts. Dr. Goldman and Dr. Lehman are professors of psychiatry and Dr. Dixon is associate professor of psychiatry at the University of Maryland School of Medicine in Baltimore.