Advanced Search

Evidence-Based Practices | September 01, 2001

Evidence-Based Practice in Child and Adolescent Mental Health Services

Kimberly Hoagwood, Ph.D.; Barbara J. Burns, Ph.D.; Laurel Kiser, Ph.D.; Heather Ringeisen, Ph.D.; Sonja K. Schoenwald, Ph.D.

Psychiatric Services 2001; doi: 10.1176/appi.ps.52.9.1179

Article

References

text A A A

Abstract

The authors review the status, strength, and quality of evidence-based practice in child and adolescent mental health services. The definitional criteria that have been applied to the evidence base differ considerably across treatments, and these definitions circumscribe the range, depth, and extensionality of the evidence. The authors describe major dimensions that differentiate evidence-based practices for children from those for adults and summarize the status of the scientific literature on a range of service practices. The readiness of the child and adolescent evidence base for large-scale dissemination should be viewed with healthy skepticism until studies of the fit between empirically based treatments and the context of service delivery have been undertaken. Acceleration of the pace at which evidence-based practices can be more readily disseminated will require new models of development of clinical services that consider the practice setting in which the service is ultimately to be delivered.

Figures in this Article

As is true with any newly popularized term, the term "evidence-based" has an almost intuitive ring of credibility to it. It brings to mind images of tree-lined and stately buildings fronted with Grecian columns and filled with persons wearing white coats, speaking in hushed tones, and offering reassurances. But this ring may be hollow. As Montaigne noted, "Nothing is so firmly believed as what we least know," and as Valery warned, "That which has been delivered by everyone, always and everywhere, has every chance of being false."

There are as many definitions of what constitutes "evidence" as there are definitions of what constitutes a "service." More important, the use of the term "evidence-based practice" presupposes agreement as to how the evidence was generated, what the evidence means, and how or when the practice can be implemented.

We suggest that before this term becomes a slogan, it may be wise to examine the presuppositions behind it, acknowledge the limitations of what is sometimes characterized as evidence-based practice, and, in the next generation of services research, attend to implementation issues at the front end.

Much of what passes for research on evidence-based practice in the field of child and adolescent mental health might more aptly be described as clinical treatment efficacy research. In this article we first describe how evidence-based practice is being defined in the field of child and adolescent mental health, the characteristics of children and of services that pose special challenges in creating evidence-based practices, and the state of research evidence for treatments and services. Finally, we explain why healthy skepticism about current evidence-based practices is not unreasonable.

Definitional applications

In the field of children's mental health services research, the term "evidence-based practice" refers to a body of scientific knowledge about service practices—for example, referral, assessment, and case management—or about the impact of clinical treatments or services on the mental health problems of children and adolescents. The knowledge base is created through the application of scientific methods that examine the impact of certain practices on outcomes for the child or adolescent and his or her family. Evidence-based practice is a shorthand term that denotes the quality, robustness, or validity of scientific evidence as it is brought to bear on these issues. Although the term can and has been applied to preventive strategies, here we focus on treatments and services for children and adolescents who have been identified as having clinical disorders.

In the child and adolescent mental health services field, the term "evidence-based" is most often used to differentiate therapies—generally psychosocial—that have been studied with varying degrees of rigor from therapies that are used but have not been studied or have not been studied well. For example, Kazdin (1) has described four domains that constitute criteria for assessing an evidence base: a theory to relate a hypothesized mechanism to a clinical problem, basic research to assess the validity of the mechanism, preliminary outcome evidence to show that a therapeutic approach changes the relevant outcomes, and process-outcome connections, which display the relationships between process change and clinical outcomes. In essence, Kazdin has defined a process for attaining a valid and substantiated theory about the impact therapy has on a patient.

Operational criteria were proposed by the division of clinical psychology of the American Psychological Association in 1998 and were applied to studies of specific childhood syndromes (2,3). According to these criteria, treatments are to be supported by either group design or single-subject experiments, and studies should clearly describe characteristics of the subjects. For a treatment to be considered "well established," two or more studies must show that it is superior to medication, placebo, or an alternative treatment or that it is equivalent to an already established treatment, or nine single-subject case studies must be conducted to establish its equivalence or superiority. Well-established treatments have been identified for attention-deficit hyperactivity disorder (ADHD)—for example, behavioral parent training and classroom behavior modification (4); for conduct problems—for example, parent training (5); and for phobias—for example, participant modeling and reinforced practice (6).

For an intervention to be considered "probably efficacious," two or more studies must show it to be superior to a wait-list control condition or one experiment must meet the criteria for a well-established treatment, or three single-case studies must be conducted. Treatments that are probably efficacious have been identified for depression and anxiety disorders—for example, cognitive-behavioral therapy (6,7)—and ADHD, conduct problems, and phobias. Studies of psychosocial interventions among children and adolescents for autism, anorexia and bulimia, posttraumatic stress disorder, bipolar disorder, obsessive-compulsive disorder, panic disorder, and substance abuse have not yet met the criteria for being considered well-established or probably efficacious.

Another approach to defining the evidence base in child research has arisen from the Interdisciplinary Committee on Evidence-Based Youth Mental Health Care, with formal participation by the American Academy of Pediatrics, the American Academy of Child and Adolescent Psychiatry, and the American Psychological Association's divisions of clinical child psychology and school psychology. This committee has built on the work of the American Psychological Association but has attempted to broaden the system to include psychosocial as well as pharmacologic treatments that are scientifically supported (8).

A manual has been developed for use by reviewers of outcome research to enable coding of studies according to highly specified criteria. For treatments to be classified as evidence based, at least two between-group design studies with a minimum of 30 subjects must be conducted across studies representing the same age group and receiving the same treatment for the same target problem, at least two within-group or single case design studies with the same parameters must be conducted, or there must be a combination of these. Further, a majority of the applicable studies must support the treatment, and the protocol must show acceptable adherence to the treatment manual.

The ultimate goal is to develop an archive of data from clinical trials for all treatment studies and to periodically update this archive such that it can provide research syntheses and meta-analyses to summarize treatment research for children and adolescents (8). This process is not unlike that proposed by the Cochrane Collaborative Group, which has been the primary standard setter for evidence-based reviews in medicine. The Cochrane Collaborative has formed the foundation for many of the projects for the Agency on Healthcare Research and Quality (9).

Finally, the term "evidence-based" has been used to refer to analytic reviews of bodies of studies on a target problem or program. For example, meta-analytic reviews of psychotherapy treatment studies—which usually meet less stringent criteria than those we have described—have been widely cited as suggestive of the strength of the evidence for these therapies (10,11,12) and as evidence that psychotherapy treatments are as effective for children as they are for adults (12). In addition, several recent reviews of interventions, such as family preservation and school-based services, have set inclusion criteria—for example, a randomized clinical trial, use of established outcome measures, and use of a comparison group—and then synthesized the strength of the evidence for or against the effectiveness of these types of services (13,14,15,16).

From a scientific standpoint, applying the same criteria to studies of pharmacological, psychosocial, or prevention interventions is probably warranted. Scientific justification rests on relatively well-accepted principles of control (17). However, there are important differences among these interventions in the ways in which they do—or do not—have regulatory backing.

In pharmaceutical medicine, evidence-based approaches have been built into the regulatory standards developed by the Food and Drug Administration (FDA) to review scientific evidence and identify effective medications. The strength of the evidence for pharmacologic treatments is regulated by the FDA, and an industry has grown up around this regulation, whereas the strength of the evidence for the effectiveness of psychotherapies and other nonpharmacological interventions lies only in the knowledge base created by researchers.

Pharmaceutical companies cannot distribute or advertise a pharmaceutical agent unless it has been approved by the FDA. The existence of regulatory authority over the distribution of effective—and therefore profitable—therapies does not exist for psychosocial treatments or services. Consequently the incentive system for the growth of these therapies is vastly different and largely academic (18).

Evidence-based practice for children's services

Children's versus adults' services

First, although to be human is to develop (in the French sense of de-enveloper or un-envelop), children undergo more rapid physiological, neuronal, and psychological changes over a briefer period than adults. The rapidity of this development implies that for evidence-based practice to be meaningful, it has to take into account developmental conditions that affect the durability of the effects of treatment. An evidence-based practice that is effective in the treatment of adolescent depression may well be ineffective or even harmful for children who have not reached puberty.

Attention to developmental increments in the creation of evidence-based practices for children and adolescents means attention not only to age-related changes but also to the complex and dynamic interactions among the child, the family, and the environmental context that accompany maturation (19). In the field of child and adolescent mental health, answers to questions about the evidence base for services are not meaningful if developmental issues have been ignored.

Second, the creation of a treatment for a child is rarely undertaken without consideration of the family context. In fact, some have argued that even the notion of what constitutes a mental illness cannot be ascertained without knowledge of the interaction between the child and his or her family (20,21). Beyond this interaction, parental perceptions of the nature of presenting problems differ substantially from the child's perceptions (22). The nature of the diagnosis itself is contextually bound for children and adolescents to a far greater degree than is true for adults (23). Although involvement of caregivers is important in implementation of evidence-based practices for adults, in child mental health research the family is central not only to the development of the treatment or service but also to the understanding of the diagnosis itself.

Third, evidence-based practices for children differ from those for adults in that the types of services for which an evidence-based practice is developed will necessarily involve substantially different service venues. For example, 70 to 80 percent of the mental health services received by children who have mental health problems are provided by schools (24). The adult analog is likely to be the workplace. Yet an evidence-based practice for treating, say, ADHD in a school setting—for example, classroom management by a teacher—is unlikely to be similar to an evidence-based practice developed for treating inattention in the workplace. Issues related to the context or setting of the service place very different demands on the provider of the treatment and on the recipient.

Because of these contextual differences, a wide range of providers will need to be trained to provide the evidence-based practice. Children who have mental health needs may come to the attention of professionals in schools, primary care offices, welfare systems, or detention facilities. The fragmentation of the mental health service system means that for evidence-based practice to reach those who provide care to children, a range of training curricula, materials, and approaches must be developed and specifically tailored for the providers in these systems.

A short history

Until recently, there has been no evidence to summarize, critique, or review in the field of child and adolescent services. In fact, from a historical standpoint it is interesting that the concept of childhood mental illnesses did not arise until the late 19th century. These illnesses were typically not seen as unique to children or distinguishable from adult mental illnesses until the early part of the 20th century. The first English-language text on child psychiatry was published in 1935 (25). The first serious attempts to assess the use of mental health services by children and adolescents were begun in the late 1980s.

Two factors galvanized developments in children's mental health services: recognition that these services were scattered across a vast array of service organizations and systems, including schools, child welfare agencies, pediatric health settings, and juvenile corrections facilities, and recognition that fewer than 20 percent of children who had identified mental health needs received help (24).

The first development was state-level activities to create coordinated points of entry for delivery of mental health services, organized largely under the auspices of the Child and Adolescent Service System Program. This initiative was given principled footing through the creation of a model for revamping children's services. The system-of-care model, developed by Stroul and Friedman (26), articulated a series of values and principles centered on maintaining children in their communities, coordinating services, involving families centrally in delivery and planning of treatments and services, and instantiating attention to the cultural relevance of services. After these principles were developed, the services included in a continuum of care were delineated.

The second development was the creation of a scientific agenda centered on examining the relationship between children's needs for psychiatric care and the availability of such care. The two major efforts in this direction were the Great Smoky Mountains study (27,28) and the study of methods for the epidemiology of children and adolescents (29,30). Both studies found that 4 to 8 percent of children between the ages of nine and 17 years had severe psychiatric disorders and that only about 20 percent of children with the most serious needs were receiving mental health services (24,27,29,31).

During the mid-1990s, the scientific research agenda redirected attention to the quality of the clinical treatments within service systems. The system-of-care studies by Bickman (32,33) showed that system coordination alone improved access to services for children and families and satisfaction with services and also reduced hospitalization and other restrictive forms of care. However, these studies also showed that clinical outcomes for children—for example, alleviation of symptoms, functioning, or reduction of impairments—were the same whether children were receiving coordinated services through systems of care or were receiving usual services.

As a result of these findings, attention was shifted away from general studies of "systemness" to the clinical effectiveness of services within these systems of care and especially to the types, dosages, and intensity of treatments delivered (34,35,36). In particular, the transportability of efficacious clinical treatments into mental health service systems was highlighted as a critical research area (37,38,39). To this end, Burns (40) proposed the creation of a research agenda on clinical interventions for youths that would accomplish four tasks: synthesize, through reviews of the evidence base, the status of science on promising interventions; assess the adequacy of quality indicators to improve standards of clinical practice; evaluate the adequacy of outcome measures; and develop a new research phase model for connecting research to practice. The latter activity was subsequently proposed by Weisz and Weersing (18) as well as by Hoagwood and colleagues (41).

As a result of this new focus on bridging science and policy and strengthening knowledge about the efficacy and effectiveness of mental health services (42), a series of studies has been undertaken by a number of investigators, focusing on questions about the effectiveness of manual-based services for children who have a range of serious psychiatric impairments and the effectiveness of practice strategies for enhancing engagement with services.

Types of practices used

Acceptability of engagement and treatment. Treatments that fail to reach those who stand to benefit from them cannot be said to be effective. Unfortunately, inaccessibility of services and termination of treatment are problems that plague the delivery of mental health care. Among children whose families do seek outpatient mental health treatment, 40 to 60 percent may discontinue services before formal completion of treatment (43). Moreover, these families typically do not use outpatient services for very long. Armbruster and Fallon (44) showed that most children who enter outpatient treatment attend for only one or two sessions. There is also evidence that children from especially vulnerable populations—children of single mothers, children living in poverty, and children from minority groups—and children who have serious presenting problems are less likely to stay in treatment beyond the first session and more likely to discontinue treatment prematurely (45,46).

However, successful efforts can be made to enhance a family's service engagement and to decrease rates of premature termination of treatment. A variety of studies have been undertaken to identify and reduce barriers to service engagement and to increase the participation of minority families in services by using brief telephone interviews before service entry (47,48,49).

Empirically supported psychosocial outpatient treatments. Meta-analyses of experimental child psychotherapy intervention trials point to a consistent beneficial effect of treatment compared with no treatment (11,50,51,52). These effects are comparable to those found for adult psychotherapy (10). A similar analysis that summarized a less extensive literature on the treatment effects of therapies conducted in clinical practice settings, as opposed to research settings, found almost no difference between treatment and no treatment (12). In fact, the effect size was negative, falling well below the effects typically found in experimental studies. Consequently, the evidence suggests that psychosocial interventions for children can successfully reduce symptoms associated with childhood mental disorders when conducted in research-based settings; however, the impact of these therapies in clinical practice settings is only now being studied (18).

Family-focused treatments. Because the family plays a major role in the social and emotional development of children, family-focused interventions have long been a part of child and adolescent mental health treatment. Meta-analysis of family-focused treatments shows the general effectiveness of such treatments (19). Controlled trials indicate the effectiveness of family-based interventions for physical child abuse and neglect; conduct problems, including ADHD; emotional disturbance, specifically anxiety, depression, and grief; toileting problems; and psychosomatic concerns (20). Effective family-based treatments are typically short-term, are offered on an outpatient basis, and have cognitive-behavioral, structural, or strategic foundations. Frequently they are combined with individual therapies and medication management.

Integrated community-based treatment. One criticism of empirically supported psychosocial interventions is that their focus on specific diagnosable disorders does not adequately take account of the heterogeneity of the psychiatric problems that a majority of children have when they present at mental health clinics (53). A series of studies on integrated service models for children who have multiple co-occurring disorders has been examining a range of service modalities, including intensive case management, treatment foster care, and home-based services.

Studies of clinically oriented, intensive case management have found that children who have specially trained case managers require fewer restrictive services, such as psychiatric hospitalizations, than children who do not (54,55). Similar reductions in the number of inpatient hospitalizations have been found for youths with substance use problems (55). The use of case managers in community-based interdisciplinary treatment teams has also been found to improve standard-practice foster care through reductions in the number of placement changes and the number of runaway episodes among older youths (56). Multiple uncontrolled studies of case management services have been conducted in the context of wraparound care (57). These studies have shown that case management services can improve children's positive adjustment, decrease negative behaviors, and improve the stability of community living environments.

A series of studies have tested the impact of therapeutic foster care services for children who have multiple comorbid mental disorders. In a therapeutic foster care environment, a child is placed in a home with foster parents who have received specialized training to work with children who have emotional or behavioral problems. Results from these studies have shown decreases in aggressive behavior and increases in positive adjustment at the conclusion of placement (56,57,58). Chamberlain and Reid (59) compared treatment outcomes for youths from a state psychiatric hospital who were placed in either therapeutic foster care or usual community care and found that those in therapeutic foster care had fewer reinstitutionalizations and more rapid behavioral improvement. In addition, youths in the experimental group had less frequent posttreatment incarcerations and criminal referrals and more frequent placements with parents or relatives in the year after treatment (60). In addition, the costs of this service were significantly lower than those of other residential placements (60).

Finally, home-based service models have been developed for children who have serious emotional disturbances. One rigorously studied home-based intervention is multisystemic therapy, the primary goal of which is to develop independent skills among youths who have behavioral problems and their parents to cope with family, peer, school, and neighborhood problems through brief (three to four months) and intense (sometimes daily) treatment (61). Treatment strategies integrate empirically based treatment approaches—for example, behavioral training for parents, cognitive-behavioral therapies, and functional family therapy—to address the problems of children and adolescents across environmental contexts.

Eight randomized trials of multisystemic therapy have been conducted, and the results have been among the strongest found for children's services. Among a group of chronic juvenile offenders, those who received multisystemic therapy had lower rates of recidivism and out-of-home placements 59 weeks after treatment and lower arrest rates more than two years after treatment (62). Similar results were found when multisystemic therapy was compared with individual therapy in a different group of juvenile offenders (63).

A recent study comparing multisystemic therapy with emergency psychiatric hospitalization among children and adolescents with serious psychiatric impairments has found that multisystemic therapy can safely reduce rates of psychiatric hospitalization and improve the functioning of youths and their families (64,65). The effects of multisystemic therapy have been further demonstrated among juvenile sex offenders (66) and abused or neglected children (67). Researchers evaluating multisystemic therapy suggest that adequate supervision, training of therapists, and institutional program support are essential to successful outcomes (65).

School-based interventions. Seventy to 80 percent of children who receive mental health services receive them in school; for many children the school system provides their only form of mental health treatment (24). About 45 studies of school-based interventions for children with emotional or behavioral problems, covering a 15-year period, were recently reviewed (15). Among these studies, a range of effective individual, classroom, and targeted interventions were identified.

The empirically supported treatments for childhood behavioral problems that are effective in school settings include targeted classroom-based contingency management for children with a diagnosis of ADHD (4) and children with other conduct problems (5). Contingency management also appears to successfully reduce aggression when implemented across entire classrooms. The "good-behavior game," a classroom-based behavior-management strategy for first-grade students, has demonstrated long-term benefits in reducing disruptive behaviors in middle school (68,69). Behavioral consultation to teachers to help them accommodate difficult students has been found to reduce the number of special-education referrals and placements and to reduce teachers' reports of students' behavioral problems (70).

School-based preventive interventions designed to target children who are at risk of emotional or behavioral problems have also been shown to alleviate symptoms and to increase the use of positive coping strategies. Cognitive group interventions to modify adolescents' depressive thinking styles have been associated with a reduced risk of the development of full depression (71). Similarly, a group intervention to teach social problem-solving skills to elementary school children with elevated depressive symptoms demonstrated reductions in reported depression, even one year after the intervention (72). There is also evidence that school-based preventive interventions reduce the risk of conduct problems. Successful interventions typically involve multiple components that target classroom, home, and peer environments (73).

Psychopharmacology. About 3.5 million child outpatient physician visits a year result in a prescription for a psychotropic medication (74). Furthermore, prescription rates for psychotropic medications for children are increasing, even among very young children (75). Clinical trials of psychotropic medications are needed for many childhood mental disorders. Despite the widespread use of these medications, surprisingly few randomized controlled studies have been conducted (16). Many medication choices and algorithms continue to be based on the experience of the individual practitioner or on standards of care for adults, with some notable exceptions (76).

In addition, proper medication selection is only one factor in successful pharmacologic treatment. The recent National Institute of Mental Health multimodal treatment study of ADHD found that clinical outcomes for participants who received careful medication management, including systematic titration to the optimal dosage, were superior to clinical outcomes associated with routine prescription of the same medication delivered in the usual manner in the community (77). The differences in outcomes seem to have been a result of management practices associated with participation in the medication treatment group.

Weisz and Jensen (16) recently reviewed evidence of the efficacy of child pharmacotherapy by using criteria established for the International Psychopharmacology Algorithm Project (78). According to these criteria, a drug is considered efficacious if its equivalency or superiority have been demonstrated through random-assignment, control group comparison and the results are replicated in one or more similarly well-controlled studies. In addition, the National Institute of Mental Health recently commissioned six scientific reviews of published studies of the safety and efficacy of psychotropic medications for children (79): psychostimulants (80), mood stabilizers and antimanic agents (81), selective serotonin reuptake inhibitors (SSRIs) (82), tricyclic antidepressants (83), antipsychotic agents (84), and miscellaneous agents (85). These reviews identified several psychotropic medications for which there is empirical support for use in both externalizing and internalizing disorders in childhood, most prominent among which are psychostimulants for children with ADHD.

For depression, the most commonly studied pharmacologic agents have been the tricyclic antidepressants, but they have not been shown to be effective in treating childhood depression. Newer and safer agents, such as SSRIs, are being studied more frequently in the treatment of childhood depression. In the largest study of the use of antidepressants among children, Emslie and colleagues (86) compared fluoxetine with placebo and found that more than half of the fluoxetine group significantly benefited from treatment, whereas only 33 percent of the placebo group showed benefit.

In the only multisite controlled trial of SSRIs for childhood anxiety disorders, fluvoxamine was found to be superior to placebo in treating children with a diagnosis of social phobia, separation anxiety, or generalized anxiety disorder (87). In addition, several pharmacologic agents have been shown to be efficacious in the treatment of children or adolescents with a diagnosis of obsessive-compulsive disorder; these include SSRIs and tricyclic antidepressants (82,83). There is also evidence supporting the usefulness of antipsychotic medications for schizophrenia with onset in childhood or adolescence; however, data on long-term effectiveness and safety are lacking (16).

Unfortunately, despite the fact that in clinical practice it is common for medication treatments to be combined with psychosocial strategies, the literature on the impact of combined treatments for disorders other than ADHD is sparse. A large multisite clinical trial of the efficacy of combined treatments for adolescent depression is under way and is expected to provide knowledge about treatment options in this area.

Potentially ineffective treatments. Recent efforts to identify empirically supported treatments for children have focused largely on the accumulation of supportive findings without an established procedure for dealing with null or even negative results (88). It is, in fact, just as important to identify treatments for which empirical studies consistently fail to show an effect on symptoms or even show worse outcomes for participants in the treatment group. Youth Violence: A Report of the Surgeon General (89) suggests that many of the services provided to delinquent juveniles have little or no evidence base.

Worse yet, a recent study indicated that peer-based, group-based interventions may actually increase behavioral problems among high-risk adolescents (90). For children who have disruptive-behavior disorders, there is no empirical justification for the use of nonbehavioral psychotherapies (91). For example, Pelham and colleagues (4), in their review of effective treatments for ADHD, found no empirical studies that tested the efficacy of many psychosocial treatments commonly used for ADHD, such as individual therapy and play therapy. In addition, although controlled treatment outcome studies have been conducted for cognitive therapy—for example, self-instruction, self-monitoring, and self-reinforcement—to treat children with a diagnosis of ADHD, these studies generally show no clinical or academic benefits from the treatment (92).

A third example of widely used but empirically unjustified services is institutional care—for example, hospitals, residential treatment centers, and group homes (13). Studies have shown that children who are placed in group homes do not maintain improvements once they return to the community (93). Yet group homes continue to be used in community practice.

A general conclusion from this review of evidence-based practices is that the literature on the efficacy of a range of child and adolescent treatments is uneven, although it is gaining strength for particular clinical syndromes. However, the evidence for the effectiveness of either clinical treatments or services within practice settings as opposed to research settings is still weak. Improving the evidence will require attention to service variables that tend to be neglected in most efficacy-based studies.

Presuppositions about evidence-based practice

Although the public health goal of developing a strong research base on effective services is to improve routine care, the implementation of research-based practice is not automatic. Rather, it requires adaptation of research design and methods to practice-related exigencies, as well as accommodation of practice settings to the incorporation of evidence-based practice. Other papers in this series have identified barriers to implementation of evidence-based practice. However, in the child and adolescent field, it is prudent to ask a different and prior set of questions. These questions have to do with whether empirically validated knowledge about treatments and services is ready to be implemented and, if not, why not. To think about these kinds of questions leads us to consider the models that typically guide development of evidence-based treatments and services and the extent to which implementation is justified given these models.

A presupposition of the evidence base is that its development has taken into account the fit between the treatment and the context of delivery. In fact, this fit has been attended to only rarely. One reason that efficacy studies, which constitute a significant portion of the evidence base in children's mental health, have not been readily deployed in service settings may be that the theory, methods, and models used to develop, refine, and test those treatments do not mesh well with the exigencies of clinic-based or community-based care (18).

The culture of psychological science as brought to bear on questions of the efficacy of treatments for children has typically involved conducting studies in controlled and somewhat rarefied environments, such as university laboratories. Over the past 40 years many controlled clinical trials and within-group studies have been published on the impact and efficacy of psychosocial treatments. Specific treatments have been identified for about two dozen clinical conditions in children. These studies have typically been conducted in or in close connection with university laboratories. Studies of conventional treatments delivered in clinics and clinical programs have demonstrated much weaker effects (94).

There has been an implicit assumption that once the laboratory studies of the efficacy of treatments have been completed, the results will be usable and relevant outside the laboratories. However, as has been noted repeatedly (13,18,94), the conditions under which most research is conducted differ in numerous ways from those under which everyday treatment is delivered. Such differences imply that treatments that have been developed through efficacy trials need to be adapted to clinics, schools, or other service settings. However, the differences also imply the opposite, namely, that the service settings or service practices themselves—that is, where, when, and how treatments are delivered—may have to adapt to accommodate delivery of evidence-based practices.

The model that has guided treatment development entails a series of controlled laboratory trials that focus on the efficacy of the treatment. Issues related to the effectiveness, dissemination, implementation, and deployment of treatments take place at the end of the testing process. This model tends to control experimentally the very "nuisance variables" that may need to be understood if treatments are to fit within clinic or community settings (18). These variables—such as comorbidity, parental substance abuse or pathology, life stresses that lead to early termination of treatment, reimbursement structures, service availability, and parental self-efficacy—may make or break the successful adoption of an evidence-based practice in a new practice setting.

Unfortunately, the development of the evidence base has rarely attended to such nuisance variables. Consequently, implementation of many treatments in clinic or community settings may be premature unless such factors are built into the long-term design and cumulative construction of new treatments and services. For example, studies of the development of combined treatments for children with depression who are first seen by primary care physicians should perhaps take into account service linkages with family practice physicians to treat maternal depression. Organizational variables, such as care management, that constrain or facilitate the ability of physicians to communicate with teachers about the impact of treatment on children should be taken into account at the front end of the treatment development cycle.

A second presupposition underlying evidence-based practice is that diffusion of the evidence base will be automatic once the strength of the evidence is ascertained. In fact, diffusion of innovative practice constitutes a researchable set of questions on its own. A major objective of most diffusion studies is to determine whether the practice is adopted as it was designed or adapted, whether the practice is sustained over time, and what factors influence sustainability (95). Diffusion is only beginning to be studied in the context of the mental health treatment of either children or adults. The literature on diffusion identifies individual and contextual factors that are potentially relevant to the effective diffusion of innovation in general, but not the specific factors most likely to predict adoption and implementation of a particular innovation, nor of a particular mental health treatment or service. Thus the evidence base needed to guide successful dissemination of effective treatments has yet to be developed (96).

The central problem is that treatments that have been validated in efficacy studies cannot be assumed to be effective when implemented under routine practice conditions. For example, the use of treatment manuals, special training for clinicians, and continual clinical monitoring to ensure treatment fidelity are characteristics of many research-based interventions but few community-based treatment practices. On the other hand, community treatment is characterized by heterogeneous populations and high caseloads, which are not features of most research-based studies (91). To enable a better fit between evidence-based treatments and community practice contexts will most likely require modifications of both the way in which evidence-based practice is created and modifications in the service delivery settings. The variables at the interface that enable a better match have been described as issues of transportability (96). Features of research protocols and practice contexts that require modification—for example, the training and background of the practitioner, the setting, the organizational culture or context, and financing—and the kinds of modification required are variables that may well influence the portability of evidence-based practices and the adaptability of practice settings (18,41,96,97).

For example, transportability issues have been examined in studies of the promotion of physicians' use of new medications, devices, or procedures. Because most medical technologies must be approved by the FDA before being marketed, the efficacy of the technologies is not in question. However, questions about the fidelity of practitioners' implementation and how it may influence the effectiveness of the intervention cannot be assumed to have been studied. Although some studies of the impact of continuing education, academic detailing, training plus follow-up procedures, and hybrid strategies on physicians' implementation of new technologies have been conducted (98), their application to the dissemination of evidence-based practices in children's services is only now being studied (41), and these practices may well require adaptation.

Attention to the fit will require modification along a variety of dimensions, which differentiate research-based treatments from community practice. The dimensions include the intervention itself; the practitioners who deliver the intervention, including their clinical training, support, and monitoring; the client population (homogeneity or heterogeneity of syndromes); the characteristics of service delivery, such as the setting and the types of services available beyond the intervention; the organizational ethos—for example, the culture or climate in which practitioners provide services, which influences motivation, attitudes, and morale; and the service system, including referral and reimbursement mechanisms and interagency relations (96). The research base that constitutes evidence-based practice for child and adolescent mental health interventions has not typically assessed these dimensions of routine practice.

Finally, factors that predict successful dissemination of evidence-based practices may overlap with those that predict effective services, may be identical to them, or may be entirely different. For example, the organizational climate was found to be a strong predictor of psychosocial outcomes among children receiving casework services in a child welfare agency, and it appears to exert its influence through the motivational attitudes of casework therapists toward their work (99). Is organizational climate an important ingredient of uptake and dissemination? The answer to this question is not yet known. If climate is associated with interpersonal variables that predict the adoption of innovation, then manipulation of climate will be important to the eventual implementation of evidence-based practices. However, if the variables that predict uptake are more closely related to systemic or interpersonal dimensions rather than to organizational dimensions, then climate may not be the active agent for implementation.

Improving implementation of evidence-based practice

Improving implementation of evidence-based practice in children's services entails the adoption of new models of treatment development and augmentation of current effectiveness studies such that dimensions of typical practice are assessed and better understood. Weisz and Weersing (18) and Hoagwood and colleagues (41) have argued for the creation of clinic and community intervention development models that, in the initial piloting and manualization phase, attend to nuisance characteristics of the practice setting, such as practitioners' behaviors, organizational variables, and the characteristics of the community.

In addition, the new report of the National Institute of Mental Health's Advisory Council, titled Blueprint for Change: Research on Child and Adolescent Mental Health (100), describes a new cyclic model of treatment development that attends to service delivery issues at the outset. The report argues that attention to these contextual variables is necessary if the intervention is ultimately to be adopted. This model of intervention development is extremely challenging and not for the fainthearted. However, without such a revolution in treatment development, the best that can be hoped for is that evidence-based practices do not gather too much dust on academic shelves.

Conclusions

As the field of mental health services research expands, it will be important to take advantage of opportunities to study new services as they arise and to do so in a timely manner. On the basis of past performance, when treatments are developed and tested via the typical medical model, ten to 20 years may be required before the treatment can be understood in terms of its effects within a practice setting. As the Surgeon General's National Action Agenda for children's mental health (101) demonstrates, this time frame is impractical and inefficient if the goal of a public health science of children's services is to improve practice. Instead, a new model is needed that will encourage studies of the effectiveness of new treatments and services in the context of the practice setting in which the treatment or service is ultimately to be delivered.

Dr. Hoagwood and Dr. Ringeisen are affiliated with the National Institute of Mental Health in Bethesda, Maryland. Dr. Burns is with Duke University in Durham, North Carolina. Dr. Kiser is with the University of Maryland in Baltimore. Dr. Schoenwald is with the Medical University of South Carolina in Charleston. Send correspondence to Dr. Hoagwood at the National Institute of Mental Health, 6001 Executive Boulevard, Bethesda, Maryland 20817 (e-mail, kh32p@nih.gov).

References

Kazdin AE: Current (lack of) status of theory in child and adolescent psychotherapy research. Journal of Clinical Child Psychology28:533-543, 1999[PubMed][CrossRef]

Chambless DL, Sanderson WC, Shohman V, et al: An update on empirically validated therapies. Clinical Psychologist49:5-18, 1996

Lonigan CJ, Elbert JC, Johnson SB: Empirically supported psychosocial interventions for children: an overview. Journal of Child Clinical Psychology27:138-145, 1998[CrossRef]

Pelham WE, Wheeler T, Chronis A: Empirically supported psychosocial treatments for attention deficit hyperactivity disorder. Journal of Child Clinical Psychology27:190-205, 1998[CrossRef]

Brestan EV, Eyberg SM: Effective psychosocial treatments of conduct-disordered children andadolescents:29 years, 82 studies, and 5,272 kids. Journal of Child Clinical Psychology 27:180-189, 1998

Ollendick TH, King NJ: Empirically supported treatments for children with phobic and anxiety disorders. Journal of Child Clinical Psychology27:156-167, 1998[CrossRef]

Kaslow NJ, Thompson MP: Applying the criteria for empirically supported treatments to studies of psychosocial interventions for child and adolescent depression. Journal of Child Clinical Psychology27:146-155, 1998[CrossRef]

Weisz JR: Lab-clinic differences and what we can do about them: linking research and practice to enhance our public impact. Newsletter of the Division of Clinical Child Psychology, 2001, in press

Cochrane Collaboration and Health Information Research Unit: Preparing and Maintaining Systematic Reviews, in The Cochran Collaboration Handbook, edited by Oxman A. Oxford, England, Cochran Collaboration, 1994

Smith ML, Glass GV: Meta-analysis of psychotherapy outcomes studies. American Psychologist32:752-760, 1977[PubMed][CrossRef]

Kazdin AE, Bass D, Ayers WA, et al: Empirical and clinical focus of child and adolescent psychotherapy research. Journal of Consulting Clinical Psychology58:729-740, 1990[CrossRef]

Weisz JR, Weiss B, Han SS, et al: Effects of psychotherapy with children and adolescents revisited: a meta-analysis of treatment outcome studies. Psychological Bulletin117:450-468, 1995[PubMed][CrossRef]

Burns BJ, Hoagwood K, Mrazek P: Effective treatment for mental disorders in children and adolescents. Clinical Child and Family Psychology Review2:199-254, 1999[PubMed][CrossRef]

Heneghan AM, Horwitz SM, Leventhal JM: Evaluating intensive family preservation programs: a methodologic review. Pediatrics97:535-542, 1996[PubMed]

Rones M, Hoagwood K: School-based mental health services: a research review. Clinical Child and Family Psychology Review3:223-241, 2000[PubMed][CrossRef]

Weisz JR, Jensen PS: Efficacy and effectiveness of child and adolescent psychotherapy and pharmacotherapy. Mental Health Services Research1:125-158, 1999[PubMed][CrossRef]

Cook TF, Campbell DT: Quasi-Experimentation: Design and Analysis Issues for Field Settings. Chicago, Rand-McNally, 1979

Weisz JR, Weersing VR: Psychotherapy with children and adolescents: efficacy, effectiveness, and developmental concerns, in Rochester Symposium on Developmental Psychopathology, vol 9, Developmental Approaches to Prevention and Intervention. Edited by Cicchetti D, Toth SL. Rochester, NY, University of Rochester Press, 1999

Shadish W, Montgomery L, Wilson P, et al: The effects of family and marital psychotherapies: a meta-analysis. Journal of Consulting and Clinical Psychology61:992-1002, 1993[PubMed][CrossRef]

Carr A: Evidence-based practice in family therapy and systemic consultation: I. child focused problems. Journal of Family Therapy22:29-60, 2000[CrossRef]

Cicchetti D, Tucker D: Development and self-regulatory structures of the mind. Development and Psychopathology6:533-549, 1994[CrossRef]

Yeh M, Weisz J: Why are we here at the clinic? Parent-child (dis)agreement on referral problems at outpatient treatment entry. Journal of Consulting and Clinical Psychology, in press

Jensen PS, Hoagwood K: The book of names: DSM-IV in context. Developmental Psychopathology9:231-249, 1997

Burns BJ, Costello EJ, Angold A, et al: Children's mental health service use across service sectors. Health Affairs14(3):147-159, 1995

Kanner L: Child Psychiatry. Springfield, Ill, Thomas, 1935

Stroul BA, Friedman RM: Caring for severely emotionally disturbed children and youth: principles for a system of care. Children Today17(4):15-17, 1988

Costello EJ, Angold A, Burns BJ, et al: The Great Smoky Mountains Study of Youth: goals, design, methods, and the prevalence of DSM-III-R disorders. Archives of General Psychiatry53:1129-1136, 1996[PubMed]

Costello EJ, Angold A, Burns BJ, et al: The Great Smoky Mountains study of youth: functional impairment and serious emotional disturbance. Archives of General Psychiatry53:1137-1143, 1996[PubMed]

Shaffer D, Fisher P, Dulcan MK, et al: The NIMH Diagnostic Interview Schedule for Children Version 2.3 (DISC-2.3): description, acceptability, prevalence rates, and performance in the MECA study. Journal of the American Academy of Child and Adolescent Psychiatry35:865-877, 1996[PubMed][CrossRef]

Lahey BB, Flagg EW, Bird HR, et al: The NIMH methods for the epidemiology of child and adolescent mental disorders (MECA) study: background and methodology. Journal of the American Academy of Child and Adolescent Psychiatry35:855-864, 1996[PubMed][CrossRef]

Leaf PJ, Alegria M, Cohen P, et al: Mental health service use in the community and schools: results from the four-community MECA study. Journal of the American Academy of Child and Adolescent Psychiatry35:889-897, 1996[PubMed][CrossRef]

Bickman L: A continuum of care: more is not always better. American Psychologist51:689-701, 1996[PubMed][CrossRef]

Bickman L: The Fort Bragg continuum of care for children and adolescents: mental health outcomes over 5 years. Journal of Consulting and Clinical Psychology68:710-716, 2000[PubMed][CrossRef]

Weisz JR, Donenberg GR, Han SS, et al: Bridging the gap between laboratory and clinic in child and adolescent psychotherapy. Journal of Consulting and Clinical Psychology63:688-701, 1995[PubMed][CrossRef]

Hoagwood K: Interpreting nullity: the Fort Bragg experiment—a comparative success or failure? American Psychologist52:546-550, 1997

Henggeler SW, Rowland MD, Pickrel SG, et al: Investigating family-based alternatives to institution-based mental health services for youth: lessons learned from the pilot study of a randomized field trial. Journal of Clinical Child Psychology26:226-233, 1997[PubMed][CrossRef]

Hoagwood K, Hibbs E, Brent D, et al: Introduction to the special section: efficacy and effectiveness in studies of child and adolescent psychotherapy. Journal of Consulting and Clinical Psychology63:683-687, 1995[PubMed][CrossRef]

Hoagwood K, Jensen P, Petti T, et al: Outcomes of care for children and adolescents: a conceptual model. Journal of the American Academy of Child and Adolescent Psychiatry35:1055-1063, 1996[PubMed][CrossRef]

Jensen PS, Hoagwood K, Petti T: Outcomes of mental health care for children and adolescents: II. literature review and application of a comprehensive model. Journal of the American Academy of Child and Adolescent Psychiatry35:1064-1077, 1996[PubMed][CrossRef]

Burns BJ: A call for a mental health services research agenda for youth with serious emotional disturbance. Mental Health Services Research1:5-20, 1999[CrossRef]

Hoagwood K, Burns BJ, Weisz JR: A profitable conjunction: from research to practice in children's mental health, in Community-Based Interventions for Youth With Severe Emotional Disorders. Edited by Burns BJ, Hoagwood K. Oxford, New York, Oxford University Press, in press

National Advisory Mental Health Council's Clinical Treatment and Services Research Workgroup: Bridging Science to Service. Pub 99-4353. Rockville, Md, National Institute of Mental Health, 1999

Kazdin AE, Holland L, Crowley M: Family experiences of barriers to treatment and premature termination from child therapy. Journal of Consulting and Clinical Psychology65:453-463, 1997[PubMed][CrossRef]

Armbruster P, Fallon T: Clinical, sociodemographic, and systems risk factors for attrition in a children's mental health clinic. American Journal of Orthopsychiatry64:577-585, 1994[PubMed][CrossRef]

Kazdin AE: Premature termination from treatment among children referred for antisocial behavior. Journal of Clinical Child Psychology31:415-425, 1993

Tuma JM: Mental health services for children. American Psychologist44:188-189, 1989[PubMed][CrossRef]

Santisteban DA, Szapocznik J, Perez-Vidal A: Efficacy of intervention for engaging youth and families into treatment and some variables that may contribute to differential effectiveness. Journal of Family Psychology10:35-44, 1996[CrossRef]

Szapocznik J, Perez-Vidal A, Brickman AL, et al: Engaging adolescent drug abusers and their families in treatment: a strategic structural systems approach. Journal of Consulting and Clinical Psychology56:552-557, 1988[PubMed][CrossRef]

McKay MM, McCadam K, Gonzales JJ: Addressing the barriers to mental health services for inner city children and their caretakers. Community Mental Health32:353-361, 1996[CrossRef]

Casey RJ, Berman JS: The outcome of psychotherapy with children. Psychological Bulletin98:388-400, 1985[PubMed][CrossRef]

Weisz JR, Weiss B, Alicke MD, et al: Effectiveness of psychotherapy with children and adolescents: a meta-analysis for clinicians. Journal of Consulting and Clinical Psychology55:542-549, 1987[PubMed][CrossRef]

Burns BJ, Farmer EMZ, Angold A, et al: A randomised trial of case management for youths with serious emotional disturbance. Journal of Clinical Child Psychology25:476-486, 1996[CrossRef]

Evans ME, Huz S, McNulty T, et al: Child, family and system outcomes of intensive case management in New York State. Psychiatric Quarterly67:273-287, 1996[PubMed][CrossRef]

Clarke HB, Prange M, Lee B, et al: An individualized wraparound process for children in foster care with emotional/behavioral disturbances: follow-up findings and implications from a controlled study, in Outcomes for Children and Youth With Behavioral and Emotional Disorders and Their Families. Edited by Epstein M, Kutash K, Duchnowski A. Austin, Tex, PRO-ED,1998

Chamberlain P, Weinrott M: Specialized foster care: treating seriously emotionally disturbed children. Children Today19(1):24-27, 1990

Clarke HB, Prange M, Lee B, et al: Improving adjustment outcomes for foster children with emotional and behavioral disorders: early findings from a controlled study on individualized services. Journal of Emotional and Behavioral Disorders2:207-218, 1994[CrossRef]

Chamberlain P, Reid JB: Using a specialized foster care community treatment model for children and adolescents leaving the state mental hospital. Journal of Community Psychology19:266-276, 1991[CrossRef]

Chamberlain P, Reid J: Comparison of two community alternatives to incarceration for chronic juvenile offenders. Journal of Consulting and Clinical Psychology4:624-633, 1998

Schoenwald SK, Brown TL, Henggeler SW: Inside multisystemic therapy: therapist, supervisory, and program practices. Journal of Emotional and Behavioral Disorders8:113-127, 2000[CrossRef]

Henggeler SW, Melton GB, Smith LA, et al: Family preservation using multisystemic treatment: long-term follow-up to a clinical trial with serious juvenile offenders. Journal of Child and Family Studies2:283-293, 1993[CrossRef]

Borduin CM, Mann BJ, Cone LT, et al: Multisystemic treatment of serious juvenile offenders: long-term prevention of criminology and violence. Journal of Consulting and Clinical Psychology63:569-578, 1995[PubMed][CrossRef]

Henggeler SW, Rowland MD, Randall J, et al: Home-based multisystemic therapy as an alternative to the hospitalization of youths in psychiatric crisis: clinical outcomes. Journal of the American Academy of Child and Adolescent Psychiatry38:1331-1339, 1999[PubMed][CrossRef]

Schoenwald SK, Ward DM, Henggeler SW, et al: MST vs hospitalization for crisis stabilization of youth: placement and service use 4 months post-referral. Mental Health Services Research2:3-12, 2000[PubMed][CrossRef]

Borduin CM, Henggeler SW, Blaske DM, et al: Multisystemic treatment of adolescent sexual offenders. International Journal of Offender Therapy and Comparative Criminology35:105-114, 1990

Brunk M, Henggeler SW, Whelan JP, et al: A comparison of multisystemic therapy and parent training in the brief treatment of child abuse and neglect. Journal of Consulting and Clinical Psychology55:311-318, 1987

Dolan L, Kellam S, Brown C, et al: The short-term impact of two classroom-based preventive interventions on aggressive and shy behaviors and poor achievement. Journal of Applied Developmental Psychology14:317-345, 1993[CrossRef]

Kellam S, Rebok G, Ialongo N, et al: The course and malleability of aggressive behavior from early first grade into middle school: results of a developmental epidemiologically-based preventive trial. Journal of Child Psychology, Psychiatry, and the Allied Disciplines35:259-281, 1994[CrossRef]

Fuchs D, Fuchs L, Bahr M: Mainstream assistance teams: a scientific basis for the art of consultation. Exceptional Children57:128-139, 1990[PubMed]

Clarke G, Hawkins W, Murphy M, et al: Targeted prevention of unipolar depressive disorder in an at-risk sample of high school adolescents: a randomized trial of a group cognitive intervention. Journal of the American Academy of Child and Adolescent Psychiatry34:312-321, 1995[PubMed][CrossRef]

Gillham J, Reivich K, Jaycox L, et al: Prevention of depressive symptoms in school children: two-year follow-up. Psychological Science6:343-351, 1995[CrossRef]

Reid J, Eddy M, Fetrow R, et al: Description and immediate impacts of a preventive intervention for conduct problems. American Journal of Community Psychology27:483-517, 1999[PubMed][CrossRef]

Jensen PS, Bhatara V, Vietello B, et al: Psychoactive medication prescribing practices for US children: gaps between research and clinical practice. Journal of the American Academy of Child and Adolescent Psychiatry38:557-565, 1999[PubMed][CrossRef]

Zito JM, Safer DJ, dosReis S, et al: Trends in the prescribing of psychotropic medications to preschoolers. JAMA283:1025-1060, 2000[PubMed][CrossRef]

Hughes CW, Emslie GJ, Crismon ML, et al: The Texas Children's Medication Algorithm Project: report of the Texas Consensus Conference Panel on Medication Treatment of Childhood Major Depressive Disorder. Journal of the American Academy of Child and Adolescent Psychiatry38:1442-1454, 1999[PubMed][CrossRef]

MTA Cooperative Group: A 14-month randomized clinical trial of treatment strategies for attention-deficit/hyperactivity disorder. Archives of General Psychiatry56:1073-1086, 1999[PubMed][CrossRef]

Jobson KO, Potter WZ: International Psychopharmacology Algorithm Project report. Psychopharmacology Bulletin31:457-459, 1995[PubMed]

Special section on current knowledge and unmet needs in pediatric psychopharmacology. Journal of the American Academy of Child and Adolescent Psychiatry38:501-565, 1999[CrossRef]

Greenhill LL, Halperin JM, Abikoff H: Stimulant medications. Journal of the American Academy of Child and Adolescent Psychiatry38:503-512, 1999[PubMed][CrossRef]

Ryan ND, Bhatara VS, Perel JM: Mood stabilizers in children and adolescents. Journal of the American Academy of Child and Adolescent Psychiatry38:529-536, 1999[PubMed][CrossRef]

Emslie GJ, Walkup JT, Pliszka SR, et al: Nontricyclic antidepressants: current trends in children and adolescents. Journal of the American Academy of Child and Adolescent Psychiatry38:517-528, 1999[PubMed][CrossRef]

Geller B, Reising D, Leonard HL, et al: Critical review of tricyclic antidepressant use in children and adolescents. Journal of the American Academy of Child and Adolescent Psychiatry38:513-516, 1999[PubMed][CrossRef]

Campbell M, Rapoport JL, Simpson GM: Antipsychotics in children and adolescents. Journal of the American Academy of Child and Adolescent Psychiatry38:537-545, 1999[PubMed][CrossRef]

Riddle MA, Bernstein GA, Cook EH, et al: Anxiolytics, adrenergic agents, and naltrexone. Journal of the American Academy of Child and Adolescent Psychiatry38:546-556, 1999[PubMed][CrossRef]

Emslie GJ, Rush AJ, Weinberg WA, et al: A double-blind, randomized, placebo-controlled trial of fluoxetine in children and adolescents with depression. Archives of General Psychiatry54:1031-1037, 1997[PubMed]

Vitiello B, Bhatara VS, Jensen PS: Current knowledge and unmet needs in pediatric psychopharmacology. Journal of Abnormal Child Psychology22:560-568, 1999

Weisz JR, Hawley KM: Finding, evaluating, refining, and applying empirically supported treatments for children and adolescents. Journal of Clinical Child Psychology27:206-216, 1998[PubMed][CrossRef]

Youth Violence: A Report of the Surgeon General. Washington, DC, US Public Health Service, 2001

Dishion TJ, McCord J, Poulin F: When interventions harm: peer groups and problem behavior. American Psychologist54:755-765, 1999[PubMed][CrossRef]

Abikoff H, Gittelman R: Hyperactive children treated with stimulants: is cognitive training a useful adjunct? Archives of General Psychiatry42:953-961, 1985

Kirigin KA, Braukmann CJ, Atwater JD, et al: An evaluation of teaching-family (achievement place) group homes for juvenile offenders. Journal of Applied Behavior Analysis15:1-16, 1982[PubMed][CrossRef]

Weisz JR, Weiss B, Donenberg GR: The lab versus the clinic: effects of child and adolescent psychotherapy. American Psychologist47:1578-1585, 1992[PubMed][CrossRef]

Van de Ven AH, Polley DE, Garud R, et al: The Innovation Journey. Oxford, NY: Oxford University Press, 1999

Schoenwald SK, Hoagwood K: Effectiveness, transportability, and dissemination of interventions: what matters when? Psychiatric Services52:1190-1197, 2001

Norquist G, Lebowitz B, Hyman S: Expanding the frontier of treatment research. Available at , 2000
http://journals.apa.org/prevention/volume2/pre0020001a.html

Wensing M, van der Weijden T, Grol R: Implementing guidelines and innovations in general practice: which interventions are effective? British Journal of General Practice48:991-997, 1998

Glisson C, Hemmelgarn A: The effects of organizational climate and interorganizational coordination on the quality of outcomes of children's service systems. Child Abuse and Neglect22:401-421, 1998[PubMed][CrossRef]

100

National Institute of Mental Health Advisory Council Workgroup Report: Blueprint for Change: Research on Child and Adolescent Mental Health. Bethesda, Maryland, National Institute of Mental Health, 2001

101

Report of the Surgeon General's Conference on Children's Mental Health: A National Action Agenda. Washington, DC, US Public Health Service, 2000