A Review of Mental Health Interventions in SAMHSA's National Registry of Evidence-Based Programs and Practices

Despite recent recommendations promoting greater adoption of evidence-based interventions in mental health (1,2), relevant information useful to practitioners and organizations interested in identifying, comparing, selecting, and implementing such interventions has been limited. Yet the need for practical information on a range of evidence-based mental health interventions will likely intensify as states, communities, and other funders encourage or even require that agencies and providers routinely implement these services (3,4).

The National Registry of Evidence-Based Programs and Practices (NREPP), developed by the U.S. Substance Abuse and Mental Health Services Administration (SAMHSA), is emerging as an objective and credible source of practical information on a range of evidence-based interventions to prevent and treat mental and substance use disorders. As described on its Web site (www.nrepp.samhsa.gov), the purpose of NREPP is to “assist the public in identifying approaches to preventing and treating mental and/or substance use disorders that have been scientifically tested and that can be readily disseminated to the field.” To further this purpose, NREPP provides descriptive information on a range of interventions, as well as independent, expert ratings of both the quality of research to support an intervention's key outcomes and the intervention's readiness for dissemination to broader service settings and populations. Since launching NREPP in March 2007, SAMHSA has been committed to the steady and continuous growth of NREPP as a decision support vehicle to assist individuals and agencies in determining which evidence-based interventions best address their needs, resources, and organizational cultures. The NREPP Web site averages more than 13,000 unique visitors per month.

This review identified and assessed the range of mental health interventions currently listed on NREPP. We highlight key characteristics of these interventions and—informed by these findings—briefly discuss the future implications that an evolving NREPP may have on research, services delivery, and mental health policy.

Methods

We conducted a search of the NREPP database to identify interventions that were posted to the Web site between March 1, 2007, and May 17, 2010, and that were classified as mental health promotion (preventing or delaying the onset of emotional or behavioral problems), mental health treatment (services aimed at reducing or eliminating symptoms or impairments from mental illness), or both. (Interventions related to substance use disorders were excluded.) For all interventions that met these criteria, further descriptive information was gathered regarding the age focus of the intervention (child and adolescent, adult, or both); the setting of the intervention (clinic based, school based, or other, such as home, community, corrections, and workplace); whether the intervention evaluation had been replicated; and the status of the intervention as proprietary (fees are charged by developers for access to implementation materials or trainings), publicly available (no fees are associated with access to implementation materials or trainings), or a mixture of proprietary and publicly available materials and trainings.

Rating information was also gathered on all mental health interventions and included the average quality-of-research score for all cited intervention outcomes, and the average readiness-for-dissemination score. Both quality of research and readiness for dissemination are scored on a scale of 0 to 4, with 4 representing the highest possible score on any given criterion. More detailed information on the criteria for both the quality-of-research and readiness-for-dissemination scores is available on the NREPP Web site.

Results

At the time the study was conducted, 91 of the 159 interventions (57%) listed on NREPP met inclusion criteria as either a mental health promotion or a mental health treatment intervention (or both). The descriptive characteristics of these interventions are presented in Table 1. Mental health interventions listed on NREPP were more likely to focus on promotion than treatment and on children and adolescents than adults. The interventions also were more frequently delivered in nonclinic settings (for example, schools, community settings, and homes), had undergone one or more replications, and were proprietary in nature.

Proprietary status, in particular, was associated with several other variables. Quality-of-research scores for key intervention outcomes were lower for interventions with exclusively proprietary materials and trainings (2.82±.55) than for interventions for which the materials or trainings were either all publicly available (3.23±.53) or a mix of proprietary and publicly available (3.14±.50) (F=4.91, df=2 and 88, p<.010); a Scheffé post hoc test indicated that the scores for the proprietary and public programs were significantly different from one another.

In contrast, readiness-for-dissemination scores were lower for interventions with exclusively publicly available materials (2.42±1.00) than for interventions for which the materials or trainings were either exclusively proprietary (3.30±.69) or a mix of proprietary and publicly available (3.22±.76) (F=6.63, df=2 and 88, p<.002); a Scheffé post hoc test indicated that the scores for proprietary and public programs were significantly different from one another.

Compared with interventions focused on adults, those focused exclusively on youths were more likely to be proprietary (χ²=6.98, df=2, p<.030). Whereas 24 (68%) child-adolescent interventions were proprietary, only eight (33%) adult interventions and 15 (50%) combined adult and child-adolescent interventions were proprietary.

Age also appeared to be associated with the setting of the intervention. Interventions for adults were more likely to take place in a traditional mental health setting (inpatient, outpatient, or residential) than those for either children and adolescents or for both adults and children (χ²=18.07, df=2, p<.001). Eighteen (75%) adult interventions took place in a traditional mental health setting, whereas nine (22%) child-adolescent interventions and ten (33%) combined adult and child-adolescent interventions took place in such a setting.

Discussion

This study provides an initial snapshot of mental health interventions contained in an important federally sponsored registry of evidence-based programs and practices. Results indicate that among mental health interventions currently included in NREPP, a larger percentage focus on the promotion of mental health than on the treatment of mental illness. The interventions emphasize a range of ages. Most interventions—especially youth-focused programs—are delivered partly or fully outside of traditional mental health settings. An overwhelming majority of the mental health interventions currently listed on NREPP are partly, if not fully, proprietary in nature—that is, the developers charge interested parties for some or all of the materials or trainings needed to successfully implement the intervention.

The finding that most child and adolescent interventions are proprietary may be a result of the inclusion in NREPP of some interventions from a predecessor registry that identified “model” substance abuse prevention and mental health promotion programs. Interventions from this system that met the NREPP criteria were identified and rereviewed for inclusion in NREPP, and many of these are proprietary school-based programs. Also, the finding that most adult interventions are designed to be delivered in traditional mental health settings is consistent with historical mental health service delivery patterns (5).

Perhaps the most striking finding is the relationship between dimension scores and program status. In general, interventions for which the necessary materials or trainings are partly or exclusively proprietary have lower quality-of-research scores but higher readiness-for-dissemination scores than interventions for which the materials or trainings are publicly available. This finding may reflect researchers' different motivations and financial incentives related to developing and marketing their interventions. In other words, this finding may suggest that some researchers—or their designated agents—become “purveyors” or marketers of a particular intervention and thus may focus less on expanding the intervention's evidence base than on facilitating the intervention's dissemination (6). Such a finding is worthy of further exploration, as it may reveal important factors that might influence the manner in which evidence-based mental health interventions are developed and disseminated in the future (7).

Nevertheless, the NREPP submission and selection processes represent important limitations of the study reported here. Interventions listed in the NREPP database have been voluntarily nominated for inclusion, and the extent to which those included are representative of all mental health interventions—or even of interventions that have been evaluated—is uncertain. Moreover, to be selected for review, interventions must meet certain minimum requirements, which are identified in annual guidance published by SAMHSA.

Legitimate questions remain as to whether the NREPP submission or selection process contributes in any systematic way to greater inclusion or exclusion of certain types of mental health interventions. Attention to these issues will be important because resource limitations, among other factors, preclude NREPP from becoming an exhaustive repository of all tested mental health interventions.

The continually evolving nature of NREPP represents another important limitation. As the registry grows by the addition of programs, aggregate data on the characteristics of included interventions are likely to change. However, the dynamic nature of NREPP also underscores the importance of periodic assessments similar to or even more extensive than this study. Such efforts might enable SAMHSA to more readily identify programmatic gaps in the overall NREPP portfolio that could be addressed during the annual open submission process for soliciting new interventions. A recent example of this during the 2010 open submission period was a call for interventions with publicly available implementation materials and trainings.

Conclusions

Many challenges confront organizations interested in successfully implementing evidence-based programs. Committed leadership, motivated and competent staff, congruence of the program with the organization's mission and values, and adequate financial resources are but a few of the drivers necessary to deliver on the promise offered by evidence-based practices (8,9). Given the complexities associated with implementing evidence-based interventions, it is not surprising that many community-based mental health organizations have yet to widely embrace their use.

Nevertheless, NREPP represents a practical tool for behavioral health organizations seeking to initiate or expand their use of evidence-based services (10). The information and ratings contained in NREPP may help agencies and others determine whether an intervention can help them to achieve desired outcomes for the individuals and families they serve. Moreover, as NREPP continues to evolve, efforts such as this study may identify gaps in the mental health services evidence base toward which future research funds might be directed.

Figures and Tables