Design Strengths and Issues of SAMHSA's Women, Co-occurring Disorders, and Violence Study

Violence is pervasive in the lives of many women, both in the formative years and later in life. Many studies link early experiences of interpersonal violence and trauma to a wide range of physical and mental health problems and to associated behaviors and risk factors (1,2,3,4,5,6,7,8,9). These studies clearly point to the substantial potential impact of adverse life experiences and highlight the need for prevention and early intervention. However, in these studies, the question of what to do to intervene successfully with women who have such a history is not answered.

Recognizing the lack of attention given to the service needs of this population, in 1998 the Substance Abuse and Mental Health Services Administration launched the Women, Co-Occurring Disorders, and Violence Study (WCDVS). The WCDVS began to address the question of what to do by developing, implementing, and evaluating the outcomes and costs of comprehensive, trauma-informed treatment programs for women with a history of violence and trauma who have substance use and mental health disorders. As the first large-scale, multisite effort to develop and evaluate comprehensive, integrated services for this population of women, the WCDVS represents an important step in exploring how services and service systems for women might be improved.

Specific details about the design of the study and the results of the outcomes and cost evaluations are found elsewhere (10,11,12,13) and in this issue of Psychiatric Services (14,15). Using a quasi-experimental, intent-to-treat design, the WCDVS compared six- and 12-month outcomes of women at nine sites who received comprehensive and integrated substance abuse, mental health, and trauma services with the outcomes of those who received usual care. The nine WCDVS sites were located across the continental United States, with six sites on the East Coast, two on the West Coast and one in Colorado. A total of 2,729 women (1,415 in the intervention condition and 1,314 in the comparison condition) were enrolled over the 13.5-month baseline accrual period (January 2001 through February 2002). This article discusses the overall design features of the study, including issues related to such a design, outcomes and cost, and suggestions for future research.

Design strengths and issues

Multisite, quasi-experimental research has many advantages, including the ability to test service models in naturalistic, real-world settings and the ability to generalize results to other settings that serve similar populations. Multisite services studies such as the WCDVS also generally allow for the rapid recruitment of large and diverse samples (16), which is not as readily possible in single-site designs. However, this type of research can also be plagued with threats to internal validity for a variety of reasons, including participant attrition, similarity or equivalence of comparison programs, and poorly standardized or nonstandardized data collection methods. Researchers must take explicit steps to minimize these threats to internal validity.

Comparison programs

If the comparison programs are not similar to the intervention programs, the threat to internal validity can be tremendous. In the WCDVS, each study site was responsible for selecting its own comparison program or programs. The intimate knowledge that each local site held about the comparability of other programs within its geographic area provided a unique basis and perspective from which comparison programs could be chosen. As such, a much more accurate comparability assessment among programs was possible, resulting in the selection of well-matched comparison programs that were similar in the nature of services delivered, modality, and traditional treatment focus (for example, mental health or substance abuse). Results of equivalence analyses between the intervention and comparison conditions are reported by McHugo and colleagues (13).

Nonrandom assignment of participants

One of the key design features that distinguishes quasi-experiments from true experimental research is the nonrandom assignment of participants to conditions. The issue of participant selection bias is inherent in this process and often plagues quasi-experimental research if strict procedures are not put into place to minimize the impact of nonrandom assignment on group composition. The multisite nature of the WCDVS made the use of random assignment procedures neither possible nor desirable because of practical difficulties at the program sites and the strong belief that was held by many members of the design committee that these procedures would create an aura of artificiality in the situation. Results of between-condition equivalence analyses conducted with the pooled, cross-site six-month (13) and 12-month (14) follow-up samples indicated that although some differences existed between the conditions at each of the two follow-up points, the differences reported are of a magnitude such that they are unlikely to be clinically significant. Furthermore, at both follow-up points there was little evidence of differential attrition between the intervention and comparison conditions (13,14).

Participant retention

We also learned a great deal about recruiting and retaining participants. Because of the sensitive nature of the interview and the often disturbing history of participants, developing a collaborative alliance between the participant and the researcher was crucial. Community outreach, utilization of research teams that worked collaboratively with program staff and that had first-hand knowledge of the programs and communities they served, as well as the use of rigorous tracking procedures, including a standard, computerized participant tracking program, ensured that a low rate of attrition was maintained throughout the follow-up periods (13,14).

Cross-site standardization

High standards of data integrity were maintained through a variety of procedures. A standardized interview schedule was used, and research interviewers and supervisors attended a two-day, centralized instrument training session before data were collected. A centralized system for monitoring data collection and quality control was used, including the development and distribution of a cross-site coding issue database, intensive review of protocols in collaboration with site research teams, double data entry and range checks, and centralized review of audiotapes of mock interviews that were submitted by each research interviewer before data were collected. The audiotapes were subject to approval of the level of the interviewer's adherence to the protocol. These procedures ensured that data were collected in the same manner and that they were equivalent in nature across the widely diverse program locations.

Features enhancing generalizability

The design of the WCDVS had at its core many features that enhanced the generalizability of study findings. Although the intervention was implemented in different ways at each of the nine study sites and although it involved different configurations of outpatient and residential settings and participating organizations, all intervention programs shared a set of core elements that defined their structure. This combination of common program elements, diversity in how each element was constituted at the program sites, and service delivery and hypothesis testing in a real-world setting increased the likelihood and degree to which the intervention may be translated into other settings. In other words, studying the effectiveness of programs in their natural context enhanced generalizability (17).

Collaborative decision making

Finally, multisite studies, especially those that are federally funded, are often subject to a collaborative decision-making process. The WCDVS had such a collaborative process at its core, but it was precisely this process that was one of the primary catalysts for success. All decisions about the research design and implementation of the intervention were guided by a cross-site steering committee, which was composed of federal representatives, staff of the coordinating center, principal investigators of the sites, clinical staff, women with experiences similar to those of research participants, and researchers. Although in many cases the necessity of engaging in this process can prolong and complicate the design process, when successful, the process can substantially enhance all facets of an evaluation.

Outcomes

Women in the intervention and the control groups showed improvement in outcomes in four areas: alcohol use, drug use, mental health, and trauma. At six months women in the intervention group scored modestly better than women in the usual-care group on outcome measures for drug use, trauma, and mental health. At 12 months women in the intervention group maintained their improvement in drug use outcomes and continued to improve in mental health and trauma outcomes. Although average treatment effects across sites were small, at both follow-up points some sites achieved significantly greater effects than others. For some sites the greater effects appeared to be caused by the receipt of more integrated counseling—that is, individual and group sessions addressing all three elements of trauma, mental health, and substance abuse—by women in the intervention group than by their counterparts in the comparison group.

After a start-up period, costs for services were not significantly different between the intervention and comparison groups at both follow-up points. Furthermore, cost of services outside the study intervention (for example, jails, hospital inpatient units, and emergency departments) was examined, but no significant differences were found between the two groups.

Discussion

Through the process of evaluating the services designed for the WCDVS, we have begun to shed light on some of the issues related to providing services for women with co-occurring disorders and a history of violence and trauma. However, there are still many unanswered questions related to clinical practice, public policy, and financing, and further research is needed. Although the following discussion is more qualitative and experiential in nature, it builds on the analytic results and offers a perspective on many of these unanswered questions.

Clinical practice

Brown and colleagues' (18,19) adaptation of Prochaska and colleagues' (20) stages of change model posits that women who are dealing with multiple needs and issues may be ready to change at any given time in some, but not all, areas. Brown (18) relates this readiness to levels of "immediacy" or time urgency felt by a woman. For example, a woman might wish to address substance abuse and any underlying physiological effects before committing to working on psychological problems. Likewise, the tentative pattern of recovery seen in the WCDVS appears to suggest that stabilization of substance abuse behaviors may be necessary before gains can be made in reducing mental health and trauma symptoms. However, practitioners must be mindful of the complexity in treating persons with three interrelated disorders and recognize how little is understood about the process and trajectory of change among people with these multiple conditions.

Furthermore, although not yet addressed in this study, practice implications include issues encountered when integrated care is adapted within different treatment modalities and settings. Outpatient organizations may seek to provide integrated counseling through one counselor or through services provided by another collaborating agency, whereas residential treatment programs may be able to provide more of an "integrated service package" at one site.

Financing and policy

The findings of the cost evaluations (11,15) have potential implications for policy and financing and suggest that there may be the opportunity for federal, state, and local funding agencies to promote broad implementation of trauma services with relatively limited investment over current levels of funding. Although costs of providing the total set of services reported by participants is substantial (averaging more than $42,000 per participant during the one-year study period), the findings by Domino and colleagues (15) indicated that the related costs of participants who receive intervention services may be no greater than the costs for those who receive usual care. The challenge of shifting resources across agencies and among payers remains, but this work indicates that efficiency gains in clinical outcomes may be possible.

In addition, it may be necessary to provide support for one-time start-up costs associated with training and supervising staff, along with the initial purchase of materials and compensation for time to recast policies, procedures, and practices (21). It is important to note that the WCDVS assessed costs within a limited time frame (12-months) after the initial implementation of services. It is possible that costs of sustaining these services over a longer period may exceed initial cost estimates, because supervisory costs and staff turnover and the associated costs of retraining could significantly affect the level of funding needed to maintain an integrated and trauma-informed service package.

Future research

The WCDVS evaluation offers many avenues for future follow-up investigations. The variation in the experiences and needs of participants, along with the diversity of intervention sites, both complicated and enriched the WCDVS results and their interpretations. This was a study of programs and program participants embedded in their natural contexts, in which the programs worked to implement comprehensive, integrated, and trauma-enhanced services.

Although the provision of integrated counseling emerged as a potentially important ingredient in interventions such as these, additional key intervention components may be at work. Future studies are needed in which other WCDVS intervention components are more fully standardized and tested, both alone and in combination with the integrated counseling component. If the integrated counseling component is still found promising, research to refine and systematically vary individual facets of this component would be warranted. This research approach would allow for an assessment of the relative contribution of distinct service elements and larger treatment milieus and would allow more direct testing of internal validity. Care should also be taken to establish measures of exposure to different treatment elements, preferably with multiple indicators of this exposure, an important factor that was not considered in the intent-to-treat model used in the WCDVS.

Conclusions

Despite their very modest nature, the WCDVS results are promising. Further study using randomized designs is still needed to isolate these effects and to confirm internal validity. Comparative assessments of intervention components in outpatient and residential settings would help to establish generalizability across settings.

Little is known about the staging of recovery, especially recovery across the multiple issues of concern. Even though the study was not intended to enhance understanding of the order of change throughout the recovery process, it does raise some unanswered questions about the potential order of recovery (substance abuse behaviors compared with mental health and trauma symptoms) that should be followed with future research. What is the trajectory and pace of change? Are there multiple trajectories? How are changes in one domain related to and affected by changes in another? These are all important questions for future research.

Acknowledgments

This study was funded by the Guidance for Applicants grant, number TI-00-003, from Substance Abuse and Mental Health Services Administration's three centers: the Center for Substance Abuse Treatment, the Center for Mental Health Services, and the Center for Substance Abuse Prevention. This grant was entitled Cooperative Agreement to Study Women With Alcohol, Drug Abuse and Mental Health Disorders Who Have Histories of Violence: Phase II.

Mr. Noether is affiliated with Policy Research Associates, Inc., in Delmar, New York. Dr. Finkelstein is with the Institute for Health and Recovery in Cambridge, Massachusetts. Ms. VanDeMark is with the department of research and evaluation at Arapahoe House, Inc., in Thornton, Colorado. Dr. Savage is with the Hunter College School of Social Work at the City University of New York in New York. Dr. Reed is with the University of Michigan School of Social Work in Ann Arbor. Ms. Moses is with the National Center on Family Homelessness in Newton Center, Massachusetts. Send correspondence to Dr. Noether at Policy Research Associates, Inc., 345 Delaware Avenue, Delmar, New York 12054 (e-mail, [email protected]). This is the third of three papers in this issue reporting results from the Women, Co-occurring Disorders, and Violence Study funded by SAMHSA.