To the Editor: An effective randomized trial comparing a novel intervention with treatment as usual requires a clearly described comparison group, reliable assessment of relevant factors, and appropriate statistical models. Difficulties in all three areas are illustrated in a recent study by Sledge and associates (1) reported in the May issue. The study monitored inpatient readmissions over nine months among patients after their discharge from an inpatient facility. Patients with at least two psychiatric hospitalizations in the prior 18 months were randomly assigned to treatment as usual or to treatment as usual plus a modified “peer companion” intervention.

Services researchers are coming to appreciate how enriched interventions can be easily contaminated by local variations in usual treatment. Efforts to address this problem require a clear and detailed account of treatment as usual, but the authors described it only as including medication, psychoeducation, case management, and supportive psychotherapy. They did not describe which outpatient services were provided at either patient or agency levels. When the study was conducted, services available to public mental health consumers in Connecticut included cognitive-behavioral therapy, assertive community treatment-level case management, illness management and recovery, and individual placement and support. Interventions at specific levels of care included medication delivery, money management, and on-site supervision. Consumers in the study also may have been receiving system-sponsored peer mentor services. Who was receiving what, and with what effects? Who benefited most from the intervention? The authors proposed that the peer companion enrichment may affect patient engagement, but their report did not include proximal outcome measures of engagement, alliance, adherence, satisfaction, symptom severity, functioning, or goal attainment.

The study reported a low effect size, which may be owing in part to its reliance on an administrative database. As the authors noted, admissions to nonpublic inpatient units are not routinely tracked. More significant, the database does not track emergency room visits. Because of severe bed shortages and complex payment structures, these visits can be lengthy, and they frequently substitute for inpatient treatment. Admissions occur for a variety of reasons, and the patient's condition at admission affects level-of-care assignments to acute or longer-stay units. Length of stay is also influenced by funding sources and by the timing and availability of community resources. Except for involuntary commitments, the administrative database does not track reasons for admission or discharge, and information about a patient's condition at admission or discharge, except for an unreliable Global Assessment of Functioning score, is not documented.

In a recent book review published in Psychiatric Services, Geller (2) noted that the vision of recovery can be so captivating that it may lead researchers to work backward, from thesis to evidence. Activities such as data gathering, description, measurement, and analysis might be viewed as mundane and inconsequential within this larger vision. Perhaps a richer and more extensive account of treatment as usual, measurement of proximal outcomes, and a greater focus on data gathering and statistical analysis might give the authors' work greater substance and credibility and facilitate efforts to improve the lives of persons who are receiving mental health services.