In Reply: We very much appreciate Professor Essock's insightful comments. As she rightly points out, all too often authors of mental health outcome studies report their findings in ways that limit a study's usefulness for practitioners in usual care settings. Our study bears this out. Authors often failed to report fundamental variables related to external validity, due in part to the lack of emphasis for many journals to require or encourage the reporting of these variables. We agree with Dr. Essock that research findings need to be reported in ways that maximize their usefulness to frontline providers. If an author fails to report on key aspects on the conduct of a study, providers and policy makers are only left to guess as to the extent that study may or may not apply to their particular patients.
We do agree with Dr. Essock that our study does have limitations. For example, we were not able to ascertain whether it was justifiable for a study not to report on or to exclude minorities; such an assessment usually takes a thorough review for a study section to examine a large body of information to make that kind of determination. Nonetheless, the fact that 71 percent of the studies entirely failed to report this important variable is a significant problem. We do report on the reliability of the variables, although a number of important ones faired poorly, such as whether or not treatment was randomly allocated. We believe that our problems with reliability reflect the flawed nature of reporting in these studies—important characteristics of the design and execution were often difficult to ascertain.
Finally, we also agree with Dr. Essock that including Psychiatric Services in a similar study would be worthwhile, and we plan to include this journal in a new study that examines changes in reported external validity from 1996 to 2005. Certainly, how generalizability has been framed and its perceived importance to the field likely have changed significantly over the last decade—but how much and in what key areas?
