Dr. Frank is affiliated with the Department of Health Care Policy, Harvard University, 180 Longwood Ave., Boston, MA 02115 (e-mail: frank@hcp.med.harvard.edu). This commentary is part of a special section on the Clinical Antipsychotic Trials of Intervention Effectiveness (CATIE) and its implications. Marvin S. Swartz, M.D., served as guest editor of the special section.

Mental health professionals live in an era that increasingly demands that clinical choices be based on evidence. Increased reliance on evidence of treatment effectiveness and cost-effectiveness to guide clinical and policy decisions about what is prescribed and paid for raises the stakes on how we assess evidence. Nowhere is this more apparent than in the debate about the value of second-generation antipsychotic drugs. Passions are running high. Millions of dollars, issues of clinical autonomy, the role of mental health care consumers, and the reputations of researchers are on the line. In this issue Rosenheck and colleagues (1) stake out a strong position by proposing to tightly restrict the use of second-generation antipsychotics.

Examining such positions forces us to consider what evidence decision makers need to support good choices. Policy development must also take account of history, ideology, and commercial interests. Rosenheck and colleagues (1) argue that cost-effectiveness analysis offers a scientific method for assessing the value of spending on second-generation antipsychotics. Great weight is given to evidence of the cost-effectiveness of second-generation antipsychotics from the Clinical Antipsychotic Trials of Intervention Effectiveness (CATIE) (2). In this commentary I discuss the evidence needed to make the types of judgments Rosenheck and colleagues propose and comment on whether research has delivered the necessary information.

Rosenheck and colleagues based their policy ideas on evidence of the relative cost-effectiveness of the first and second generations of antipsychotics. The enthusiasm for cost-effectiveness analysis is striking, but in identifying the "right" course to follow, it is prudent to recall the admonition of the late Edward Gramlich (3): "Although the benefit-cost framework is simple and useful as an organizing device, it is easy to see more in benefit-cost analysis than there really is. It could naïvely be felt that benefit-cost analysis provides more answers than it really does, or makes questions easier than they really are."

Rosenheck and colleagues emphasize results from CATIE, arguing that other studies comparing second-generation antipsychotics and first-generation antipsychotics are methodologically weaker or should be regarded with suspicion because they were funded by the pharmaceutical industry, whereas CATIE was sponsored by the National Institute of Mental Health. So there are two central questions: Does CATIE offer sufficient evidence about effectiveness and cost-effectiveness to provide a strong platform for making choices, and What might decision makers sensibly consider in making choices about the rationing of second-generation antipsychotics?

A clinician wants to know what can be expected in terms of the clinical and functioning outcomes for new patients with schizophrenia if he or she prescribes a second-generation antipsychotic instead of a first-generation antipsychotic. A policy maker wants to know what extra benefits (clinical, functioning, subjective, and social) the community collects in return for paying the extra costs of a second-generation antipsychotic to treat schizophrenia.

CATIE is one of the most important and sophisticated studies of the treatment of schizophrenia. Yet because it sought to provide evidence of effectiveness in relatively natural clinical settings it has some inferential shortcomings. The most significant limitation for answering the questions posed above relates to high rates of discontinuation of drugs to which patients were randomly assigned. Overall, about 75% of the patients ended the study on a different medication than the one to which they were initially assigned. Moreover, the median patient spent well under 50% of the 18 months of the trial on the randomly assigned medication.

For the primary endpoint, time to discontinuation, switching does not pose an inferential problem (4). For the cost-effectiveness analysis it may. CATIE investigators followed the standard intent-to-treat approach to data analysis. They compared outcomes according to which drug participants were randomly assigned. This method works well when only modest numbers of patients switch treatments. When large portions of participants move to treatments other than those assigned, clinicians have greater difficulty answering the question, What will happen to my patient if I treat him or her with drug X? A high rate of switching also limits the policy maker's ability to judge the social benefits and costs of different treatments. In fact, this approach increases the likelihood of finding no differences.

Rosenheck and colleagues argue that finding similar clinical benefits and lower costs for second-generation antipsychotics and first-generation antipsychotics justifies strictly limiting access to second-generation antipsychotics at current prices or lowering their prices. It may be premature to adopt such policies primarily on the basis of CATIE results. This does not mean that some of the ideas proposed by the authors are not worthy of careful consideration. Adopting "fail first" policies for people with first episodes of schizophrenia may be quite sensible. Nevertheless, in considering such a policy it is important to heed concerns raised by advocates rather than dismissing them as being unduly influenced by left-wing ideology or commercial interests. The advocacy community is painfully aware that payers have frequently seized on uncertainty about evidence of effectiveness to limit mental health insurance coverage, driven partly by economic incentives to avoid enrolling costly patients. Thus a reluctance to support strict rationing of potentially important treatments may reflect a sound reading of history. It would therefore be wise not to overemphasize a single set of results and to understand the importance of nonscientific input about how policy works in practice.

CATIE was supported by grant N01-MH-90001 from the National Institute of Mental Health. A list of study locations and principal investigators can be found at www.catie.unc.edu/schizophren ia/locations.html. The author gratefully acknowledges discussions with Sherry Glied, Ph.D., Howard H. Goldman, M.D., Ph.D., and Tom McGuire, Ph.D., as well as financial support from the National Institute of Mental Health (grant R01-MH-069721) and the John D. and Catherine T. MacArthur Foundation.