To the Editor: The report describing changes in the process of care in Utah's prepaid mental health plan by Michael Popkin and his colleagues (
+1) in the April 1998 issue demonstrates the perils of drawing conclusions about changes in enormously complex systems. As clinical director in the largest of the prepaid community mental health systems, Valley Mental Health, serving Salt Lake County, I have numerous technical questions and concerns about the report. However, I would like to focus on four major issues.
The first issue is the difficulty of inferring causality in a study that, despite the use of exceedingly complex statistical procedures, could not possibly control for all the variables that might account for the differences observed. A case in point is the discussion of the increase in the number of clients lost to follow-up. Dr. Popkin concludes that the changes in process "led" to the increased loss. However, the changes noted were consistent with assertive community treatment (
+2) and were based on research that would predict reduced loss. A confounding variable not controlled for was the enormous increase in the cost of housing in Salt Lake County between 1989 and 1993, which appears to have forced a number of consumers to relocate to other communities, with the result that they were lost to follow-up. The conclusion that the process change led to the outcome is scientifically not demonstrable.
The second issue is the failure to focus on variables that have previously been shown to be most predictive of outcome, and more predictive than the variables that this study attempted to evaluate. A study by Sullivan and colleagues (
+3) demonstrated that medication noncompliance, comorbid alcohol abuse, and high levels of criticism by family members were predictive of relapse. Use of traditional outpatient services was not. It is striking that the variables monitored in Utah, other than medication dosage, were not those associated with relapse.
The third issue is the authors' failure to consider the context of the changes in Utah's mental health system. The decision to implement a model for cost containment was a strategic one, the goal being survival of the system. Given the pressures for cost containment that are faced by community mental health systems, it is myopic not to consider the possible consequences of failure to implement a cost-containment strategy. The changes that Dr. Popkin and his colleagues focused on pale in comparison to the potential results of system disenfranchisement and failure, as was documented recently in Tennessee by Chang and associates (
+4).
The fourth issue is the imprecision and pejorative implication of the central conclusion of the article—that the data "raise pointed concerns about the vigor of care." Although the authors state that "only longitudinal data will afford a definitive answer," I am dubious that longitudinal data about process, in the absence of data about outcome, will answer questions of substantial importance.
Exciting new treatment technologies are dramatically increasing our ability to relieve the symptoms of severe mental illness and make rehabilitation and recovery a real possibility for our patients. These advances should change our process of care. I have great respect for our caregivers, who do far more with fewer resources than seems possible. Commitment to clinical excellence is a central component of our mission. I am not persuaded that the authors were rigorous in their efforts to measure meaningful process variables, or that their findings, which are mixed, support their pejorative and imprecise conclusion. In my opinion, their conclusion suggests a vigorous effort to negatively critique those systems that have implemented a prepaid financing arrangement.