Diagnoses of Patients Treated With ECT: A Comparison of Evidence-Based Standards With Reported Use

A substantial body of research has documented the efficacy and relative safety of electroconvulsive therapy (ECT) for selected psychiatric disorders (1,2). Despite these findings, ECT remains one of the most controversial procedures in medicine, as reflected by a variety of data.

The use of ECT varies widely across the United States; the extent of variation is greater than that for most other medical procedures (3,4). Less than 8 percent of U.S. psychiatrists provide ECT; the likelihood of use varies with where and when the psychiatrist was trained as well as with gender and proximity to an academic medical center (5). Surveys show that mental health clinicians are divided over the utility of ECT (6,7,8,9).

ECT has become one of the most commonly regulated procedures in medicine, subject to divergent laws and regulations across the country (10). ECT use in the U.S. declined from the 1970s to the mid-1980s, although evidence suggests this decline may have leveled off in recent years (11,12).

Among the reasons for ECT's controversial status is the belief that the procedure has been used for conditions other than those for which it has been proven efficacious. Few data are available on diagnoses of patients treated with ECT during its first decades of use, but several authorities have observed that ECT was used indiscriminately in early years (13,14,15). More utilization data are available from the 1970s, showing that ECT was principally used to treat major depression, mania, and schizophrenia but also that a substantial proportion of use (11 to 27 percent of samples in published reports) was for other conditions such as dysthymia, neurotic depression, anorexia nervosa, and certain personality disorders (16,17,18,19).

Over the last ten years, American health care has increasingly focused on the evidentiary foundation of clinical practices (20,21). Major components of the evidence-based medicine movement include the following:

• Standardized assessments of research literature and expert opinion to determine which conditions are best treated with which interventions

• Examination of clinical practice patterns to determine where clinical practices diverge from evidence-based standards

• Efforts to educate clinicians and guide their practices to follow evidence-based standards.

With regard to ECT, the psychiatric community has made significant progress in the first and third of these tasks—reviewing the research literature and producing educational guidelines—but less has been done to evaluate the extent to which ECT use follows evidence-based standards. Several groups have reviewed studies evaluating the efficacy of ECT. In 1985 a National Institute of Mental Health (NIMH) consensus conference determined that ECT was efficacious for delusional and severe endogenous depression, mania, and certain schizophrenic syndromes (1). In 1990 the American Psychiatric Association (APA) task force on ECT concluded that "compelling data . . . or a strong consensus" exists to support use for major depression, bipolar depression and mania, schizophrenia, and schizoaffective and schizophreniform disorders (2). Practice guidelines issued by APA and the U.S. Agency for Health Care Policy and Research (AHCPR) on the treatment of major depression have supported the use of ECT, particularly for severe or refractory depression (22,23).

Since these reviews and guidelines were published, no studies have examined the diagnoses of patients treated with ECT and compared them with these evidence-based standards. Studies by Reid (24) and Thompson and associates (11) reported use of ECT for broad diagnostic categories, but these reports contained insufficient detail to compare utilization to published recommendations. The absence of knowledge about how ECT is used in the U.S. limits our ability to evaluate whether its use follows evidence-based standards. If, in fact, the use of ECT closely follows research findings, such knowledge would inform ongoing debates over use and regulation of the procedure. Alternatively, if a gap was found between clinical use of ECT and the research evidence, then the extent and characteristics of the gap would provide an important resource for improving the use of ECT.

In this study we used claims data provided by a large New England insurance company to examine the diagnoses of patients treated with ECT in 1994-1995 in a multistate sample of patients. Our focus was to assess the extent to which diagnostic indications for the procedure, as reported on insurance claims, corresponded to the psychiatric disorders that ECT has been found to treat efficaciously. We also examined characteristics of psychiatrists and hospitals associated with the use of ECT for diagnoses outside of evidence-based standards.

Methods

Sample and data

Our sample consisted of all courses of ECT received by beneficiaries of a large New England insurance company during 1994 and 1995. The company insured approximately 1.2 million covered lives during the period studied. Administrative claims for each course of ECT were abstracted to record the number of treatments per course, the DSM-III-R diagnostic code, information about the clinician administering ECT, and the facility where treatment was provided.

During 1994-1995, a total of 996 patients in the sample were treated with ECT. They received 1,532 courses of ECT treatments. For courses of ECT for which the diagnosis was identified, we analyzed the proportion that met diagnostic criteria for which ECT had been proven efficacious and the proportion that did not meet the criteria. A diagnosis was identified for 94.7 percent of the treatment courses and for 96.4 percent of the patients treated.

We linked information for each patient to training and certification data about the treating psychiatrist from the APA membership directory, the physician master file of the American Medical Association, physician profiles from the Massachusetts Board of Registration in Medicine, and information from the American Board of Medical Specialties. Data included the treating physician's year of medical school graduation, whether the physician graduated from a U.S. or a non-U.S. medical school, and whether the physician had specialty certification from the American Board of Psychiatry and Neurology.

We also linked patient information to data on the treating hospital from the American Hospital Association's 1992 Annual Survey of Hospitals. Data included whether the hospital was a general medical hospital versus a psychiatric specialty hospital, the hospital's teaching status (a nonteaching hospital, a hospital with residencies, or a hospital with residencies and a medical school), and hospital size (0 to 150 beds, 151 to 350 beds, or 351 to 1,000 beds).

Information about the treating psychiatrist and the hospital was complete for a subsample of 824 ECT courses (53.7 percent of courses). The subsample was used for multivariate analysis of the association between provider characteristics (physician and hospital) and whether the patient's diagnosis was within the criteria for indications for ECT.

Diagnostic categories

Criteria were established to categorize psychiatric disorders in terms of the efficacy of ECT. The modality was judged to be an efficacious treatment for a disorder if one or more randomized controlled trials published in a peer-reviewed journal found clear evidence of efficacy, or if the efficacy of ECT for the disorder was supported by a combination of peer-reviewed studies of nonrandomized cohorts and the findings of one or more of the national panels of experts established by NIMH, APA, and AHCPR (1,2,22,23).

Diagnoses that met these criteria were major depressive disorder, bipolar disorder, schizophrenia, schizoaffective disorder, and schizophreniform disorder. Table 1 summarizes peer-reviewed research published between 1970 and 1998 on the efficacy of ECT by diagnosis. A bibliography of the 134 studies summarized in the table is available from the authors on request.

Descriptive analyses

Insurance claim forms were submitted for each ECT treatment session in a treatment course. The claim recorded the primary diagnostic indication for ECT. In some cases, the reported diagnosis differed in one or more of the claims for a single course of ECT. In these cases, the criteria were conservatively applied—that is, if any of the primary diagnoses reported for ECT treatments within a course were within evidence-based indications for ECT, the course was categorized as within the criteria. The number and proportion of ECT courses within and outside the criteria were calculated.

Multivariate analyses

Logistic regression was used to determine the association between the likelihood of a patient's diagnosis being within the criteria (the dependent variable) and characteristics of the psychiatrist and hospital providing ECT (the independent variables). The odds ratio and 95 percent confidence interval are reported for each independent variable. Courses of ECT were likely to be correlated within hospitals because of unobservable hospital and psychiatrist characteristics. To account for these effects, the regression model was estimated using generalized estimating equations methods (25,26).

The regression model tested four hypotheses about ECT use outside the criteria. First, we hypothesized that use outside the criteria is less likely to be by psychiatrists more recently trained or board certified and by those who attended medical school within the U.S., because each of these groups would potentially be more familiar with U.S. practice recommendations. Second, use outside the criteria is less likely in teaching hospitals because of the oversight and emphasis on evidence-based medicine at these institutions. Third, use outside the criteria is less likely in psychiatric specialty hospitals than in general medical hospitals because of the greater dissemination of information specific to psychiatry within the former institutions. Fourth, use outside the criteria is less likely in large hospitals than in small hospitals because the larger environment provides greater opportunities for the dissemination of clinical information.

Results

During 1994 and 1995, a total of 996 individuals received 1,532 courses of ECT treatments, for an average of 1.6 courses per treated individual. Based on a denominator of approximately 1,200,000 insured beneficiaries, an annual utilization rate of 4.2 treated individuals per 10,000 covered lives was calculated.

A diagnosis was reported for 1,451 ECT courses (94.7 percent of the sample). As shown in upper part of Table 2, the diagnoses given for 1,276 of these courses (86.5 percent) were within the criteria for diagnoses treated efficaciously with ECT. A substantial majority of this use was for major depressive disorder, followed by bipolar disorder, schizoaffective disorder, and schizophrenia.

As shown in the lower part of Table 2, for a total of 199 ECT courses (13.5 percent), the diagnoses reported were outside the criteria. The majority of them were depressive disorders, including depressive disorder not otherwise specified and dysthymia. Other diagnostic categories in this group included anxiety disorders, substance use disorders, eating disorders, and personality disorders.

Unadjusted associations between provider characteristics and the use of ECT within or outside of the criteria are shown in Tables 3 and 4. Table 5 shows the results of multivariate analysis. When the analysis controlled for other psychiatrist and hospital characteristics, psychiatrists who graduated from medical school between 1940 and 1960 and between 1961 and 1980 were significantly more likely to use ECT for diagnoses outside the criteria than psychiatrists who graduated between 1981 and 1990 (Table 5). Psychiatrists who received their medical education outside the U.S. were less likely to perform ECT for diagnoses outside the criteria. Board certification status and hospital characteristics, including hospital type, teaching status, and size, were not significantly associated with the use of ECT for diagnoses outside the criteria.

Discussion and conclusions

This study provides preliminary data on how ECT use conforms to the evidence-based diagnostic indications for ECT. It represents the first such assessment since the dissemination of evidence-based diagnostic indications in the 1980s and practice guidelines in the 1990s. The principal finding is that the vast majority of ECT use in the sample was for disorders that can be efficaciously treated with ECT. In 86.5 percent of ECT courses, the reported diagnoses were for conditions for which the efficacy of ECT is well supported by research data. Most ECT was used for major depression and bipolar disorder, for which the research evidence is strongest. The evidence supporting use for schizophrenia and schizoaffective disorder is less compelling (1). Only 2.6 percent of ECT use in our sample was for these disorders.

These findings should provide reassurance to those concerned that ECT may be used indiscriminately. They also provide empirical data to inform often-heated debates in state legislatures about a variety of proposed regulations for ECT (10,24).

For 14 percent of the ECT courses in the sample, compelling research evidence was lacking to support the efficacy of ECT for the reported diagnosis. More than half of these conditions were depressive disorders, such as depression not otherwise specified or dysthymia, or disorders that likely had associated depressive symptoms, such as dementia and organic mood disorder. However, no controlled trials have shown the efficacy of ECT for these conditions or for the other disorders reported for this subgroup of the sample.

Use in these cases may nonetheless have been appropriate. Other factors can justify use of ECT, including a patient's treatment history and the severity of a patient's condition. ECT may reasonably be considered for certain life-threatening conditions, particularly when other treatment options have been exhausted or when the patient has a history of a positive response to ECT. Claims data do not provide enough clinical information to evaluate these possibilities. A study based on more detailed clinical data, such as that found in the medical record, would be needed to conclude whether use of ECT in these cases was appropriate.

We conducted analyses of psychiatrist and hospital characteristics associated with the use of ECT. As we hypothesized, psychiatrists educated in earlier decades were more likely to use ECT for diagnoses outside the criteria. Compared with psychiatrists who graduated from medical school between 1981 and 1990, psychiatrists who graduated between 1940 and 1960 were eight times more likely to use ECT for diagnoses outside the criteria, and psychiatrists who graduated between 1961 and 1980 were five times more likely.

These results may reflect differences in practice styles among psychiatrists trained in earlier eras. Clinicians may be less influenced by clinical trials or practice recommendations published after their training. Neither formal certification nor regular recertification are currently required for the delivery of ECT, leaving practitioners to assimilate new research findings through voluntary activities of varying quality. Certification programs have been advocated, as have improvements in ECT education in residency training programs and more questions testing knowledge of ECT in American Board of Psychiatry and Neurology certification examinations (27,28). Each of these recommendations could increase appropriate use of ECT.

An alternative interpretation of the association between training era and disorders treated with ECT may be related to differences in the reporting of psychiatric diagnoses rather than differences in utilization. Psychiatrists trained in earlier eras may be less likely to use contemporary psychiatric diagnostic terminology. Survey data suggest that psychiatrists trained in earlier decades used DSM-III and DSM-III-R diagnoses less rigorously and enthusiastically than more recently trained clinicians (29,30). Further study of psychiatrists' diagnostic practices, use of ECT, and knowledge of indications for the procedure would provide more insight in this area.

The estimated annual ECT utilization rate of 4.2 treated individuals per 10,000 covered lives is consistent with utilization rates found in other U.S. samples. Using results from a 1988 national psychiatrist survey, Hermann and associates (3) estimated a U.S. rate of 4.9 per 10,000 population. Using data from a 1980 national U.S. hospital sample survey, Thompson and Blaine (31) found an annual rate of 1.6 individuals treated with ECT per 10,000 population.

Several limitations of this study should be noted. First, these findings may not be generalizable to other populations. The data are from a commercially insured population, consisting largely of employed or retired individuals and their dependents. Privately insured patients may be more likely to receive ECT than beneficiaries of Medicare or Medicaid. Previous studies have found that ECT is more commonly used in private hospitals than in public hospitals (11,32). Our finding of a small proportion of patients treated with ECT for schizophrenia is also likely to be related to the privately insured status of our sample. Many patients with schizophrenia are publicly insured. Most of the patients in our sample were treated in Massachusetts; utilization of ECT may differ in other parts of the U.S.

Another limitation of our study is that the multivariate analysis included only the 54 percent of ECT courses for which the psychiatrist and hospital characteristics could be identified. In the remaining cases, information on the provider or institution was not available. Utilization of ECT in this group could have differed from our results.

It should also be noted that our data were limited to the primary diagnosis; patients may have had secondary psychiatric diagnoses. The insurance claim requested the primary diagnostic indication for ECT; thus to the extent these instructions were followed, it is unlikely that a patient's primary diagnosis, which was reported, would have been outside of the criteria while the secondary diagnosis, which was not reported, was within the criteria. However, some reporting error is likely.

Another concern is that the validity and accuracy of diagnoses reported in this study are subject to limitations. Unstructured diagnoses by clinicians are less valid than diagnoses obtained using an instrument such as the Structured Clinical Interview for DSM-III-R (33). In contrast to an outcome study in which the underlying condition is of primary interest, in this study we were interested in the clinician's diagnostic determination and its correspondence with the subsequent choice of treatment.

More problematic is the uncertain accuracy of diagnoses reported on administrative claims (34,35,36,37). Administrative claims are increasingly used in health services research to investigate the utilization and quality of health services. Studies of diagnoses reported on claims have shown good agreement with other sources of information, such as physicians' reports and abstractions from medical records, and evidence suggests that the accuracy of claims data is improving. Nonetheless, diagnoses reported on claims remain subject to errors. Our use of claims data in this study was undertaken cautiously, in the absence of other published data on specific diagnoses treated with ECT and in concordance with Iezzoni's conclusion (38) that "administrative data are probably most useful as screening tools that highlight areas in which quality should be investigated in greater depth."

Accordingly, our findings should be considered preliminary. Further study of ECT using established methods of evaluating the appropriateness of medical treatments should be conducted. These methods rely on clinically detailed sources of treatment data and rigorously developed assessment criteria (39,40,41,42). Studies of the use of coronary artery bypass grafts, hysterectomies, and tympanostomy tubes for childhood ear infections have found that rates of inappropriate use of these common medical and surgical procedures range from 15 to 30 percent.

Further study of ECT use should examine patient, clinician, and hospital characteristics associated with appropriate use. If subsequent studies confirm that psychiatrists trained in earlier decades are less likely to use ECT for evidence-based indications, then this finding would provide an opportunity for targeted intervention to improve the use of ECT. Our findings also suggest areas for further clinical efficacy trials—for example, in the use of ECT for depressive symptoms associated with dementia.

Finally, it should be noted that potential overuse is only one concern associated with ECT. Based on the procedure's efficacy in the treatment of severe and refractory depression, concerns have been raised about underuse, particularly in treating depression among elderly patients and in geographic regions with few psychiatrists or more stringent state regulation of the procedure (3,43).

Acknowledgments

This research was supported by grants K08-MH-001477 to Dr. Hermann, K02-MH-01177 to the late Dr. Dorwart, and R29-MH-53698 to Dr. Ettner from the National Institute of Mental Health. The authors thank Lynne Miller and Kimberly Santillo for assistance with data analysis.

Dr. Hermann is affiliated with the department of psychiatry at Harvard Medical School and Cambridge (Mass.) Hospital, where the late Dr. Dorwart was chairman. Dr. Ettner is with the division of general internal medicine and health services research in the department of medicine at the University of California, Los Angeles. Dr. Kleinman is with the department of psychiatry at Harvard Medical School. Ms. Langman-Dorwart is vice-president for medical affairs of the Oak Group, a developer of medical decision support systems in Wellesley, Massachusetts. Send correspondence to Dr. Hermann at the Department of Psychiatry, Cambridge Hospital, 1493 Cambridge Street, Cambridge, Massachusetts 02139 (e-mail, [email protected]).

Enlarge table

Enlarge table

Enlarge table

Enlarge table

Enlarge table