Depression is a significant problem in the Medicare population (1,2). Depressive conditions are highly prevalent and associated with considerable morbidity (3). Despite substantial improvements in recent years, only about two-thirds of Medicare beneficiaries who have a diagnosis of depression receive appropriate treatment (4–6).

Medicare or other claims have been used as case-finding tools for a variety of purposes, such as quality measurement or improvement, comparative effectiveness research, and economic assessments (5,7–9), that are aimed at improving depression outcomes. However, the quality of such research depends on how well administrative claims data identify individuals with depression and those who receive care. For example, if the goal is to determine the quality of treatment among those treated, case-finding methods based on pharmacy claims or procedure codes may be adequate. The needs of researchers also likely differ from those of administrators seeking to use claims data for public reporting or quality monitoring. However, because claims may cover large populations and are readily available, using claims data is considerably less expensive than alternative approaches requiring primary data collection. The key is to understand when and how these data can be used appropriately.

The purpose of this study was to examine how closely codes for depression from claim files correlate with symptoms and identification of depression as measured by independent research assessment instruments. We chose to use two types of data that are frequently available in clinical and policy research settings: Medicare administrative claims and patient self-assessed depressive symptoms.

Data sources

This study used participant self-reported data from the Medicare Primary and Consumer-Directed Care demonstration (10) and Medicare claims for demonstration participants. This demonstration was a randomized controlled trial to test the acceptability and effectiveness of a health promotion nurse intervention; a voucher that reimbursed for services, supplies, and equipment not covered under traditional Medicare; and a combination of the voucher and the nurse interventions.

Patient recruitment and the demonstration study design have been described elsewhere (10). In short, 1,605 primary care patients in 19 counties in New York, West Virginia, and Ohio enrolled in the demonstration between July 1998 and June 2000 and completed the demonstration between July 2000 and June 2002. This was a relatively select Medicare population because the demonstration included only beneficiaries who were enrolled in both Medicare Part A and Part B, who needed or received help for two or more activities of daily living or with three or more instrumental activities of daily living, and who had recent significant health care use before enrollment. Information on patient health status, including depression, sociodemographic characteristics, and other measures, was collected before randomization (we refer to this data collection time point as baseline) and at 22 months postbaseline. A limited array of information, including depression measures, was collected at 12 months postbaseline. About 12% of the demonstration participants were enrolled in a Medicare managed care plan at baseline. Thus they had no Medicare claims and were excluded from the analysis.

Identifying depression cases from Medicare claims

We identified participants with depression from the Medicare claims by using ICD-9-CM depression-specific codes 296.20–24 and 296.30–34. We reviewed claims submitted 12 months before through 12 months after baseline. A person was considered depressed if he or she had at least one claim with at least one of the selected diagnoses, which is similar to previous approaches (11–13). Beneficiaries for whom Medicare claims were not available were excluded from the study, for a final sample of 1,551 persons.

To account for possible variation in coding practices, we also repeated all analyses after adding more codes associated with dysthymic disorder, adjustment disorder with depressed mood, and depressive disorder not elsewhere classified (ICD-9-CM codes 300.4, 309.0, and 311).

Provider assessments of depression identification and symptoms

We used the data from two depression scales, the Mini-International Neuropsychiatric Interview-Major Depressive Episode Module (MINI-MDE) (14) and the 15-item Geriatric Depression Scale (GDS) (15), which were administered as part of the baseline and 12-month questionnaires that were completed during July 1998 through December 2000. These instruments have been validated (16,17) and extensively used in studies of elderly patients with chronic illnesses. It is important to note that no interviewer judgment is required for the administration of the MINI-MDE or GDS questionnaires. All of the items are simple yes-no questions answered by the respondent; for the MINI-MDE, an informant answered the questions when the demonstration participant was cognitively impaired (N=263). Patients with missing MINI-MDE or GDS information were excluded from the analysis.

The MINI-MDE (14) consists of nine items (two main questions and seven additional questions) assessing a person's depressive sxymptoms in the past month and necessary to make a DSM-IV diagnosis of a major depressive episode during the past month (18). The patient-rated (that is, self-rated) version of the MINI-MDE has been shown to be valid and reliable compared with the patient version of the Structured Clinical Interview for DSM-III-R Mental Disorders (SCID-P) (19). Specifically, agreement between the MINI-MDE and the SCID-P was good for major depressive disorder, with a kappa of .55 and a sensitivity and specificity of .77 and .79, respectively (20). Time periods for several individual MINI-MDE questions were extended from the past two weeks to the past month to match the time frame of other study questionnaires (similar to the approach used in the SCID). For the base case analysis, depression was defined as the presence of a major depressive episode at baseline. For sensitivity analyses, we considered depression in two ways: as marked by the presence of major depressive disorder at either baseline or 12 months postbaseline, and at both baseline and 12 months.

The GDS (15) is a self-report measure of depressive symptoms during the prior week, with higher scores indicating greater severity of depression. It is a validated and reliable scale (20) and is recommended as a screening tool for depression among primary care patients (21). The GDS had a sensitivity of 92% and a specificity of 81% when a cutoff of 5 is used (21). In this study depression was defined in two ways: as a GDS score of ≥6 (denoted as mild, moderate, and severe depression) or of ≥11 (severe depression). We used the GDS as an independent research assessment of depression identification and symptoms because it includes few somatic symptoms in addition to the assessment of psychological and emotional issues. For many elderly patients depressive symptoms may be attributable to their chronic physical illnesses.

Duration of follow-up

For the base case analysis, the observation window included six months before administration of the baseline MINI-MDE and GDS and 6 months after their administration (a period of 12 months). The MINI-MDE and GDS identify current depressive symptoms during the past month, whereas the claims indicate the presence of depression over a period of time (in our case, 12 or 24 months). Moreover, there is usually a lag period between the onset of depressive symptoms and a visit to a health provider to address these symptoms (22,23). Therefore, we tested two additional 24-month observation windows: one from 12 months before the baseline assessment to 12 months after, and the other from six months before the baseline examination to six months after the 12-month postbaseline examination.

Because depression is a chronic recurrent illness and because it is common to expect a lag period between several weeks to several years from symptom onset to diagnosis, we tested depression as defined by the presence of a major depressive episode at either baseline or 12 months postbaseline or at both baseline and 12 months and as a GDS score of ≥6 (mild or moderate depression).

Analysis

We compared demographic characteristics between patients with depression identified by either of the two instruments and depression identified by Medicare claims by using the t test for continuous variables and the chi square test for categorical variables. We calculated sensitivity, specificity, and positive and negative predictive values for each comparison approach (the MINI-MDE and the GDS) for all Medicare claims files combined (with the exception of the hospice file). The analyses were also repeated separately for each of the six types of claims files (carrier, inpatient, outpatient, home health agency, skilled nursing facility, and durable medical equipment). To quantify the agreement between different measures, we calculated Cohen's kappa statistic for each of the two pairs of depression case identification strategies (Medicare claims and baseline MINI-MDE and GDS score of ≥6).

Study population and prevalence of depression

Overall, the claims files identified 321 beneficiaries with depression-related diagnostic codes. Nearly 60% of the beneficiaries identified as depressed by Medicare claims had depression diagnoses only in one type of claim (for example, inpatient or outpatient), whereas about 40% had depression diagnoses present in multiple claim types.

Based on the MINI-MDE, 18% (N=276) of participants had major depression at baseline (Table 1). The mean±SD GDS score at baseline was 5±3 for the subsample of 1,288 cognitively intact individuals (that is, 1,551–263=1,288). Of these, 36% (N=462) were considered to be at least mildly depressed (a GDS score ≥6). About 7% (N=91) had severe depressive symptoms (a GDS score of ≥11).

For the entire sample of 1,551, the mean±SD age was 77±11years, more than two-thirds were female, and less than 4% (N=57) were nonwhite. The overall health status of the study population was poor, with 58% reporting fair or poor health at baseline. The average study participant needed or received help with 2.30 activities of daily living and 3.49 instrumental activities of daily living.

The individuals identified by the MINI-MDE as having a depressive episode within the past month reported worse self-rated health, more chronic conditions, and more deficiencies in activities of daily living than those not identified as depressed by the MINI-MDE (Table 1). Beneficiaries with any depression (mild, moderate, or severe), as identified by a GDS score ≥6, reported being in worse self-rated health and had more comorbid conditions and deficiencies in activities of daily living and instrumental activities of daily living than those who were not identified as depressed on the basis of the GDS. They were also significantly less satisfied with their lives those who were not identified as depressed (Table 1).

Validity of claims in identifying depression cases

The sensitivity of the Medicare claims was below 20%, whether the MINI-MDE or the GDS was used for comparison, and the specificity of the claims ranged from 93% to 96% (Table 2). The positive predictive values for these algorithms ranged from 37% to 55%. The negative predictive value of Medicare claims was 84% compared with the MINI-MDE and 66% compared with the GDS (Table 2).

When the broader set of ICD-9-CM codes was used, the sensitivity for all scenarios above increased to 28%–46%, but the specificity decreased to 75%–83% compared with the more restricted set of codes (Table 3). For participants who had two consecutive positive MINI-MDE evaluations for major depressive disorder, the sensitivity and specificity with the core set of ICD-9-CM codes were 47% and 75%, respectively, with the negative predictive value of 95% and the positive predictive value of 12% (Table 3). Expanding the observation window to a total of 24 months did not have a substantive effect on the accuracy of the depression identification process.

Identification of individuals as depressed on the basis of claims was in agreement with identification by baseline MINI-MDE score in 74% of cases (κ=.15). Identification of depressed individuals on the basis of claims was in agreement with identification by baseline GDS scores of ≥6 in 64% of cases (κ=.12). The agreement between the two self-reported measures, the MINI-MDE and GDS, was 76% (κ=.41) (Table 4).

Claims data are used for a variety of purposes, and the desired accuracy depends largely on the use. The key is to understand when and how they can be used appropriately. Our findings demonstrate that Medicare administrative data are more accurate for ruling out depression among older adults than for identifying those with depression. The 10%–46% sensitivity we found is similar to the 17% reported for chronic kidney disease (24) and 35% for stroke (25) when medical records and Medicare data were used. At the same time, the sensitivity estimates we report are lower than the 50%–90% sensitivity reported for breast cancer (26,27) and Parkinson's disease (28). Our findings were robust regardless of the length of the follow-up period and were consistent across different codes used to define depression.

A number of studies have examined the accuracy of administrative or claims data for identifying depression (29–33). Overall, they have reported a substantial discrepancy between the presence of depression diagnoses in medical claims and those in patient charts. Moreover, claims and charts did not correlate very well in identifying the time of symptom onset (30,32). Requiring the presence of two or more depression-related visits to a health care provider decreased the number of identified cases and improved the specificity of the identification strategy compared with a less restricted approach (29). However, the small sample size of several validation studies raises concerns about generalizability, patient and provider self-selection, and statistical robustness (34).

Spettell and colleagues (31) identified several additional challenges of health plan data presented by depression in addition to those normally encountered in administrative data: lack of recognition of depression by the physician, physician's or patient's unwillingness to report depression because of stigma, and the role of medical comorbidity in confounding the diagnosis (competing risks). They also mentioned three commonly encountered challenges of health plan data: submission by providers of incomplete data; limited clinical detail in the ICD-9-CM, Current Procedural Terminology (CPT), and diagnosis-related group (DRG) systems; and inaccurate demographic data in administrative files.

In this study we required one health care visit or encounter with a depression code for any of six types of Medicare claims files, including outpatient visits and hospitalizations, which we thought was more realistic for the current pattern of care for patients with depression. Also, our study used a Medicare population exclusively, in which the vast majority of participants (88%) were not members of managed care health plans and were aged 65 or older (90%).

There are several possible explanations for the limited accuracy of the claims-based identification approach for depression in our study. First, despite recent efforts, depression remains underdiagnosed in primary care (5,35,36). Even with the best clinical expertise and even when there are no time or system constraints, it has been shown that unaided physician detection correctly recognizes only about 45% of depressed patients in primary care (37). This is comparable to the positive predictive values that we report. If a large portion of depressed patients are not diagnosed and are most likely not being treated, we cannot expect that using claims to identify depression would yield better results (that is, positive predictive value >45%). Because many patients do not receive care for their depression, or receive it late, the number of cases identified via claims is expected to be lower than number of persons with depressive symptoms. Second, the diagnosis of depression is often complicated by the presence of comorbid health problems (38,39), which may either overlap with depressive symptoms or simply take priority during provider visits because of the limited visit time.

Third, physicians may even underreport depression among poor elderly persons, assuming that these patients would not be able to afford treatment (40,41). Also, some older patients with clinically significant depression may underreport their symptoms (42). There is also evidence that some physicians deliberately substitute other diagnostic codes for major depression to ensure that their patients are not penalized by high insurance copayments for depression (34). Finally, it is possible that even when a physician recognizes and treats depression, a depression diagnosis may not appear on the medical claim because of stigma concerns.

We must acknowledge a number of limitations. First, the Medicare demonstration project was not intended to be representative of all Medicare patients but rather of seniors with substantial disability and health care use. Second, using the MINI-MDE, with a kappa of .55 compared with the SCID, as the gold standard comparator is a limitation. However, it is better than any other gold standard previously used. Third, although there was a lack of agreement between the baseline MINI-MDE and GDS, Adams (43) recently reported that the GDS at standard cutoff points had 70%–75% agreement with the MINI-MDE. However, some GDS items did not adequately represent depression among functionally impaired senior or those in the oldest-old group. Fourth, a beneficiary could have been diagnosed as having depression before the baseline or 12-month administration of the MINI-MDE or GDS and have been treated successfully; thus the independent research assessment at baseline or 12 months would have incorrectly resulted in a false positive and would have lowered the specificity and the positive predictive value. Fifth, prescriptions for outpatient antidepressants were not covered by Medicare at the time of the demonstration and therefore Medicare pharmacy claims data were not available for inclusion in a depression identification algorithm. Sixth, asking the study participants and their caregivers questions about depression at baseline may have resulted in participants' perceptions of less stigma related to depression at 12 and 22 months (44). Therefore, study participants may have been more willing to answer “yes” to questions about depression symptoms at 12 and 22 months. A similar effect could have occurred in the nurse and combination groups because the nurses also asked questions about depression. However, we do not know whether these effects occurred.

In addition, although using the presence of two claims for identifying a condition is common (29), our base case identification of depression required the presence of only one depression diagnosis. Using two claims may have led to different findings. Finally, the HIPAA (Health Insurance Portability and Accountability Act) 4010 standard for electronic claims transfer limits the number of diagnoses that can be reported to the Centers for Medicare and Medicaid services. This may partly explain the low sensitivity we found and may also help mitigate the concern that requiring only a single claim with a depression diagnosis limits specificity.

These findings have important implications for health care administrators and researchers. Because it appears that the timing and frequency of depression-specific billable diagnoses do not correlate well with the presence of depressive symptoms, health care administrators and decision makers should be cautious about using administrative claims for punitive or financial decisions, such as for developing quality indicators or contracting with individual practitioners. Researchers should recognize the limitations of claims data for services and research related to identification and treatment of depression. In many ways, our ability to “chart the course of mental disorders over the lifespan in order to understand ideal times and methods for intervention” (45) will depend on the accuracy of identifying depression cases and the timing of symptom onset.

The Medicare Primary and Consumer-Directed Care Demonstration was sponsored by contract CMS 95-C-90467 from the Centers for Medicare and Medicaid Services. This study was partially funded by grant K01 AG20980 from the National Institute on Aging and grant K01 MH64718 from the National Institute of Mental Health.

The authors report no competing interests.