Congress passed the nursing home reform provisions of the 1987 Omnibus Budget Reconciliation Act (OBRA-87) in response to a study by the Institute of Medicine that assessed the quality of care in nursing homes (1). One requirement stipulates that nursing home applicants receive preadmission screening for mental illness and mental retardation to identify the proper residential settings to accommodate their needs (2,3,4). The Centers for Medicare and Medicaid Services (CMS—at the time, the Health Care Financing Administration) developed what is now called Preadmission Screening and Resident Review (PASRR) as the interpretive guidelines regulating how states implement the preadmission requirements (5). The provisions state that individuals with mental illness should not be admitted to nursing facilities if they do not have other health care needs that require nursing services (4). For residents with both a mental illness and a need for nursing assistance, the regulations require that facilities provide active mental health treatment.

Since passage of OBRA-87, a limited number of studies have used different data sets to document the size of the mentally ill population of nursing facility residents and examine the care that these patients receive (6,7,8). Mechanic and McAlpine (7) analyzed the 1985 and 1995 National Nursing Home Survey (NNHS) and the 1987 and 1996 Medical Expenditure Survey (MES) to assess the initial impact of OBRA-87. The NNHS codes diagnoses on the basis of ICD-9-CM, whereas the MES includes indicators for a number of "active" mental disorders, including schizophrenia, bipolar disorder, depression, dementia or organic brain disorder, and anxiety. Mechanic and McAlpine excluded from their analyses residents with mental retardation but included all other residents with diagnoses corresponding to ICD-9-CM codes in the range 290–317. They found that 58% of residents in the 1995 NNHS versus 68% in the MES had a diagnosed mental disorder. In both cases, dementia and related disorders accounted for a large share of the recorded diagnoses (41% of residents in the NNHS and 50% in the MES).

A 2001 Office of the Inspector General report used 1998 Medicaid Statistical Information System records from 34 states and the 1999 Minimum Data Set (MDS) from 39 states to document the size of the nursing home population between ages 22 and 64 who had a mental illness (6). Although the report also used ICD-9-CM codes to identify residents who had mental disorders, the analyses were limited to a small range of diagnoses (codes 293–301, 311, and 312) and provided no prevalence estimates.

Shea and colleagues (8) used the Medicare Current Beneficiary Survey for 1992 and defined mental illness with the use of indicators for whether respondents had ever been told they had a mental illness or were receiving Medicare benefits because of disability caused by mental illness. Excluding residents with Alzheimer's disease, they found that 20% of the 732 nursing facility residents in their sample reported having a mental illness.

These studies demonstrated that data sets vary in their methods for identifying mental illness diagnoses, which can lead to vastly different estimates of the size of the nursing home population with mental illness. With this study we sought to build on those findings by identifying differences in estimates of the number of residents with a mental illness from three nationally representative data sets (the NNHS, the MDS, and CMS Medicaid Analytic eXtract [MAX] files), determining which of these data sets provides the best estimate of the size of the national population of nursing home residents with mental illness, and identifying the types of policy and monitoring questions that can best be answered with each data set.

Because the statutory definition of mental illness for PASRR specifically excludes persons with dementia and related conditions (if their primary diagnosis is not mental illness), and because individuals with mental retardation are treated separately under the screening and review process, our analyses excluded these diagnoses. We also excluded substance use disorders because our intention was to focus on the population with mental disorders. Preliminary analyses found that alcohol- and drug-induced mental disorders account for less than 2% of primary mental illness diagnoses among nursing home residents. Thus mental illness was defined in our study as conditions corresponding to ICD-9-CM codes ranging from 295 to 302 and from 306 to 314.

The NNHS is a nationally representative survey of certified and state-licensed nursing facilities conducted periodically by the National Center for Health Statistics. The 1999 NNHS used a two-stage probability design that first sampled facilities with at least three beds and then identified six current residents and six residents who had been discharged within a designated month (9). Nursing facility staff provided surveyors with written descriptions of patients' illnesses at admission and at the time of survey administration as recorded in patients' medical records or the facility's MDS file. These descriptions were later assigned ICD-9-CM codes by an independent vendor.

The MAX files represent the more than 65% of nursing facility residents whose care is paid for by Medicaid. Analyses for this study included all types of facilities except inpatient psychiatric services for individuals aged 21 and under and intermediate-care facility services for persons with mental retardation. Each claim in the MAX long-term-care files may include up to five diagnoses, reported as ICD-9-CM codes. However, analysis of the 1999 files found that 42.5% had no valid diagnosis code; another 30.7% had only one, and the remaining 26.8% had two or more valid codes (71.5% of these records had only two codes; the remaining had three or more). Among cases with a valid code, the diagnoses listed often are the original admitting diagnoses rather than the most current. In order to improve the identification of relevant diagnoses and to ensure greater comparability between the findings from the MAX, MDS, and NNHS, our analysis included mental illness diagnoses from the inpatient and outpatient Medicaid claims in the MAX files. Inpatient claims were included if they occurred after the date of the long-term-care claim, and outpatient claims were included if they occurred in the same month as the long-term-care claim. Studies that examined the reliability of Medicaid claims data for use in research on psychiatric disorders have found the primary and secondary diagnosis variables to be reliable (10,11).

All long-term-care facilities that are certified to participate in Medicare or Medicaid are required to complete the MDS for all residents at the time of their admission and at least quarterly thereafter. The MDS includes two ways to record diagnoses. The staff member filling out the form can either mark checkboxes for specific physician-documented diagnoses (including schizophrenia, depression, bipolar disorder, and anxiety disorder) or record up to five conditions by their ICD-9-CM codes in a separate space on the form. It is not possible to determine a patient's primary diagnosis with the MDS because the staff member recording diagnoses is not required to list them in any particular order. Reliability estimates indicate that the psychiatric and mood diagnostic items in the MDS have an average kappa (a statistic that compares the agreement between raters, where 1.0 is perfect agreement) of .83. Kappa ranged from .79 for anxiety disorder to .85 for bipolar disorder (12,13,14).

As detailed in Table 1, each of the data sets evaluated has strengths and weaknesses for policy analysis. Given the methodological variations in the data sets, several modifications were necessary to create more comparable analytic files. First, we used data from 1999 because this was the most recent year for which research-ready data were available for all three data sources at the time of writing. Second, we restricted the MDS to residents who had either an admission or an annual assessment in the latter half of the year, because that is the period covered by the NNHS. We restricted the MAX to residents with a claim during the same period. Finally, we restricted the NNHS and MDS files to residents for whom Medicaid was the primary or a secondary payer (to match the payer status of residents represented in the MAX). Limiting the data to residents with Medicaid-covered expenses improved the comparability of the data files for analysis but also limited the generalizability of our findings to the national population of nursing home residents. Residents excluded because of the Medicaid restriction (about 10% in the MDS and 25% in the NNHS) were less likely to have a diagnosis of mental illness than those included in the study sample. Those in the excluded group with a diagnosis of mental illness were less likely to have a diagnosis of schizophrenia and were more likely to have depression. Demographically, residents excluded from the analyses were older and more likely to be white than residents in the study subpopulation.

The analyses distinguish between primary diagnoses of mental illness and any diagnosed mental illness. In the NNHS and the MAX, the primary diagnosis is the first listed diagnosis code; residents were counted as having any mental illness diagnosis if at least one of the remaining diagnosis variables contained an ICD-9-CM code within the specified range (specifically, 295–302 and 306–314). Because it is impossible to identify a resident's primary diagnosis with the MDS, we focused the analyses on residents with any mental illness diagnosis (defined as those for whom schizophrenia, depression, anxiety, or bipolar disorder was indicated on the checkboxes or for whom any of the "other diagnosis" variables included the specified ICD-9-CM codes). Therefore, we were able to compare the NNHS and MAX by primary diagnosis and to compare all three data sets for any mental illness diagnosis.

Limiting the MDS and MAX to the population of residents in the facility in the last half of the year reduced the number of residents represented in the MDS to 51.8% of the original total and that of the MAX to just under 83.0% of the original (Table 2). The disproportionate drop in the number of residents from the MDS compared with the MAX likely reflects sampling bias; the MDS is based on only annual and admission assessments (either of which should occur once in the year), whereas MAX includes all residents with a claim (which could occur multiple times across the entire year). Restricting each of the data sets by Medicaid payer reduced the NNHS to 63.1% of the original number of residents and left the MDS with less than a quarter of the original number of residents. However, the distribution of residents by demographic characteristics was quite similar across data sets even after introducing these restrictions (Table 3). Although the differences were statistically significant with the large file sizes, there were few meaningful differences by demographic subgroup; a majority of residents in all three data sets were white women of age 75 or older.

Table 4 highlights significant variations in the prevalence of mental illness for all residents and by age group, 64 or younger versus 65 or older (because of space limitations, standard errors are not reported in Tables 4 and 5 but are available from the authors on request). Within the NNHS, 7.0% of the entire nursing home population had a primary diagnosis of mental illness, compared with 4.3% in the MAX. By comparison, although 33.1% of residents in the NNHS had any mental illness diagnosis, the percentage was much lower in the MAX (at only 18.3%) but larger in the MDS (46.0%). Across all three data sets, a majority of residents with any diagnosed mental disorder had only one diagnosis of mental illness (84.0% in the NNHS, 72.0% in the MDS, and 94.0% in the MAX; data not presented).

In both the NNHS and the MAX, the percentage of younger residents with a primary diagnosis of mental illness was nearly three times that among older residents (for example, in the NNHS, the proportions were 14.8% for residents aged 64 or younger and 5.1% for residents aged 65 or older). The prevalence of any mental illness diagnosis was more comparable across age groups but was still twice as large for younger versus older residents in the MAX (34.2% versus 15.9%, respectively).

Table 5 shows that the NNHS, MAX, and MDS data sets also varied significantly by recorded diagnosis. The distribution of ICD-9-CM codes for primary diagnoses was similar across the NNHS and MAX. There were similar percentages of the primary mental illness diagnoses identified in these data sets by code 295, schizophrenic disorders; 296, affective psychoses; 310, mental disorders resulting from brain damage; and 311, depressive disorders. However, a significantly higher percentage of residents in the MAX had a diagnosis of "other nonorganic psychoses" (code 298) (20.9% versus 7.7% in the NNHS), and a larger percentage of residents in the NNHS were diagnosed as having neurotic disorders (code 300) (8.7% versus 4.7% in the MAX).

The distribution of mental illness diagnoses was far less comparable in the case of any diagnosed mental illness. Most significantly, the four mental illness diagnoses captured with checkboxes in the MDS (that is, schizophrenia, bipolar disorder, anxiety disorder, and depression) accounted for nearly all mental illness diagnoses recorded in that data set. Although the prevalence of any diagnoses of schizophrenia disorders was similar between the NNHS and MDS (13.8% and 13.0%), it was higher in the MAX (18.1%). Similarly, although the NNHS and MDS indicated higher percentages of residents with diagnoses of a depressive disorder (34.3% and 61.4%, respectively) than the MAX indicated (17.7%), the percentage in the MDS was nearly twice that of the NNHS.

Because Medicare is the primary payer for residents who are also eligible for Medicaid and have stays less than 90 days and because Medicaid pays lower reimbursement rates, Medicaid may not be billed for mental health services for these short stays. As a result, diagnoses for these residents may be missed in the MAX claims. As a sensitivity test, we ran separate analyses with the NNHS and MDS for short-stay residents (those with stays of 90 days or less). Although the prevalence rates of mental illness were slightly lower (46.0% versus 50.0% for any diagnosis in the MDS and 28.0% versus 33.0% in the NNHS), the distributions of diagnoses were similar.

Our review of the data suggests that because the NNHS included the most complete diagnostic information and was representative of the entire population of nursing home residents, it is the preferred data set from which to draw national prevalence estimates of the number of residents with a primary diagnosis of mental illness. In an analysis of the 2004 NNHS file (the year for which the most recent data were available), which included all residents and all payer types, we found that 101,667 (6.8%) of the 1.49 million nursing home residents represented in the data had a primary diagnosis of mental illness. The estimate was significantly higher among residents aged 65 or younger (N=22,564, 12.9%) and lower for those aged 65 or older (N=79,038, 6.0%). These estimates are similar to those for 1999 shown in Table 4.

This study highlighted how discrepancies in the timing and methods of data collection for the NNHS, MDS, and MAX affect estimates of the number of nursing home residents with diagnosed mental illness as well as variations in the frequencies of specific diagnosis codes. The percentage of nursing home residents identified as having a primary mental illness was much lower in the MAX than in the NNHS, which may reflect the fact that 43.0% of the MAX claims were missing diagnosis codes in the long-term-care claims file. Because listing diagnoses in MAX does not directly affect nursing home payment, incentives to fully report diagnoses are limited.

Estimates of the percentage of residents with any diagnosed mental illness varied widely across data sets, at 33.1% in the NNHS compared with 46.0% in the MDS and 18.3% in the MAX. These estimates cannot be directly compared with estimates from the literature because of differences in the methods used to record mental illness diagnoses, the subpopulations of nursing home residents included in the data sets chosen for analysis, and the specific diagnoses included for analysis. Our prevalence estimate from the NNHS was lower than that reported by Mechanic and McAlpine (7) because they included a wider range of diagnoses in their analyses. Not surprisingly, replicating their methodology with the 1999 NNHS yielded a similar prevalence estimate (53.0% in 1999 versus 58.0% in 1995).

The low estimate of mental illness from the MAX may reflect the fact that only 26.8% of nursing home residents had two or more diagnoses noted in their long-term-care claims (and the majority of these, 71.5%, had no more than two). The higher percentage seen in the MDS may reflect the fact that some nursing facility staff members use diagnoses as an aid in care planning, which may lead them to check all potentially relevant diagnoses. The results for the MDS also showed a bias toward the four mental illness diagnoses represented in the MDS checkboxes. A 2007 report by the Medicare Payment Advisory Commission included a recommendation to CMS to require skilled nursing facilities to use full five-digit ICD-9-CM codes in the MDS and to revise data collection procedures to be able to distinguish between primary and secondary diagnoses (15).

Assessing the reliability of nationally representative data sources for identifying nursing facility residents who have mental disorders can help researchers and policy makers evaluate the effectiveness of the PASRR process and the appropriateness of care provided in nursing facilities for those with mental illness, including their use of psychotropic drugs. Although prior studies have produced comparative estimates of the size of the nursing home resident population with mental illnesses, none to date has compared the NNHS, MAX, and MDS data sets. This study highlighted variations in the percentages of residents with diagnosed mental disorders in nursing homes and raises questions about the reliability of the diagnoses that may be used in the screening and review process, as well as the diagnoses being assigned in the MDS and MAX. Further analysis is needed to determine how much of the variation in the MDS and MAX diagnoses may be due to differences in the population that is covered and how much to differences in facility-level coding and reporting practices. Evaluating the reliability of the NNHS has additional benefits, because the federal government relies heavily on the NNHS to estimate spending for mental health services (16,17).

This article was prepared for the Substance Abuse and Mental Health Services Administration of the U.S. Department of Health and Human Services under contract 280-03-1501. The authors thank Miki Satake, M.A., and Deo Bencio, B.S., who provided programming support for data analyses. Judith L. Teich, M.S.W., served as the government project officer, and Jeffrey A. Buck, Ph.D., served as adviser.

The authors report no competing interests.