Convergence of Clinical Staff Ratings and Research Ratings to Assess Patients With Schizophrenia in Nursing Homes

With the aging of the population and the downsizing of psychiatric institutions in the United States, nursing homes and equivalent settings have become an increasingly common residence for patients with schizophrenia in the later stages of life. Although the prevalence of schizophrenia might be underestimated in these settings because the diagnosis might not have been transferred to a nursing home chart ( 1 ), as long ago as 1986 an estimated 200,000 individuals with chronic schizophrenia were nursing home residents ( 2 ). We know from one study that over 1,000 patients with schizophrenia have been discharged to such residences from one psychiatric center in New York alone ( 3 ). However, the amount of research on nursing home patients with schizophrenia has not matched the pace of these discharges to nursing homes. As a result, not enough is known about the cognitive, clinical, and functional status of patients after discharge, the capability of nursing home staff to identify and treat symptoms of schizophrenia, or the quality of care in these settings.

Schizophrenia is a disease that requires a high level of care across the lifespan, including symptom and behavioral management, medication monitoring, social support, severe cognitive impairments, and, particularly in later stages of life, support with daily living skills and comorbid medical burden. Nursing homes seem best suited to treat late-life schizophrenia in regard to daily living and comorbid medical burden. However, it is still unknown whether nursing home staff have the expertise or the instrumentation to accurately assess and treat the other symptoms common to late-life schizophrenia. The scant existing data suggest that both inadequate training and lack of instrumentation may be problematic in nursing homes. For example, Timko and colleagues ( 4 ) reported that staff in public-sector nursing homes were treating patients with greater disability and had less training than staff in Veterans Affairs (VA) nursing homes. Sherrell and colleagues ( 5 ) examined the charts of 570 nursing home residents who had a history of chronic psychiatric illness. They found that no systematic method for assessment of cognitive or functional status was in place, which led to the conclusion from chart narratives that most patients had cognitive impairments in the "none to mild" range. The accuracy of this assessment is quite unlikely, given the results of direct assessment of the cognitive and functional status of patients with schizophrenia in late life who are residents in these homes ( 6 ). Identification of the presence and severity of symptoms by direct care staff may be even more critical in nursing home settings than in other settings, such as acute care and long-stay psychiatric institutions, where the voluminous caseloads for physicians preclude detailed individual assessments and evaluations often rely on chart notes and nursing staff input.

Failure to identify symptoms may result in inadequate treatment. Linn and colleagues ( 7 ) randomly assigned 403 patients with schizophrenia at discharge from a long-stay psychiatric center to another ward within the VA hospital, a community nursing home, or a VA nursing care unit. Patients discharged to nursing homes experienced functional deterioration and symptomatic exacerbation, which were associated with the level of training and experience of the staff. Perhaps more striking, the mortality rate for patients with schizophrenia living in nursing homes is lower when mental health specialists are available to provide treatment ( 8 ).

Assessment and treatment guidelines for nursing homes are regulated by the congressional Omnibus Budget Reconciliation Act (OBRA) of 1987. This act established medication guidelines and mandates administration of the Minimum Data Set (MDS) to assess the medical, functional, and cognitive status of residents. The MDS is an instrument rated by direct care staff on a quarterly basis. There has been considerable controversy about these regulations. Eichmann and colleagues ( 9 ) argued that forcing nursing homes to provide more mental health care would lead to reductions in levels of care. Llorente and colleagues ( 10 ) reported that most antipsychotic medication dosages are within OBRA guidelines, although patients with schizophrenia were much more likely to receive dosages that exceeded the standards. Other work suggests that nursing home residents with chronic schizophrenia are five times more likely than patients receiving care at a state or VA long-term psychiatric center to receive no antipsychotic medications, even with a similar severity of psychosis ( 11 ). In addition, nursing home patients were twice as likely as patients in the psychiatric center to receive first-generation antipsychotic medications.

Although the appropriateness of the medication guidelines has been called into question, less is known about the appropriateness of the MDS for use in nursing home assessment of persons with chronic mental illness. Yet, it is the raters of the MDS who have the most contact with the residents and are able to provide prescribing clinicians with information that is critical for accurate diagnosis and the effectiveness of pharmacotherapy. The MDS has been well validated in the general nursing home population as a measure of cognition ( 12 , 13 ), mood ( 14 , 15 ), behavior problems ( 14 ), and communication ( 14 ). However, it has not been validated for patients with schizophrenia. This is noteworthy because patients with schizophrenia are likely to differ from the general nursing home population in symptomatology and cognitive profiles. Although nurses tend to have a good understanding of dementia ( 16 ), their knowledge of specific domains of cognitive function has not been reported. Having detailed knowledge is particularly relevant in schizophrenia, because it is associated with a cognitive profile that is different from Alzheimer's disease or vascular dementia ( 17 ). Schizophrenia symptoms also differ from those typically displayed by nursing home residents. Recognition of psychiatric symptoms not specific to schizophrenia may be more commonly encountered in nursing homes; still, nursing home staff often fail to recognize symptoms such as depression ( 18 ).

Therefore, the purpose of this report was to examine the concurrent validity of nursing home staff ratings on the MDS with objective clinical and cognitive measures. We also examined the criterion validity of the MDS as it relates to functional status. In so doing, we used the results of a comprehensive cognitive, clinical, and functional assessment of nursing home residents with schizophrenia as the reference point against which to compare the MDS results.

Methods

Participants

Participants in this study were patients with schizophrenia, all over the age of 60, who had been previously recruited for longitudinal studies of cognitive and functional status potentially culminating in postmortem neuropathological examination. In this project, the institutional review board granted a waiver of written informed consent, and patients' verbal assent was obtained before study procedures. All participants were residing in a nursing home in New York City at the time of the current assessments, although all had been contacted because they were at one time a long-stay patient in a New York State Office of Mental Health psychiatric facility. All participants are part of a longitudinal study; all data for these analyses came from consecutive assessments in May to November 2003. Each participant met DSM-III-R criteria for schizophrenia on the basis of a structured diagnostic interview and a review of the patient's lifetime medical history to confirm the diagnosis of schizophrenia. Exclusion criteria were substance dependence, evidence of a concurrent neurological condition, or unstable medical conditions.

MDS cognition

Morris and colleagues ( 12 ) developed the Cognitive Performance Scale (CPS) from items rated on the MDS in order to estimate the level of cognitive impairment. Building on the CPS, Hartmaier and colleagues ( 13 ) used a summative approach to rating cognitive impairment, resulting in the MDS-COG, which produces a continuous scale with scores ranging from 0 to 10 and shows better agreement, sensitivity, specificity, and predictive value than the CPS. Several studies have confirmed that the MDS-COG provides a valid method of estimating the level of cognitive impairment in the general nursing home population ( 14 , 19 ). We used the refined MDS-COG to examine concordance with objective neuropsychological performance, as described below.

Performance-based assessment of cognition

Cognition was objectively assessed with the neuropsychological battery developed by the Consortium to Establish a Registry for Alzheimer's Disease (CERAD) ( 20 ). This battery consists of the following objective measures.

Word list learning and delayed recall. A ten-item list of words was presented to the participants over three separate learning trials. After each trial, free recall of the list was required. After a delay, filled by the Praxis drawings test described below, a delayed recall of the word list was required. The dependent variables were the total number of words correctly recalled over the three learning trials and the number of words recalled at the delayed recall.

Praxis drawings. Four stimuli (circle, diamond, overlapping rectangles, and three-dimensional cube) were presented to each participant, who was instructed to copy the shapes exactly. Reproductions were scored according to predetermined criteria, and the dependent measure was the total score for the four drawings.

Modified Boston Naming Test. Participants were presented with 15 line drawings and asked to name the object depicted. The drawings consisted of five objects with high frequency of occurrence in spoken English, five of moderate frequency, and five of low frequency. The dependent variable was the total number of correct responses.

Verbal fluency. Participants completed phonological and semantic fluency tests. In the phonological fluency test, patients were asked to name as many items as possible that begin with a certain letter (F, A, and S) over three one-minute trials. In the semantic fluency test, participants were instructed to name as many different animals as possible in one minute. The dependent variables were the number of unique words and animals named for the phonological and semantic fluency tests, respectively.

The cognitive portion of the Alzheimer's Disease Assessment Scale, Late Version (ADAS-L Cog) yields error scores ranging from 0, indicating no impairment, to 36, for no correct responses. The scale includes six subtests: immediate recall of three words presented over three trials, for a score of 0 to 9; remote memory, with 2 points for each error in stating the year born, naming the first president, naming two other presidents, and naming the letter that comes after "ABC," for a score of 0 to 8; orientation, which requires the participant to say his or her full name, age, the current month, the current year, the country, and the name of the place of assessment for 1 point each, for a possible score ranging from 0 to 6; naming, which requires the patient to identify toy versions of six three-dimensional objects for 1 point each, for a maximum of 6 points; commands, which requires participants to respond to commands to make a fist, point to the floor, put their hands together, and then pull them apart for 1 point each, for a score from 0 to 3; and expressive language, which requires participants to make a statement that begins with "I wish…" and is scored from 0 to 3.

We have previously demonstrated validity of the ADAS-L Cog subscale in this population even among patients with Mini-Mental State Examination (MMSE) scores below 10 ( 17 ). However, the validity of the CERAD subtests is questionable in these profoundly impaired patients because of floor effects, as patients tend to make very few correct responses ( 17 ). Therefore, in this study, to examine concordance between MDS ratings of cognition and objective neuropsychological performance, we used the results from patients who had MMSE scores greater than 10.

Clinical symptoms

We examined concordance of researcher ratings and MDS ratings for three symptom domains: depression, thought disorder, and hallucinations. Research ratings were based on direct observation of and interview with the patient and review of chart notes to complete the Positive and Negative Syndrome Scale (PANSS) ( 21 ). This scale comprises seven items measuring positive symptoms, seven items measuring negative symptoms, and 16 items measuring general aspects of psychopathology. Interrater reliability of these ratings for our patients was found to be acceptably high ( 22 ), with intraclass correlations for a sample of 30 ranging from a low of .86 to a high of 1.00 (p<.001 for all comparisons). We also examined the affect and depression rating from the ADAS-L for convergence with the MDS.

The MDS contains 16 items for rating indicators of depression or anxiety. We used a depression rating scale from a validation study by Burrows and colleagues ( 15 ). This scale consists of seven of the MDS mood disturbance items. Because we previously found depression to be present for only a minority of our chronically ill patients with schizophrenia ( 23 ), we expected a positively skewed distribution in this sample, which could lead to high correlations resulting from the absence of this symptom. Therefore, we performed an additional analysis to examine concordance in ratings of severity of depression when it was evident by selecting only patients who showed at least mild depression, as rated on the PANSS.

The MDS items that assess thought disorder and hallucinatory behavior were rated as present or absent. These items were examined for concordance with the researcher-rated PANSS conceptual disorganization and hallucinatory behavior items. Because of the dichotomous nature of these ratings on the MDS, the PANSS ratings of thought disorder and hallucinatory behavior were dichotomized to "present," with a rating of 3, mild, or higher, or "absent," with a rating of "questionable" or "absent." Because the data were dichotomized, concordance was assessed with McNemar's test and the likelihood ratio.

Functional status

The Social Adaptive Functioning Evaluation (SAFE) ( 24 ) scale was used as a measure of social-interpersonal, instrumental self-care, and impulse-control skills. This scale was designed to be rated by an observer after observation of and interaction with the participant and after a caregiver interview. The SAFE scale includes 17 items that are rated from 0 to 4, indicating increasing impairment. This scale has suitable reliability, with interrater reliabilities of the items all exceeding .88 for a sample of 60. We used the total score for this study as a measure of instrumental functional status.

The ADAS activities of daily living subscale (ADAS-ADL) includes items assessing toileting, feeding, dressing, and physical ambulation rated on the basis of the best source of information. The items are scored on a graded scale from 0 to 4, where 0 represents little or no impairment and 4 represents the most severe impairment. We used the total score as a measure of basic daily living skills. These functional skills are considered to be at a lower level than instrumental functional skills. Both of these functional measures rely heavily on information from the nursing home staff, either through direct interview or from reading chart notes. Therefore, we did not examine concordance between the MDS and these measures because of shared variance.

Results

Demographic data

The 77 participants in this study were between the ages of 61 and 88 (mean±SD=73.8±7.2 years), with 10.3±2.3 years of education. Fifty-four patients were women, and 23 were men. At the time of the assessment, these patients had resided in the current nursing home for three to 138 months (mean=45.2±37). Most patients were taking second-generation antipsychotic medications (81 percent), fewer were taking first-generation antipsychotic medications (4 percent), and 15 percent were not taking antipsychotic medications at the time of testing.

Convergent validity

Means and standard deviations for all cognitive, symptom, and functional variables are presented in Table 1 . Correlational results are presented in Table 2 .

Enlarge table

Enlarge table

Cognition. Among patients with MMSE total scores greater than 10, the MDS-COG was significantly correlated with the ADAS-L Cog and the MMSE, although these two objective measures were more highly correlated with each other. However, objective neuropsychological tests, although all highly correlated with the MMSE and ADAS-L Cog, were generally not significantly correlated with the MDS-COG (See Table 2 ). Tests of the difference between the correlation coefficients revealed significantly stronger correlations for the MMSE than for the MDS-COG with the other global measure (ADAS-L Cog) and the specific neuropsychological measures ( Table 2 ).

Symptoms. The MDS depression ratings were not significantly correlated with the PANSS depression item (Pearson's r=.176, p=.08) or the ADAS-L affect and depression item (Pearson's r=.02, p=.84). Selecting only the 19 patients whose PANSS ratings indicated at least mild depression resulted in inferior concordance between MDS and clinician ratings with the PANSS (Pearson's r=-.02, p=.47).

Hallucinatory behavior was profoundly underestimated with the MDS, resulting in a lack of concordance between MDS and PANSS ratings on this item (McNemar's χ ² =.35, df=1, p=.006; likelihood ratio=.599, p=.43). Thought disorder was similarly underestimated with the MDS (McNemar's χ ² =.59, df=1, p=.001; likelihood ratio=.94, p=.33).

Criterion validity

Correlations between the MDS items and functional status were examined to determine the criterion validity of the instrument in comparison with performance-based measures. In the sample of patients with MMSE scores greater than 10, correlations were significant between the MMSE and the ADAS-ADL (r=-.40, p<.003) and SAFE total score (r=-.25, p<.05) but not between the MDS-COG and those items (r=.22 and .21, respectively, p values >.05). In the more cognitively impaired sample, which was examined because these analyses did not involve instruments with floor effects, correlations with the ADAS-ADL were significant with both the MMSE (r=.78, p<.001) and the MDS-COG (r=.54, p=.004).

Discussion

The MDS is a mandatory instrument used in nursing homes and has shown adequate psychometric characteristics in general nursing home populations. However, no published studies have examined its validity for patients with schizophrenia. The results of our study suggest that the MDS is not a valid assessment tool for understanding the characteristic signs and symptoms of schizophrenia as the disorder presents itself in nursing home care.

The MDS has several psychometric flaws as a method for assessing patients with schizophrenia. It clearly lacks face and content validity; that is, several symptoms and deficits common to schizophrenia are either entirely absent—delusions, for example—or not comprehensively assessed—cognition, for example. Although the MDS correlates moderately with other cognitive measures, unlike those other measures, it is only modestly related to specific cognitive functions, suggesting poor concurrent validity. The assessment of cognitive status in schizophrenia with the MDS appears inadequate in breadth and depth and provides a score that has little functional relevance.

For patients with schizophrenia, the MDS has poor criterion validity in that the MDS-COG is associated with functional status only when cognitive deficits are profound and when functional status measures gross impairments. Perhaps the staff in nursing homes, who are familiar with dementia, are more adept at rating the patients with this level of impairment, the MDS-COG has ceiling effects with patients who have schizophrenia, or the truncated range of possible scores affects the likelihood of finding higher and statistically significant correlations. Although many geriatric patients with schizophrenia who have poor outcomes have profound cognitive impairments, these impairments are not uniform, and the MDS appears inadequate in measuring the diversity of cognitive deficits and symptoms found among patients with schizophrenia.

Lack of concordance between clinically oriented and performance-based estimates of cognitive status is not unique to the nursing home or assessments with the MDS. Studies of the relationships between clinical ratings of cognitive deficits and the results of neuropsychological assessments in chronic ( 25 ) and first-episode ( 26 ) patients with schizophrenia have shown that only 10 percent of the variance in performance-based test scores was accounted for by clinical ratings. Thus the lack of concordance may be a result of poor instrumentation rather than flawed ratings by nursing home staff. We know from this study and others that staff validly contribute to the assessment of functional status. With appropriate instrumentation, they might be good informants of cognition and symptomatology among schizophrenia patients.

We must note that this recruited sample was small and from a single site. Future work across sites and with other instruments might help clarify which instruments would be best suited for assessment of patients with schizophrenia in nursing homes. The ADAS-L has suitable psychometric properties for assessment of cognition and functional status in cognitively impaired patients with schizophrenia in the nursing home ( 17 ), and several other instruments aim to assess cognition ( 27 ) or symptoms ( 21 ) specifically in schizophrenia.

Conclusions

Although measurement of cognition requires performance-based assessment, the assessment of clinical symptoms could likely be done by nursing home staff. Accuracy is important, as the nursing staff are typically the primary informants for the prescribing clinicians. The MDS is flawed because it provides only a dichotomized description of symptoms common among nursing home residents. Little is known about nursing home staff's ability to accurately recognize and report the severity of symptoms of schizophrenia. The results of this study suggest that symptoms such as depression, thought disorder, and hallucinations go almost completely unrated by nursing care staff when they are asked to generate a present-absent rating. Further exploration of staff recognition and ratings of symptoms is an important area for future research as these facilities become an increasingly common residence for patients with schizophrenia. With so many patients with schizophrenia being transferred to full-care nursing residences late in life, the search for valid instrumentation for assessing cognitive deficits, functional impairments, and symptoms common to schizophrenia is an important topic for research and should be an objective for policy makers and individual nursing homes. The MDS should be treated as named, meaning it assesses symptoms at a minimum level, and illness-specific symptoms should be assessed and managed with appropriate instrumentation.

Acknowledgments

This research was funded by the Mount Sinai Mental Health Clinical Research Center on Late-Life Schizophrenia; the Silvio Conte Neuroscience Center grant MH-36692 from the National Institute of Mental Health (NIMH); National Institutes of Health grant M01-RR-00071; NIMH grant 63116 (Dr. Harvey); the VA Veterans Integrated Service Network 3 Mental Illness Research, Education, and Clinical Center; and a young investigator award from the National Alliance for Research on Schizophrenia and Depression to Dr. Bowie.