Reliable Self-Report of Health Service Use by Individuals With Serious Mental Illness

Medical morbidity and mortality rates are higher among persons who have serious mental illness than in the general population (1). In addition, recent research suggests that psychiatric populations have different patterns of health service use than nonpsychiatric populations (2,3). Because data on service use are commonly self-reported (4,5), research is needed to evaluate the reliability with which individuals who have serious mental illness report their use of health care services.

Previous research has suggested that reliable self-report among psychiatric populations, particularly self-assessment of symptoms, may be compromised by the cognitive deficits and lack of insight associated with serious mental illness (6). Whether this observation extends to the self-report of behavioral events such as use of services has been tested less frequently. We reviewed the literature and found only two studies that examined this question with a psychiatric population (7,8), and only one of them assessed test-retest reliability (7). Reporting test-retest correlation coefficients ranging from .87 to .89, Calsyn and colleagues (7) concluded that homeless individuals with mental illness can provide reliable information about their use of mental health services. However, no study has provided detailed information about the extent to which individuals who have serious mental illness reliably report their use of medical services with instruments developed for nonpsychiatric populations.

Using a modified version of a well-known national health services survey, we conducted this study to determine whether individuals with schizophrenia could consistently report their use of medical services. On the basis of the findings of Calsyn and colleagues (7), we hypothesized that this patient population would report service use consistently.

Methods

The participants in this study were part of a larger study of quality of care in schizophrenia. A modified version of the household component of the Medical Expenditure Panel Survey (MEPS), a scientifically valid and widely used survey of medical service use and expenditures developed by the Agency for Healthcare Research and Quality (AHRQ) (4), was administered to 171 outpatients who had been diagnosed as having schizophrenia. Our review of the literature and MEPS-related documentation from the AHRQ found that no test-retest results were available for the national survey.

Twenty-nine participants completed a one-week test-retest protocol using the modified MEPS. The instrument—read aloud by the interviewer—included items about medical services received in outpatient settings (for example, "During the past six months, did you go to a doctor's office, outpatient clinic, health center, or some other outpatient setting for any kind of medical care?"), hospital emergency departments (for example, "During the past six months, did you go to a hospital emergency room for medical care?"), and inpatient hospital settings (for example, "During the past five years, approximately how many times were you a patient in the hospital because of a medical condition or problem?"). There were also items related to dental, vision, and hearing services.

For dichotomous variables, we measured agreement across administrations by using a simple (unweighted) kappa statistic. On the basis of the standard error, a 95 percent confidence interval was also calculated as a measure of the precision of the estimate of agreement. Because low-prevalence phenomena can influence the kappa statistic, percent agreement scores were also calculated. For ordinal variables with three or more levels, a weighted kappa statistic was used to allow for maximum weighting of the more discrepant cases of disagreement. To increase the cell counts and improve statistical power, some response levels were combined. In accordance with the method of Landis and Koch (9), estimated kappa values below .40 were considered poor to fair; .41 to .60, moderate; .61 to .80, substantial; and .81 to 1.00, almost perfect.

Results

Twenty-seven (93 percent) of the study participants were men; 25 (86 percent) were African American, and four (14 percent) were Caucasian. The mean±SD age of the patients was 40±7.4 years (range, 26 to 56 years). Efforts were made to administer the two interviews as closely as possible to seven days apart. The mean period between administrations was 5±2 days.

Response distributions across 17 items for each administration of the survey, kappa values and related confidence intervals, a strength-of-consistency classification based on the criteria of Landis and Koch, and total agreement percentages are listed in Table 1. Consistency across items was generally strong: of the 16 variables for which kappa values could be classified, three were classified as almost perfect, six as substantial, six as moderate, and one as fair.

Consistency was strongest for items that required respondents to report whether they had used a particular service in the past six months—for example, for six-month use of an emergency department, kappa=.91; for six-month use of outpatient medical services, kappa=.86. Kappa values and percent agreement scores were generally somewhat lower for items that sought information about the number of visits—for example, for the number of outpatient visits in the past six months, kappa=.50; for the number of visits to an emergency department in the past five years, kappa=.30. Respondents were moderately to substantially consistent in reporting receipt of medications (kappa=.45) and use of dental services (.61 and .62). Substantial agreement was found for receipt of a vision test (.79) and receipt of a hearing test (.63).

Discussion and conclusions

The purpose of this study was to determine whether individuals with schizophrenia could consistently report their use of medical services. Overall, the kappa values and percent agreement scores indicated strong consistency. Respondents were most consistent about whether they had used a particular service and were somewhat less consistent in reporting their actual level of service use. These results both support and extend the findings of Calsyn and colleagues (7). Although previous research has shown that the cognitive deficits and lack of insight associated with schizophrenia may compromise the reliability of self-assessments of symptoms (6), our results suggest that people with schizophrenia can reliably report behavioral events such as use of medical services.

It is not known, however, how a longer interval between the two administrations of the survey would have affected consistency scores. Nor do we know how our results compare with those from other samples of individuals who responded to the MEPS, as we did not find reports of test-retest reliability for the instrument in our review of the literature and related MEPS documentation. Also, our findings do not address the validity of the study participants' reports. Although the results of recent research have suggested that the validity of self-report among individuals with serious mental illness may vary by the type of service use reported (7,8,10), the services assessed in those studies were primarily mental health services.

Thus future research also needs to assess the validity of self-reported use of medical services among individuals with serious mental illness. Once the reliability and validity of such information can be established, researchers can use national surveys of medical service utilization, such as the MEPS, to better understand the types and quality of medical care received by people with schizophrenia. Such information will also enable services researchers and policy makers to learn more about the integration of psychiatric and medical services for this vulnerable population.

Acknowledgments

Support for this project was provided by grant MH-57707 from the National Institute of Mental Health and from the Mental Illness Research, Education, and Clinical Center of the Veterans Affairs Capitol Health Care Network (VISN 5).

The authors are affiliated with the School of Medicine at the University of Maryland at Baltimore. Send correspondence to Dr. Goldberg at the Center for Mental Health Services Research, University of Maryland at Baltimore, 685 West Baltimore Street, MSTF Building, Room 300, Baltimore, Maryland 21201 (e-mail, [email protected]). This paper is one of several in this issue focusing on the health and health care of persons with severe mental illness.

Enlarge table