Agreement Between Seriously Mentally Ill Veterans and Their Clinicians About Medication Compliance

Antipsychotic medication is known to be an efficacious treatment for schizophrenia, with the majority of patients showing symptomatic improvement (1). Fifty to 75 percent of stable patients who discontinue antipsychotic medications relapse within a year (2,3,4,5), and current practice guidelines recommend long-term maintenance treatment for most patients (6).

Medication noncompliance is common among seriously mentally ill patients (7) and is a predictor of rehospitalization (3,8,9,10). Noncompliance may result from patients' confusion about the prescribed regimen, disagreement about the need for treatment, medication side effects, or a focus on other life issues (11,12,13). Compliance has been found to decrease when the medication regimen is long term, patients are experiencing chronic rather than acute symptoms, and treatment is preventive—all characteristics of antipsychotic maintenance treatment (14,15). Concurrent substance abuse, a lack of awareness of the psychiatric illness, and thought disorganization are also associated with decreased compliance (8,16,17).

Studies of compliance among seriously mentally ill patients have typically used patients' self-reports or ratings by clinicians and family members as compliance measures (7,10,18,19). Other types of measures, such as systematic pill counts or determinations of plasma drug levels, have advantages, but they are expensive, difficult to employ in naturalistic settings, and still subject to error (12,20,21). Studies that have used pill counts or plasma levels as compliance measures have found approximately 25 to 65 percent of patients with schizophrenia to be noncompliant (13,22,23).

Researchers have attempted to improve the accuracy of compliance reports by obtaining ratings from multiple observers—patients, family members, and clinicians. However, few studies have commented on agreement between concurrent assessments, and those that have commented found both "no significant correlation" (17) and "adequate concordance" (16).

Investigators have used concurrent compliance reports in a variety of ways. They have combined reports into an overall compliance score (24), analyzed reports separately (17), or selectively used the reports, often selecting the one indicating poorer compliance as the study measure (9,19). Examining the relationship between reports by the patient, the clinician, and family members may help determine if the different assessments tap the same underlying behaviors and may also shed light on how to combine or otherwise use these measures.

Agreement between patients and clinicians on medication use has implications for clinical care as well as compliance research. Clinicians have been urged to engage seriously mentally ill patients in treatment partnerships (11,25) and to "elicit the patient's cooperation in monitoring participation in treatment" (25). Open disclosure about compliance has been thought to be "a product and a catalyst for practitioner-patient relationships of high quality" (26), and poor compliance has been viewed as a failure of the patient-clinician partnership.

In clinical partnerships characterized by accurate patient self-monitoring and open discussions about medication, a common understanding about medication use should logically emerge. Disparate reports of medication use by clinicians and patients would reflect ineffective communication and suggest that these partnerships have not yet developed.

In 1991 Congress appropriated special funds to establish enhanced treatment programs for seriously mentally ill veterans at 14 Veterans Affairs medical centers. The enhanced programs include day treatment programs, intensive community case management programs, and STAR II programs, which are intensive rehabilitative programs designed to assist patients in adjusting to community living (27). All enhanced programs were designed to decrease hospital admissions and lengths of stay for patients with high service utilization. Eleven of the 14 enhanced programs list "increasing compliance" as a major programmatic goal, and all programs include treatment components thought to increase compliance, such as increased contact with patients, increased continuity of care by a single treatment team, and coordinated educational programs on medication and symptom management.

In addition, all the enhanced treatment programs were supplied with the Brentwood modules (28), a structured educational intervention designed to help patients "become progressively more self-reliant in their use of antipsychotic medication." The Brentwood modules emphasize the development of skills for self-monitoring, medication management, and negotiation among seriously mentally ill patients (29).

We conducted a secondary analysis of data from the longitudinal cohort study of seriously mentally ill veterans enrolled either in enhanced treatment programs or in comparison programs of standard care. Our objectives were to determine agreement about compliance in a large cohort of seriously mentally ill patients and their clinicians, whether agreement increased with exposure to intensive programming that emphasized compliance, and whether compliance reports by patients or clinicians were associated with hospital admission.

Methods

Study population

Veterans were eligible for enrollment in enhanced treatment programs or comparison standard care programs if they qualified for VA hospital care, had a DSM-III-R diagnosis indicating psychosis (schizophrenia, schizoaffective disorder, bipolar disorder with psychosis, chronic organic psychosis, or other psychoses), and had either 150 or more documented days of hospitalization or five or more inpatient admissions in the year before enrollment.

A total of 1,258 eligible veterans were enrolled in the enhanced programs, and 342 were enrolled in the comparison programs, in a four-to-one ratio. Evaluation coordinators at each site assigned patients to the programs based on the enrollment ratio and clinical judgment. A total of 1,600 veterans were enrolled between January 9, 1991, and December 19, 1995, at 14 VA medical centers. Due to rolling enrollment and patient mortality, 1,516 veterans were eligible for a one-year follow-up, and 1,281 veterans were eligible for a two-year follow-up by February 4, 1997.

Assessments and measures

Veterans and treating clinicians were asked to rate medication compliance at the time of enrollment and at one and two years after enrollment. Clinicians' compliance reports were collected by surveying clinicians, and patients' compliance reports were collected through structured interviews conducted by evaluation coordinators, who were research staff located on site and not directly involved in patient care. Evaluation coordinators received two days of formal training in the use of the structured interview and the interview procedure manual, as well as annual retraining and regular consultation during monthly conference calls.

Members of clinical treatment teams, including psychiatrists, social workers, psychologists, and nurses, were trained in use of the clinician survey. Team members viewed a video of a patient interview, completed clinician surveys, and compared scores with other team members and an expert rater. Clinicians on the team with the closest relationship with the patient were responsible for completing the survey. The clinician's ratings of the patient were completed in close conjunction with the patient's visits. Patients were scheduled to see their clinicians when the surveys were due, unless a clinical visit was already scheduled at the appropriate time.

Both patients and clinicians rated medication compliance on a 5-point ordinal scale with anchors of taking medication none of the time, some of the time, quite a bit of the time, most of the time, or all of the time. Ordinal ratings were dichotomized. Clinicians' and patients' reports were considered to indicate compliance if the clinician or the patient reported that medication was taken all or most of the time. The reports were considered to indicate noncompliance if the clinician or patient reported that medication was taken quite a bit, some, or none of the time. This dichotomy was chosen to reflect a point at which a reasonable clinician might be concerned about adverse consequences arising from the reported level of medication use.

Patients assessed their compliance over the previous three months, whereas clinicians assessed compliance over the previous six months. Patient-clinician assessments with overlapping time frames were used in the analyses.

Data on hospital admissions were obtained from the centralized data files maintained by the Department of Veterans Affairs.

Statistical analyses

Analyses were conducted on data collected between January 9, 1991, and February 5, 1997. Differences in ordinal assessments of compliance by patients and clinicians were analyzed with the median sign test.

Agreement between patients and clinicians was determined with four measures: overall agreement; Cohen's kappa, a statistical test that corrects for agreement expected by chance; positive agreement, or agreement that patients were compliant; and negative agreement, or agreement that patients were noncompliant. Documentation of positive and negative agreement supplements information contained in the omnibus test of agreement, the kappa, and is useful in understanding low kappas coupled with high levels of observed agreement (30,31). Further information about these measures can be found elsewhere (30,31).

Associations between overall agreement and time in the program were analyzed with McNemar's test for paired data. The significance of differences in kappa scores at baseline, one, and two years was determined with the chi square test described by Fleiss (32).

Multivariable analyses were conducted using generalized estimating equations (GEEs). The first analysis looked at the relationship between agreement (yes or no) and the predictor variable (program type), with covariates of age and initial score on the Global Assessment of Functioning (GAF). The second analysis looked at the relationship of agreement to predictor variables of age, GAF scores, and scores on the Brief Psychiatric Rating Scale (BPRS). BPRS items were scored using a rating scale of 0 to 6; a score of 20 is equivalent to a score of 38 on the commonly used scale, on which ratings of 1 to 7 are given.

The final GEE analysis examined the relationship between hospitalization within the 30 days after assessment (yes or no) and clinicians' and patients' compliance reports.

All GEE analyses accounted for the fact that observations would be correlated for patients with more than one observation. Statistical analyses were completed using SAS software, version 6.12.

Results

Compliance data

Compliance reports with appropriate time overlaps were available for 1,369 of the 1,600 patient-clinician pairs at baseline (86 percent of eligible pairs), for 829 of the 1,001 pairs with one-year follow-up assessments (83 percent of the pairs with one-year data, and 55 percent of eligible pairs), and for 676 of the 793 pairs with two-year follow-up assessments (85 percent of pairs with two-year data, and 53 percent of eligible patients).

Patient characteristics

The mean±SD age of patients at study entry was 51.3±12.9 years, with a range of 21.3 to 86 years. Ninety-six percent of patients were male. Seventy-six percent had a qualifying diagnosis of schizophrenia; 13 percent, schizoaffective disorder; 4 percent, bipolar disorder; 5 percent, another mood disorder; and 2 percent, organic psychotic disorder. Patients were seriously impaired, with mean±SD baseline BPRS scores of 20±12 and mean GAF scores of 45±15.

Assessments of compliance

Table 1 summarizes data on assessments of medication compliance by clinicians and patients in enhanced programs at baseline and at one and two years after enrollment. Assessments differed significantly at baseline (sign test, p<.001), with patients considering themselves far more compliant than clinicians. At enrollment, 95 percent of patients in the enhanced programs stated that they were taking their medications all or most of the time, while only 71 percent of clinicians judged these patients to be taking medications all or most of the time. Significant differences between patients' and clinicians' compliance assessments remained at one and two years for patients in the enhanced and comparison programs.

Agreement

Figure 1 summarizes data on overall observed agreement and agreement about compliance and noncompliance between clinicians and patients in enhanced programs. Kappas for the patient-clinician pairs in enhanced programs at enrollment and at one and two years after enrollment are also illustrated.

Although relatively high levels of overall agreement at enrollment were found for the clinician-patient pairs in enhanced programs, there was little agreement beyond that expected by chance (Cohen's kappa=.095). This low kappa value was the result of a low level of agreement about noncompliance. On average, patients and clinicians agreed only 17 percent of the time when one rater indicated noncompliance. The majority of raters indicating noncompliance were clinicians. In the enhanced programs, when clinicians indicated noncompliance (N=322), patients agreed only 10 percent of the time, and when patients indicated noncompliance (N=51), clinicians agreed 61 percent of the time.

Overall agreement increased over time for patients in enhanced programs, as did agreement about both compliance and noncompliance. At two years, patients and clinicians agreed an average of 34 percent of the time when one rater indicated noncompliance.

Significant differences were found between kappa scores at baseline, one year, and two years for patients in enhanced programs (kappas=.095, .21, .29, respectively; χ²=13.1, df=2, p< .01). Kappas at one and two years were significantly greater than the baseline kappa (χ²=4.6, df=1, p<.05, and χ²=10.4, df=1, p<.01, respectively). However, even the largest kappa of .29 indicated only poor-to-modest agreement. No significant differences between kappa scores were found for patients in the comparison group at baseline, one year, and two years (kappas=.096, .08, and .21, respectively).

Although patient-clinician pairs in enhanced programs showed increased overall agreement with time, they did not differ significantly from patient-clinician pairs in the comparison program when the analyses adjusted for age and initial GAF scores (odds ratio=1.3, 95 percent confidence interval=1 to 1.6).

The above conclusions remained essentially unchanged when ordinal compliance reports for patients and clinicians in enhanced programs were analyzed with weighted kappas. Again, there were low levels of agreement that increased modestly with time.

Effects of patient attrition

Some evidence was found of differential patient attrition. Patients considered noncompliant by clinicians at baseline were less likely to complete the one-year follow-up assessment than those judged compliant at baseline (χ²=18.4, df=1, p<.001). Analyses using data only from patients with complete data from all assessments (N=569) continued to show significant increases in agreement at one and two years for patients in enhanced programs (McNemar's test, p<.02 at one year and p<.001 at two years) but not for patients in comparison programs.

Other predictors of agreement

Higher BPRS scores and lower GAF scores, indicating a higher level of symptoms and impairment, were associated with less patient-clinician agreement about compliance (GEE analysis for BPRS scores, z=-5.4, p< .001; for GAF scores, z=4.4, p<.001). Thus more impaired patients were less likely to agree with clinicians. Agreement increased significantly with patient age (z=3.1, p<.01).

Relationship with hospitalization

Assessments of medication compliance by both patients and clinicians were strongly associated with hospitalization in the 30 days after the assessment. Patients judged compliant by clinicians had half the odds of admission in that period as patients judged noncompliant (OR=.50, 95 percent CI=.37 to .68, p<.001). Patients who reported they were compliant also had approximately half the odds of admission as patients who reported they were noncompliant (OR= .54, 95 percent CI=.32 to .91, p<.05).

Discussion

Although this study found high levels of agreement between seriously mentally ill patients and clinicians about patients' medication compliance, there was little agreement beyond that expected by chance. Low levels of agreement resulted primarily from poor agreement about noncompliance, with patients seldom reporting noncompliance even when their clinicians considered them to be noncompliant. Agreement increased over time for patients and clinicians participating in enhanced programs; however, even in enhanced programs, agreement beyond that expected by chance remained in the poor-to-modest range (kappas<.4).

The low levels of agreement suggest that patients and clinicians may have difficulty achieving a common understanding about medication use. If patients and clinicians are engaged in working partnerships, with accurate patient self-monitoring and open discussions about medication, one would expect that clinicians' assessments of compliance would be influenced by credible reports about medication use from their patients, and that patients' compliance reports would be influenced by their clinicians' supportive but careful questioning about medication use. The fact that this common understanding did not emerge suggests that working partnerships that include effective communication about medication compliance may be difficult to establish.

In this study, poor agreement occurred primarily because patients reported higher levels of medication use than clinicians—a finding that can be interpreted in two ways. If patients' and clinicians' compliance reports are assumed to be equally valid, with neither report being considered more correct or more closely approximating actual compliance, then the observed discrepancies could be attributed either to patients' overestimation or to clinicians' underestimation of compliance. This position could be taken because the reports of both patients and clinicians predicted hospitalization. One would then conclude only that agreement about compliance is low and that it is affected little by intensive programming.

A second approach to the data would be to assume that clinicians' assessments of medication compliance are the more correct assessments—a quasi-gold standard. Discrepancies in compliance reports would then be interpreted as arising from patients' overestimation rather than clinicians' underestimation of compliance. This position might be taken for several reasons. First, the majority of published studies suggest that patients (21) as well as clinicians (26) generally overestimate medication compliance. Patients' overestimation may arise from a desire to give a socially acceptable response, inaccurate perceptions of their own behavior (33), problems with recall, or inadequate understanding of the prescribed regimen. Patients who have neurocognitive impairments (17) and whose medication consumption is more variable (20) may be particularly prone to overestimation.

Patients in this study assessed medication use during interviews with research staff, and they may have been reluctant to acknowledge low levels of use. They were also seriously impaired and may have had neurocognitive problems, prominent psychiatric symptoms, and sleep disruptions, resulting in increased variability in medication consumption and increased difficulties in recalling consumption. Only 5 percent of these seriously mentally ill patients assessed themselves as noncompliant at enrollment. This rate was below noncompliance rates in the literature for this group; however, clinicians' assessments of noncompliance—29 percent at enrollment—were within reported ranges (13,16,23).

If clinicians' assessments of compliance are assumed to be the more correct assessments, difficulties in reaching a common understanding can then be tentatively attributed to patients' difficulties in perceiving or disclosing noncompliance. Actual compliance is seen as increasing over time as clinicians increasingly report good compliance, even though agreement about compliance remains poor to modest. These conclusions, in turn, suggest that agreement might be increased by facilitating more accurate or open patient disclosure of medication use. These conclusions also suggest that reaching a common understanding about medication use may not be essential in increasing compliance and that other strategies, such as "eyes-on monitoring" or intensive case management, might increase compliance in the absence of high levels of patient-clinician agreement.

In this study both patients' and clinicians' compliance reports were associated with hospital admission in the 30 days after assessment, indicating the predictive validity of the reports. Because only modest agreement was found between the reports, information in the measures was not redundant—that is, patients' and clinicians' reports tapped somewhat different behaviors and attitudes, and both predicted hospitalization. Thus both patients' and clinicians' reports can be used in models attempting to predict relapse and readmission.

Several factors may limit the generalizability of these conclusions. First, the study population was predominantly male, older, and seriously impaired. As a result, study patients may have been less compliant, less accurate in their compliance reports, and less likely to agree with clinicians than persons with serious mental illness in general. Second, the comparison group was small, and assignment to groups was not random, which may have limited our ability to detect increased agreement resulting from specialized programming. Third, patients assessed their compliance over the previous three months in structured interviews with research staff, whereas clinicians assessed compliance over the previous six months in self-administered surveys. This difference in data collection methods may have resulted in systematic biases in compliance reports that were not corrected by use of overlapping time frames in the analyses. Finally, we were unable to determine if agreement was independently associated with hospital admission because increases in agreement were necessarily confounded by increases in assessments of good compliance.

Conclusions

Seriously mentally ill patients and their clinicians may have difficulty reaching a common understanding about medication use. Intensive programming appears to have little effect on agreement. If clinicians' assessments of compliance are considered to be more correct than patients' assessments, the results of this study suggest that patients seldom perceive or disclose noncompliance. Patients' and clinicians' reports are not redundant, and both can be used in research models that attempt to predict relapse and readmission.

Acknowledgments

Funding for this research was provided by the Mental Health Strategic Health Group of the Veterans Health Administration. The authors thank Myra Kim, Ph.D., for statistical assistance and Ray Bingham, Ph.D., Gregory Dalack, M.D., Tim Hofer, M.D., Steven Katz, M.D., and Alan Mellow, M.D., for reviewing earlier versions of the paper.

The authors are affiliated with the Serious Mental Illness Treatment Research and Evaluation Center, Health Services Research and Development, Veterans Affairs Medical Center, P.O. Box 130170, Ann Arbor, Michigan 48113-0170 (e-mail, [email protected]). Dr. Valenstein, Dr. Barry, and Dr. Blow are also with the department of psychiatry at the University of Michigan in Ann Arbor.

Enlarge table