Validity of Electronically Monitored Medication Adherence and Conventional Adherence Measures in Schizophrenia

Most studies of medication adherence among patients with schizophrenia have relied on clinicians' judgments or patients' self-reports, which may have limited validity ( 1 ). Clinician assessment of adherence has limited ability to detect nonadherence in comparison with testing drug levels in urine ( 2 ). Even medication markers, such as in urine and blood levels, may be invalid, because patients may ingest medications shortly before sample collection ( 3 ). Other assessment methods, including self-report, prescription renewals, and pill counts, can be easily manipulated.

Given these limitations, use of electronic monitoring to assess adherence in general medical populations has increased ( 3 ). Electronic monitoring captures the date and time that a bottle of medication is opened. A period in which a medication bottle is not opened identifies an episode of likely nonadherence. However, the opening of the bottle does not ensure medication ingestion. Thus electronic monitoring is unlikely to underestimate adherence, though it may overestimate adherence.

Although several studies have used electronic monitoring to measure the adherence of outpatients with schizophrenia ( 4 , 5 , 6 , 7 ), only one report has compared the performance of electronic monitoring with another adherence assessment method. In a small study of 25 outpatients with schizophrenia or schizoaffective disorder, a highly experienced research coordinator dramatically underestimated the proportion of patients with antipsychotic nonadherence as compared with electronic monitoring (0% versus 48%) ( 6 ).

In this study, we compared ratings by prescribers, patients, and research assistants with ratings of electronically monitored adherence to antipsychotic medication among outpatients with schizophrenia or schizoaffective disorder.

Methods

The study was approved by the institutional review board of the University of Texas Southwestern Medical Center at Dallas. Written informed consent was obtained from all participants. Data were collected from March 2003 to April 2004.

Participants

Participants were outpatients treated at Dallas public mental health clinics and usually self-referred from flyers posted in the clinics. The flyers made no mention of medication adherence status or the investigational focus of this study. Patients were recruited and studied at their outpatient clinic sites. Eligibility criteria included being at least 18 years old, having a DSM-IV diagnosis of schizophrenia or schizoaffective disorder as established by the Structured Clinical Interview for DSM-IV ( 8 ), and taking a single oral antipsychotic medication. Patients receiving a depot antipsychotic within one treatment cycle and those using a pillbox were excluded.

Demographic characteristics

Of 61 participants, 35 (57%) were diagnosed as having schizophrenia and 26 (43%) as having schizoaffective disorder. Thirty-one participants were women, and 30 were men. Mean±SD age was 44.3±9.1 years; mean illness duration was 21.2±10.7 years. Most participants (51 of 61, or 84%) remained on the same dosing schedule throughout the study (41 once daily, nine twice daily, and one three times daily). Six caps were lost during the trial (five were misplaced, and one was damaged).

Measures

Adherence. Adherence ratings were completed at six consecutive, monthly study visits. Fifteen prescribers (ten psychiatrists and five advanced nurse practitioners) rated adherence during the study. One prescriber did not provide ratings for a small number of his patients. In these instances, totaling 6% (24 of 385) of all prescribing clinician ratings, the patients' case manager completed the rating.

Electronic monitoring was performed with the medication event monitoring system, a medication vial cap that electronically records the date and time that a medication bottle is opened. All raters (including patients) were aware of the purpose of the monitoring cap, but they did not have access to cap-generated adherence results. If patients with multiple-dosing regimens (such as twice or three times daily dosing) opened the cap at least the number of times prescribed each day, they received full credit for adherence for that particular day. If patients opened the medication cap fewer than the prescribed number of times per day, missed doses were counted as nonadherence and subtracted from prescribed doses to calculate monthly adherence. To have nonadherence denote insufficient use of medication rather than a mismatch with a prescribed regimen, openings of an amount greater than the number of prescribed doses in a given day did not count toward or against overall adherence. A questionnaire was used at each visit to account for extra openings (such as opening the medication bottle by study staff) or for the lack of bottle openings for valid reasons. These openings (or lack thereof) were added or subtracted accordingly from the electronic monitoring data to obtain a final adherence calculation.

Prescriber ratings, research assistant ratings, and patient self-ratings were completed on visual analog scales (0%–100%), where adherence was defined as the proportion of prescribed doses taken in the prior month. Before completing the visual analog scale rating, the research assistants asked patients three questions regarding their adherence behavior. Adapted from a lengthier adherence questionnaire used in the Clinical Antipsychotic Trials of Intervention Effectiveness (CATIE) study ( 9 ), the three questions inquired about patients' knowledge of their medication regimen and episodes of missed doses or incomplete adherence. All adherence ratings were made blind to and independent of the prescriber-monitored and electronically monitored estimates of adherence. The research assistants who completed these ratings were aware that patients were also rating their own adherence.

Patients and their prescribers were free to switch antipsychotics during the trial, as long as patients remained on a single, oral agent.

Baseline characteristics. Baseline measures included age, gender, race, education level, age at onset of illness, length of illness, presence of a caregiver, dosing schedule, whether antipsychotic dose was within an acceptable guideline range ( 10 ), medication side effects (ordinal severity scale rating of 1 to 10), number of medications, alcohol abuse (Alcohol Use Scale) ( 11 ), drug abuse (Drug Use Scale) ( 11 ), insight (Schedule for the Assessment of Insight) ( 12 ), and symptom severity (Positive and Negative Syndrome Scale total score) ( 13 ).

Analysis

Adherence: dichotomous outcome. For the primary outcome of the study, the likelihood of one method to be significantly more likely to assess nonadherence than another was tested by pairwise comparisons using McNemar's test.

We sought a definition of nonadherence that would be both conservative in detecting nonadherence and consistent with definitions from prior published reports ( 1 ). Patients who were rated as nonadherent (<70% adherence) for two or more months during the study were classified as nonadherent patients. A finding that patients with schizophrenia who were less than 70% adherent to their antipsychotic medication are at greater risk of hospitalization than those who were adherent 70% or more of the time provides additional empirical support to the nonadherence definition used in this study ( 14 ).

Adherence: continuous outcome. The difference between level of adherence as rated by electronic monitoring versus by prescriber was assessed in a random regression analysis. Each baseline characteristic was tested individually for significance as a covariate in this model.

Results

Adherence: dichotomous outcome

Nonadherence was far lower on the basis of prescriber (four of 60, or 7%) and patient self-ratings (three of 61, or 5%) than by electronic monitoring (35 of 61, or 57%) or research assistant ratings (33 of 61, or 54%) ( Table 1 ). For the primary outcome of the study, where adherence defined by electronic monitoring disagreed with adherence ratings by prescribers (occurring with 34 of the 60 participants, or 57%), electronic monitoring was significantly more likely than prescribers to classify participants as nonadherent (32 of 34 versus two of 34; p<.001). Where disagreement occurred between electronic monitoring and patient self-ratings (32 of 61 patients), all 32 patients were classified as nonadherent by electronic monitoring but adherent by patient self-report (p<.001). There was no directional bias when differences occurred in classification of adherence ratings by electronic monitoring versus research assistant or prescriber versus patient. However, disagreements between electronic-monitoring ratings and research assistant ratings were common, occurring in 22 of 61 (36%) cases.

Enlarge table

Adherence: continuous outcome

As a secondary analysis defined a priori, the difference in adherence between electronic monitoring and prescriber assessment was significant for the overall patient group (t=-5.32, df=59, p<.001). Estimated adherence by electronic monitoring, prescriber, patient, and research assistant ratings, respectively, was 67%, 85%, 91%, and 69%. Education was the only baseline parameter that had a significant effect on the difference between electronic monitoring and prescribers' estimation of adherence (t=3.84, df=58, p<.001), with greater education associated with greater adherence.

Discussion

This is the first prospective study, to our knowledge, to compare prescriber, patient, research assistant, and electronic-monitoring methods to estimate adherence for patients with schizophrenia. Compared with electronic monitoring, prescribers dramatically underestimated nonadherence (57% versus 7%, respectively). Prescribers were significantly more likely than electronic monitoring to classify patients as adherent when disagreement occurred between these ratings. In addition, prescribers and patients provided remarkably high estimates of antipsychotic medication adherence (85% and 91%, respectively). Reasons for such optimistic estimates of adherence warrant further investigation, particularly considering that concurrently collected objective adherence rates were dramatically lower as measured by electronic monitoring (67%).

These findings suggest that overestimation of adherence by prescribers and patients might contribute to the underutilization of long-acting injectable antipsychotic medications, which are recommended for patients with nonadherence ( 15 ). The findings also may have implications for clinical research studies. It is possible that participating in a research study inflates prescriber and patient estimates of adherence, which would have implications for clinical studies that use subjective measures of adherence. Finally, these findings raise the possibility that nonadherence is more common than appreciated in clinical research studies and may confound the interpretation of findings from such studies.

Our findings and the results of two prior studies ( 16 , 17 ) suggest that prescribers overestimate adherence (that is, underestimate nonadherence) of outpatients with schizophrenia. Prescriber-determined rates of nonadherence in these three studies (7%, 17%, and 33%) are considerably lower than the 50% mean nonadherence rate reported in a recent review ( 1 ) of studies using mostly clinician ratings in schizophrenia. In addition, findings of nearly complete agreement between prescriber and patient ratings of nonadherence in our study (7% and 5%, respectively) suggest that usual-care interactions strongly influence prescriber estimates of individual medication-taking behavior.

Greater agreement was seen with estimated adherence of research assistant (69%) and electronic-monitoring ratings (67%) compared with those of prescribers (85%). The superior fit between research assistant and electronic-monitoring ratings of adherence could have resulted from several factors. Of particular note, research assistants, but not prescribers, asked patients three adherence-related questions before completing visual analog scale ratings of adherence. Thus research assistants may have had access to valuable adherence information unavailable to prescribers. In contrast to prescribers, research assistants also had access to patients' visual analog scale ratings of adherence. However, the very high degree of adherence reported in these patient self-ratings (estimated adherence of 91%) is unlikely to account for the relatively low adherence described by research assistants.

Potential study limitations include a relatively short assessment period, modest sample size, and the somewhat arbitrary categorical definition used for nonadherence in the primary analysis. Finally, use of a convenience sample could have introduced bias into the study. In particular, volunteers may be more likely to be adherent to medications, and thus the findings of the study may overestimate adherence in the actual population.

Conclusions

Study results suggest that when medication management decisions, such as changing the dosage or type of medication, are based on clinical judgment or patient self-report, they may be invalid for up to 50% of patients. The use of electronic monitoring to identify patients who are nonadherent to medication regimens or the use of long-acting agents may provide substantial benefits not achievable with common clinical approaches to addressing such adherence issues. Additional studies are needed to further elucidate the potential impact of providing more valid adherence estimates to patients and clinicians.

Acknowledgments and disclosures

This study was supported by K-Award grant 5-K23-MH-064930 from the National Institute of Mental Health, by a grant from Janssen Medical Affairs, LLC, and by the Stanley Medical Research Institute.

Dr. Byerly is a paid consultant for Bristol-Myers Squibb, is on the speaker's bureau for Janssen and Bristol-Myers Squibb, and receives current grant funding from Janssen and AstraZeneca. Dr. Rush is on the speaker's bureau for Cyberonics, Inc., Forest Pharmaceuticals, Inc., and GlaxoSmithKline. He is a consultant for Advanced Neuromodulation Systems, Inc., AstraZeneca, Best Practice Project Management, Inc., Bristol-Myers Squibb, Cyberonics, Inc., Eli Lilly and Company, Forest Pharmaceuticals, Inc., Gerson Lehman Group, GlaxoSmithKline, Jazz Pharmaceuticals, Merck and Company, Neuronetics, Ono Pharmaceuticals, Organon Pharmaceuticals, Inc., Pamlab, Personality Disorder Research Corp., Urban Institute, and Wyeth-Ayerst Laboratories, Inc. Dr. Rush owns stock in Pfizer, Inc. Ms. Thompson is an employee of Janssen Medical Affairs, Inc. The other authors report no competing interests.