Prescriber Fidelity to a Medication Management Evidence-Based Practice in the Treatment of Schizophrenia

A variety of evidence indicates that medication management in public mental health systems is often suboptimal, both in the ways medications are used ( 1 ) and in the assessment and documentation of medication effects ( 2 ). Unlike other evidence-based practices in mental health, where there are low rates of utilization relative to need, use of medications in the treatment of severe mental illnesses is ubiquitous in developed countries. Thus the task in implementing evidence-based medication management practices is not to promote the adoption of medication use. Rather, the task is to ensure that the knowledge we have about medications, medication effects, and patient factors that predict or determine medication effects is incorporated into medication management decisions.

In 2003 the Substance Abuse and Mental Health Services Administration (SAMHSA) awarded several grants to implement six evidence-based practice toolkits in community mental health settings throughout the United States. One toolkit focused on a medication management program—Medication Management Approaches in Psychiatry (MedMAP)—in the treatment of schizophrenia ( 3 , 4 , 5 , 6 ). The MedMAP toolkit focuses on providing prescribers with tools that are helpful for gathering, organizing, and accessing evidence about medications and medication-related consumer characteristics ( 3 ). These tools include guidelines on medications (for example, dosing, side effects, and recommended sequences), specification of patient-related information needed for medication decisions (for example, medication history, symptom levels, and side effects), and standardized medical record forms for documenting this information.

MedMAP was developed by a national panel of experts in psychopharmacology and is based on four principles of good medication management, which include the use of psychotropics in ways that are consistent with the evidence about them, assessment of patient characteristics that affect medication selection, measurement of medication-related outcomes, and patient involvement in medication selection. A further underlying premise of MedMAP is that documentation of the observations and rationales that have gone into medication management decisions is essential in a world in which patients almost always have multiple providers over the course of their illnesses. In developing the MedMAP Prescriber Fidelity Scale, these principles of medication management were conceptualized into seven different domains and 22 measureable items. These domains and items are illustrated in the box on this page.

Thus MedMAP provides a comprehensive set of evidence-based medication management guidelines for treating schizophrenia. However, the medical literature shows that although explicit guidelines generally improve clinical practice, the impact of guidelines varies considerably ( 7 , 8 , 9 , 10 ). Many organizational and systems factors influence the success of implementing evidence-based guidelines, including organizational culture, leadership, practitioner attitudes, and funding ( 11 , 12 , 13 , 14 , 15 ).

This is the first report on the use of MedMAP in a longitudinal intervention study. The overall goals of this multifaceted study were to assess the feasibility of implementing MedMAP in a community mental health setting, evaluate the usefulness of assessment tools included in the Implementation Resource Kit provided for this study ( 3 ), and measure prescriber fidelity to MedMAP principles. The purpose of this article is to report on findings related to prescriber fidelity to MedMAP and discuss implications for further efforts in implementation.

Methods

Overview

MedMAP was implemented in six community mental health clinics in Kentucky. Four of the sites were located in a densely populated urban area of the state; these urban sites served approximately 1,154 unduplicated adult clients diagnosed as having schizophrenia during the three-year period of the study. Services for clients at the urban sites included several programs developed specifically for those with severe mental illnesses, such as individual and group counseling, medication assessment and monitoring, case management, peer support, and psychoeducation. Two additional participating sites were located in a sparsely populated, mountainous rural area and served approximately 226 unduplicated adult clients diagnosed as having schizophrenia during the three-year study. Similar to the urban sites, the rural sites provided individual and group counseling, education, medication management, and case management.

The MedMAP Implementation Resource Kit ( 3 ) provided the foundation for study implementation at the participating clinical sites. Fidelity assessments were conducted over 30 months to assess the degree of prescriber adherence to the MedMAP principles at the participating sites. Data were collected from December 2003 to May 2006. The fidelity assessment team was composed of a co-investigator of the MedMAP Project and six graduate students from two universities in the state. The institutional review board of the University of Kentucky and the Kentucky Cabinet for Health and Human Services approved all study procedures. Written informed consent was obtained before data collection.

Participants

Study participants consisted of 14 prescribers, who represented 100% of the prescribers at the six participating sites; nine were psychiatrists and five were advanced practice psychiatric nurses with prescriptive authority. All but three were employed on a full-time basis. The prescribers were generally evenly distributed among the six sites; four sites each had two prescribers, and two sites each had three prescribers. The inclusion criterion for prescriber participation in the study was employment at the community mental health center for at least six months before the study. Inclusion criteria for the medical records reviewed in the fidelity assessments were a diagnostic code of schizophrenia subtypes (295.1, 295.3, 295.6, and 295.9), services provided for at least six months before a fidelity assessment, and no inpatient hospitalizations of more than one month during the six months before a fidelity assessment.

Thirty medical records were reviewed at each site at each fidelity assessment. The number of records selected per prescriber was based on the number of hours the prescriber worked per week. For example, in a fidelity assessment that audited 30 records of two prescribers at a site, 20 records were audited for the prescriber who was employed on a full-time basis (40 hours per week), and ten were audited for the prescriber who was employed on a half-time basis (20 hours per week).

Instruments

The MedMAP Prescriber Fidelity Scale is a 22-item instrument that assesses documentation in medical records of medication practices ( 3 ). The scale is based on the assumption that adequate documentation is a necessary ingredient for evidence-based medication management. Scoring of the prescriber fidelity scale requires review of the medical records to identify documentation that indicates adherence to MedMAP guidelines. Items on the prescriber fidelity scale are scored as yes when documentation indicating adherence is found, and they are scored as no when it is lacking. Percentages of the total number of medical records with documented adherence are calculated, and the percentages for each item are then transformed into a 1 to 5 Likert scale, with a rating of 5 indicating excellent adherence to MedMAP principles and a rating of 1 representing complete or almost complete lack of adherence ( 3 ). Adherence on 90% or more of the medical records was required for a fidelity rating of 5, 70%–89% was coded as 4, 50%–69% was coded as 3, 11%–49% was coded as 2, and 10% or less was coded as 1.

Psychometric evaluation

Early in the study, the MedMAP Steering Committee agreed to collaborate with the developers of MedMAP to conduct psychometric testing on the 22-item MedMAP Prescriber Fidelity Scale. This psychometric testing was conducted in conjunction with a larger study that included 50 prescribers in 26 community mental health centers in four states ( 16 ). Two raters completed all assessments. Findings from the psychometric analysis of data from the larger study indicated that for the 22-item scale, item-level interrater reliability, determined by assessing the intraclass correlation coefficient (ICC), generally exceeded .80. Interrater reliability for the total scale was .89. One-month test-retest reliability for data collected in Kentucky for the total scale was .94. For the study described here, the item-level interrater reliability ranged from .71 to .97, with most ICC values clustered around .80. Interrater reliability for the total scale based on this study's data was .97, and one-month test-retest reliability was .93.

It is important to note that four items on the Prescriber Fidelity Scale required modification during the course of the study to more accurately measure prescriber practice. These modifications included adding a "not applicable" choice to the response set, when that choice was not available in the original tool. Because of the changes made to the response sets, these items are not included in this article.

Procedure

Training. Prescriber education was based on a train-the-trainer model. Designated trainers, who were prescribing physicians at a local academic health center, participated in a one-day educational session that focused on the use of the medication algorithms and symptom rating scales included in the MedMAP Implementation Toolkit. Subsequently, physician trainers conducted similar training sessions for prescribers and clinical staff at all participating sites. Training was conducted one month after baseline.

Follow-up training was conducted in annual one-day sessions at the sites throughout the study to reinforce MedMAP principles. In addition, the fidelity assessment team regularly provided consultation and feedback to prescribers and administrators during fidelity assessment visits; feedback consisted of summary reports of fidelity scores to the directors of the participating sites after each fidelity assessment visit.

Fidelity assessments. Fidelity assessments were conducted approximately every four months at each of the six participating sites, for a total of 30 fidelity assessment visits. Each assessment was conducted by four fidelity assessors. The fidelity assessments involved comprehensive review of the medical records. Sources of documentation included narrative progress notes, treatment plans, medication administration records, and intake assessments. Interrater reliability among fidelity assessors was enhanced by developing chart review guidelines, which specified which sections of the medical records should be included in the review.

Before training, baseline chart reviews were conducted on 30 randomly selected medical records at each of the six sites, for a total of 180 medical records. Posttraining chart reviews were conducted on 30 randomly selected medical records at each of the six sites for each of the four posttraining fidelity assessments, for a total of 720 medical records.

Analysis. At each posttraining fidelity assessment visit, prescriber-specific scores for fidelity were calculated for each of the 18 items on the MedMAP Prescriber Fidelity Scale by determining the percentage of medical records reviewed for a prescriber that had documented adherence. These percentages were converted to the scores on the Likert scale that ranged from 1 to 5. In addition, overall fidelity scores were determined for each prescriber by calculating the average of the 18 individual fidelity scores obtained at each fidelity assessment visit.

Scores above 4.0 were classified as high fidelity, scores of 3.0 to 4.0 were classified as moderate fidelity, and scores below 3.0 were classified as low fidelity. These classifications are consistent with recommendations from McHugo and colleagues ( 6 ), which are based on findings from studies that examined fidelity to implementation of five psychosocial evidence-based practices in the National Implementing Evidence-Based Practices Project.

Mixed-effects modeling was used to test for changes in fidelity over time and by prescriber type, with separate models for overall fidelity and for each of the 18 items on the scale. For each model, the fixed effects included time (ranging from 0 to 4, with 0 indicating baseline, 1 indicating the first posttraining fidelity assessment, 2 the second assessment, 3 the third assessment, and 4 the fourth assessment) and prescriber type (psychiatrist versus advanced practice psychiatric nurse), as well as the interaction between time and prescriber type. The random effects were prescriber nested within site, and the prescriber-by-time interaction. Possible values ranged from 1 to 14 for prescriber and from 1 to 6 for site. Post hoc pairwise comparisons of significant fixed effects were accomplished using Fisher's least significant difference procedure. These mixed models determined which fidelity scores exhibited significant changes over time and which were relatively unchanged despite the intervention, while simultaneously testing for differences by prescriber type. Data analysis was conducted with SAS for Windows. An alpha level of .05 was used throughout.

Results

Overall fidelity scores

The mean overall fidelity score at baseline was 2.93 ( Table 1 ). Posttraining mean overall fidelity scores ranged from 3.12 to 3.43; the mixed model for overall fidelity demonstrated a significant main effect of time (F=6.6, df=4 and 49, p<.001). The prescriber-type variable and the interaction between time and prescriber type were not significant. Post hoc analysis of the time main effect indicated that overall fidelity at baseline was significantly lower than at each of the four posttraining fidelity assessments (p<.04 for each pairwise comparison).

Enlarge table

Individual item fidelity scores

Separate mixed-effects models were constructed to test for differences in each of the 18 individual items over time, by prescriber type and among the combinations of time and prescriber type. None of these models had a significant prescriber main effect or a prescriber type-by-time interaction. The main effect of time was significant for simplification of medication regimen (F=3.9, df=4 and 28, p=.01), patient education (F=3.9, df=4 and 18, p=.02), diagnosis (F=5.0, df=4 and 11, p=.02), illness history (F=5.3, df=4 and 17, p=.006), and past medication history (F=4.6, df=4 and 50, p=.003). For these five items, there was a significant improvement between baseline and the first posttraining fidelity assessment that was at least maintained over the subsequent assessments ( Figure 1 ). For the diagnosis, past medication history, and illness history items, there was additional improvement in fidelity from the first to second posttraining assessment that was significant.

The remaining 13 items did not change significantly between baseline and the posttraining fidelity assessments ( Table 1 ), although there was considerable variation in these scores. For example, at the fourth and final fidelity assessment, five items scored in the high range (above 4.0), three scored in the moderate range (3.0 to 4.0), and five scored in the low range (below 3.0).

Scores for eight of these 13 unchanged items—summary within the past four to six months, recommended dose range, patient involvement in treatment planning, documentation of medication regime (current medication, side effects, and medication adherence), prescriber attendance at treatment team meetings, and medication visit frequency—were in the moderate to high range both at baseline and throughout the study. In addition, fidelity scores for some of these items were higher at baseline than they were for items that showed significant improvement over time ( Table 1 ). For example, the mean fidelity score for illness history improved significantly over time, but scores were in the low range (baseline score of 1.50 and final score of 2.43). In contrast, scores for patient involvement in treatment planning did not change significantly over time, but the scores were consistently high (baseline score of 4.50 and final score of 4.57).

Discussion

The average attainment of MedMAP fidelity at the six sites was moderate, even with the training. Findings suggest that training improved prescriber adherence to some items of MedMAP early in the study; however, fidelity scores for all items were static later in the study. The trajectories for improvement in overall fidelity scores are similar to those reported by McHugo and colleagues ( 6 ) for five psychosocial evidence-based practices studied in the National Implementing Evidence-Based Practices Project. McHugo and colleagues found that for all five practices, fidelity scores increased in the first year of implementation and stabilized between 12 and 24 months of implementation. The overall mean fidelity score at the conclusion of our study was 3.27, compared with 3.89 for the earlier study ( 6 ).

In this study, some items showed improvement over time, whereas others did not. Significant improvements were seen in simplification of medication regimen, patient education, documentation of diagnosis, and documentation of illness and medication history. However, scores for most of these items remained in the moderate to low ranges at the final fidelity assessment ( Table 1 ). Although scores for these items remained moderate to low at the final fidelity assessment, significant improvements over time suggest that these aspects of MedMAP were most amenable to change resulting from initial training.

It is encouraging that fidelity scores for eight items were in the moderate to high range both at baseline and at all four posttraining fidelity assessments. This finding suggests that prescribers were already providing treatment consistent with some aspects of MedMAP best-practice guidelines, even before training and implementation. This has several implications for future implementation efforts. Implementation can be facilitated by using the baseline assessment data to evaluate an agency's current practice and by comparing current practice to the best-practice guidelines. These baseline data would allow administrators and prescribers to concentrate implementation efforts toward improving weak areas of practice and continuing to support strong areas of practice. In addition, the process of fidelity assessment could be streamlined by focusing assessments primarily on areas with low scores at baseline, with less frequent review of areas with high scores at baseline.

Several items showed low fidelity scores at baseline and throughout the study, which suggests that training was ineffective in improving fidelity to these items. Furthermore, findings suggest that the current structure of the mental health delivery system does not support some practice changes and implementation of these evidence-based guidelines in the clinical setting. For example, consistently low prescriber fidelity was found in the measurement and use of outcomes, specifically documentation of outcomes and the provision of treatment guided by outcomes ( Table 1 ). Fidelity to these items involved the use of quantitative rating scales at each medication visit. Throughout the project, participating prescribers indicated that although they agreed with the need to use quantitative scales to monitor and document treatment outcomes, organizational factors, such as time and billing constraints, often interfered with their ability to implement the outcome measurement guidelines included in MedMAP.

Low rates of fidelity to items related to the measurement and use of outcomes are concerning. Before quality and safety initiatives in the mental health service system ( 17 ), medication management was strongly guided by subjective reports of symptoms and functioning, rather than by the use of validated symptom rating scales. For over a decade researchers, clinicians, and policy makers have stressed the need for practices that are guided by empirical evidence and supported by outcome assessment ( 17 , 18 ), with the ultimate goal of demonstrating efficacy of mental health treatment and reducing ambiguity in communication between providers and service systems. Ganju ( 19 ) concurred, maintaining that performance measurements of mental health treatment and assessment of outcomes resulting from treatment are integral to the process of implementing evidence-based practices.

Because training appeared to be ineffective in improving fidelity scores related to the measurement and use of outcomes, the identification of alternative strategies to facilitate implementation of outcome assessment is crucial. For example, fidelity may be improved by regularly reporting fidelity scores to administrators and individual prescribers, thereby providing ongoing feedback about areas of practice. Furthermore, organizational-level changes in the structure of service delivery may improve implementation. However, strategies to improve fidelity must be tailored to conform to the structure and the workflow of the individual sites that implement MedMAP. At the participating sites, the time allotment for medication checks with prescribers was 15 minutes. It appears that this time constraint prevented successful implementation of outcome monitoring, yet administrators and policy makers may question the feasibility of increasing the time allotment for a service encounter, given the financial constraints that state mental health budgets currently face.

As an alternative, other members of the multidisciplinary team can improve fidelity by participating in implementation efforts. McHugo and Drake ( 20 ) have observed that the multidisciplinary nature of contemporary health care delivery is often overlooked in the development and implementation of evidence-based medicine. MedMAP implementation focused solely on prescribers, to the exclusion of other members of the multidisciplinary treatment team, such as nonprescribing nurses, social workers, psychologists, and case managers. The complexity of implementing an evidence-based practice, based on the number of elements a prescriber must address in a short service encounter, may warrant a change in mental health service delivery to a system that is more consistent with a primary care service system. In such a system, nonprescribing clinical staff would be trained to use quantitative rating scales to conduct a preliminary assessment of patients' symptom severity and provide that data to prescribers immediately before they see the patients.

An additional strategy to improve implementation of outcome measurement involves the use of patient self-report regarding progress in meeting treatment goals, rather than clinician ratings of symptoms and functioning. For example, Deegan and colleagues ( 21 ) created a peer-support decision center in the waiting room of a community mental health clinic; resources included an Internet-based software program that patients used to rate their symptoms and functioning while waiting to see their prescriber. The program generated a report that patients and prescribers used as a guide to shared decision making about medications.

Ongoing fidelity assessments, both at the prescriber level and organizational level, are essential in future MedMAP implementation efforts. Timely feedback to clinicians and administrators regarding trends in fidelity over time is equally important. However, it is important to note that the fidelity assessment procedures described here were complex and required several assessors to collect data. Additional research is needed to examine strategies for collecting, analyzing, and summarizing fidelity data in a streamlined manner to facilitate the process of providing regular feedback to clinical sites. The assessments were lengthy, in part because paper medical records were reviewed. Tsai and Bond ( 22 ) conducted fidelity assessments with electronic medical records by using a checklist adapted from the MedMAP Prescriber Fidelity Scale; they found that retrieval of data from electronic medical records was 20% faster than when paper records were used.

Some study limitations are evident. Four items on the MedMAP Chart Audit Instrument required modification to the response sets during the study; therefore, findings related to these items were not included in the data analysis. An additional limitation is the lack of a control group of sites that offered usual care. Finally, although the longitudinal analysis modeled prescribers nested within site, no organizational characteristics were assessed so as to maintain site anonymity in light of the relatively small number of participating sites. We recommend continued testing and refinement of the instrument, with future studies designed to address these limitations.

Conclusions

The MedMAP project represents the first longitudinal effort to implement a medication management program in a community mental health treatment setting. Study findings have implications for research, practice, education, and policy. Medications are essential in the treatment of schizophrenia, and the success of all other evidence-based practices depends on the prescriber's ability to effectively prescribe medications to treat the symptoms of schizophrenia. Continued development and refinement of MedMAP are crucial for prescribers and patients. This study represents an early point in the trajectory of knowledge regarding medication management evidence-based practices; additional implementation projects will provide direction for the refinement of MedMAP assessment instruments and protocols and will ultimately allow clinicians to provide care to individuals with schizophrenia that is guided by best practices.

Acknowledgments and disclosures

This study was funded by grant CFDA5-HD9-SM56151 from the Substance Abuse and Mental Health Services Administration (SAMHSA). Development of the fidelity scale was initially supported by a SAMHSA contract. Further refinements and testing of the scale were funded through unrestricted educational grants from AstraZeneca Pharmaceuticals, Bristol-Myers Squibb, Eli Lilly and Company, Janssen Pharmaceuticals, and Pfizer.

Dr. Miller has received grants or research support from AstraZeneca Pharmaceuticals, Bristol-Myers Squibb, Eli Lilly and Company, GlaxoSmithKline, Janssen Research Foundation, Organon Pharmaceuticals, Pfizer, and Sanofi Aventis. Dr. Cooley has received grants or research support from or served as a consultant for Janssen Research Foundation, Abbott Laboratories, AstraZeneca Pharmaceuticals, Bristol-Myers Squibb, and SmithKline Beecham. The other authors report no competing interests.