Indicators of Conformance With Guidelines of Schizophrenia Treatment in Mental Health Services

Appropriateness of care is increasingly used as a tool to monitor and improve clinical practice. Often the best clinical evidence of appropriateness of care is derived from published clinical guidelines and adapted by researchers and clinicians to the local context of care ( 1 ). On this basis indicators are developed to assess the adherence of clinical practice to the proposed recommendations. Indicators are applied with individual patients and thus measure quality of care more precisely than studies where quality is inferred from the relative frequency of specific patterns of care ( 2 ).

Banking on previous studies ( 3 , 4 , 5 ), we have developed a set of indicators covering pharmacological treatment, general care, and rehabilitation of patients with psychosis. A critical point in the evaluation of care for patients with psychosis is that recommendations rarely apply to all patients regardless of social abilities, family context, and phase of the disorder. Accordingly, we have developed a method that tailors each indicator to the main patient characteristics and takes into account factors that may justify the lack of compliance with a recommendation. This study reports on the development of indicators of conformance with recommended practices and the feasibility of the application of these indicators in the context of treating psychotic disorders in the public mental health care system in Italy.

Methods

In order to construct the indicators of conformance, we reviewed several available clinical guidelines for the treatment of schizophrenia as well as published reviews of the effectiveness of various treatments. In particular, we looked at the Schizophrenia Patient Outcomes Research Team (PORT) guidelines ( 6 ), the guidelines developed by McEvoy and colleagues ( 7 ), and the guidelines developed by the National Institute for Health and Clinical Excellence ( 8 ). Although these guidelines were developed in different contexts of service systems and they varied in breadth of subjects covered, the core aspects of the recommendations largely overlapped.

In Italy the Department of Mental Health (DMH) is a public institution covering a population of approximately 100,000. It has complete responsibility for mental health care in its catchment area. Each DMH includes facilities for acute inpatient care and long-term residential care and provides for outpatient treatment and rehabilitation. A research team composed of methodologists and senior clinicians from three DMHs discussed the recommendations proposed by the guidelines in the areas of pharmacological treatment, general care, and rehabilitation and decided which components of care to retain. A preliminary draft containing for each indicator the source of the recommendation, the temporal window to be studied, eligibility criteria, conformance, and possible moderators was prepared by two of us and discussed with the study team. Two criteria were applied to identify the components of care: relevance in the context of community mental health care in Italy and the possibility of operationally defining each recommendation only by looking at clinical records and interviewing professionals in charge of the patient. Of 19 indicators originally proposed (including family psychosocial intervention, psychotherapy, tardive dyskinesia, and regular checks of patients' health), the team eventually retained 15.

For each component of care, we constructed the corresponding indicators of conformance (or review criteria, according to Baker and Fraser [1]) by means of three binary variables amenable to computation for each patient: eligibility, meaning the requirements to be met in order for a patient to be a candidate for evaluation according to a given indicator; conformance, or the criteria to be satisfied for the treatment strategy to comply with the recommendations; and moderators, which are factors that could justify nonadherence to given recommendations. For instance, PORT guideline recommendation 3, "Persons experiencing their first acute symptom episode should be treated with an antipsychotic medication …, but dosages should remain in the lower end of the range … (300–500 mg/equivalents of chlorpromazine per day)" was operationalized as follows: eligibility, "Patient has his or her first acute episode"; for conformance, "The daily dose prescribed today is between 300- and 500-mg equivalents of chlorpromazine"; and for moderators, "The patient has serious comorbidity or has experienced severe adverse reactions to at least two antipsychotic drugs," or "The patient or the family refuses treatment with antipsychotic drugs."

Whenever the guidelines differed, we chose the one that best fit the local expert consensus (for example, concerning length of maintenance treatment). Finally, whenever the guidelines offered only a generic statement, we adopted criteria for operationalization that seemed sensible for the local context. For example, the McEvoy recommendation "Consider cognitive and social skills training; support group; training in activities of daily living" was considered to be fulfilled when an eligible patient attended at least ten rehabilitation sessions of any kind in the month preceding the survey.

The daily dose of antipsychotic medication prescribed to each patient was converted to milligram equivalents of chlorpromazine according to conversion factors derived from the literature. Since the early 1980s, established conversion factors to chlorpromazine have been available for traditional antipsychotic drugs. For second-generation antipsychotics, comparative clinical trials (the basis to establish equivalent dose) are fewer and more controversial ( 9 ). Therefore, we examined various sources, including randomized clinical trials, meta-analyses, and guidelines, and considered both efficacy and adverse reactions. Whenever more than one conversion factor was available, we chose the one most frequently recommended. For instance, the dose of quetiapine equivalent to 100 mg of chlorpromazine varies from 75 mg to 200 mg, but the most frequently reported, which we have adopted, is 100 mg. [A table of conversion factors to chorpromazine used in the study is provided in an online supplement to this article at ps.psychiatryonline.org.] In a few cases, when randomized clinical trials or meta-analyses were not available, we relied on the consensus of expert clinicians.

We developed data collection forms to cover all data items needed to operationalize the indicators. Each data item was collected with relevance to a definite temporal window to satisfy the corresponding indicator (to define eligibility, conformance, or moderators). The forms were developed to assemble the required information in such a way that the person filling them in could not recognize the criterion to be applied to assess conformance. They covered the following areas: sociodemographic details and living arrangements; annual income, history of employment, evaluation of employment skills, and current employment or training, if any; diagnosis of schizophrenia or schizoaffective disorder according to DSM-IV-TR ( 10 ), assigned by the psychiatrist in charge of the patient; length of psychiatric history; current level of disability, as measured by the Italian version of the Social and Occupational Functioning Assessment Scale (SOFAS) ( 11 ); history of somatic disorders possibly interfering with treatment and rehabilitation programs; current presence of acute psychotic symptoms and extrapyramidal side effects, according to the judgment of the psychiatrist in charge; pharmacological treatment prescribed on the day of recruitment; history of acute episodes in the past 12 months; episodes of noncompliance to pharmacological treatment in the past five years; outpatient visits over the past two months; meetings of mental health professionals with the patient's family in the past 12 months; rehabilitation sessions over the past month, including sessions of social skills training, support groups, and socialization activities; and interventions at the community level (home visits, assistance with activities of daily living, ensuring compliance with medication, and so on) over the past month.

A preliminary evaluation of a sample of medical charts at the three DMHs showed that some items were likely to be reliably recorded (including diagnosis, current pharmacological treatment, or episodes of noncompliance) whereas other information (such as interventions at the community level, family meetings, reasons for changing medication, or presence of side effects) was usually limited to brief and haphazard notes, making it impossible to define conformance only on the basis of medical records. An interview with the professionals in charge of the patients was therefore conducted for all patients in the study to complement data extraction from the charts. The study evaluated quality of care by retrieving information from medical charts and by interviews with professionals. Anonymity was maintained, and data were used only in aggregate form: accordingly, participating institutions did not require the patient's consent to the use of these data. The study was submitted to and approved by the institutional review boards of each of the three local authorities whose DMHs participated in the study.

A complete data collection exercise was started in order to assess the feasibility of the proposed approach. The study was conducted in three DMHs of the Piedmont region of Italy as part of a project on quality of care and costs in schizophrenia that was sponsored by the Italian Ministry of Health. In order to estimate with sufficient precision the true proportion of conformance achieved for each indicator, we needed a sample size of at least 720 patients with schizophrenia currently in the care of the participating DMH. To compute sample size, we hypothesized a worst-case scenario level of conformance of about 35%, to be estimated with a 95% confidence interval of 7 percentage points ( 12 ). Six research assistants were trained to perform the data collection under the supervision of a senior clinician at each DMH and with the help of a manual that detailed the procedures to be followed. To minimize the bias in data collection, neither the research assistants nor the professionals in charge of the patients were aware of the algorithms to derive conformance. Data management procedures were developed to centrally check the data collected every month and to request clarifications if needed.

Site visits conducted at each DMH during the study to reexamine a random sample of patients showed good agreement between clinical records and abstracted data. Data were collected over ten months, from June 2002 to March 2003. Because patients with psychosis under the care of the three DMHs could seek services at any site according to their degree of disability, level of symptoms, and family ties, we sampled all of the facilities available in the catchment area for acute inpatient care, long-term residential care (both public and private), outpatient clinics, and rehabilitation centers and day hospitals. Each facility was visited three or four times. The research assistants were trained to draw random samples of six to eight patients by means of a predefined list of random numbers to be matched to the alphabetical list of patients having a visit or in the hospital on the index days of the survey.

Results

The research team derived a set of 15 indicators, nine that concerned pharmacological treatment and six on general care and psychosocial rehabilitation interventions. A careful discussion was necessary to define both eligibility and moderators. The source guidelines and the criteria making up the indicators for pharmacological treatment are detailed in Table 1 . They covered essential principles of treatment with antipsychotic drugs, namely adequate dosage, length of treatment, monotherapy, use of depot antipsychotics, and management of extrapyramidal adverse reactions. Among antipsychotic drugs, we evaluated current prescription of clozapine, because of its particular indication and the severity of possible adverse reactions. In creating the definition of moderators of pharmacological treatment, we gave special consideration to comorbidity for its potential to interfere with antipsychotic medications and to the history of severe adverse reactions, which might have justified the lack of prescription of an antipsychotic medication or the prescription of an inadequate dose. We also considered refusal (by the patient or the patient's family, acting for the patient) of antipsychotic medication, which may prevent an appropriate prescription. Finally, with regard to monotherapy with antipsychotic drugs, we used as a moderator the concomitant prescription of two antipsychotics in less than six weeks (or eight weeks for clozapine) to allow for switching from one drug to another with cross-titration or overlap and taper.

Enlarge table

Table 2 summarizes the indicators of general care and psychosocial rehabilitation. We considered outpatient visits lasting at least 30 minutes with the psychiatrists in charge, collaborative decision making with the patient's family, attendance at psychosocial rehabilitation sessions, work and vocational rehabilitation, and assertive community treatment. As expected, a more complex set of conditions was considered to define eligibility with rehabilitation indicators than with pharmacological treatment. In particular, to define eligibility we looked at the patient's level of disability, family relationships, current living situation, age, and recent or current hospitalization. For conformance, we defined the minimum frequency of psychiatric visits, family meetings, assessment of work potential, and the number of rehabilitation sessions or interventions in the community according to local standards. For all general care and rehabilitation indicators except the one on psychiatric visits, we allowed for the patient's level of disability as a moderator, considering that patients with preserved social functioning may not need some of the interventions, whereas those with severe malfunctioning may have been too disabled to qualify for them. SOFAS scores of 55 or higher and 30 or below, respectively, corresponded to such characteristics.

Enlarge table

The indicators were applied in a sample of 807 patients randomly selected in three DMHs of the Piedmont region. Patients were recruited at outpatient clinics of community mental health centers (550 patients, or 68%), outpatient rehabilitation centers (107 patients, or 13%), long-term residential facilities (134 patients, or 17%), and hospital units for acute inpatient care (16 patients, or 2%). Table 3 summarizes the main characteristics of the patients recruited. The overall picture indicates a patient population with considerable disability. Most patients had low levels of education and were unmarried, had an average SOFAS score of 46, and had a psychiatric history of approximately 14 years. Three-quarters of the patients lived at home, independently or with their family (parents, spouse, children, or other relatives), and the remaining patients lived in sheltered apartments or in highly supervised residential facilities.

Enlarge table

Table 4 details for each of the 15 indicators the number of eligible patients, the percentage of patients with conformant care and with nonconformant care, and whether patients with nonconformant care had moderators to explain the nonconformance. Eligibility criteria considerably reduced the number of patients evaluated for pharmacological indicators (except monotherapy with antipsychotic drugs). The level of conformance of pharmacological treatment to treatment guidelines varied according to the indicators considered.

Enlarge table

Of all patients, 774 (96%) were treated with antipsychotic drugs either during acute episodes (very few were recruited at the first episode) or during the maintenance phase. Of these, 364 patients (47%) received only first-generation antipsychotics, 269 patients (35%) received only second-generation antipsychotics, and 141 patients (18%) received both. Monotherapy with antipsychotic drugs was correctly prescribed to two-thirds of patients. However, the dose prescribed was often below the range recommended by the respective indicators, both for treatment of acute episodes and for maintenance treatment. It is worth noting that the indicator on maintenance treatment restricted eligibility to patients who experienced an acute episode in the year before the study. However, the distribution of dosages among the 508 patients without an acute episode in the 12 months before the study closely mirrored the dosages observed among the 219 patients with an acute episode in the same period (6% versus 2% of patients not receiving antipsychotics, 53% versus 49% receiving less than 300-mg equivalents of chlorpromazine, 28% versus 31% receiving 300- to 600-mg equivalents, and 13% versus 18% receiving more than 600-mg equivalents).

The percentage of conformant prescription of clozapine was particularly low, which can be attributed in most cases to lack of previous trials with two second-generation antipsychotics. The presence of moderators, in particular the patient's or the family's refusal of antipsychotic drugs, helped to justify the lack of conformance of antipsychotic drug dose in a large proportion of cases ( Table 4 , indicators 2, 3, and 5). Severe comorbidity, which possibly interfered with treatment, included conditions such as cancer, diabetes with complications, cardiovascular disorders (often coupled with obesity), and hepatitis.

A substantial proportion of patients were eligible for evaluation according to general care and rehabilitation indicators. A high level of conformance was reached for psychiatric visits, work, and collaborative decision making with the patient's family. The presence of moderators explained a considerable proportion of nonconformant care for psychosocial rehabilitation, assessment of the patient's work potential, work, assertive community treatment, and collaborative decision making with the patient's family. Moderators involved were good social functioning (SOFAS score at or above 55) and comorbidity that prevented psychosocial rehabilitation, poor social functioning (SOFAS score at or below 30), and comorbidity that prevented assessment of the patient's work potential, conformance to the indicator on psychosocial rehabilitation for assertive community treatment, and good social functioning (SOFAS score at or above 55) for collaborative decision making with the patient's family. The work indicator was moderated mainly by the fact that patients did not wish to work, were too disabled, or had serious comorbidity.

Overall, the information needed to define the indicators was easy to retrieve, with the exceptions of patients who had recently had a change of treating psychiatrist and patients with a very long treatment history. The care of psychotic conditions typically requires that patients be referred to several services within the DMH. Because medical records are not centralized and coordination among the DMH staff is not always optimal, research assistants had to elicit information from several sources and resolve conflicts and inconsistencies through consultation with the various professionals in charge of each patient. In addition, despite the initial training and the availability of a manual detailing the information to be retrieved, throughout the data collection some ambiguity had to be clarified during regular meetings with the team of research assistants.

Once data were collected and thoroughly verified, we turned to the last phase of this feasibility study, the practical application of the proposed algorithms. Data were input into a specially developed database, and a series of computer routines were written in Visual Basic for Applications to classify each patient on the basis of three binary variables describing eligibility, whether conformance was achieved, and presence of moderators for each indicator. No specific difficulty was encountered, suggesting that the data grid that had been used was both sufficient and pertinent.

Discussion

Assessing the appropriateness of care in schizophrenia has been the topic of several studies that have used different methodologies. A common concern underlying the measurement of the quality of treatment provided is that patients with psychosis are very vulnerable and have limited advocacy capacity, thus being at risk of inadequate care or even neglect ( 13 , 14 , 15 ). Our study documents the development of a set of core indicators to measure conformance of care with treatment guidelines for patients with schizophrenia and schizoaffective disorders. The rationale behind the development of indicators of conformance of care is that randomized clinical trials offer a generally accepted anchor point from which to rate the quality of medical procedures. The results of such trials are synthesized in clinical guidelines, which outline the evidence-based procedures (and when evidence is lacking, the best experience-based procedures) that will maximize positive outcomes for specific groups of patients. Accordingly, measuring the gap between clinical practice and recommended guidelines is sufficient to assess the quality of care, without the need to evaluate patient outcomes, which may greatly vary across settings because of differing levels of illness severity, comorbidities, and ability to comply with the physician's suggestion ( 16 ). This approach seems even more appealing in schizophrenia treatment, where nonconformant care cannot be surmised from long-term outcome, so that the potential negative effects of poor standards of care, like unnecessary exposure to high-dose antipsychotic medications, may go unnoticed for years ( 17 , 18 ).

Throughout the study we were aware that indicators could cover neither all facets of the care provided to patients nor the complexity of environmental and personal aspects. Rather, the indicators represented a minimal set of requirements to be met in the care of patients with psychosis ( 19 ). The study, for instance, did not develop indicators for the interventions that were difficult to document, such as health checks by general practitioners or general hospitals, because they are rarely coordinated with the DMHs; that were not routinely applied within the participating DMHs, such as family therapy or cognitive therapy, because the participating DMHs lacked expertise in these areas or had little faith in their usefulness; or that required a standardized evaluation by a psychiatrist, which would have been difficult to organize for logistical reasons (for example, the assessment of tardive dyskinesia). Excluding these components of care does not mean that they are not important but that within the resources of this study we preferred to concentrate on dimensions measurable and relevant to the local practice. Our approach, similarly to all data collection exercises embedded in clinical practice, was affected by issues of reliability and completeness of data. These limitations should be kept in mind when applying the proposed methodology in order to minimize possible biases.

The three measures developed for each indicator—namely eligibility, conformance, and moderators—are similar to criterion, standard, and exception presented in the NICE guidelines to facilitate clinical audit ( 8 ). Moderators are close to the notion of "patient-related factors" (such as noncompliance or refusal of treatment) used by Young and colleagues ( 4 ) to explain the poor-quality medication management received by some patients ( 4 ). We found that the dimension of flexibility introduced by the moderators of quality made the indicators more acceptable to clinicians, because they allowed, albeit partially, for the complexity of choices to be made in caring for patients with psychosis ( 1 ). Furthermore, moderators helped in the interpretation of the quality achieved by providing potential explanation for departures from expected results. It is important to stress that the level of conformance measured in this study is meaningful only in the light of its determinants of an organizational and clinical nature. These determinants include availability of structures and personnel, training and motivation of health professionals, absolute level of resources, and their efficient use ( 16 ).

According to the literature, three core elements help decide whether care is appropriate: evidence, context, and values ( 20 ). Evidence refers to the availability of high-quality research results, context to the circumstances where care is provided, and values to the preferences of patients, families, and caregivers. This framework can be applied to discuss strengths and weaknesses of the proposed approach.

Evidence

Overall, the indicators of pharmacological treatment were easier to operationalize than the indicators of psychosocial rehabilitation. The indicators were derived from published guidelines, where detailed recommendations concerned pharmacological treatment more frequently than rehabilitation interventions. This partially mirrors the imbalance of research, and hence of evidence, toward purely pharmacological treatment of psychotic disorders ( 3 , 21 ). Although rehabilitation aspects are equally if not more important than medication in the care of patients with psychosis, clinical trials testing nonpharmacological interventions are very few in comparison with hundreds of drug trials ( 22 ). Along these lines, the guidelines that we considered strongly supported the use of antipsychotic medication at the first episode, whereas alternative nonpharmacological strategies were not supported, although concern is increasingly raised about long-term efficacy and side effects of antipsychotic drugs ( 18 , 23 ). Any indicator of conformance is developed with reference to a specific set of norms, and when these norms change, because of updated evidence and consensus, the indicators should be adapted accordingly.

Context

Standards should be realistic and set according to local context and patient circumstances ( 24 ). The guidelines from which we derived the indicators reflected the organization of primary and specialist care both in the United States and the United Kingdom. Accordingly, the translation of rehabilitation recommendations into review criteria adapted to the Italian mental health care system was more complex than the same exercise for pharmacological recommendations. For example, the indicator on assertive community treatment was intuitively appealing in the Italian context, where a community-focused system has been developed. However, the definition of assertive community treatment requires independent multidisciplinary teams that provide intensive services to patients in order to enhance coordination, integration, and continuity of care in the community (for example, PORT recommendation 29 [6]). In Italy each DMH directly provides or coordinates all rehabilitation activities in its catchment area, thus ensuring coordination and continuity of care. Mental health teams usually are based in outpatient facilities or in day treatment centers located in the community, where all ranges of services are available (including home visits), whereas mobile teams devoted solely to assertive outreach are very unusual. Accordingly, the operationalization of the assertive community treatment indicator, in particular criteria for conformance and moderators, had to be adapted to the local situation. This example brings up the issue of the treatment model when effectiveness depends on it: an adapted criterion may bypass fidelity to the model, but in so doing the indicator of conformance may lose the intrinsic measure of effectiveness.

Values

Preferences and values of patients and families—such as the patient's desire to work or refusal of pharmacological treatment—were considered as moderators for specific indicators. The refusal of pharmacological treatment actually justified a sizeable proportion of cases with a nonconformant medication dose, which tended to be lower than recommended. Lack of desire to work moderated almost all the instances of nonconformance to the corresponding indicator. Values also inform some of the choices of mental health professionals. For instance, caring for the most disabled patients is a strong value entrenched in Italian mental health care reform ( 25 ). This may help us understand the concentration of rehabilitation activities among more disabled patients, as compared with those with fair social functioning. Similarly, other projects measuring quality of care revealed that the values of the treating physicians may explain departure from conformant treatment. For instance, McLaughlin and colleagues ( 26 ) described insufficient use of potentially lifesaving therapies among patients with myocardial infarction, possibly resulting from a conscious decision by physicians to withhold these therapies from patients unlikely to benefit in the long term.

Conclusions

In summary, this study proposed a set of indicators that is not sensitive to the organization of the health care system itself and that is useful for documenting the quality of care for patients with psychosis, identifying areas of treatment that need improvement, and setting goals for progressive change. Coupled with a study of the determinants of care (organizational, cultural, and ethical factors; level of knowledge and motivation of the staff; and availability of human resources, equipment, and structures), indicators of conformance can help mental health teams and departments to implement quality improvement programs, the results of which may in turn be evaluated with the same methodology, within a continuous cycle of quality improvement. Tending to appropriate care is an incremental process that requires the identification of factors that affect quality of care and feasible interventions to improve it. Indicators such as the ones developed in this project represent a crucial step to objectively measure and monitor achieved quality. Further research on these aspects should be encouraged. Efforts to estimate the costs involved in implementing the proposed process within any given health system for actual management purposes, rather as an academic exercise as done in this study, should also be pursued.

Acknowledgments and disclosures

This study was supported by Health Planning Funds (ex art. 12 d.lgs. 502/1992) from the Italian Ministry of Health. The authors thank Thomas McLaughlin, D.Sc., Mauro Nannini, M.D., Maja Strakovà, Ph.D., Emanuele Lomonaco, M.D., Albina Rispoli, M.D., Roberto Merli, M.D., and Lucio Ghio, M.D., for their contribution to the development of the indicators. They also thank Federica Grottaroli, Barbara Riviera, Luana Ceccarini, Alessandra Mandriani, Elena Macchiarulo, and Paola Minacapelli, who were research assistants, as well as Christian Schneider, who was responsible for data management. Finally, the authors thank the health professionals who patiently shared essential information about their practice.

The authors report no competing interests.