Using Patient-Reported Outcomes in Schizophrenia: The Scottish Schizophrenia Outcomes Study

Despite considerable research interest in the past 20 years, outcome measures for assessing clinical effectiveness in mental health care have rarely been used systematically in routine clinical settings. There is a lack of consensus about which outcome measure to use in clinical practice, and it has become evident that outcome measures used in research have limited relevance in measuring treatment effectiveness in clinical practice. Rating scales such as the Positive and Negative Syndrome Scale ( 1 , 2 ) are widely used in pharmaceutical registrational trials, but they are of limited utility in clinical settings, given the importance of functional as well as symptomatic outcomes. There is a need for brief measures that clinicians can use in their own practice.

The importance of involving patients in their own health care and of patient-reported assessments is increasingly recognized in the United Kingdom ( 3 , 4 ) and in the United States, where the Food and Drug Administration has recently advocated the use of patient-reported outcomes in clinical trials ( 5 ). Most outcome measures in mental health, such as the Health of the Nation Outcome Scale (HoNOS), are rated by clinicians, rather than patients. Most clinicians would consider such assessments to be objective, although all ratings in mental health are essentially subjective. In studies where patient and clinician assessments have been compared, patients and clinicians tend to agree better on clinical issues, whereas social and functional aspects are emphasized by service users ( 6 , 7 ). In chronic conditions such as schizophrenia, there are advantages to using patient-reported measures to understand complex needs and improve alliances between service users and clinicians. Patient-reported outcomes have gained acceptance in other areas of health care ( 8 , 9 ), but there is skepticism about their use in schizophrenia where reduced insight commonly occurs. Service user involvement has been championed by advocacy groups, and recovery approaches to improving mental health ( 10 , 11 ) are influencing service development in both state and independent sectors.

The primary aim of the Scottish Schizophrenia Outcomes Study (SSOS) was to assess the feasibility and utility of routinely collecting outcome data in everyday clinical settings. Data were collected over three years in the Scottish National Health Service (NHS). During this period national standards of care for schizophrenia ( 12 ) were introduced by the Scottish Health Department. There were two secondary aims of SSOS: first, to compare data from patient-rated, objective, and clinician-rated outcomes, and second, to describe trends in outcome data and service use across Scotland over the three years of the study.

Methods

Study design

A prospective naturalistic observational design was utilized, which is described in detail in the SSOS study report published by NHS Quality Improvement Scotland ( 13 ). Ethics committee approval from the Scottish Mulitcentre Research Ethics Committee was obtained for the study, and written informed consent was obtained.

Recruitment of participants

Participants were recruited from all NHS Health Boards in Scotland and from a number of national advocacy organizations. Approximately 9,000 people with schizophrenia were known to be in contact with NHS mental health services in Scotland ( 14 ), and we aimed to recruit a 10% sample of this population. Patients with an ICD-10 F20–F29 diagnosis (schizophrenia, schizotypal disorders, or delusional disorders) ( 15 ) were identified by key worker clinicians from their caseloads, from which a random sample was invited to participate. A total of 748 clinicians participated: nurses (N=620, 83%), occupational therapists (N=74, 10%), social workers (N=37, 5%), and psychiatrists (N=17, 2%). Clinicians received standardized training in the use of the outcome measurement tools and training on how to use the information obtained in order to continuously improve practice at individual, team, and service levels. HoNOS training was accredited by the Royal College of Psychiatrists.

Assessments

The core measures used in SSOS were HoNOS ( 16 ) and the Avon Mental Health Measure (Avon) ( 17 ). The validity and reliability of HoNOS and Avon have been established among patients with severe mental health problems ( 7 , 16 ). HoNOS was designed for use by U.K. clinicians, and it is now used in Europe, Australia and Canada. It comprises 12 items rated on a severity scale of 0 to 4. Possible scores range from 0 to 48, with higher scores indicating a higher severity of mental health problems. The 12 rated items can further be categorized into the following four subscales: behavior problems (aggression, self-injury, and substance use), impairment problems (cognitive dysfunction and physical disabilities), symptomatic problems (depression, hallucinations and delusions, and other psychological problems), and social problems (personal relationships, overall functioning, residential and living conditions, and occupation and activities). HoNOS was designed for use in routine clinical practice as a record of a patient's progress. A HoNOS rating was made by clinicians before collecting the completed Avon, which was self-rated by patients.

Avon is a patient-reported needs assessment tool designed by service users and health professionals in the United Kingdom for use by service users. The scale comprises five categories with a total of 29 items rated on a 5-point severity scale. Possible scores range from 0 to 145, with higher scores indicating less need over five categories. The categories (and items) are physical (food, accommodation, physical health, self-care, and ill effects of treatment), social (support, discrimination, daily routine, community involvement, and participation), behavior (sleep, risk to self, substance misuse, suicide, and anger), access (transport, use of transport, information availability, information understanding, communication, income, and managing money), and mental health (mood swings, depression, unusual thoughts, anxiety, obsessive thinking, and forgetting). In this study Avon was completed by patients separately from their clinician to ensure that their responses on Avon reflected their needs. When patients needed help to complete the assessment, this was provided by caregiver, friends, or advocacy workers. The completed assessment was then collected by clinicians.

Data collection

The study began in 2002 and concluded in 2005. Data were collected during three time periods (phases 1, 2, and 3), where phase 1 represented baseline and phases 2 and 3 occurring at approximately 12 and 24 months after baseline, respectively. The data collection form ( 13 ) was brief to allow accurate completion within the time constraints of normal clinical practice. Completed data forms were then forwarded to the study center in Glasgow, Scotland, where data were verified and entered in an Access database.

Statistical analysis

Statistical analysis was conducted by using SAS 9.1 for Windows. Baseline data were tabulated, and trends over time in outcomes, treatments, and services—including psychiatric hospitalization, detention, imprisonment, enrollment in the Care Programme Approach (CPA), attempted suicide, and self-harm (all in the previous 12 months)—were estimated by using statistical models. (The CPA is a U.K.-wide system of care for patients with complex needs.) For binary outcomes a mixed-effects logistic regression model was used, with subject as a random effect and time as a fixed categorical effect (phases 1, 2, and 3) to estimate the odds ratio [OR] (with 95% confidence interval [CI] and associated p value) for phase 2 versus phase 1 and phase 3 versus phase 1. An autoregressive correlation structure across the years was assumed, and a generalized estimating equations approach was used to fit the model. Phase 2 and 3 were compared by using a likelihood ratio test. Finally, a number of covariates were introduced that might be predictive of the outcome in question to adjust the estimated effect over time. The values of covariates as measured at each phase were used rather than those at baseline. ORs for covariates that were jointly significant at p<.001 are selectively reported. The covariates utilized are described in the footnote to Table 2. For the continuous outcomes (Avon and HoNOS), a similar modeling strategy was used—a standard repeated-measures linear model.

Results

Study population at baseline

The ICD-10 diagnoses of 1,015 participants consisted of the following: F20.0–F20.9 (schizophrenia) (N=945, 93%), F21.0–F24.0 (schizotypal disorder, persistent delusional disorders, acute and transient psychotic disorders, and induced delusional disorder) (N=20, 2%), F25.0–F25.9 (schizoaffective disorder) (N=50, 5%). Of the 1,015 persons recruited, 91% (N=919) completed phase 2, and 78% of the cohort (N=789) completed all three phases. Of the total cohort 710 (70%) were male. A total of 995 (98%) were white, seven (1%) were Asian, three (<1%) were Chinese, one (<1%) was black African, and nine (1%) did not list their race or ethnicity. A total of 781 (77%) were outpatients. The mean±SD age was 43±11 years (males, 42±11 years; females, 44±11 years; range 18 to 78 years). A total of 579 (57%) were diagnosed ten or more years ago, and 133 (13%) had a comorbid diagnosis of a substance misuse disorder. A total of 916 (90%) were registered with a primary care physician. A total of 862 (85%) received treatment voluntarily, 42 (4%) were detained in a hospital under criminal detention procedures, and 107 (11%) were detained in a hospital under civil detention (Mental Health [Care and Treatment] [Scotland] Act 2003). A total of 493 (49%) were living alone, and 318 (31%) were living with a caregiver. A total of 725 (71%) were living at home, with 188 (19%) a resident in the NHS (that is, living in long-term rehabilitation units) and 102 (10%) living in a supported accommodation. Only three were employed in a competitive job; 18 (2%) were in a sheltered work program, and 29 (3%) worked as a volunteer.

Objective outcomes in the year before baseline

A total of 595 (59%) had no psychiatric admission to a hospital during the year before recruitment, 224 (22%) had one admission, 65 (6%) had two admissions, and 37 (4%) had three or more. Of these admissions, 202 (20%) were legal detentions, and 21 individuals (2%) had recently been in prison. A total of 187 (18%) were enrolled in the CPA. In the year before recruitment, 54 (5%) attempted suicide and 90 (9%) had self-harmed.

Trends over time and predictors of outcome

Table 1 shows the marginal proportions over the three phases of selected objective outcomes, and Table 2 gives the ORs and CIs comparing phase 2 and phase 3 against phase 1 (baseline) and shows the influence of selected prognostic covariates. These tables show significant improvements across the study period in the categories of hospitalization, attempted suicide, self-harm, and civil detention. Tables 3 and 4 are analogues of Tables 1 and 2 for continuous outcomes (HoNOS and Avon).

Enlarge table

Enlarge table

Enlarge table

Enlarge table

Patient-rated assessments: Avon

Mean Avon scores increased significantly across the three phases (representing reported improvement), and this was true for all five subsections of Avon ( Table 3 ). Higher total Avon scores were associated with being male and taking clozapine. Self-harm and the use of discretionary antipsychotic medication were associated with lower Avon scores ( Table 4 ).

Clinician-rated outcomes: HoNOS

In contrast to Avon, reduction in HoNOS scores represents improvement, and across the duration of the study, total HoNOS scores and scores on the behavioral problems and symptomatic problems subscales remained unchanged, while scores on the impairment problems subscale significantly increased (indicating increased impairment). Only scores on the social subscale decreased significantly, representing improved functioning ( Tables 3 and 4 ).

Discussion

This study has demonstrated that it is feasible within the Scottish NHS to routinely collect meaningful outcomes data in schizophrenia. Furthermore over three years, more than 1,000 patients assessed their own needs regularly by using the Avon measure.

Because this was an effectiveness study rather than an efficacy study, observational methodology was utilized and participants were recruited from ordinary clinical services. All had an ICD-10 clinical diagnosis of a schizophrenia-related disorder. The cohort represents 10% of people with schizophrenia known to be in contact with secondary services in Scotland and is representative of those with a chronic schizophrenic illness. It is clear that this approach provides useful data for clinicians, because it incorporates patient perspectives on their needs into care plans.

There are very few services anywhere where the routine collection of outcome data in mental health has been successfully achieved ( 13 , 18 , 19 ). Furthermore we believe that SSOS is the first report of a patient-reported measure being utilized routinely, along with other outcome measures, in psychiatric services. Several factors prevent the routine collection of outcomes data, including uncertainty about which measure to employ, use of inappropriate efficacy measures often better suited to research, and a lack of information systems and administrative support for clinicians. Psychiatrists, unlike nurses and occupational therapists, appear reluctant to use outcome measures ( 20 ). In SSOS a majority of raters were nurses; this is appropriate given the key role nurses play in chronic care and their close working relationship with psychiatrists and social workers. With appropriate training, acceptable interrater reliability can be achieved with HoNOS ( 7 , 16 ). Given the numbers of clinicians involved and the high turnover of clinicians during the study, a rolling training program was utilized. This experience is consistent with centers in Australia ( 18 ), Canada ( 13 ), and England ( 19 ) where efforts are also being made to establish the routine collection of outcome data.

Clinician engagement

The importance of clinicians' use of outcomes data in the management of their own patients was incorporated into the training and educational sessions delivered by the research team. This local emphasis is fundamental for the acceptance and engagement of clinicians in the routine use of outcomes data ( 19 ). During the course of SSOS, information on aggregated outcomes was fed back to clinicians at three national meetings, demonstrating the way in which outcomes data can be utilized to configure services to meet the needs of specific populations. When individual clinicians were using the Avon needs assessment for their patients, and not just the patients included in the study, they were asked to share their practice experiences at these meetings. At the end of the study a series of focus groups were also held with patients who had participated, and study findings were reported back to them. Their views on using the patient-reported needs assessment tool were also elicited and will be available in a separate report.

Patient-rated assessments

Patient-rated outcomes have been defined as the measurement of any aspect of a patient's health status that comes directly from the patient ( 21 ). The systematic assessment of the patient's perspective may provide valuable information that could be lost if relying only on clinical evaluation. The Food and Drug Administration has recently issued guidance for the use of patient assessments as effectiveness endpoints in clinical trials ( 5 ). This study shows that patients with schizophrenia can use Avon effectively, except perhaps a few individuals with markedly reduced insight or severe cognitive impairment. Reliable evidence exists about the robustness of the predictive value of patients' perception of their own health status ( 22 , 23 ). In mental health a number of instruments have been proposed, including Outcomes of Problems of Users of Services (OPUS) ( 7 ), Avon ( 17 ), Carers and Users Expectation of Services (CUES) ( 24 ), the Camberwell Assessment of Need ( 25 ), and FACE ( 26 ). Given the increasing evidence of the advantages of patient involvement ( 7 , 19 , 27 , 28 , 29 , 30 ), patient-rated assessments have the potential to improve therapeutic alliances and treatment adherence.

Over three years of SSOS, all objective outcomes improved (reduced number of hospitalizations, incidents of self-harm, and detentions through the Mental Health Act and increased number of persons receiving benefits from the CPA), consistent with Avon patient self-assessments; HoNOS scores, however, remained constant, although scores indicated impairment significantly worsened and social functioning significantly improved. One reason for this disparity may be that clinicians tend to focus on symptomatic and behavioral problems, while functioning is more important to patients. This illustrates the value of using both patient-rated and clinician-rated measures. The approach used in SSOS is consistent with recovery models that emphasize functional rather than symptomatic improvement ( 10 , 11 ).

In Scotland, after this study, the NHS introduced Standards for Integrated Care Pathways in Mental Health ( 31 , 32 , 33 ). The standards incorporate the routine collection of outcome measures for all patients with a severe and enduring mental health condition and include patient-rated assessments, including Avon. As a result of this policy, routine recording and analysis of outcomes data will become embedded within clinical practice ( 33 ). This will encourage reflective practice for clinicians and will provide data for health care providers to utilize in the redesign of services to meet patients' needs.

Conclusions

This study has demonstrated that it is feasible to routinely collect mental health outcomes data from patients with schizophrenia who are using U.K. services. Clinical and objective outcomes and patient-reported assessments were successfully collected and used in care plans. The pattern of outcomes and patient assessments shows that despite the introduction of guidelines, new treatments, and new services, people with schizophrenia continue to have high levels of chronic disability. The routine collection of outcomes data allows clinical teams to review systematically whether patient needs are being addressed and optimize the effectiveness of their interventions. After the SSOS was completed, the Scottish NHS has promoted the use of systematic routine assessment, using both clinician-rated and patient-reported measures, through national standards of care incorporated in Integrated Care Pathways in Mental Health.

Acknowledgments and disclosures

The study was funded by a grant CEPS P01/6 from NHS Quality Improvement Scotland. The authors thank patients, caregivers, and health professionals who participated and are especially grateful to members of the SSOS Advisory Groups, who provided expert advice throughout the study. Members of the advisory groups are listed by name in the SSOS project report at www.nhshealthquality.org .

The authors report no competing interests.