Developing an International Standard Set of Patient-Reported Outcome Measures for Psychotic Disorders

A patient-reported outcome (PRO), as measured by patient-reported outcome measures (PROMs), is any aspect of a patient's health status that comes directly from the patient (1). PROMs can be used to improve clinical care (2, 3); to inform clinicians and patients about treatment progress; to create, compare, and aggregate scores at a high level to inform policy; and to inform approval of drugs and devices (4, 5). Research and application of PROMs in health care, particularly in the management of chronic disorders, have increased over the past 20 years (6). The use of PROMs is a focus of patient-centered care (7). Notwithstanding the challenges to implementing large-scale PROMs systems in health care, factors that increase the rate of PROM collection include provider training in the use of PROMs, use of software to register and work with PROMs in daily practice, administrative surveillance of collection rates, and the presence of local clinical champions (3). International application of PROMs requires high standards for translation (8).

We identified two examples of large-scale implementation of recommended PROMs to mental health services in specific programs in the United Kingdom (9) and in routine outcome measurement in Australia (10). However, a Cochrane review of routine monitoring that used PROMs for improving treatment of common mental disorders among adults found insufficient evidence to support routine monitoring and identified the need for “more research of better quality,” including measuring a range of relevant outcomes (11).

The outcomes important to patients can be identified through focus groups, in-depth interviews, target population surveys, qualitative synthesis of the literature, and content analysis of available data sources (12). Ideally, PROMs display strong psychometric properties, including a conceptual and measurement model, reliability, validity, responsiveness, interpretability, alternative modes of administration, and cross-cultural and linguistic adaptations (13, 14). Practical implementation of measures warrants consideration and includes identification of the goals for collecting PROs, selection of patients, and determination of the setting and timing of assessments (15).

Psychotic disorders, including schizophrenia and bipolar disorder, represent significant burdens to service users, families, and health systems worldwide (16). Although schizophrenia is a low-prevalence disorder, with an estimated population prevalence of <1% (17), it is associated with adverse mental and general medical health outcomes, a high degree of disability, high health care costs, and a 15-year reduction in life expectancy (18–23). Bipolar disorder has a slightly higher prevalence, at approximately 1%, with psychiatric as well as general medical burden resulting from its early onset, severity, and chronicity (21–25). Although clinical recovery is defined as the remission of symptoms and the return of functioning (26), the meaning of personal recovery to service users is broader and a process that encompasses many aspects of life and promotion of an individual’s strength and potential (27). Evidence supports recovery as a feasible outcome for schizophrenia spectrum disorders and bipolar disorder (28, 29). At the health systems level, there has been a shift in focus toward a recovery orientation and personalized care (30, 31). The broad impact of psychotic disorders has spurred investigators to examine a range of PROs in schizophrenia and patient-reported quality of life (32) and functional outcomes (33) in bipolar disorder.

Through use of a consensus-building process, the International Consortium for Health Outcomes Measurement (ICHOM) developed and implemented standard sets of PROMs for use in routine clinical practice for various medical conditions. The process is supported by identification of the evidence and by systematic, critical evaluation of measures and their psychometric properties. ICHOM organized a working group of psychosis experts, including clinicians, researchers, and service users, to identify a set of PROMs to monitor individual treatment outcomes or to compare outcomes of similar mental health services, with a view to establishing the value of these services. The value of health care can be defined as the patient outcomes relative to the costs for obtaining those outcomes (34). Outcome assessment can be guided by a set of PROs for a specific disorder, as exemplified by the set for depression and anxiety developed by ICHOM (35). The specific aims of this study were to develop a standard minimum set of PROMs for psychotic disorders that can be used anywhere in the world and to identify a set of risk adjustment factors to enhance utility of comparisons across treatment modalities, institutions, and systems.

Methods

The research team included a chair, project manager, research fellow, and five research associates. The working group (N=19 members) included service users (N=3), and its members were selected to represent diverse professions and geographic areas. Ten areas of expertise were represented: psychometrists, psychiatrists, mental health nurses, psychologists, health economists, epidemiologists, national clinical quality programs, health service researchers, social workers, and service users, with members from 11 countries (Australia, Canada, Denmark, Greece, India, Israel, Japan, Mexico, the Netherlands, the United Kingdom, and the United States). No institutional review board approval or informed consent was necessary for this study.

Service User Focus Groups

Before commencement of the working group meetings, two focus groups were held with four service users to identify outcomes of highest importance to them. Service user recruitment occurred through recommendations from patient organizations and from working group members.

Systematic Literature Review to Identify Outcomes

Between January and March 2019, systematic literature searches were conducted to identify outcomes related to schizophrenia and bipolar disorder type I in adolescent and adult populations. The databases MEDLINE, PubMed, and PsycInfo were searched for publications between January 2013 and January 2019 (the search strategies are detailed in an online supplement to this article).

The Cochrane highly sensitive search strategy for identifying randomized trials (36) was first used to identify outcomes for schizophrenia spectrum and bipolar I disorders. Additional searches that excluded randomized trials were conducted in PsycInfo and MEDLINE. This included qualitative research that examined service users’ perspectives on outcomes of importance and the impact of schizophrenia or bipolar disorder on service users’ lives. Supplementary sources from working group members and reference lists in identified papers were used to find additional PROs and PROMs.

Consensus Process

A modified Delphi consensus process was used to select the outcome set (37). The process involved reaching consensus in five main areas: scope, to determine which psychotic disorders, patient populations, and treatments to include; outcomes, to identify a minimum number of outcomes for inclusion in the set; measures, to assess each outcome; definitions and time points, to assess outcomes; and risk adjustment factors, to enable comparisons among providers implementing the set. The research team prepared and distributed presentations for review before each videoconferencing meeting. The five main areas were discussed during the meetings, and feedback from the working group was incorporated. Members rated and provided feedback on each item under review in the five main areas in online surveys.

On the basis of ICHOM processes outlined a priori, inclusion in the set required that at least 80% of the working group voted an item as “essential” (score of 7–9) in the first or second Delphi round (the scores in the Qualtrics survey ranged from 1, not recommended, to 9, essential). When consensus was not reached by voting, the item was discussed and revisited in the next meeting and survey. Outcomes were excluded if at least 80% of the working group voted an item as “not recommended” (score 1–3). The working group voted on all inconclusive outcomes in the third and final survey round, in which response options were “include” or “exclude” and inclusion required only 70% consensus.

Identification of Potential Outcome Measures From the Systematic Literature Review

Publications identified in the systematic literature review were the source of potential outcome measures. After development of a comprehensive measures list, a search filter was used to facilitate measure selection (38) that identified studies in PubMed reporting psychometric properties of measures. The research team excluded measures that had a cost associated with use.

Assessment of PROMs

The COnsensus-based Standards for the selection of health outcomes Measurement Instruments (COSMIN) checklist was used to assess the psychometric properties of measures, including reliability (test-retest and internal consistency), validity (content validity, face validity, and construct validity), and responsiveness (sensitivity to change) (14). In addition, the working group considered the feasibility of collecting the measures, including length of time to administer, training needed, and international applicability.

Breakout Sessions to Narrow the List of Measures

After the COSMIN checklist was applied, four breakout sessions were held to review and reduce the number of potential measures. The sessions were held with a small number of working group members with lived experience or professional expertise in the areas of functioning, personal recovery, symptoms, and treatment. Participants narrowed the list of potential measures and established a proposal for the wider working group to discuss and endorse. Any measures that took >20 minutes to complete were removed.

Risk Adjustment Factors

A preliminary list of risk adjustment factors and definitions was developed on the basis of risk adjustment factors identified from the systematic literature searches, national registries, and review of existing ICHOM standard sets. Factors were identified according to evidence of their effect on the outcomes selected. Demographic and clinical factors were assessed on relevance and feasibility of measurement. Further, to harmonize across ICHOM mental health standard sets, ICHOM mental health working group chairs developed a list of factors to reduce burden of implementation for service users with more than one diagnosis. The harmonized list was presented to each mental health consensus working group. Factors voted for inclusion reached 70% consensus.

Open Review and Patient Validation

After development, the set was distributed to international stakeholder groups, including professionals, adult service users, and caregivers in two separate stakeholder surveys to obtain feedback on outcomes, measures, risk adjustment factors, and timing of outcome collection. Respondents were recruited via networks identified by the research team and working group members through e-mail and social media and national and international patient organizations. Service users and caregivers were asked to rate the importance of each outcome using a 9-point Likert scale and were provided space to suggest additional outcomes.

Results

Scope

The working group selected both schizophrenia spectrum disorders (as classified in ICD-11 and DSM-5) and bipolar disorder type I (as classified in ICD-11). The set is limited to the adolescent (ages ≥12 years) and adult populations.

Service User Focus Groups

The core outcomes identified as important to service users included improvement in positive and negative symptoms, general medical health, and medication side effects; personal recovery; and prevention of relapse.

Identifying PROs

The literature searches identified 9,118 references, of which 758 were eligible for full-text review (see diagram in online supplement). Additional sources were recommended by working group members. After removal of duplicates and measures with associated costs and in languages other than English, a total of 131 measures were examined that assess symptoms, personal recovery, functioning, and treatment in psychosis (see online supplement).

Domain, Outcome, and Measure Selection by Consensus Working Group

The development process used by the consensus working group is illustrated in Figure 1. The working group identified four outcome domains (symptoms, recovery, functioning, and treatment) that encompassed 14 outcomes (Table 1) (39–47). Symptoms included depressive symptoms, suicidal ideation and behavior, positive symptoms, negative symptoms (schizophrenia), mania or hypomania (bipolar I), sleep quality, and relapse rate as measured by hospitalizations. Personal recovery included quality-of-life measures. Functioning included global, social, and role functioning and general medical health. Medication side effects were included under treatment outcomes. Each core outcome identified by service users in the focus groups was included in the final set.

A total of 57 measures were presented to the working group for a vote (see online supplement). Outcome measures were reviewed in their entirety, with consideration given to psychometric properties, previous use in the specified population, the number and wording of items, and time taken to complete (Table 2) (39–50). Response rates were 80%, 85%, 80%, and 80% for the first through fourth modified Delphi processes, respectively.

Service users provided feedback regarding item wording and appropriateness and their opinion about the ability of the measure to capture the outcome for individuals with a psychotic disorder. This feedback was summarized in tables and presented alongside feedback from other working group members. A total of 147 measures were initially mapped to the 14 outcomes, and 39 measures were presented via short lists in breakout sessions.

Evaluation of Measures

Measures were assessed on psychometric properties, including acceptable interrater reliability (r≥0.70) (51), internal consistency (Cronbach’s α≥0.70) (52), and evidence of sensitivity to change, reflected by change in scores measured over time, consistent with a priori hypotheses about anticipated treatment outcome (Table 2). Measures rated as strong had acceptable interrater reliability, internal consistency, and evidence of sensitivity to change. Measures rated as mixed had only a single evaluation identified, mixed evidence from several sources, or strong evidence only for certain items. Measures rated as weak had below-threshold evidence. The working group selected nine measures in an outcome set that takes approximately 35 minutes to complete. The selected measures are freely available in English to users.

Time Point Recommendations

The outcome assessment time line was proposed by the working group to best achieve a balance between the clinically relevant times when outcomes may be expected to change and pragmatic concerns in data collection (53). The working group recommended assessment of outcomes before treatment as a baseline and then every 6 months and assessment of risk adjustment factors at baseline and annually thereafter.

Risk Adjustment Factors

The preliminary list of risk adjustment factors included demographic factors, such as age and socioeconomic status; clinical factors, such as comorbid conditions (54) and hospitalizations; and intervention factors, such as the setting. Harmonization of risk adjustment factors across mental health sets resulted in the addition of two factors: trauma, as assessed by adverse childhood experiences, and contact with law enforcement (Table 3).

Open Review and Patient Validation

Ninety-five professionals living in Australia, Canada, Chile, Nigeria, Sweden, the United Kingdom, and the United States responded to the open review survey. Service users and caregivers (N=25) were from Australia, the United Kingdom, and the United States. Overall endorsement of the set and its elements exceeded the required 70%. Of the 106 participants with professional experience, support for outcome domains ranged from 77% to 93%, and support for outcome measures ranged from 61% to 84%. Of the 25 participants with lived experience, 92% agreed that the measures are useful to collect, and 91% stated that the list captured all important outcomes. Endorsement of included outcomes ranged from 72% to 92%.

Discussion

The research team identified numerous PROMs for both schizophrenia spectrum disorders and bipolar disorder. The process involved a review to identify the measures and a subsequent review to assess the measures’ psychometric properties (55), a process similar to that used by other reviewers of PROMs. The consensus process was successful in reducing the number of measures to a pragmatic set for use in routine practice. Service users were integral to the development of the set, from initial identification of core outcomes that mattered most to them to assessment of measures’ face validity and comprehensibility of items. The open review and patient validation phase helped ensure the interpretability and cultural sensitivity of the set.

In addition to the well-recognized symptoms associated with psychosis, sleep problems are common among people with schizophrenia spectrum disorders and bipolar I disorder; they have a negative impact on functioning and well-being and are associated with a reduced ability or opportunity to participate in valued activities (56). Sleep quality, as assessed by the PROMIS-Sleep measure (43), is included in the set.

Personal recovery was very important to service users. The personal recovery measure in the outcome set has good psychometric properties and has been used in published research on populations of mental health service users (39, 57, 58). We did not find a positive symptom measure that has been developed and tested exclusively in samples of people with schizophrenia. However, the modified Colorado Symptom Index has been used in large-scale studies of populations with severe mental illness (40). We did not identify a self-report measure for negative symptoms. Because of its length and the availability of only one language version, the Clinical Assessment Interview for Negative Symptoms (59), a 30-item measure, was not recommended for inclusion in the outcome set. As a best alternative, the working group suggested a PROM, the Recovering Quality of Life (39). Historically, a clinician-rated outcome measure (CROM), the Quality of Life Scale (60), has been used as a negative symptom measure. There was less research to support decision making on adolescent measures.

Limitations of this work include a low number of service users in the working group and no service users with lived experience of bipolar I disorder. The inclusion of service users in developing PROMs is important yet remains challenging. Few studies include them at all stages of development (61). In this study, service users were recruited after the design stage and before the decision to include PROMs for bipolar disorder. At project commencement, two service user–only focus groups were held. Additionally, we paid specific attention to recovery measure studies that involved service users in their development and evaluation. A limitation of the final set is redundancy in some items across measures. For example, sleep is assessed in the PROMIS-Sleep measure and in the depression and mania measures. This commonly encountered issue could be addressed in future research by using statistical methods to address overlap across the entire outcome set. Consistent with a systematic review of PROMs and CROMs for assessing youth outcomes (62), we found broader outcome measures developed for adolescents, including measures for quality of life. Targeted measures for symptoms and treatment side effects were often developed for adults and rarely tested with or adapted for adolescents. This research gap highlights the need to validate outcome measures for the adolescent population.

Important psychometric properties were not included in our selection criteria. These properties included meaningful change thresholds (63) and severity thresholds, such as mild, moderate, and severe, which can be linked to treatment decisions (64). The properties were present to varying degrees in the selected measures, but they were not used as selection criteria. A critical evaluation of each measure was therefore beyond the scope of this project. We did not address alternative modes of PRO administration, an important consideration in implementation. However, a meta-analytic review concluded substantial evidence indicating the equivalence of computer- and paper-administered PROs (65).

Conclusions

A standardized set of nine PROMs was identified in this study. The set can be used to support measurement-based care and, in combination with risk adjustment factors, to compare program outcomes. Finally, the set can be used to support development of value-based health care for people with psychotic disorders.