Personal Recovery Among Service Users Compared With Siblings and a Control Group: A Critical Note on Recovery Assessment

A recovery-oriented approach has become increasingly prominent in mental health care services throughout the world (1–5). One way to boost the implementation of a recovery-oriented practice is by including routine outcome measures assessing recovery (6–8). “Personal recovery,” as opposed to “clinical recovery” (9), has often been defined as living a meaningful and contributing life beyond the catastrophic effects of the illness (10–12). In this definition, recovery clearly applies to people with a psychiatric disability.

However, the five core processes of recovery, as identified in the empirically based CHIME framework (13), are not service user–specific. These core processes—connectedness, hope and optimism about the future, identity, meaning in life, and empowerment (13)—can also be considered relevant to nonusers of services. Any individual, whether they experience psychiatric symptoms or not, can feel variation in hope, optimism, empowerment, strength of identity, and sense of meaning in life. Nevertheless, the extent to which these core processes of recovery are rated differently by service users with a mental illness and persons without a mental illness has never been investigated. The primary aim of this study was to investigate whether recovery among service users with a psychotic disorder, both those in remission and not in remission, differs from that of their siblings and members of a control group and, if so, to describe how the groups differed. Recovery was measured with the Recovery Assessment Scale (RAS) (14, 15), and all participants were part of a large Dutch sample. Because the Dutch version of the RAS had not yet been validated, we first examined the psychometric qualities of the Dutch RAS.

Methods

Participants

Data derive from the second follow-up measurements of the 2004–2013 longitudinal Genetic Risk and Outcome in Psychosis (GROUP) study (16). At baseline, the full sample consisted of 1,119 service users with nonaffective psychotic disorder, 1,057 siblings, and 589 persons serving as a control group. Inclusion criteria for service users were ages 16 to 50 years, diagnosis of nonaffective psychotic disorder, and good command of the Dutch language. Inclusion criteria for siblings and the control group were ages 16 to 50 years, no lifetime psychotic disorder, and good command of the Dutch language. For the control group, an additional inclusion criterion was having no first-degree family member with a lifetime psychotic disorder, as established by the Family Interview for Genetic Studies (17). DSM-IV diagnoses were assessed in all three groups with the Comprehensive Assessment of Symptoms and History interview (18) or the Schedules for Clinical Assessment for Neuropsychiatry (19). The diagnoses for service users were schizophrenia and related disorders (N=945, 84%), other psychotic disorders (N=149, 13%), and psychotic illness in the context of substance abuse or somatic illness (N=25, 3%). At the second follow-up, 6 years after baseline, the GROUP sample consisted of 660 service users with nonaffective psychotic disorder, 710 siblings (unaffected by a psychotic disorder), and 394 persons in the control group (white, N=1,560, 89%; black, N=26, 2%; Asian, N=2, <1%; and mixed race-ethnicity, N=176, 10%).

Ethical Statement

The study protocol was approved centrally by the Ethical Review Board of the University Medical Centre Utrecht and subsequently by local review boards of each participating institute and was conducted in accordance with the Helsinki Declaration of 1975, as revised in 2008. All study participants provided written informed consent.

Measures

We used the 24-item RAS (15) with the factors of personal confidence and hope, willingness to ask for help, goal and success orientation, reliance on others, and no domination by symptoms. The RAS has a 5-point agreement scale, with higher scores indicating more recovery. We added the response option of “not applicable” to the items that loaded on the last factor (items 15 [”Coping with mental illness is no longer the main focus of my life”], 16 [“My symptoms interfere less and less with my life”], and 17 [“My symptoms seem to be a problem for shorter periods of time each time they occur”])—given that this factor could have been irrelevant to the siblings and members of the control group. The English RAS-24 was translated to Dutch by two researchers (LvdK, SS) and back-translated by a qualified translator.

The World Health Organization Quality of Life–Bref (WHOQoL-Bref) (20), a 26-item self-report questionnaire, assesses four domains of subjective quality of life over the past 2 weeks—the physical, the psychological, the social, and the environmental. Higher scores on the WHOQoL-Bref indicate higher levels of subjective quality of life.

The Positive and Negative Syndrome Scale (PANSS) (21), including the positive, negative, and general psychopathology subscales, assesses symptom severity. PANSS scores were also used to distinguish between service users in remission and those not in remission, given that for some service users, symptoms were of such low intensity that they no longer interfered significantly with daily functioning (22). Remission was operationalized as a PANSS score of 3 or less on the PANSS items for delusions, unusual thought content, hallucinatory behavior, conceptual disorganization, mannerism/posturing, blunted affect, passive/apathetic social withdrawal, and lack of spontaneity and flow of conversation for a minimum of 6 months (23).

The Social Functioning Scale (SFS) (24) is a 76-item, self-report instrument measuring withdrawal, interpersonal behavior, prosocial activities, recreation, independence-performance, independence-competence, and employment/occupation. Higher scores on the SFS indicate higher levels of social functioning.

The Camberwell Assessment of Need (CAN) (25, 26) was used to assess the need for care on a 3-point scale (no need, met need, and unmet need). In the GROUP study, the original 22 CAN items were extended by two additional items for paid work and undesired effects of medication. In this study, we focused on the number of unmet needs.

The Recovery Style Questionnaire (RSQ) (27) consists of 39 dichotomous items used to distinguish between individuals with an integration recovery style, characterized by active attempts to manage the illness, and those with a sealing-over recovery style, characterized by cognitive and behavioral avoidance of the illness. Higher scores on the RSQ indicate a sealing-over recovery style.

Analysis

GROUP database version 5.0 was used for all analyses. All statistical analyses were performed by using SPSS software for Windows, version 24.0, and MPlus, version 6.12 (for factor analysis only).

Psychometric evaluation.

Internal consistency of the RAS was investigated by using the Cronbach’s alpha coefficient (α≥0.9=excellent, 0.9>α≥0.8=good, 0.8>α≥0.7=acceptable, 0.7>α≥0.6=questionable; 0.6>α≥0.5=poor; and α<0.5=unacceptable (28). To measure the correlation between RAS items and the total scale, we used the criterion that corrected item-total correlation should not be below 0.3 (29). Internal consistency was calculated for service users, siblings, and members of the control group separately (30, 31).

Concurrent validity of the RAS was investigated by relating recovery to quality of life, a concept that is closely related to recovery (14). A positive association between the RAS and the WHOQoL-Bref was expected. Divergent validity of the RAS was examined by relating recovery to severity of psychotic symptoms measured with the PANSS. The RAS is said to measure ”personal” recovery, whereas the PANSS assesses symptom alleviation and can be considered a measure of “clinical” recovery. A trivial to small association between the RAS and the PANSS was expected. For the service user sample only, Pearson’s correlation coefficients were calculated for the correlation between total score on the RAS on the one hand and the subscale scores of the WHOQoL-Bref and the PANSS on the other hand. Correlations were interpreted according to Cohen (32): a coefficient of 0.10 is considered small; 0.30, moderate; and 0.50, large.

In addition, we explored associations between the RAS and the SFS, CAN unmet needs, and the RSQ for the service user sample. We expected a positive association between the RAS and the SFS and trivial to small associations between the RAS and CAN unmet needs and the RSQ, given that the RSQ measures recovery style and the RAS measures the degree of recovery.

A confirmatory factor analysis (CFA) was performed on the 24 RAS items for service users only (15), using Mplus 6.0. Model fit was assessed by using the root mean square error of approximation (RMSEA) and the Tucker-Lewis Index (TLI). An RMSEA below .06 and a TLI close to .95 can be considered good (33).

Comparisons.

The RAS total score and the other factor scores were compared between service users not in remission, service users in remission, siblings, and control group members in terms of statistical significance, effect size, and clinical significance. To determine statistical significance we used analysis of variance. Cohen’s f, defined by the ratio of population standard deviations (32), was calculated as an effect size measure of mean difference. According to Cohen, values of 0.10, 0.25, and 0.40 represent small, moderate and large effect sizes, respectively (32). Clinical significance was calculated by comparing the ratio of recovered versus not recovered persons between the four subgroups, using the three criteria provided by Jacobson and Truax (34). With criterion A, an individual moves above two standard deviations (SDs) from the mean of the reference population (or “dysfunctional” population, the term used by the original authors). In this study, recovery was indicated by whether the RAS score was greater than the mean score among services users not in remission +2 SDs. With criterion B, an individual moves below two SDs of the mean of the “functional population” (original authors’ term). In this study, recovery was indicated by whether the RAS score was lower than the mean score of the control group members –2 SDs. With criterion C, an individual is closer to the mean of the “functional population” than to the mean of the reference population ([criterion A + criterion B]/2).

Which criterion is best depends on one’s standards (34). We chose to use all three criteria and compare the results to get a comprehensive view on the data. For the reference population, the subgroup of service users not in remission was used. In all comparison analyses, the factor “no domination by symptoms” was left out because the items in this factor were considered potentially inapplicable to the siblings of service users and the members of the control group.

Results

A total of 581 service users, 630 siblings, and 372 control group members completed the RAS. This was 88%, 89%, and 94%, respectively, of the GROUP sample at the second follow-up measurement. Of the 561 (97%) service users for whom PANSS remission items were available, 336 (60%) were in remission and 225 (40%) were not in remission.

Psychometric Evaluation

Internal consistency.

Cronbach’s alpha coefficient for the total score of the RAS was 0.90 for service users, 0.90 for siblings, and 0.92 for persons in the control group, which can be considered excellent. For the service users and siblings, one item (“I have a desire to succeed”) had a corrected item-total correlation of 0.28, which is below the criterion (α=0.30). However, dropping the item would hardly increase the overall reliability of the factor (if item deleted, α=0.90 and 0.91, respectively). Internal consistency scores of the other RAS factors were good to acceptable (see online supplement).

Concurrent and divergent validity.

Among service users, the RAS total score showed a large positive correlation with the WHOQoL-Bref psychological subscale and a moderate positive correlation with the WHOQoL-Bref physical, social, and environmental subscales. A small negative correlation was found between the RAS total score and the PANSS subscales. (The correlation coefficients are available in an online supplement.)

Exploratory associations.

Among service users, the RAS total score showed small to moderate positive correlations with the SFS subscales and total scale and small negative correlations with the CAN unmet needs and with the RSQ (see online supplement).

Factor structure.

Results of the CFA for service users showed a RMSEA of 0.047 (90% confidence interval=0.041–0.052), indicating a good model fit. The TLI was 0.910, which can be considered a reasonable model fit (see online supplement for details about the CFA).

Comparisons

Statistical significance and effect sizes.

A one-way, between-subjects analysis of variance was conducted to compare the RAS scores between service users not in remission, service users in remission, siblings of service users, and persons in the control group. A significant and moderate difference was found between the four groups for RAS total score (F=31.73, df=3 and 1,559, p<0.001; Cohen’s f=0.25) and personal confidence and hope (F=44.91, df=3 and 1,559, p<0.001; Cohen’s f=0.29). A significant and small difference was found between the four groups for goal and success orientation (F=10.77, df=3 and 1,561, p<0.001; Cohen’s f=0.14) and reliance on others (F=28.67, df=3 and 1,560, p<0.001; Cohen’s f=0.23). A significant but trivial difference was found between the four groups for willingness to ask for help (F=4.27, df=3 and 1,557, p=0.005; Cohen’s f=0.09). Post hoc comparisons, using the Tukey honestly significant difference test, indicated that for all RAS factors, except willingness to ask for help, the mean score for both service user groups was lower than the mean score for siblings and control group members, whereas the scores for the latter two groups did not differ from each other. Moreover, for all factors, the mean score for service users not in remission was lower than the mean score for service users in remission (Table 1).

Clinical significance.

The clinical significance of the group differences in RAS total score (excluding the factor for no domination by symptoms) and in personal confidence and hope, the most discriminating factor, is presented in Tables 2 and 3. For the RAS total factor and the personal confidence and hope factor, the proportion of persons who had recovered was significantly lower among service users not in remission, the reference group, than among siblings and control group members, according to criteria B and C, but not criterion A. However, the proportions of persons in the four groups who had or had not recovered was rather remarkable. According to criterion A, almost none of the participants, regardless of group status, had recovered. According to criterion B, almost all participants, regardless of group status, had recovered. According to criterion C, 50% to 70% of the service users had not recovered, but almost 40% of siblings and control group members also had not recovered. For the remaining factors of the RAS, the application of criteria A, B, and C to the other subscales showed even more extreme differences between the four groups in the proportion of persons who had recovered (see online supplement).

Discussion and Conclusions

Psychometric Quality

The Dutch version of the RAS showed high overall internal consistency and satisfactory concurrent and divergent validity. Higher recovery levels coincided with higher quality of life and were associated, although not strongly, with lower levels of psychiatric symptoms. This is in line with previous research (14, 35, 36). In addition, higher recovery levels were associated with better social functioning, with less unmet care needs, and with an “integration” recovery style, although the latter two effects were very small. The original factor structure of the RAS is a reasonable to good fit with our data. These psychometric findings demonstrate that the Dutch translation of the RAS assesses personal recovery as adequately as other RAS translations (35, 36). Yet, that is not to say that the RAS is a meaningful instrument for routine outcome monitoring.

Key Findings

Our comparison between RAS scores of service users not in remission, service users in remission, siblings, and members of a control group shines a new light on this frequently used measure of recovery. To find out whether the RAS discriminates between service users, siblings, and control group members, we compared the groups in terms of statistical and clinical differences. The differences in recovery scores between service users versus siblings and control group members were mostly statistically significant with trivial to moderate effects. However, going beyond the analysis of variance and zooming in on the clinical significance of differences on the RAS between service users, siblings, and control group members, we found that the differences between the four groups, for the factors evaluated, were less clear.

Criterion C, the least arbitrary criterion according to Jacobson and Truax (34), distinguished most clearly between service users and siblings and control group members. Looking at the RAS total score, we found that 36% of service users who were not in remission and 48% of service users who were in remission had recovered, according to this criterion, compared with 63% of siblings and 64% of control group members. In other words, 64% of service users who were not in remission, 52% of service users who were in remission had not recovered, but this was also true of 37% of siblings, and 36% of control group members. These findings indicate that the RAS total factor discriminates between service users and nonservice users if, and only if, the service users score at the lower extreme of the RAS.

Implications

On the one hand, the fact that we did not find large differences in recovery scores between service users, siblings, and control group members may be considered encouraging. For instance, the finding that service users did not differ from their siblings and control group members on the factor for willingness to ask for help suggests that service users with symptoms of a mental disorder can be just as assertive in asking for help as other people, although the kind of help they are looking for or the resources involved are probably different. On the other hand, the similarity in the proportion of recovered persons among service users, siblings, and control group members raises questions about the usability of the recovery construct as measured with the RAS. If service users report a recovery level similar to that of siblings and control group members, what is the meaning of recovery? It does not make sense to expect recovery scores to rise more among service users than among control group members. For service users who score very low on personal recovery, the RAS, especially the factor for personal confidence and hope, may be useful for measuring progress on personal recovery throughout the clinical care trajectory. Its sensitivity to change should be investigated in a future study.

Nonetheless, for most service users, the RAS will probably not detect changes in personal recovery. Therefore, if we want to include recovery measures in routine outcome assessment, as has been suggested (6–8), the results of this study indicate that including the24-item RAS would be of questionable usefulness. An alternative method may be to use more personalized assessment procedures, which measure change in terms of subjective experiences relevant to the person involved, such as the Discan approach (37).

Strengths and Limitations

This is the first study that compares recovery ratings among service users and nonusers. Moreover, the study was conducted in a large national sample. A limitation was that we could not evaluate the sensitivity to change of the RAS and test-retest reliability. Furthermore, in the comparison between service users, siblings, and control group members, we had to exclude the RAS factor for domination by symptoms, given that the items of this factor were less relevant to siblings and control group members. In fact, this factor may be more service user–specific and, therefore, discriminate better between groups. Finally, it may be argued that the service users included in this study form a relatively healthy group, given that they managed to participate in the second follow-up of the GROUP study, 6 years after baseline. Service users with a poorer condition may have dropped out. However, we accounted for this potential limitation by distinguishing between service users in remission and those not in remission.