Advancing Recovery Science: Reliability and Validity Properties of the Recovery Assessment Scale

Recovery is the driving philosophy underlying the development of national, state, and local mental health policy and services and promises to remain a guiding influence for decades to come. This recovery era requires a new generation of measures to be added to the current mental health measurement toolbox. Current recovery-related measures that have been discussed in various studies and compendia (1–5) fall into three broad categories. First, measures of the subjective experience of recovery are those that focus on the various ways in which people experience recovery and the recovery process. These measures typically assess the consumer-oriented perspective on recovery rather than taking a clinical perspective that focuses on reduced symptoms, increased functioning, and so on as indicators of recovery (6). A consumer-oriented perspective includes the extent to which people are living a satisfying, fulfilling, and hopeful life and developing meaning and purpose regardless of the presence of mental health issues.

Second are recovery-oriented attitudes, beliefs, and knowledge measures. These measures examine attitudes about the extent to which respondents (consumers, providers, and others) are favorable toward the idea that achieving recovery is possible. These measures also assess knowledge and beliefs about recovery and recovery-oriented practices. Third are measures of respondents’ perceptions of recovery-promoting environments, which include their opinions about the extent to which recovery-oriented policies, programs, and practices are in place.

Recovery-oriented research is still at a relatively early stage, especially compared with research examining other constructs, such as symptoms or cognitive functioning. And just like what happened in the early days of clinical outcome studies, thorough development and examination of evidence will be important to ensure that these measures meet rigorous measurement standards. This scrutiny will help researchers select credible recovery instruments for use in future studies, assist policy makers and providers who are also using various types of recovery measures to evaluate the effectiveness of policies and programs associated with recovery-oriented transformations, and heighten confidence in the results of these studies in order to advance recovery science.

This study examined the psychometric properties of what is plausibly the most commonly used recovery measure in the published literature, the Recovery Assessment Scale (RAS). The RAS was developed by mental health consumers in the mid-1990s, relatively early in the recovery era, through an analysis of recovery stories that resulted in the identification of 39 themes of the subjective experience of recovery. These themes were used to create the original 41-item measure (7). Using data from more than 1,800 respondents, a factor analysis of the original 41-item version identified five factors: personal confidence and hope, willingness to ask for help, goal and success orientation, reliance on others, and not being dominated by symptoms (8).

Several narrative reviews of recovery measures have been highly favorable toward the RAS. In a study by Law and colleagues (9) the RAS and five other measures met three criteria out of 25 recovery-oriented measures they examined: scrutinized in a peer-reviewed journal; offers a self-report questionnaire that yields quantitative results, thus enhancing its clinical utility; and aims to measure factors relating to what the researchers called “personal recovery.” The RAS was described as the most acceptable and valid measure available, although authors stated that no gold-standard measure of recovery had yet been developed. Burgess and colleagues (3,10) identified the RAS as the most suitable of four scales for measuring recovery concepts in Australia because it meets all nine criteria that they established. Cavelti and colleagues (4) concluded their review by describing the RAS as possibly the top measure of recovery available. And the RAS is listed in an assessment guide as the only highly recommended “subjective appraisal” recovery scale based on its adequate test-retest reliability, construct validity, treatment sensitivity, and clinical utility, with good internal consistency and content validity and excellent validity generalization (11). Finally, another review concluded that the RAS can facilitate dialogue between consumers and clinicians and is relevant to consumers’ lives. The review also noted that consumers expressed willingness to complete the assessment again and that clinicians thought that the instrument was better than other recovery scales at capturing illness management (5).

This study is the first to offer an in-depth review of the quantitative properties of the RAS to complement the aforementioned narrative reviews. The specific questions we set out to answer are as follows: Is there substantial evidence to conclude that the RAS has relatively consistent means and standard deviations and reliability across populations? What is the nature of the evidence pertaining to construct validity of the RAS? Is the RAS sensitive to change in intervention research?

Methods

A search was conducted in August 2012 on Google Scholar and PubMed with the search term “Recovery Assessment Scale.” This search resulted in the identification of 222 documents. From that list, we excluded articles for the following reasons: duplicates or translations (N=26); articles could not be translated from other languages by the authors (N=5); there is no discussion of the RAS or the article refers to a neurological recovery measure (N=66); the article does not present results or was not published in peer-reviewed journals, dissertations, book chapters, books, or compendia (N=19); the article reviews the concept and measurement of recovery but presents no data (N=16); and the article mentions the RAS, but the study used another measure of recovery for data collection (N=13). The remaining 77 articles that report use of the RAS in data collection and present results are included in this review.

These 77 studies came from 11 countries: United States, Australia, United Kingdom, Japan, Canada, Portugal, Spain, Israel, Sweden, Switzerland, and Taiwan, with most studies being from the United States and Australia. The sample sizes varied greatly across the studies, from a minimum of ten to a maximum of 1,827. Most studies used the RAS-41 or RAS-24 versions, but some used variations, such as the RAS-20, RAS-22, RAS-42, and RAS-50. Most studies used a 1–5 Likert scale, where 1 is strongly disagree and 5 is strongly agree. Some studies calculated scores as a sum across all items, whereas others presented means.

Because studies used different versions of the RAS, different response scales for the items, and different ways of computing composite scores, we needed to convert the results to allow for comparisons across studies. Summed results were converted to average scores by dividing the means and standard deviations of the reported summed scores by the number of items on the version of the RAS that was used. For instance, a RAS-24 sum score of 84 would be equivalent to an average score of 84/24=3.5. Also, if studies used a 0–4 Likert scale versus a 1–5 scale, we added 1 to the mean score; standard deviations remained the same, as shifting by a constant does not affect variance. In addition, there were several studies that did not report mean scores for the entire sample but reported them separately for different groups (experimental group and control group, for example). In those instances, we calculated weighted overall means, using sample size of groups as weights, and pooled standard deviations for the entire sample.

Results

Means

Twenty-eight of the 77 studies reported RAS means and standard deviations, the RAS version, and the response scale used for the items. The means in the 28 studies ranged between 3.14 and 4.12, according to the relatively standard scoring system of 1 (low) to 5 (high), and the interquartile range (IQR) was 3.72–3.90. The average of all reported mean scores was 3.78±.19. Figure 1 shows the distribution of the means and standard deviations.

Reliability

Nineteen of the 77 studies reported on the internal consistency of the RAS (7,12–29). Cronbach’s alphas ranged between .76 and .97. Four studies reported test-retest reliability ranging from .65 to .88 (7,13,21,30). Two studies examined the inter-interviewer reliability of the RAS and found intraclass correlation coefficients to be .94 and .98 (31,32).

Factor structure

Corrigan and colleagues (8) were the first to subject the RAS to a factor analysis. Using data from 1,824 individuals with serious mental illnesses, they found a five-factor solution that incorporated 24 of the 41 items on the scale. The five factors were personal confidence and hope, willingness to ask for help, goal and success orientation, reliance on others, and no domination by symptoms. The five other studies that used American, Australian, and Japanese samples to conduct exploratory factor analysis or principal-components analysis (13,16,21,27,33) identified the same or highly similar factors or principal components. In addition, multiple other studies used these factors as subscales without rechecking the factor structure. A dozen studies (8,16,24,28,29,33–39) reported acceptable or good internal consistency (Cronbach’s α >.7) for most of these subscales, although some of the subscales in these and other studies had alphas near the lower limit of acceptability (α=.58–.70) (16,24,28,29,34,39,40).

Validity

A total of 49 studies examined point-in-time associations between RAS and other constructs; 47 of the 49 reported simple bivariate associations (Pearson or Spearman correlations, t tests, or one-way analyses of variance), and 14 used the RAS in a regression analysis (12 as a dependent variable and two as a predictor). Table 1 summarizes these studies.

Characteristics examined

Psychological well-being (12 studies).

Studies reported significant positive associations between measures of individuals’ quality of life, meaning of life, empowerment, self-esteem, sense of mattering, hope, and resilience and the RAS total score or specific RAS subscale scores (7,18,21,28,34,36–38,41–44).

Other recovery measures (11 studies).

The studies that looked at the associations between the RAS and other putative recovery measures, including those attempting to assess stages of recovery, found positive moderate correlations (30,33,36,45–52).

Social functioning and support (nine studies).

Higher levels of social functioning, social network size, and greater social support were associated with higher RAS scores (7,13,23,25,42,53–56).

Psychiatric symptoms, distress, and poor health (eight studies).

Greater symptoms and distress, as well as poor health, were associated with lower scores on the RAS (7,21,30,33,34,40,56,57).

Stigma (four studies).

Perceived stigma and self-stigma were found to be negatively correlated with the RAS (15,23,39,42).

Community participation (four studies).

Greater self-determination and motivation for leisure, community participation, and activity space area, and ability to perform activities of daily living and importance assigned to these activities were positively associated with the RAS (14,16,58,59).

Perceived community inclusion (three studies).

Greater sense of belonging and greater perceived community integration corresponded to higher RAS scores (14,34,41).

Positive coping (three studies).

Individuals who had a more positive outlook on their illness and were less affected by negative events had higher RAS scores (36,57,60).

Religiosity and religious support (three studies).

Findings were mixed when it came to religiosity and religious support. In one study, RAS scores were found to be higher for those who considered themselves to be spiritual and religious (61), and in another study there was a positive association between RAS and perceived support from the congregation and God (25). However, a third study showed that lower RAS scores corresponded to greater religious support and religiosity (19).

RAS and other constructs (15 studies).

Studies have shown that the RAS total score or RAS subscale scores are positively associated with the relationship between consumers and their mental health providers (38); active involvement in treatment (36); acknowledgment of the need for change, as assessed by the Stages of Change Questionnaire (16,34); behavior activation (62,63); use of psychological acceptance, defined as “actively contacting psychological experiences while behaving effectively” (64); and progress toward goal attainment (65). On the other hand, loss attributed to mental illness (defined as a reduction in tangible or intangible resources in which a person has a significant emotional investment) (18), financial deprivation (34), and unmet needs (41) were negatively associated with the RAS.

Inconsistent results were found between RAS scores and employment status: one study showed that consumers engaged in paid employment had higher RAS scores than those receiving Social Security benefits only (41); however, another article reported no significant difference in total RAS scores between workers and nonworkers (12). Two studies found a relationship between the RAS and diagnosis: one found that persons with substance dependence had higher RAS scores than those with a serious mental illness or co-occurring mental and substance use disorders (66), and another showed that persons with bipolar disorder had higher RAS scores than those with schizophrenia or depression (41). Furthermore, individuals receiving peer support (66,67) or partnered with community volunteers (68) were not shown to have higher RAS scores than those receiving treatment as usual. Finally, one study (69) found a marginally significant positive association between RAS scores and neighborhood socioeconomic status.

RAS and change over time

Six randomized controlled studies looked at changes in the RAS over time to examine the effectiveness of an intervention that aimed to improve ratings of subjective recovery. In three of the six studies, individuals in the experimental group compared with those in the control group had a greater improvement in RAS total or subscale scores. These interventions were Building Recovery of Individual Dreams and Goals through Education and Support (BRIDGES) (28), Wellness Recovery Action Planning (29), and the Recovery Workbook (70). There were no significant differences in RAS scores between the experimental and control groups in the other three studies, which examined the effectiveness of Illness Management and Recovery (IMR) (71), Internet peer support (72), and participation in treatment that combines traditional services with a religious and spiritual assessment (73).

RAS total and subscale scores were used as outcome variables in 12 pretest-posttest longitudinal studies. Eight of the 12 studies found an increase in RAS scores over time that was associated with achieving goals (65); participating in a peer-led BRIDGES education intervention (74); participating in Compeer, a program that matches people with a serious mental illness to community volunteers in order to increase social support (75); participating in the adapted version of group positive psychotherapy (“positive living”) for people with schizophrenia (26); participating in Australian Salvation Army Alcohol Rehabilitation programs for people with co-occurring disorders or with a substance use disorder only (24); participating in assertive community treatment (76,77); and participating in IMR (77,78). On the other hand, participating in a program that uses psychoeducation through the combination of sociodrama and e-learning (39), being in a peer education and support program for veterans (20), and completing homework as part of therapy (17,22) were not associated with changes in RAS scores over time.

Other quantitative analyses

Not discussed here are studies that used the RAS for other types of analyses, including structural equation modeling and path analysis (23,34,65,79–81), Rasch analysis (32,82), and an attempt to find a cutoff score to indicate achievement of recovery (83). These studies were not relevant to the key issues we reviewed.

Discussion

An impressive number of quantitative studies using the RAS have been published around the world since its emergence in the late 1990s, leading us to the following answers to our research questions. First, there is substantial evidence to conclude that the RAS has solid and consistent psychometric properties, as indicated in findings pertaining to internal consistency, test-retest and interrater reliability, and the mean±SD results presented in Figure 1.

Second, there is solid evidence supporting the construct validity of the RAS. RAS scores were positively associated with other recovery measures, psychological well-being, positive illness outlook, higher levels of social participation and support, sense of belonging and community participation, and a whole range of other constructs, including positive relationships with mental health providers and active involvement in treatment. RAS scores were negatively correlated with perceived and self-stigma, presence of psychiatric symptoms, distress, and poor general medical and emotional health, as well as other constructs such as financial deprivation, the presence of unmet needs, and loss attributed to mental illness.

Third, the RAS appears to be sensitive to change in intervention research. Three out of six RCTs showed changes on the RAS, and eight of 12 pretest-posttest intervention studies also showed statistically significant changes over time. Of course, such results beg the question of why changes were not found in all studies. This certainly could reflect a lack of sensitivity of the measure but also could be the result of the ineffectiveness of the intervention under study in producing changes on the measure. The sensitivity of the RAS could also be affected by plausible ceiling effects that may be evident in our examination of the cross-study means reported in Figure 1. Over the 28 studies that reported mean scores, the subjective experience of recovery appeared fairly high on average, although there was certainly room for change. For use as an outcome measure, it might be advisable for intervention studies to enroll individuals whose score falls below a certain baseline threshold on the RAS to increase RAS sensitivity in detecting statistically significant changes over time. The studies that did not find changes on the RAS may have enrolled people who scored relatively high in their recovery at baseline.

Significant effort was made to include all studies that reported RAS quantitative results. The fact that we eliminated nearly two-thirds of all articles identified with our search term suggests that we cast a broad net in our search. Nonetheless, some relevant articles may have been missed. We discovered that many versions of the RAS exist and are being used in research. Although we do not believe that this fact affected our conclusions, more work is needed to establish a standard version of the measure. Good psychometric properties were demonstrated with the RAS-20 and RAS-24 versions of the RAS; thus these shorter versions of the scale may be the most parsimonious and may best uphold confidence in the scale’s psychometric properties.

Conclusions

The RAS is likely the most commonly used recovery measure in research. The thorough review of the data reported here, along with the favorable comments about the measure made by others in narrative reviews, afford confidence that the RAS can be justifiably used in future research, evaluations, and clinical assessments to advance recovery science. Such a conclusion is important as mental health research increasingly seeks to examine the impact of interventions on the subjective experience of recovery. Researchers and policy makers have greater assurance that recovery can be reliably and validly measured, which will further increase research in this critically important area. On the basis of our findings, we recommend that researchers who use this measure in the future be more consistent in reporting psychometric information; the greater use of standardized approaches to the numbers of items used, scaling, and scoring will further ensure that the RAS is a sound measure.

The results of this review also support the use of the RAS in clinical settings, especially to assess one’s current state of recovery and improvement over time. Although we were unable to find reports about the use of the RAS in assessment, service planning, and individual-level evaluation in the peer-reviewed literature, such research could be useful to providers and consumers in making decisions about recovery-oriented outcomes that they would like to target. For example, the RAS subscales could be used to determine whether targeted efforts are needed to address personal confidence and hope, support people in seeking help from others, and enhance coping strategies to reduce the effects of symptoms on their quality of life. Overall, the RAS appears to be a valuable tool for enhancing recovery science and practice.