Assessing the Recovery Assessment Scale Across Time

Recovery has become a central tenet of mental health services, fostering “a process of change through which individuals improve their health and wellness, live a self-directed life, and strive to reach their full potential” (1). Under this guiding vision, recovery-oriented services need clear operationalization of the recovery construct and a set of measurable milestones to evaluate service effectiveness. Without reliable and valid recovery measures, effectiveness of recovery-oriented practices cannot fully be tested. Thus, establishing psychometrically sound recovery measures is essential for advancing mental health practice, research, and policy.

Several recovery measures have been developed and systematically reviewed to evaluate their utility (2–4). Among the existing recovery measures, the Recovery Assessment Scale (RAS) (5) has been identified as one of the most used in recovery-oriented practice evaluation, and its psychometric properties have been extensively studied (6, 7). The RAS has good reliability (i.e., internal consistency, test-retest reliability, interrater reliability), validity (i.e., consistent factor structure across different samples, expected associations with relevant constructs), and utility for intervention (6).

However, one critical piece of information that is missing is the longitudinal factorial invariance of the RAS, showing that the recovery construct measured by the RAS is consistent over time. If this assumption fails, the observed changes in score do not represent the true changes in the same recovery construct. For example, if the recovery experience has different meanings across time, the measurement score changes may not be comparable in the same structure or metric. Therefore, longitudinal factorial invariance is required to ensure that the RAS measures the same recovery construct over time, which is a prerequisite for longitudinal assessments of change (8). The current study tested the longitudinal factorial invariance of the RAS with data longitudinally collected at three time points from people with severe mental illness. Given the growing emphasis on measurement-based practices (9), this study could increase confidence in the RAS as a tool for longitudinal evaluation of recovery-oriented practice.

Methods

We conducted secondary analyses of data originally collected from a study on CommonGround, an intervention designed to increase shared decision making in psychiatric treatment (10) (see the original study for the details). Consumers were recruited during their psychiatric visits at a community mental health center. A total of 167 consumers participated in the study (pretest-posttest with a follow-up experimental-group-only design). Most of the participants were male (N=95, 57%), Black or African American (N=100, 60%), and had completed high school or some college (N=97, 58%). The data were collected before the intervention and 12 and 18 months after the intervention. Because of study dropout, 167 (baseline), 105 (12-month follow-up), and 83 (18-month follow-up) participants were available for the analyses. All procedures for the original study were approved by the Indiana University Institutional Review Board.

The RAS was developed through consumer narratives and item reviews to capture self-perceptions of a sense of recovery for people with psychiatric disabilities (5). Five factors with 24 items were identified through exploratory and confirmatory factor analyses (4, 6). The five factors tap distinct domains and are correlated with psychosocial (e.g., empowerment, hope, quality of life) and symptom variables. The items use a 5-point scale ranging from 1, strongly disagree, to 5, strongly agree. For the current study, Cronbach’s alphas for each RAS domain were as follows: personal confidence and hope, 0.86–0.89; willingness to ask for help, 0.86–0.88; goal and success orientation, 0.79–0.83; reliance on others, 0.75–0.77; and no domination by symptoms, 0.54–0.75. The alphas varied slightly across different time points.

Measurement and structural invariance testing (11) of the RAS was conducted to evaluate whether the recovery concept measured by the RAS is consistent over time (time 1, baseline; time 2, 12-month follow-up; and time 3, 18-month follow-up). We created parcel-level indicators (i.e., averaging item scores within each domain) that represent the recovery construct (i.e., five indicators under the recovery construct). A parceling method can establish a parsimonious model and reduce the chance of residual correlations, dual loadings, or sampling errors that are often present in item-level data (12). Invariance testing was conducted sequentially (from the least to the most restrictive models), in the following order: configural longitudinal invariance (the factor structure is identical over time), metric invariance (corresponding factor loadings are equal, and the common factors have the same meaning over time), scalar invariance (corresponding parcel-level indicator intercepts are equal, and no different additive influences exist as a result of time that systematically act to raise or lower response patterns over time), factor variance and covariance invariance (corresponding factor variance and covariance are equal, and the recovery construct is structurally comparable over time), and factor mean invariance (latent factor mean is equal, and the mean of the recovery construct is consistent over time).

Each model was evaluated with the multi-index approach (13), on the basis of the comparative fit index (values >0.95 are preferred), Tucker-Lewis Index (values >0.95 are preferred), and root mean square error of approximation (values <0.05 are preferred). The appropriateness of the parameter constraints of the RAS over time was assessed by means of a chi-square difference test between nested models (the likelihood ratio test), sequentially testing the least to most restrictive models of invariance. If the difference is statistically significant (a significant drop in model fit), the model constraints are not considered tenable (the corresponding parameters are not equal across time). When the constraints were not tenable, we used modification indices to obtain partial invariance. Analyses were performed using Mplus, version 7.11. The full information maximum likelihood estimation method was used to accommodate missing data.

Results

The results are summarized in Figure 1 and the online supplement to this report. First, the configural invariance model revealed good model fit (the factor structure of the recovery construct was the same over time). Second, the metric invariance model revealed good model fit. A significant drop in model fit from the configural invariance model was not observed (the corresponding factor loadings were equal over time). Third, the scalar invariance model showed significant worsening of fit compared with the metric invariance model (p<0.01). On the basis of the modification indices, the invariance constraint on the intercept for “no symptom domination” at time 1 was relaxed. The parcel-level scores at times 2 and 3 were significantly higher than those at time 1. Reexamining fit with the partial scalar invariance model showed good model fit, with no significant drop in model fit from the scalar invariance model (p=0.57). Fourth, the factor variance and covariance invariance model revealed good model fit, with no significant drop in model fit from the partial scalar invariance model (p=0.48). Finally, the factor mean invariance model showed that the mean of the recovery latent construct (alpha) was higher at time 3 (α=0.20) than at time 1 (α=0).

Discussion

This study tested the longitudinal factorial invariance of the RAS by using data collected longitudinally at three time points. We confirmed the viability of the scale for longitudinal assessment, with supporting evidence of several indices.

The configural and metric invariances indicate that the recovery construct measured by the RAS is consistent in terms of the components (five factors): the RAS captures the same conceptual meaning of the recovery construct over time. The finding is similar to prior research testing the factor structure of the RAS across different samples that shows the consistency of components across groups of people (6). Our analysis extends this consistency across time—that is, it affirms that the RAS measures the same recovery construct longitudinally.

We confirmed the scalar invariance of four of the subscales (people’s response patterns for these domains are equal across time) except for “no symptom domination” (which met criteria for partial scalar invariance). People may evaluate “no symptom domination” differently across time (it may become easier or more difficult to obtain a high score on the scale’s domain items at different time points). An interesting finding is that “no symptom domination” is the only domain that uses time-relevant wording (e.g., “My symptoms interfere less and less with my life”). People may weigh their symptom experiences differently across different time periods of the recovery journey. Although we cannot make direct comparisons, previous research has also found that this domain functions a little differently among the RAS domains (i.e., it is more sensitive to change) (14). Another possibility includes the potential lack of unidimensionality. The symptom domain consists of only three items, and its internal consistency is lower than that of the other four RAS domains, including in our sample. This factor might introduce inconsistent item-response patterns within the domain.

Finally, we confirmed the factor variance and covariance invariance of the RAS, showing that the variance of the RAS does not significantly vary across time. Together, the series of invariance testing indicates that the recovery construct measured by the RAS is comparable across time, which allows for rigorous tests in longitudinal assessment. In our data, the mean recovery scores increased from baseline (time 1) to 18 months (time 3) at the construct level, which is consistent with the manifest level outcome assessment in the original study (10).

This study had some limitations. First, we used a sample participating in an intervention study from one community mental health center, which limits the study’s generalizability as well as some conclusions that might be drawn. For example, we cannot determine whether the partial scalar invariance of “no symptom domination” was due to the intervention (which might change the item-response patterns after the intervention) or to the nature of the domain itself (e.g., symptom experiences in recovery are influenced by time factors). Second, the available sample size was limited and, along with attrition, might compromise the power and parameter estimates. Although the five RAS domains are correlated with one another, each domain also represents a unique construct (15). Securing larger samples will permit additional assessments of each subconstruct in future studies.

Despite these limitations, this study has important implications for the field of mental health recovery evaluation. The RAS is the most used measure in recovery-oriented research and practice evaluation, with the presumption that it can capture the same recovery construct across time. Our study empirically tested this assumption, providing evidence that RAS scores can be compared over time with consideration that the symptom domain may function a little differently. We are not aware of any other recovery measures whose longitudinal factorial invariance has been tested. The importance of measurement-based practices has been emphasized, especially for routine use of symptom measures (9). Recovery is a personal and individualized phenomenon, yet our findings suggest that the RAS can also reliably measure recovery over time, which is an important measurement trait that can inform recovery-oriented practices at the client, provider, agency, and policy levels.

Conclusions

Our study empirically confirms that the RAS measures the same recovery construct over time, which is an important prerequisite for longitudinal assessment. Personal recovery and clinical recovery are both important constructs, and the symptom domain may be an important domain of the RAS that bridges them; thus, further examination of this domain in longitudinal assessment may be important.