Measuring Clients' Satisfaction With Self-Help Agencies

The self-help movement represents one of the most significant developments in the provision of mental health services (1,2,3,4). Involving consumers as active participants in addressing their treatment needs has become a key element in health care practice (5).

Self-help agencies are mental health service organizations operated primarily by consumers with psychological disabilities. These agencies have been increasingly used under managed care as a source of inexpensive services. They are proliferating throughout the country, and state legislatures and private foundations are lending them increased support (6,7). Given cost-cutting trends, the self-help model seems to be one of the few components of the mental health system exhibiting consistent growth.

However, questions have been raised about the effectiveness and the types of outcomes that self-help agencies can deliver (8). As part of the effort to enhance treatment outcomes, clients' satisfaction with services has become one of the criteria most frequently used to evaluate program success. This paper describes the development of a scale measuring clients' satisfaction with being actively involved in a self-help agency and the services they receive.

Two of the instruments most frequently used in service satisfaction research, the Client Satisfaction Questionnaire (9) and the Service Satisfaction Scale (10), focus primarily on satisfaction with types of services. Ruggeri (11) reviewed 16 studies in which mental health organizations assessed clients' satisfaction with services as a whole and with specific service characteristics, such as the physical environment and types of intervention. All but two studies examined only satisfaction with the types of services rather than clients' satisfaction with their role in making care-related decisions. A similar focus was reflected in the majority of research surveyed for this paper (12,13,14,15,16,17).

Elbeck and Fecteau (18) conducted a factor analysis of a 50-item questionnaire about service satisfaction that was developed on the basis of focus-group discussions with psychiatric inpatients. They found that the items diverged into two main factors, behavioral autonomy and supportive care. The primary objective of the self-help agency is to help clients develop behavioral autonomy—to help them become actively involved in their care. The instrument described in this paper, the Self-Help Agency Satisfaction Scale (SHASS), is used to measure both clients' satisfaction with their involvement in the helping process—the autonomy dimension, and their satisfaction with the services received—the support dimension.

Methods

Subjects

This study is part of a larger investigation of users of mental health self-help agencies and the agencies themselves (19,20,21,22,23). Participants in the survey were long-term adult users of four client-run self-help agencies in the San Francisco Bay area. Staff—who were themselves clients—volunteers, and other members were interviewed between 1992 and 1993. Only participants who had used the self-help agency at least 12 times over a three-month period were included. During the month before the initial interview, the study participants visited the self-help agency an average of 16.6 times. At the six-month follow-up, they had visited the agency 13.1 times in the previous three months.

A total of 321 baseline interviews were initiated, and 310 were completed (97 percent). A total of 248 participants completed six-month interviews (80 percent of the original sample). No participant who could be located refused a follow-up interview.

The mean±SD age of the 310 participants was 38±8.44 years. A total of 222 (72 percent) were male, and 269 (87 percent) had confirmed DSM-III-R diagnoses. At the time of their first interview, 143 participants (46 percent) were living on the streets or in a shelter. The remaining 167 (54 percent) were often unstably housed.

Interviews were conducted by both mental health professionals and clients not currently active in the target agencies. The interviewers were trained at the Center for Self-Help Research at the Public Health Institute in Berkeley, California.

Study sites

The four self-help agencies were located in urban settings. They provided mutual support groups, drop-in space, survival resources, assistance in obtaining shelter, case management, financial planning, payeeship services, substance abuse and peer counseling, advocacy, employment assistance, and information and referral. Most paid staff members were drawn from the ranks of program clients or had shared similar experiences of poverty, homelessness, and institutionalization.

Volunteer jobs within the agencies were provided with the aim of offering members the opportunity to help others, to develop mainstream work habits, and to participate in organizational decision making. Staff, volunteers, and clients at all four agencies also engaged in a variety of ad hoc political and social-change activities.

Instruments

The Center for Self-Help Research developed a structured interview schedule for the study. The items were based on discussions between consumers in self-help agencies and mental health professionals. The items were reviewed by the center's consumer advisory group and the staff of the four agencies.

Four standard outcome measures were used to assess the predictive utility of the SHASS.

The Independent Social Functioning Scale. The Independent Social Functioning Scale (ISFS) is a modified version of Segal and Aviram's External Social Integration Scale (24), which measures "the extent to which an individual participated in and made use of the community in a self-initiated manner and without the help of others" (25). Segal and Kotler (25) reported construct validation of the ISFS and good internal consistency (Cronbach's alpha=.95).

The Assisted Social Functioning Scale. The Assisted Social Functioning Scale (ASFS) assesses the same types of community involvement as the ISFS while assessing whether the involvement was in some way enabled by a helper. The internal consistency of the ASFS has been reported to be good (Cronbach's alpha=.94) (25).

The Brief Psychiatric Rating Scale. The Brief Psychiatric Rating Scale (BPRS), developed by Overall and Gorham (26), is a symptom-based index that has frequently been used in drug trials (26,27) and was used by Segal and colleagues (19,24) in their studies of former psychiatric patients in board-and-care homes. Good interrater reliability has been reported (r=.9) (24), with internal consistencies varying from .79 (24) to .86 (25).

The Personal Empowerment Scale. The Personal Empowerment Scale (20) measures the degree of perceived control over common life activities, including obtaining shelter, income, and services. It also measures the ability to minimize the occurrence of unwanted events such as assault or homelessness.

Statistical methods

Item analysis for the SHASS. Factor analysis of item content related to satisfaction with various aspects of the self-help agency's program—including the physical environment, organization of services, staff competence and attitudes, types of intervention, and clients' involvement in the helping process—was used to construct the SHASS. The items used a 5-point Likert format, with 1 indicating very dissatisfied and 5 indicating very satisfied. The factor analysis, which employed principal-axis factoring with Eigenvalues set at 1, was conducted at baseline and at six months. Simple structure was sought via rotation with varimax and Kaiser normalization procedures. The communality estimate used was multiple R².

Reliability and stability analyses. Internal consistencies (Cronbach's alpha) and stability coefficients (Pearson's product-moment correlation coefficient) were computed across a six-month period.

Construct validation. One concern that has been raised about using satisfaction as an outcome measure is that respondents' ratings of satisfaction with the organization and services may be confounded by their global sense of well-being. To ensure that our measure of satisfaction with the agency was distinct from a respondent's general level of life satisfaction, we included a measure of satisfaction with quality of life based on three subscales—housing, income, and social network. We then factor analyzed the subscale scores of both the SHASS and the quality-of-life measure, using the same procedures described above, to determine whether the subscales represented different dimensions of satisfaction—that is, whether they demonstrated discriminant validity.

Predictive utility. We used partial correlation coefficients to determine the relationship between client satisfaction at baseline and each of the four outcomes measured at six months, after controlling for baseline status on each outcome measure.

Results

Item analysis

Table 1 shows that the item content on the SHASS breaks down into two distinct factors. The first measures satisfaction with active involvement in the agency, and the second measures satisfaction with the services or support received. Principal-axis factoring accounted for 60.7 percent of the variance extracted from the two factors at baseline and 66.8 percent of the variance at follow-up. Eigenvalues were equal to or greater than one. Varimax, with Kaiser normalization rotations to simplify structure, was achieved with three iterations. As Table 1 shows, the analyses yielded almost identical factors at baseline and follow-up. Results confirmed our two-factor hypothesis.

Reliability and stability analyses

SHASS. The SHASS includes 11 items. Six items constitute the service subscale and five the involvement subscale. Table 2 presents the reliability and stability characteristics of the full scale and both of its subscales. Internal consistencies varied from .87 to .90 at baseline and from .91 to .93 at follow-up. Stability coefficients varied from .44 to .61.

Quality-of-life measure.Table 2 also reports measurement characteristics of the quality-of-life measure and its three subscales. Internal consistencies varied from .79 to .90 at baseline and from .83 to .91 at follow-up. Stability coefficients ranged from .39 to .59.

Discriminant validity

Table 3 shows results of the factor analyses of the quality-of-life subscales and the SHASS subscales at baseline and follow-up, with factor loadings reflecting a rotated factor matrix. Both at baseline and at follow-up, two different factors emerged—a SHASS subscale factor and a quality-of-life factor. These results support the hypothesis that the SHASS measures a construct distinct from general life satisfaction.

Utility in predicting outcomes

The reliability and stability coefficients of the four status-outcome measures are reported in Table 2. Table 4 shows the zero-order correlations and partial correlations of the two SHASS subscales and the four status-outcome measures. Six of the eight zero-order correlations indicated a significant relationship between baseline satisfaction and follow-up outcome scores. However, only three relationships were significant when the analyses controlled for baseline scores on the outcome variable.

The partial correlation between scores on the SHASS service subscale and the ISFS, adjusted for baseline scores on the ISFS, was significant, as was the correlation with the ASFS score. For the SHASS involvement subscale, only the partial correlation with the ISFS was significant.

Discussion

The findings support the utility of a new measure of client satisfaction for use in self-help agencies and suggest the need for further research on the Self-Help Agency Satisfaction Scale to determine its usefulness in other mental health settings, including outpatient settings, community-based care, and inpatient settings. It was in an inpatient setting that Elbeck and Fecteau (18) first distinguished the behavioral autonomy and support dimensions of satisfaction that we have validated with the development of the SHASS.

The SHASS is brief—11 items—and easy to administer. It measures clients' satisfaction with services and with their involvement in their own care. These two factors have been identified as the most salient in evaluating the provision of mental health services (18). Particularly in the context of inpatient care, which can erode an individual's sense of control over many life domains, maximizing involvement in treatment decisions might prove useful in mitigating some of the effects of institutionalization. The SHASS and its two subscales showed a high degree of internal consistency. The moderate degree of stability across a six-month period attests to its reactivity to change in clients' organization-related experiences.

With the deletion of a single item—satisfaction with the opportunity to do a job—and removal of the word "peer" preceding "counseling," this scale may work in most mental health settings. The elimination of the job item has a minimal effect on the scale's internal consistency (Cronbach's alphas of .90 at baseline and .93 at follow-up for the total scale and .89 at baseline and .92 at follow-up for the scale without the item).

A key factor in the validation of the scale was our ability to show its independence from more global measures of satisfaction with quality of life at baseline. The distinction between these measures was less strong at six months, possibly because participation in a self-help agency, especially satisfaction with the social network, increasingly defines the quality of life for long-term users of self-help agencies.

If the SHASS has a weakness, it is what Ruggeri (11) called sensitivity—the ability to assess satisfaction with specific program components. The SHASS could be expanded, although that might affect its convenience of application.

Finally, consumer satisfaction as an outcome that has value in and of itself is open to question. Research has demonstrated cross-sectional relationships between satisfaction measures and status-outcome variables, showing that status measures representing the primary objectives of the helping effort are associated with satisfaction levels at follow-up. However, such studies have yet to fully explore the role of satisfaction with services in mediating treatment outcomes (28,29,30,31,32,33,34,35,36,37,38). Because participants were recruited into this study after a minimum of three months of participation in the self-help agencies, the relationship of satisfaction at the time of recruitment and outcomes measured six months after enrolling in the study was explored. Dissatisfaction with traditional mental health programs is one of the primary reasons that self-help agencies have been established. Thus the discovery of an association between satisfaction during a service period and outcomes has special significance.

The study found a modest relationship between the satisfaction subscales and two treatment outcomes. Participants' scores on the SHASS service satisfaction subscale were significantly associated at follow-up with assisted and independent social functioning. Scores on the SHASS involvement subscale were significantly associated with independent social functioning at follow-up. Further study is needed to determine whether these two dimensions of satisfaction have a mediator effect on client outcomes.

Conclusions

Central to the operation of self-help agencies is the active involvement of clients in the helping process. This study focused on the development of a multidimensional measure of satisfaction with self-help agencies that included subscales reflecting both satisfaction with active involvement in the agency and satisfaction with the support or services received. The SHASS exhibited high internal consistencies, moderate stability, and discriminant validity with measures of satisfaction with quality of life. In addition, the SHASS subscales showed modest associations with two of four measures of client outcomes—assisted and independent social functioning.

Acknowledgments

This study was supported by grant R01-NM-47487 and training grant HH-18828 from the National Institute of Mental Health.

Dr. Segal is professor and director of the Mental Health and Social Welfare Research Group in the School of Social Welfare, 120 Haviland Hall (MC#7400), University of California, Berkeley, California 94720-7400 (e-mail, [email protected]). Ms. Redman is research associate and Dr. Silverman is program director of the Center for Self-Help Research of the Public Health Institute in Berkeley.

Enlarge table

Enlarge table

Enlarge table

Enlarge table