Use of a Consumer-Led Intervention to Improve Provider Competencies

There are pervasive problems with the quality of care provided to people with severe and persistent mental illness, and few interventions have been empirically demonstrated to improve the structure or process of care for these disorders (1,2,3). For example, treatment guidelines strongly recommend that clients have access to vocational rehabilitation and caregiver services, but these services are seldom used (4,5,6).

The Institute of Medicine examined prevalent problems with health care and identified client-centered care as one of six major targets for quality improvement (7). Client-centered care refers to collaboration between clinicians and informed, empowered clients to ensure that treatment respects clients' needs and preferences (8). Client-centered care is particularly important for chronic illnesses, including chronic psychiatric disorders (9). People who actively participate in their care are known to have better outcomes (10,11); yet it has been hard to move this principle into practice. This study evaluated a consumer-led intervention to improve provider quality and empower consumers. The intervention is grounded in an emerging national movement of consumers with severe mental illness. This study is one of the first to examine whether consumers can drive an intervention to improvecare for chronic illness.

Most efforts to improve care have focused on best practices, which are defined by scientific evidence and clinical experience. In mental health, treatment guidelines have been developed. However, there are barriers to using these guidelines to improve care. For one, shortcomings of existing medical records make it very difficult to know when clients are receiving guideline-concordant mental health services (12). A complementary approach to improving care and the mental health workforce focuses on clinicians' competencies that are important but are not targeted by standards-based approaches—for example, attitudes (13). Competencies are attitudes, knowledge, and skills that are causally related to effective job performance (14). Competencies have been shown to affect the delivery of appropriate care (15) and can be accurately assessed by using direct observation, standardized simulations, and written surveys (16,17). Competency assessment can inform efforts to improve care by using clinician recruitment, training, and profiling. Although core competencies have been defined for clinicians who provide care to people with severe and persistent mental illness (18), little research has evaluated interventions to improve these competencies.

A lack of competencies among many clinicians has been implicated as a critical barrier to providing high-quality mental health care (19). Currently, more than three-quarters of mental health clinicians in the United States have a bachelor's degree or less education (20), and a large proportion of clinicians lack the skills necessary to deliver evidence-based practices (18). For example, professionals often have negative attitudes toward rehabilitation and mutual support (21) and underestimate consumers' interest in collaborative treatment planning (22). Negative attitudes such as these have hindered the adoption of innovations (23).

The objective of this study was to evaluate the effectiveness of a novel, consumer-led intervention designed to improve provider competency and empower consumers. Although a variety of interventions have the potential to make care more client centered, one promising approach begins with the increasingly influential movement of mental health consumers (24,25,26). Consumer leaders have influenced care through advocacy, mutual support, and consumer-run services (27,28). This study evaluated the intervention Staff Supporting Skills for Self-Help, which was developed and implemented by consumers. This intervention promotes client-centered care and recovery, which has been defined as living "a satisfying life within the limitations caused by illness" (29). Staff Supporting Skills for Self-Help focuses on established principles of rehabilitation (30,31) and includes education, structured dialogues between clinicians and clients, and ongoing technical assistance. It promotes mutual support. We report on a prospective, controlled evaluation of the effectiveness of this intervention over a one-year period at five large community mental health organizations in two states.

Methods

Project sites were part of provider organizations that provided care by using Medicaid funds managed by ValueOptions, a large behavioral health care organization. Provider organizations were selected for our study if they served a large population with severe and persistent mental illness, were willing to participate, and provided similar types of services. Two organizations were selected in Arizona, and three were selected in Colorado. In Arizona, both organizations provide housing and treatment in a large urban area. One organization has three clinical sites and about 100 clinicians and 140 clients. The other organization has two sites and about 100 clinicians and 80 clients. In Colorado, the three organizations provide case management, psychotherapy, and housing in small urban and rural areas. The first organization has two sites and about 25 clinical staff and 130 clients; the second organization has two sites and about 20 clinical staff and 120 clients; and the third has three sites and about 50 clinical staff and 160 clients.

Among the clients served by the participating organizations, 72 percent were white, 17 percent were Hispanic, 7 percent were black, and 2 percent were Asian. Sixty-four percent were male, 25 percent had at least one psychiatric hospitalization during the past year, and 77 percent had a primary diagnosis of a psychotic disorder.

This study used a quasi-experimental design. One organization in each state was assigned to the intervention for one year. Remaining organizations continued with usual care. Each state included both intervention and control organizations, ensuring that external events (for example, changes in state policy) would not be confounded with the intervention. In Colorado, to reduce travel costs for intervention staff, the closest organization to the ValueOptions offices was assigned to the intervention. This organization had three sites. In Arizona, the organization that appeared most enthusiastic was selected for the intervention. This organization had three sites. The study was approved by the RAND institutional review board.

At each site, clinicians were recruited to participate in our study if they provided care to people with severe and persistent mental illness. The only exception was clinicians from homeless outreach and day treatment programs; they were excluded because they represented a small number of staff who performed unique, more specialized types of job duties. Written informed consent was obtained from participants.

Intervention

The intervention was developed and delivered by two of the authors, Knight and Vogel. Both are consumers of mental health services. Knight is vice-president for recovery, rehabilitation, and mutual support for ValueOptions Healthcare, and Vogel is executive director of Double Trouble in Recovery, a national self-help organization. They developed Staff Supporting Skills for Self-Help by involving consumers and providers from across the United States in structured dialogues and focus groups and by modifying successful preexisting mutual support programs (26,32).

Manuals for implementation of Staff Supporting Skills for Self-Help are available from Knight ([email protected]) or Vogel ([email protected]). Table 1 describes the six components of the intervention that involve clinicians. The components Scientific Presentation on Self-Help and Structured Dialogues were conducted in January and February 2001, Rehabilitation Readiness was conducted in May and June 2001, Strategies for Independence in November 2001, and Professional Support Skills for Self-Help in January 2002. Detailing, another component of the intervention, included four full-day visits to Arizona sites and three full-day visits to Colorado sites during the intervention year. An additional 16 hours were spent meeting at various times with staff at the sites.

The intervention also has components that involve only consumers. Technical assistance was provided to consumers to encourage and facilitate the formation of consumer-operated services, such as mutual support groups, drop-in centers, and health education programs. There was a focus on two prepackaged mutual support groups: Double Trouble in Recovery and Wellness in Numbers. In Double Trouble in Recovery the 12-step Alcoholics Anonymous model was tailored to the needs of people with severe and persistent mental illness. Participation in Double Trouble in Recovery has been associated with better medication adherence, less substance use, and greater well-being (33,34). Wellness in Numbers is a manualized group model that includes examination of one's own behaviors, open sharing or guided meditation, and reading about or hearing speakers discuss recovery. At the start of the intervention, a Consumer Mutual Support Day was held at each site that included research presentations, structured questions, and small group discussions. Consumer leaders were identified and charged with starting mutual support groups. Encouragement was provided to local consumer leaders throughout the year, and a fund was established to provide logistical support for mutual support. About $1,000 was spent on refreshments, supplies, and travel.

Measures

We measured the effect of the intervention on clinicians' competencies, care processes, and the formation of mutual support. Competencies were assessed at baseline and at one year with a written survey questionnaire that clinicians confidentially completed. The questionnaire included 55 items from the Competency Assessment Instrument (CAI), an established instrument that has good internal consistency, test-retest reliability, concurrent validity, and construct validity (16). The CAI is available at www.mirecc.org/product-frames.html and includes scales that measure 16 competencies selected from a set of 37 competencies identified by a literature review and national consensus process (18). The competencies measured by the CAI can be found in Table 2. These competencies were selected because the national consensus panel viewed them as central to recovery-oriented care, not prevalent among providers, and potentially modifiable.

An overall competency score was obtained by averaging the 16 scale scores. The questionnaire also included items about job duties and seven items from a scale that assesses attitudes toward recovery (35). In preparation for data analyses, all scales were transformed to range from 0 to 1, with 1 indicating the highest competency.

Treatment processes were evaluated with semistructured interviews with managers and clinicians at baseline and at the one-year follow-up. Interviews were conducted by one of the authors and focused on rehabilitation, recovery-oriented services, and mutual support groups. Questions addressed barriers to and facilitators of service provision and organizational change.

Data analysis

The effect of the intervention on clinicians' competencies was evaluated by comparing change over time between the intervention and control groups. Effects were evaluated by using maximum-likelihood, mixed-model general linear analyses of covariance. Separate analyses were done for each competency by using the intervention condition as the fixed independent variable and controlling for baseline score. To address possible nonindependence from clustering within organizations, analyses included the five treatment organizations as a random design factor. To evaluate the possible confounding effects of clinician type between the intervention and control groups, we used analysis of covariance, coding the clinician variable as a dichotomy, contrasting clinician or therapist with residential staff or mental health worker. We tested whether this covariate interacted with the intervention effects for each competency and found that none were significant. Therefore, we analyzed each competency by using simple analyses of covariance, evaluating the intervention effect while statistically controlling for clinician type.

When a significant effect was found for a competency, a secondary analysis evaluated whether there was a dose-response relationship. Analyses examined whether competency change was associated with the number of intervention components participated in by the clinician (zero to six). Stepwise multiple regression was performed with competency change as the dependent variable, forcing baseline score into the model in the first step and then adding the participation score. Analyses were performed with SAS statistical software.

Not all of the participating clinicians had data at follow-up. To confirm results, we ran analyses on variables that showed significant intervention effects, using multiple imputation to replace missing data. Imputations were based on the full set of baseline and one-year follow-up competency variables.

Data from the semistructured interviews were analyzed with the constant comparison method (36). This method involved continually moving between initial and later interviews, identifying units of meaning, coding the data, and interpreting the data. First-level coding was completed by observing similarities and differences between data segments from the interviews. After these units of meaning were identified, they were grouped into categories. Interpreting the more abstract meaning of the first-level categories yielded second-level coding, which is presented in the Results section. Reliability was ensured by sampling individuals with a wide range of employment positions and perspectives, collecting and analyzing data simultaneously with a "macro-micro" perspective to move from single bits of data to overarching themes and back to the data to confirm emerging concepts (37), and having an independent expert in mental health professional training and systems change review coding and findings to determine dependability, credibility, and degree of fit (38).

Results

Of 340 clinicians eligible for our study, 269 (79 percent) chose to participate: 151 in the intervention group and 118 in the control group. Participation rates did not differ significantly between intervention and control groups (151 of 186 clinicians, or 81 percent, in the intervention group, compared with 118 of 154 clinicians, or 77 percent, in the control group). As is often the case at agencies such as these, clinician turnover was substantial. During the intervention, 59 clinicians at the Arizona sites (37 percent) and 15 clinicians at the Colorado sites (14 percent) left their jobs. Because of this turnover, 195 of 269 participants (72 percent) completed the one-year follow-up. Clinicians in the intervention group were somewhat more likely to complete the one-year follow-up survey than those in the comparison group (117 of 151 clinicians in the intervention group, or 76 percent, compared with 78 of 118 clinicians in the control group, or 66 percent; χ²=4.30, df=1, p=.04).

At each site, a number of managers and clinicians were selected to complete the semistructured interview. At baseline, interviews were conducted with two upper managers, six middle managers, nine direct supervisors, and ten front-line clinicians. At the one-year follow-up, interviews were conducted with two upper managers, five middle managers, seven direct supervisors, and eight frontline clinicians. Sixteen people participated in interviews at both time points.

Table 3 presents characteristics of clinicians participating in our study. At baseline, more than two-thirds of participants stated that they taught daily living skills (216 clinicians, or 80 percent), interacted with clients' family and friends (199 clinicians, or 74 percent), accompanied clients into the community (191 clinicians, or 71 percent), provided crisis intervention (187 clinicians, or 70 percent), or taught clients medication skills (179 clinicians, or 67 percent). Less than a quarter stated that they helped clients find housing (64 clinicians, or 24 percent) or work (62 clinicians, or 23 percent) administered medications (57 clinicians, or 21 percent), engaged clients who have dropped out of care (43 clinicians, or 16 percent), provided alcohol or drug treatment (27 clinicians, or 10 percent), or performed physical exams (three clinicians, or 1 percent). The mean±SD number of years of mental health work experience was 8.5±7.9. Clinicians in the intervention and control groups did not differ significantly on demographic variables, educational level, or number of years in mental health. The intervention group had more participants who were clinicians or therapists, whereas the control group had more persons who were the residential staff and mental health workers (χ²=58.7, df=6, p<.001). As shown in Table 4, the percentage of clinicians who participated in each intervention component varied among sites. Most clinicians in the intervention group (121 clinicians, or 80 percent) participated in at least one component.

In the unadjusted analyses, the intervention group's competency scores improved significantly more than those of the control group on teamwork, medication management, holistic approach, education about care, rehabilitation methods, natural supports, overall competency, and recovery orientation. Competency regarding stigma worsened equally and significantly in both groups. As shown in Table 5, when the analyses adjusted for clinician type, the overall pattern of results was similar. The differences were that scores on medication management, holistic approach, and natural supports no longer showed improvement that was significant at p<.05. However, in within-group analyses, the scores of the intervention group for these three scales improved significantly over time, whereas those of the control group did not (Table 5). Analyses were repeated by using imputed instead of missing data, and the pattern of results was the same.

Exposure to the intervention was correlated with improvement in teamwork (r=.28, p=.003), holistic approach (r=.17, p=.06), education about care (r=.22, p=.03), rehabilitation methods (r=.25, p=.007), natural supports (r=.24, p=.02), and overall competency (r=.21, p=.02). There was no dose-response relationship for medication management.

Preintervention interviews with managers and clinicians revealed substantial variation in the organizations' understanding and use of recovery-oriented services. Some staff recalled trainings in daily living skills, and others talked about client-focused care. Many mentioned support of vocational rehabilitation. Attitudes and behaviors that foster psychiatric rehabilitation and client-centered care were not well understood. Staff conveyed an interest in supporting client-run activities but often did not know how to follow through. Staff at most sites reported that their organizations espoused a shift toward recovery-oriented services and that training toward this end would be beneficial.

At one year, interviews revealed that intervention sites were providing more recovery-oriented services than control sites. At intervention sites, clinicians reported more support from management for implementing new rehabilitation services. They commonly stated that providing rehabilitation is hard work and takes a new kind of thoughtfulness. One said, "We thought we knew what we were doing, but it's much more complicated than we thought." Many staff were relieved that a foundation for psychiatric rehabilitation is helping clients take personal responsibility for their lives and their treatment. One stated, "Everything we do has to focus on the small steps, the next thing we can help each client do for himself or herself." Staff at control sites said that not much had changed. One clinician stated, "We're not doing rehabilitation, we're just trying to keep things under control."

Before the intervention, few self-help groups existed. More often, naturally occurring, informal support networks were described. For instance, some clients lived in the same neighborhood and socialized together. At the one-year follow-up, staff at all sites reported difficulties interesting clients in mutual support groups. They reported that many clients found mutual support groups cliquish or uninteresting. However, intervention staff were more supportive of mutual support and more determined to find ways to support mutual support groups. Intervention staff reported an increased use of existing client-run services.

Despite the challenges, eight new consumer-run mutual support groups were started during the study period, all at intervention sites. In Arizona four groups were started: two current topics groups, a Double Trouble in Recovery group, and a Wellness in Numbers group. The format of the current topics group was determined by consumers at the site and focused on news items. The groups existed for three, seven, five, and nine months, respectively. In Colorado, three groups started: one Double Trouble in Recovery group, one Wellness in Numbers group, and one relaxation group. The format of the relaxation group was determined by consumers and focused on choosing and playing relaxation tapes. All support groups in Colorado continued beyond the intervention year. As groups became established, they increased their efforts to recruit new members and began to provide transportation so that more consumers could get involved.

Discussion

In this project, a one-year consumer-led intervention led to the formation of mutual support and improved clinicians' competencies, which were assessed with a research survey. The greatest improvements were observed in clinicians' competencies that are critical for client-centered care. In addition, clinicians at intervention sites stated that they were providing more recovery-oriented services. Changes in treatment provision must be interpreted with caution, because we did not quantify service use. However, changes in treatment process were consistent with improvements seen in the structure of care. Although most competencies improved under the intervention, stigma worsened among clinicians in both the intervention and control groups during the year. Clinicians may have been influenced by several violent incidents committed by individuals with mental illness that received national attention at that time.

Although the effectiveness of Staff Supporting Skills for Self-Help needs to be replicated, improvement in competency is notable, because interventions targeting the organization of care for people with severe and persistent mental illness have often failed to generate change at the clinical level (3). Staff Supporting Skills for Self-Help differs from most previous interventions in that it involves both the clinician and the client, focuses on client-centered care, and targets competencies. This approach is consistent with organizational theory, which suggests that resistance of staff to change can be overcome by assigning value to staff characteristics that are related to client outcomes (39). Also, in this project an incentive for organizations to change was provided by ValueOptions, which managed their funding and was supportive of the project.

Interventions are easier to disseminate if they are acceptable, feasible, and useful within the context of usual care arrangements. The intervention performed well in each of these domains. As expected, staff varied in the extent to which they accepted the intervention and believed that client-centered care is important. Staff participation was moderate. Resources allocated for the intervention were modest and consisted primarily of the time of the intervention leaders and of the clinicians who were released from usual activities to participate in intervention components. It may be possible to strengthen provider change by linking personnel practices to possession of valued competencies. Further benefit should also be possible by linking the client-centered focus with implementation of specific evidence-based services, such as supported employment.

Certain limitations should be noted. First, we did not measure change in the appropriateness of care or client outcomes. Studying downstream effects on outcomes would require a much larger sample size. Also, the generalizability of our findings is limited by the number of sites. Because selection of intervention organizations was based in part on willingness to participate, findings could have resulted from clinicians' being predisposed to the care model. However, this potential for bias is somewhat mitigated by the fact that any baseline differences between competencies of clinicians in the control and intervention groups were statistically controlled. In addition, the positive dose-response relationships demonstrate a role for exposure to the intervention. Finally, the CAI is a self-report measure; therefore, there could be a discrepancy between clinicians' responses and their actual level of competencies. However, self-report survey methods have been successfully used to assess competencies of groups such as medical residents (40), nurses (41), and social workers (42). In all cases, surveys similar to the CAI were reliable and valid and led to the identification of educational priorities. Further research should evaluate broader dissemination of Staff Supporting Skills for Self-Help and provide more information about links between changes in competencies and service delivery.

Conclusions

The principle of client-centered care is said to be vitally important in the care of persons with mental illness. Yet there have been few efforts to make care more client centered. This intervention is one of the first such efforts to be evaluated. Staff Supporting Skills for Self-Help is designed and implemented by consumers. It is feasible at busy clinics and has a positive effect on the structure of care, particularly in domains supporting recovery. This study focused on provider competencies that are critical to client-centered care, evaluated them by using a validated survey method, and found improvement in multiple domains. Because there are numerous barriers to improving the process of care and outcomes for people with severe mental illness, adequate progress will require a number of simultaneous approaches. Although having a competent workforce does not assure provision of appropriate care, it represents a critical step forward.

Acknowledgments

Support was provided by grant ML-580 from the Center for Health Care Strategies (which was made possible through a grant from the Robert Wood Johnson Foundation); ValueOptions; grant RCD-00-033 from the Department of Veterans Affairs (VA); the VA's Desert Pacific Mental Illness Research, Education, and Clinical Center; and grant MH-068-639 from the National Institute of Mental Health to the UCLA-RAND Center for Research on Quality in Managed Care. The authors thank Midge Starzynski, Ph.D., R.N., for her contributions.

Dr. Young, Dr. Chinman, and Dr. Mintz are affiliated with the Department of Veterans Affairs Greater Los Angeles Healthcare System, Desert Pacific Mental Illness Research, Education, and Clinical Center, 11301 Wilshire Boulevard, 210A, Los Angeles, California 90073 (e-mail, [email protected]). Dr. Chinman and Dr. Rowe are with RAND in Santa Monica, California. Dr. Young and Dr. Mintz are also with the department of psychiatry at the University of California, Los Angeles. Dr. Knight and Dr. Miller are with ValueOptions in Colorado Springs. Dr. Forquer is with Comprehensive Neuroscience in Colorado Springs. Mr. Vogel is with the Mental Health Empowerment Project in Albany. Preliminary results of this study were presented at the Department of Veterans Affairs Health Services Research and Development national meeting, held March 9 to 11, 2004, in Washington, D.C., and at the Academy Health annual research meeting, held June 26 to 28, 2005, in Boston.

Enlarge table

Enlarge table

Enlarge table

Enlarge table

Enlarge table