The Effects of Collaborative Care Training on Case Managers’ Perceived Depression-Related Services Delivery

Depression is common across racial, ethnic, and socioeconomic groups (1), it affects health, cost, and productivity outcomes (2,3), and it is a leading cause of disability (4–6). Compared with whites, African Americans report lower lifetime prevalence of major depression, but they have more severe symptoms (7,8), use fewer services, and terminate treatment early (7,9,10). African Americans and Latinos are less likely than whites to receive evidence-based depression care and have worse outcomes (8,9,11–13).

Although evidence-based practices for depression exist, their implementation in community settings has been limited (11,14). Quality improvement (QI) programs for depression that are based on the collaborative care model improve depressive symptoms, quality of life, and social outcomes and reduce disparities in outcomes among racial and ethnic groups (12). However, these programs are often not implemented in underresourced communities, where there is low availability of specialty care and historical distrust of services (13,15). In such settings, depressed individuals may seek support from alternative sectors that are not included in health care QI efforts and where unlicensed providers may be the primary supports for depressed individuals. The Affordable Care Act (ACA) provides incentives for health homes to utilize case managers (16,17), but these roles in collaborative care settings are typically filled by licensed providers (18,19). Little is known about how depression QI affects implementation outcomes, such as the use of therapeutic practices, among case managers who primarily are unlicensed providers (20).

This study addressed this gap by analyzing data from Community Partners in Care (CPIC), a cluster-randomized trial of two implementation conditions of a depression QI program (21). The two conditions were Community Engagement and Planning (CEP), an intervention to support networks of health and community-based agencies, and Resources for Services (RS), an intervention that provides technical assistance to individual agencies. Following community feedback, health and community-based programs were included as sites for the depression QI programs (21). Inclusion of unlicensed in addition to licensed case managers was motivated by the communities expressed awareness of shortages of mental health professional in underresourced communities (22,23) and by a desire to increase provider diversity and community trust. Chung and others (24) reported that compared with RS, the CEP condition was associated with greater increases in participation in depression training sessions among staff estimated as eligible by participating program administrators. However, change in practice outcomes for predominantly unlicensed case managers was not reported.

We sought to replicate findings of increased training participation associated with CEP compared with RS among case managers who were participating in a provider outcomes substudy. We also sought to examine the interventions’ effects on case managers’ use of depression case management and therapeutic problem solving and on the amount of time they spent delivering community services (primary outcomes). We also sought to examine the interventions’ effects on case managers’ depression knowledge, attitudes, and perceived system barriers (secondary outcomes). We hypothesized that case managers in programs assigned to CEP would participate more in training activities, spend more time providing community services, and report higher use of depression case management and therapeutic services compared with case managers in programs assigned to RS. We expected that CEP would have greater effects than RS on improving depression knowledge and attitudes and perceived system barriers, owing to a greater focus in CEP on network development.

Methods

Data were from the provider substudy of CPIC, a group-level, randomized trial implemented by using community-partnered participatory research (CPPR) (25,26), which supports academic and community partners as equal decision makers in research design and implementation (27). As described elsewhere (21,24,28), the study was conducted in South Los Angeles, with a population of roughly 1.5 million, and Hollywood–Metro Los Angeles, with a population of roughly 500,000—both areas with high rates of unemployment, homelessness, and lack of insurance (29).

The interventions represented two ways of implementing a depression QI training program based on the collaborative care model (30–32) that was adapted for licensed and unlicensed providers (33,34) from health and community-based services programs. Toolkits for case managers included guidelines for case management, depression screening, care coordination, outreach strategies, problem-solving therapy, behavioral management, and activation skills. Materials were introduced in kick-off conferences prior to enrollment and randomization and were available to participants in both conditions as hard copies, in flash drives, and on a Web site (www.communitypartnersincare.org/community-engagement/cep/). The interventions encouraged, but did not require, use of these resources by eligible providers.

The Interventions

CEP invited participating program administrators to attend two-hour, bimonthly council meetings for four months to adapt depression QI toolkits to their community and collaborate as a network, following a workbook based on principles of CPPR. Councils were asked to develop and implement a written plan for adaptation of the toolkit, including training and monitoring practices, supported by $15,000 from the study. Final plans featured conferences, follow-up with programs, telephone and Webinar supervision by intervention experts for cognitive-behavioral therapy (CBT) and case management, and innovations such as provider self-care and depression book clubs.

CEP case manager training sessions were co-led by academic and community leaders. Eligible providers were invited to attend half-day or all-day conferences as well as follow-up and make-up sessions at individual programs. The six-hour case manager training sessions reviewed the study’s purpose, defined terms, and presented resources. Participants were taught about client engagement and outreach, use of the Patient Health Questionnaire–9 (PHQ-9) for depression screening, behavioral activation, and making referrals and problem solving, all of which was reinforced by role-play. Participation in other components, such as medication management, was encouraged. Telephone supervision of case managers (two to three sessions per community) by intervention experts was offered.

The RS intervention offered outreach and technical assistance to individual programs by using a “train-the-trainer” approach. A team made up of psychiatrists, a nurse care manager, a CBT trainer, a QI expert, support staff, and a community-engagement specialist offered 12 Webinars lasting 90 to 120 minutes on team management, CBT, care management, and patient education as well as visits to primary care sites to support medication management. Case managers in each community received four Webinars, each lasting one to hours. Topics included client engagement, depressive symptom recognition, screening using the PHQ-9, making referrals, and use of problem-solving strategies.

Participants

County lists and community nominations were used to identify the names of 149 agencies for possible inclusion. Of those, 29 were ineligible, 41 refused, and 19 could not be reached. Among 60 eligible participating agencies, there were 194 programs, from which 133 potentially eligible programs (serving ≥15 clients per week, having one or more staff, and not focusing on psychotic disorders or home services) were randomly assigned to the CEP or RS intervention (21).

Within each community, programs were paired into units or clusters of smaller programs and were paired based upon location, service sector, size, population served, and other program characteristics; two larger agencies were their own single stratum. Within pairs, one program or cluster was randomly assigned to CEP and the other to RS (21,24,35).

At site visits to confirm eligibility and participation, 20 programs were ineligible, 18 programs refused to participate, and 95 (84% of the 113 eligible programs) enrolled. [A flow diagram summarizing provider participation throughout the study is available as an online supplement to this article.] Site visits were conducted by staff blinded to program assignment (21). Program administrators were informed of intervention status by letter prior to screening. Zip code–level data for the neighborhoods of participating and nonparticipating programs were comparable (21).

Provider Outcomes Substudy

In eligible programs having more than one permanent staff member in addition to the administrator (N=84), providers with direct patient contact, including volunteers, were invited to participate in the provider substudy through agency presentations and distribution of recruitment packets by administrators as well as through telephone follow-up and study site visits. The goal was to achieve a baseline sample of 300 providers and a one-year follow-up sample of 200 providers. From a pool of 370 providers who had provided verbal and written consent at baseline, 326 (88%) from 77 programs completed baseline surveys. New providers were permitted to enter after baseline. At one-year follow-up, 392 providers from 84 programs were approached for the online survey, 297 were considered eligible, 92 had left the program, one was on medical leave, and one was deceased. Of 297 eligible, 237 (80%) participated, representing 75 of the 84 eligible programs. The sample for this substudy included 117 case managers (CEP, N=58; RS, N=59) from 52 programs. Institutional review boards at the RAND Corporation, the University of California, Los Angeles, and participating agencies approved this study, which was registered as a clinical trial after baseline enrollment.

Measures

The main independent variable was program intervention status (CEP or RS). Service sector (health or community-based program) was assigned for each program based on the services it provided. Other measures were obtained from baseline and follow-up surveys of providers, and sign-in logs were used to document participation in study-provided training. Self-report measures were based in part on measures used in the Partners in Care study, adapted for licensed and unlicensed providers (36). Personal depression stigma was assessed by the mean score for agreement with three items adapted from Link’s Devaluation and Discrimination Scale (37). Standardized alpha coefficients were calculated for scales with two or more items. The relation of self-report training participation to the training logs was evaluated as an indicator of validity for that item, and we examined the pattern of correlations among provider outcomes as a preliminary indicator of convergent and discriminant validity [see online supplement], with an expectation of higher agreement among measures related to depression practices and lower agreement of these items to measures not directly reflecting use of depression techniques, for example, attitudes, knowledge, and system barriers.

Sample characteristics.

Case manager characteristics included age, sex, education, race-ethnicity, and license status.

Primary outcomes.

Participation in training, including study-provided and in-house training events, was assessed by sign-in logs or by self-report at follow-up.

Depression care techniques were measured by the mean score for nine items assessing how often respondents performed the following tasks in the past six months for people with symptoms of depression: encourage positive thinking, discuss costs of alternative mental health treatments, encourage pleasurable activities, discuss ways to improve social skills, determine depression treatment preferences, recommend ways to take care of oneself, reframe or clarify the individual’s problems, discuss benefits of treatments, and help the individual feel better about his or her life. Responses were rated on a 5-point scale ranging from 1, never, to 5, always (α=.929). Responses by case managers who did not provide services for depressed clients were set to “never.”

Depression case management was measured by the mean score for five items assessing how often respondents who had provided services for depressed clients in the past six months had performed the following tasks: explain what depression is, ask the individual what he or she thinks depression is, ask about prior treatment, make a referral, and ask about barriers to depression care. Responses were rated on a 5-point scale ranging from 1, never, to 5, always (α=.917). Responses by case managers who did not provide services for depressed clients were set to “never.”

As a measure of provision of community services, respondents were asked to indicate on a 6-point ranked scale how many hours in a typical week they spent providing services to individuals in community settings, with 0 indicting 0 hours; 1, 1–10 hours; 2, 11–20 hours; 3, 21–30 hours; 4, 31–40 hours; and 5, >40 hours.

Secondary outcomes.

Perceived depression knowledge was assessed by the mean score for agreement with three items adapted from Partners in Care (“Depression is a medical condition,” “Depression runs in families,” and “Depression can cause physical changes like aches and pains”) (10,36). Responses were rated on a 5-point scale ranging from 1, strongly agree, to 5, strongly disagree (α=.627).

Perceived depression skill was assessed by the mean score for seven items assessing the following specific skills: case finding, depression screening with a standardized instrument, educating individuals or families about depression, depression counseling, referral to mental health specialty care, providing social support for depression (support groups), and engaging in community outreach for depression. Responses were rated on a 4-point scale ranging from 1, not at all skilled, to 4, very skilled (α=.890).

Personal depression stigma was assessed by the mean score for agreement with three items adapted from Link’s Devaluation and Discrimination Scale (37) (“I have no patience with a person who is always feeling 'blue' or depressed,” “I would be embarrassed if people thought I was depressed,” and “Most people think less of a person who has been depressed”). Responses were rated on a 5-point scale ranging from 1, strongly agree, to 5, strongly disagree (α=.579).

Perceived system barriers were assessed by a count of four items assessing the extent to which optimal depression care services were limited in the past six months by difficulty obtaining treatment, unavailability of mental health professionals, poor reimbursement, limited insurance, other benefits, or other barriers. Responses were dichotomized as limited a great deal versus limited somewhat or not limited.

As expected, measures of depression care techniques and care management were strongly positively correlated, supporting convergent validity [see online supplement]. Measures of depression knowledge and attitudes were not significantly associated with depression care techniques, supporting discriminant validity. Further, perception of depression skill was positively associated with depression care techniques and depression case management. Personal depression stigma was positively associated with depression case management, and community services provision was associated with number of system barriers.

Analysis Plan

We conducted intent-to-treat, comparative-effectiveness analyses of outcomes for case managers in the provider substudy with one-year follow-up data. We used logistic regression models for dichotomous variables and multiple linear regression models for continuously scaled variables, both adjusted for baseline status of the dependent variable, sector (heath versus social-community), and provider type (licensed versus unlicensed). There was no baseline status for training participation. We compared baseline characteristics of case managers by intervention status.

We used an extended hot-deck technique to impute missing values for nonresponse, using five imputed data sets for baseline and follow-up responses and multiple imputation inference for all analyses (21,38). To control for potential nonresponse bias, we used nonresponse weighting (39) to address missing data for the 20% of providers who did not complete one-year follow-up. The objective of nonresponse weighting is to extrapolate from the observed one-year sample to the original eligible sample. Nonresponse weights were constructed by fitting logistic regression models to predict follow-up status from baseline characteristics. Separate models were fitted for each intervention group. The final logistic regression model included predictors that were significant (p<.10) for either CEP or RS groups (service sector [health vs. community based], education, and baseline perception of depression attitude and skill). Five versions of the weight were created corresponding to imputed data sets (39,40). Significance of comparisons by intervention status is based on regression coefficients. Results are presented as between-group differences for linear regression and as odds ratios (ORs) with 95% confidence intervals (CIs) for logistic regression.

Average results for an intervention group were adjusted for all covariates by using standardized predictions generated from the fitted regression model (39). To account for client clustering within programs, the variance estimation was based on the Taylor series linearization method (40) All analyses were conducted by using SUDAAN, version 11.0 (Software for the Statistical Analysis of Correlated Data [www.rti.org/sudaan/]), which contains a design specification for sampling with replacement in the first stage of sample selection (programs), accounting for attrition weights.

For primary outcomes, to account for multiple comparisons, we calculated the false discovery rate (FDR), for comparing observed significance findings with expected order statistics from a uniform distribution (41).

Results

Demographic and Descriptive Characteristics

As shown in Table 1, 47% of case managers self-identified as Hispanic, 45% as African American, 6% as non-Hispanic white, and 2% as Asian, Pacific Islander, or other race-ethnicity. The mean age was 43.6 years, and 71% were female. Over 70% were unlicensed, with over half (62%) working in the health sector. There were no significant baseline differences by intervention status, except CEP participants had higher mean scores for personal depression stigma compared with RS participants (p<.05).

Outcomes

As shown in Table 2, 27% of case managers in RS and 74% in CEP participated in CPIC-sponsored training (OR=7.78, p<.001). CEP case managers reported use of more depression care techniques compared with RS case managers (3.1 and 2.8, respectively; difference=.32, CI=.03–.61, p<.05). CEP case managers had higher mean scores for provision of community services compared with RS case managers (1.2 and .7, respectively; difference=.51, CI=.13–.89, p<.05). Findings remained significant after applying the FDR for multiple comparisons. Differences between mean scores for depression case management for participants in CEP or RS were not statistically significant.

As shown in Table 3, there were no significant differences in secondary outcomes (p≥.05) between participants in CEP or RS.

Discussion

We examined effects of two interventions to implement depression QI across underresourced communities—CEP for supporting networks and RS for providing technical assistance to individual agencies. Case managers were predominantly unlicensed and from racial-ethnic minority groups, representing a resource for expanding workforce diversity given known disparities in services access. We found that the percentage of case managers in CEP programs who participated in training was nearly three times higher than the percentage in RS programs, confirming similar findings for all eligible providers (24). Compared with RS case managers, CEP case managers reported delivering community services for a greater number of hours and greater use of therapeutic problem-solving skills for depression. However, we found no differences between interventions in case management tasks, which may be standard competencies of case managers or may be easier to influence with technical assistance (42). The stronger CEP effect on increasing use of therapeutic strategies may be due to the fact that those skills are not considered standard competencies for case managers and to the role modeling and supervision provided by CEP. Contrary to expectations, we did not find significant intervention effects on attitudes or perceived skill. CEP participants demonstrated greater knowledge and reported fewer system barriers compared with RS participants, but the differences were of borderline significance, suggesting areas for future research.

Others have noted that providing staff with information prior to training can improve self-efficacy and motivation (43), which can increase participation (44). In CPIC, providers in both interventions could participate in conferences providing an overview of depression toolkits prior to randomization, representing equivalent preassignment and preintervention exposure. The RS intervention supported knowledge exchange, whereas CEP supported in-person guidance, skills building (self-efficacy), modeling (observational learning), peer support, networking, and collaboration. Under self-efficacy theory (45), these CEP features may have increased case managers’ confidence in initiating and sustaining behavior change rather than avoiding new tasks (46). Torrey and others (47), for example, found that clinicians were motivated to change practice if the change was perceived as clinically helpful and reinforced through observation, supervision, and feedback. Further, the CEP focus on administrator involvement and community feedback may have enhanced collective efficacy (48,49) to motivate an expansion of case managers’ roles.

Future research should focus on ways to further build and sustain capacity for depression services, as required for medical homes (16,17), given that CEP supported a case manager workforce predominantly from racial-ethnic minority groups in underresourced communities. Although we did not focus on the link between provider change and client outcomes, we have previously reported that CEP improved client mental health–related quality of life and reduced behavioral health hospitalizations over six to 12 months compared with RS, with significance sensitive to alternative statistical methods (24). In addition, it will be important to determine whether differences between interventions in provider outcomes resulted from differences in training participation.

Several limitations should be acknowledged. Data were from provider self-report in two communities, meriting replication with service records or observation and further research on the validity and content of provider outcome measures. Many of the survey questions were worded in one direction, so that a higher score represented more appropriate practice, for example. Knowledge and attitude measures could be expanded to include theoretical constructs (50–52) related to intervention adoption and sustainability. The case manager sample was modest for observing small effects that are typical for public health implementation strategies. There was attrition at follow-up primarily because of turnover, which may be expected in underresourced communities and which was accounted for with weights.

Conclusions

This study suggests that depression QI programs can feasibly include predominantly unlicensed, racially and ethnically diverse case managers. Case managers in CEP had greater participation in training, spent more time in delivering community services, and made more frequent use of therapeutically oriented problem-solving skills compared with case managers in RS. As one CEP community leader noted, “This offers hope for underresourced communities.” It may be important to identify policy mechanisms to sustain such efforts by working with the Health Resources and Services Administration, the Substance Abuse and Mental Health Services Administration, the Centers for Medicare and Medicaid Services, and health plans or foundations to certify unlicensed case managers as health workers, develop formal partnerships, and access funding.