Factors Associated With Caregiver Attendance in Implementation of Multiple Evidence-Based Practices in Youth Mental Health Services

Publicly funded mental health systems have enacted policies supporting the implementation of evidence-based practices (EBPs) as a strategy to improve the quality of care for the estimated 20% of youths with a mental disorder (1, 2). Despite these efforts, the effects of EBPs are consistently weaker when implemented in the community compared with their effects in controlled trials (3). This “implementation cliff” is explained chiefly by differences between experimental and usual care conditions, for example, the greater complexity of client profiles and the decreased intervention integrity in usual care (4, 5). Given the variability in success of EBP implementation, monitoring quality indicators to identify improvement targets is essential. To address the “quality chasm,” the Institute of Medicine has called for federal agencies to develop indicators grounded in the evidence base (6). Previously studied quality indicators include patient satisfaction and continuity of care following hospitalization (7). Yet the National Committee on Quality Assurance reported that neither the scope nor the design of extant indicators is sufficient to optimally measure quality of care (8).

Current quality monitoring tools focus largely on detailed measures of adherence to multicomponent therapy protocols. These measures index the extent to which providers deliver treatment elements as tested in controlled trials (9). Such assessments require laborious data collection and review, often involving observer ratings. Large-scale EBP implementation requires quality indicators that are relevant to a wide range of EBPs and that can be feasibly obtained at low cost and with minimal burden to the workforce (10, 11).

Caregiver attendance is an empirically informed, feasibly assessed quality indicator (12, 13) that can help gauge the quality of care within large-scale EBP delivery in youth mental health services. Indeed, caregiver attendance is a relevant quality indicator in publicly funded services, given that caregivers are often considered the primary agents of behavior change across many treatments for youths (14–16). Caregivers help reinforce skills and generalize desired youth behaviors to new settings (17). Moreover, participation by caregivers in treatment is linked to improved clinical outcomes (12).

There are three strong indications for caregiver attendance. First, systematic reviews list parent training as a frontline treatment for externalizing behaviors, including disruptive behavior and attention-deficit hyperactivity disorder (ADHD) (18–20). In contrast, meta-analyses and reviews show inconsistent evidence of incremental benefits of caregiver treatment participation for child internalizing disorders (21–24). Second, because early intervention generally produces the largest effects in preventing behavioral problems (25), children ages 12 and under are more likely than adolescents to benefit from caregiver treatment participation (19, 20). Third, caregiver attendance is a critical quality indicator in the delivery of EBPs that prescribe caregiver participation. Many EBPs for various disorders have been explicitly designed to include caregivers to drive therapeutic change or to improve caregiver-child relationships as the primary mechanism of change. Examples include attachment-based treatments for trauma (26).

Although caregiver attendance is not sufficient for high-quality implementation of an EBP, it is a necessary precondition to deliver EBPs that require caregiver involvement. Without caregiver attendance, a provider could not deliver caregiver-directed components, thereby compromising EBP integrity profoundly. Thus caregiver attendance should be considered a quality indicator in youth mental health treatment sessions in which a child ages 12 or under presents with an externalizing behavior problem or a therapist delivers an EBP that prescribes caregiver participation.

The complexity of translating research innovations into practice has spurred the examination of multilevel factors that are related to implementation outcomes. Provider-level factors—such as educational attainment, training background, and experience—have been linked to EBP implementation quality (27). Importantly, positive attitudes by therapists toward EBPs are associated with use of EBP posttraining and fewer self-reported modifications that reduce EBP content (28, 29). At the client level, studies demonstrate that compared with boys, girls are less likely to obtain needed treatment for mental health problems, especially externalizing problems (30, 31). The continued examination of therapist and youth factors in EBP implementation can identify quality improvement targets that could become areas of focus in efforts to scale up EBPs.

In this study, we examined caregiver attendance as a quality indicator in psychotherapy sessions delivered within youth mental health services in Los Angeles County, the nation’s largest county mental health system. The study addressed two research questions: In the large-scale implementation of multiple EBPs, does preadolescent age, externalizing problems, or type of EBP delivered predict the occurrence of caregiver attendance? Within sessions in which caregiver attendance is indicated, are therapist or youth factors associated with significant differences in caregiver attendance?

Methods

Current Study Context

Data were collected as part of the Knowledge Exchange on Evidence-based Practice Sustainment (4KEEPS) study, which examined determinants of EBP sustainment following a system reform by the Los Angeles County Department of Mental Health (LACDMH) (32). In 2009, LACDMH enacted the Prevention and Early Intervention (PEI) transformation, which utilizes a state revenue stream funded by a voter-approved ballot initiative to promote EBP uptake. LACDMH offers reimbursement to community agencies for the delivery of EBPs from an approved list and facilitated the rapid scale-up of six EBPs (33). This study examined caregiver attendance in sessions in which one of the EBPs was delivered (child-parent psychotherapy [CPP], Positive Parenting Program [Triple P], trauma-focused cognitive-behavioral therapy [TF-CBT], Managing and Adapting Practices [MAP], and Seeking Safety; although LACDMH provided implementation support for Cognitive-Behavioral Intervention for Trauma in Schools, community therapists in this sample did not report on their delivery of this EBP.

CPP addresses trauma in children ages 6 and under, primarily with conjoint child-caregiver sessions, although some sessions are caregiver focused (26). Like CPP, Triple P also requires caregiver attendance in all sessions; it targets conduct problems among youths ages 17 and under (34). TF-CBT addresses trauma among youths ages 3 to 18. Optimal implementation of TF-CBT prescribes caregiver participation in every session, although it can be delivered effectively without a caregiver participant (35). Not considered a standalone EBP, MAP is a system of decision support tools that guide evidence-based treatment planning and delivery; therapists can select practice elements for anxiety, depression, conduct problems, and trauma for individuals ages 21 and under (36). Although most MAP practice elements that target conduct problems are caregiver directed, some strategies may be exclusively youth-directed (e.g., problem-solving skills training). Seeking Safety is youth directed and targets trauma and substance use in individuals ages 3 to 20 (37).

This study examined EBP implementation as usual and did not assess therapists’ specific training histories. The EBP training requirements established by LACDMH in collaboration with EBP developers are briefly described elsewhere (38–41) and are described in detail in the 2016 PEI Implementation Handbook; these guidelines were active during the time of data collection for this study (42). Generally speaking, after the initial training, Seeking Safety and Triple P do not require ongoing consultation, but CPP, MAP, and TF-CBT require consultation at varying lengths and formats. CPP requires 18 months of consultation calls with external trainers. MAP requires 6 months of ongoing consultation, which can be provided by a certified external or internal trainer; MAP therapists also submit case-based portfolios for certification. TF-CBT requires 16 consultation calls with external trainers along with audiotape review of two sessions.

Sample

Therapist participants worked across 14 LACDMH-contracted agencies. Eligible therapists delivered one of the five EBPs of interest to a youth client. The sample included 101 therapists who delivered care to 267 youth clients. Most therapists were female (N=89, 88%), had a master’s degree (N=86, 85%), and were not yet licensed (N=82, 81%); the mean±SD age was 33.9±8.7. The sample was racially and ethnically diverse, with 56 (55%) identifying as Latinx, 22 (22%) as white, and 23 (23%) as other (e.g., Asian, African American); 53 (52%) reported being able to deliver treatment in Spanish. Data were collected on 685 sessions, with each therapist contributing an average of 6.78 sessions and an average of 2.57 sessions per client.

Procedure

All procedures were approved by the institutional review boards at the University of California, San Diego; University of California, Los Angeles; and LACDMH. Data collection occurred between 2015 and 2017. Informed consent was obtained at recruitment meetings held during staff meetings at each community mental health agency. Therapists completed a baseline survey about their background and perspectives on EBPs and submitted questionnaires describing their delivery of EBPs for three sessions for up to three clients. Therapists obtained permission from caregivers for session recording, but no identifying information was collected about clients or caregivers beyond basic demographic characteristics and presenting problem. Therapists received $20 for completion of the survey and $5 for completion of each session questionnaire.

Measures

Therapist characteristics.

Therapists reported their personal characteristics (age, gender, and race-ethnicity) and professional history (licensure status, highest degree obtained, ability to deliver services in a language other than English, and caseload information) on the Therapist Background Questionnaire (43).

Perceived Characteristics of Intervention Scale (PCIS).

An adapted version of the PCIS was used to measure therapist perceptions about each EBP that they delivered (44). To mitigate participant burden, the 20 original PCIS items were pared to eight, including items measuring perceptions related to relative advantage (e.g., “[The EBP] is more effective than other therapies I have used”), compatibility (e.g., “[The EBP] is aligned with my clinical judgment”), complexity (e.g., “[The EBP] is easy to use”), and potential for reinvention (e.g., “[The EBP] can be adapted to meet the needs of my patients”). Items were rated on a 5-point Likert scale, from 1 (not at all), to 5 (a very great extent); a higher mean score reflected more positive views of the EBP. The eight-item scale had excellent internal consistency for all EBPs (Cronbach’s α=.92–.96).

Session Questionnaire.

For each session, therapists reported client demographic characteristics (gender, race-ethnicity, and age), targeted presenting problem, language used in session, and the EBP delivered.

EBP and expected caregiver attendance.

Based on a review of intervention materials, EBPs were categorized according to expected level of caregiver attendance. EBPs were classified into the following three-level ordinal scale: caregiver attendance not routinely expected (Seeking Safety and MAP targeting nonconduct problems [MAP-other]); caregiver attendance expected in at least some sessions (TF-CBT and MAP targeting conduct problems [MAP-conduct]); and caregiver attendance expected in every session (Triple P and CPP).

Sessions in which caregiver attendance was indicated.

On the basis of the evidence base, we identified sessions in which caregiver attendance was indicated as those in which the child was age 12 or under and the reported presenting problem was disruptive behavior or ADHD or those that involved the delivery of TF-CBT, MAP-conduct, Triple P, or CPP.

Caregiver attendance.

Caregiver attendance was a dichotomous, session-level variable based on therapists’ report of who attended the session. Therapists provided responses to, “Who was involved in the session on which you are responding?” Caregiver attendance was considered to occur if the session was attended by only a collateral caregiver or by both a caregiver and a youth.

Statistical Analysis

Multivariable binomial logistic regressions that accounted for a three-level data structure were conducted in Stata, version 15.1; sessions were nested within youth clients, who were nested within therapists (45). To address research question 1, analyses were conducted in the full sample to examine the relationship between caregiver attendance and the youth’s age, presenting problem, and caregiver attendance expected for the EBP (no sessions, some sessions, or all sessions); youth- and therapist-level covariates were included. To address research question 2, the model was conducted with the subsample of sessions involving an EBP in which caregiver attendance is indicated, with the goal of examining differences in caregiver attendance by youth and therapist attributes.

Results

Table 1 displays characteristics of the youths and sessions for the 685 sessions in which an EBP was provided. Table 2 shows overall rate of caregiver attendance and how caregiver attendance varied by client age, gender, presenting problem, and EBP type in the full sample of sessions and in the sessions in which caregiver attendance was indicated. Caregiver attendance occurred in 42% of sessions in the full sample and in 49% of sessions in which caregiver attendance was indicated. The rate of caregiver attendance was higher in sessions with children ages 12 and under compared with sessions with adolescents (13 and over). Caregiver attendance occurred more often in sessions with boys than with girls. In the full sample, caregiver attendance occurred in 53% of sessions targeting an externalizing problem, 45% of sessions targeting trauma, and 29% of sessions targeting internalizing problems. Caregiver attendance also varied by EBP, ranging from 28% of Seeking Safety and MAP-other sessions to 99% of Triple P and CPP sessions.

Predictors of Caregiver Attendance

Table 3 presents adjusted coefficients (b) or odds ratios (ORs) and 95% confidence intervals (CIs) for the full-sample model. Results showed significant associations between caregiver attendance and youths’ age, youths’ gender, and the type of EBP delivered. The odds of caregiver attendance were higher for sessions with children ages 12 and under compared with adolescents (OR=3.67, CI=1.53–8.84, p<.01); for sessions with boys compared with girls (OR=2.12, CI=1.08–4.16, p<.05); and for sessions with Triple P and CPP compared with Seeking Safety and MAP-other (OR=988.31, CI=45.52–21,459.11, p<.01). The other therapist and youth factors were nonsignificant.

Determinants of Caregiver Attendance at Indicated Sessions

Youths’ age, youths’ gender, EBP type, and presenting problem showed a significant association with caregiver attendance at sessions involving an EBP with an expectation of at least some caregiver attendance (Table 3). Similar to full-sample results, the odds of caregiver attendance were higher in sessions with children compared with adolescents (OR=4.10, 95% CI=1.30–12.88, p<.05) and with boys compared with girls (OR=3.06, CI=1.39–6.75, p<.01). Once again, sessions in which Triple P or CPP was delivered had increased odds of caregiver attendance compared with sessions in which Seeking Safety or MAP-other was delivered (OR=2,545.52, 95% CI=51.68–125,392.40, p<.001). Sessions targeting externalizing problems had increased odds of caregiver attendance compared with those focusing on internalizing problems (OR=4.21, 95% CI=1.22–14.61, p<.05). Caregiver attendance was not significantly associated with other therapist factors and youths’ race-ethnicity.

Discussion

This study evaluated one quality indicator, caregiver attendance, in the delivery of multiple EBPs within a systemwide implementation effort. The analyses consistently demonstrated that sessions with younger children were associated with increased odds of caregiver attendance compared with sessions with adolescents. From a developmental perspective, this finding is not surprising, given that treatment with young children tends to involve caregivers (46). Moreover, adolescents may decline caregiver involvement because of feelings of discomfort when discussing emotional topics or because the nature of the parent-child relationship contributes to their presenting problem (47). Additionally, the delivery of Triple P and CPP, which prescribe caregiver attendance at every session, demonstrated increased odds of caregiver attendance compared with practices in which caregiver attendance is not routine. This finding is a promising indicator for the implementation of caregiver-mediated EBPs because it suggests some success in engaging caregivers.

Results show that few therapist and youth factors were associated with significant differences in the occurrence of caregiver attendance, with the exception of youths’ gender. Sessions with boys had increased odds of caregiver attendance in both models; this gender disparity was most pronounced in sessions targeting externalizing disorders and trauma. Haine-Schlagel and colleagues (48) previously found that community therapists were more likely to involve caregivers in the treatment of children with disruptive behavior if the patient was a boy. More generally, studies show that girls with externalizing problems are less likely to obtain necessary treatment (30, 31). The benefits of caregiver treatment participation for child trauma are unclear (49), and no previous studies document gender differences in caregiver attendance for trauma. However, girls may be less likely than boys to have caregivers who are available to support trauma treatment and who are themselves not a perpetrator, given that girls are more likely than boys to come from families where the parents are the perpetrators of abuse (50).

This study cannot narrow explanations for this observed gender disparity in caregiver attendance, which may be driven by therapist or youth and family factors. Given that most therapists were female, they may perceive a stronger alliance with girls, and view caregiver attendance as less necessary for treatment progress. Indeed, gender-matched therapist-youth dyads have reported stronger alliance than gender-mismatched dyads (51). Therapists may also expect productive individual therapy with girls based on gendered norms of boys being comparatively less expressive (52). Considering that the youths in our sample were predominantly Latinx, cultural factors may also shape this gender disparity. Traditionally, more restrictions are placed on girls in Latinx families; Latinx caregivers may be more motivated to participate in the treatment of sons because they are less equipped to set limits on boys’ problem behaviors (53). Although the explanation is unclear, the implication of the finding is that quality improvement efforts should focus on reducing the gender disparity in caregiver attendance. Clinical supervisors could monitor caregiver attendance, discuss how symptoms may manifest differently by gender and culture, and encourage caregiver inclusion through psychoeducation when it is indicated (54).

Overall, caregiver attendance occurred in 42% of all sessions and at only a slightly greater rate (49%) in sessions in which it was expected. This rate compares unfavorably to another community study, which found caregiver attendance in 70% of sessions with predominantly non-Hispanic white youths presenting with externalizing problems in usual care (55). The lower rate of caregiver attendance observed may be due to our majority Latinx sample, which is a population that faces increased barriers to treatment, such as a shortage of bilingual therapists (56). In addition, for EBPs in which at least some caregiver attendance was indicated, sessions involving externalizing problems had increased odds of caregiver attendance compared with sessions involving internalizing problems. However, the lack of association between caregiver attendance and externalizing problems in the full sample is concerning and requires further study.

It is unclear what accounts for suboptimal levels of caregiver attendance. Factors that may reduce caregiver attendance in community services are the inclusion of child welfare–involved youths who are not in the care of a long-term primary caregiver and low caregiver motivation (12, 16, 57). Another possibility is that system-driven EBP implementation has inadvertently shaped services. An unintended consequence of implementation is that the EBPs adopted by the agency and provider tend to dictate the content and form of services perhaps to a greater extent than the child’s presenting problem (58).

Altogether, the lower than optimal rates of caregiver attendance observed in this study are a reminder of the challenges of caregiver engagement in youth community treatment. These and other findings suggest the need for caregiver engagement interventions, which have gained increasing attention in the field (59). Tracking caregiver attendance as a quality indicator can help align community practice with evidence-based recommendations for caregiver treatment participation and can help to identify children who could benefit from caregiver engagement strategies. The results may have implications for the adoption of quality assurance procedures. For example, an agency can conduct a rapid quality assessment by collecting data on caregiver attendance through routine audits of billing for family or collateral services for youths; if rates of caregiver attendance are suboptimal for services with indications for caregiver engagement, focal quality improvement efforts could work to pinpoint reasons for low penetration of caregiver attendance. Clinical supervisors and providers can then attempt to resolve the problems that have circumvented efforts to involve caregivers.

This study adds to the literature examining caregiver engagement as a quality indicator; however, the findings should be considered in light of limitations (14, 16). First, caregiver engagement is only a proxy of treatment engagement, and assessment of the quality of in-session caregiver participation is needed (54). Second, it is unknown whether caregiver engagement was driven by youth, caregiver, or therapist preferences or decisions. For therapists, data were not collected on ongoing engagement in training activities, such as fidelity monitoring, which could have influenced caregiver engagement. Third, our sample included an overrepresentation of unlicensed therapists; thus it may not generalize to a more experienced workforce. Last, because of the client sample composition, we were unable to explore racial-ethnic disparities; further research is needed in this domain.

Conclusions

Measurement of caregiver engagement can be utilized as a pragmatic quality indicator to help identify quality gaps in EBP community implementation. Study findings suggest the need for improvement in the quality of treatment for girls and youths being treated for externalizing problems. Monitoring the quality of care remains essential, given that implementation of EBPs is likely to remain a priority of publicly funded behavioral health system reform.