Trainer Fidelity as a Predictor of Crisis Counselors’ Behaviors With Callers Who Express Suicidal Thoughts

Suicide is a significant public health problem in the United States. It is the tenth leading cause of death for adults overall, with higher rates among youths and older adults (1). Telephone crisis lines are an important component of the national suicide prevention strategy (2). The National Suicide Prevention Lifeline is a network of over 160 community crisis centers that have served over three million callers since the network’s inception in 2005. Crisis counselors must be well trained to assess and intervene with callers who communicate suicidal thoughts and feelings (3).

Gould and others (4) conducted the first national randomized controlled trial (RCT) to study whether a crisis center training program increased the effectiveness of Lifeline’s telephone crisis services. The program, LivingWorks’ Applied Suicide Intervention Skills Training (ASIST) (5), is a broadly disseminated suicide prevention training program that uses a train-the-trainer (TTT) model.

The study used a dynamic waitlisted “roll out” design (6) in which ASIST TTT was offered during three different time frames to 17 crisis centers; the centers were randomly assigned to the first, second, or third time frame. Two staff members from each center participated in a five-day training program to learn to be ASIST trainers during two-day programs. These trainer pairs then delivered the group-based program to crisis counselors at their home centers. Crisis calls were reliably “silent monitored” at each center before and after training. The silent monitors were blind to the identity of the center and to whether the monitored crisis counselors had received ASIST. The monitoring tool assessed behaviors by the counselors, including global positive behaviors (for example, empathic behavior), negative behaviors (for example, judgmental behavior), overall effectiveness, and behaviors consistent with the ASIST program, as well as duration of the calls and caller behaviors. Multilevel modeling was used to analyze outcome data derived from 1,507 monitored calls from 1,410 suicidal individuals nested within the crisis centers.

Results showed that counselors who had received ASIST were more likely than those who had not received ASIST to link the callers’ invitations (verbal cues about suicide) to suicidal thoughts, explore reasons for living, explore ambivalence about dying, and explore whether informal support contacts were part of the callers' safeplans. Callers who spoke with ASIST-trained counselors were rated as less depressed, suicidal, and overwhelmed by the end of their calls compared with callers who had not spoken with an ASIST-trained counselor. However, ASIST-trained counselors were not more likely to carry out more comprehensive suicide risk assessments compared with counselors who had not received ASIST training, nor were they more likely to ask about or explore the callers’ current suicide plans, ask about or explore preparatory behaviors or actions, ask about or explore intent, or ask about or explore prior suicide thoughts or attempts. Also, they were not rated as having engaged in more positive behaviors overall or as having provided more effective interventions.

Simultaneous with this outcome study, Cross and others (7) investigated fidelity to ASIST among trainers who were trained, via TTT, to deliver subsequent training to counselors in their home centers. Cross and colleagues analyzed videotapes of the trainers as they delivered the two-day ASIST program to counselors at their centers. Consistent with the literature (8,9), trainer fidelity was defined as adherence to the manual content and competence in program delivery (quality of presentation). Seven segments of the program (exploring invitations, asking about suicide, listening to reasons for living/dying, contracting a safeplan, process of intervention, ambivalence simulation/role-play, and bridge simulation/role-play) were coded by using a segment-specific trainer adherence measure (reflecting content for each of the seven segments) and one trainer competence measure was used for all segments. Almost 50% of the videotapes were double-coded for interrater reliability, which was high for both adherence and competence (average intraclass correlation, .80–.98 and .71–.92, respectively).

Ratings were analyzed for 66 training sessions conducted by 34 trainers at the 17 centers, for a total of 324 recorded observations representing the seven training segments. Results showed that the majority of trainers were adherent to the program content and delivered two-thirds or more of the intervention content to counselors. Competence ratings were lower than adherence ratings, with 18% of observations rated as solidly capable. Ratings were lowest for interactive training activities (role-playing). Previous suicide prevention training was associated with lower levels of trainer adherence.

To address a critical gap in our understanding about the implementation of TTT programs in real-world settings, this study combined data from these two published studies to examine the relationship between trainer fidelity (measured by adherence and competence) and program outcomes (measured by counselor behaviors with callers expressing suicidal thoughts). We hypothesized that higher levels of trainer adherence and higher competence ratings would be associated with improved counselor behaviors on calls after ASIST training.

Methods

Only posttraining outcome data were included in the analyses because outcome data that preceded the ASIST training was not relevant to linking outcomes and fidelity. We first aligned the silent-monitored outcome variables from 764 calls with the relevant segments of the training and the fidelity ratings of the trainer who trained counselors in the center. For example, the segment of the training that focuses on engaging suicidal individuals in a discussion about reasons for living corresponds to a silent-monitored outcome variable. Each segment yielded trainer fidelity scores (adherence and competence) that were examined in terms of associated outcome variables. Four of the seven segments produced significant results: asking about suicide, contracting a safeplan, process of intervention, and bridge simulation. As the final segment of ASIST, bridge simulation is a highly interactive role-playing technique that incorporates all of the knowledge and skills imparted by the trainers. Therefore, we examined all of the outcome variables with this segment’s fidelity scores.

The RCT protocol was approved by the Institutional Review Board (IRB) of the New York State Psychiatric Institute and Columbia University. This study was approved by the University of Rochester IRB.

The generalized estimating equation (GEE) analysis was employed to address clustered data resulting from the nested study design (counselors nested within centers) and because it is considered preferable to a parametric alternative such as a generalized linear mixed-effects model (10). Variability in number of calls received by counselors was addressed by weighting each center by using the reciprocal of the number of calls from the center. The effect of trainer adherence with and competence in specific segments on counselor behaviors during the calls (for example, asking about current suicidal thoughts and safeplan elements), positive counselor behaviors, negative counselor behaviors, and a global rating of overall effectiveness were examined. The identity, logit, or log link was used depending on whether the outcome was treated as a continuous, binary, or count response. If a categorical outcome had more than two categories, we fit the multinomial response model or the proportional odds model, the latter of which is a simplified version of the multinomial response model, if assessment by goodness-of-fit tests indicated that the latter model was more appropriate (10). Some outcomes had a large percentage of 0 scores (for example, 80% of outcomes for negative counselor behaviors were rated 0, or “not at all”); the hurdle model was used to fit such outcomes. The hurdle model has two components: the logistic component models the effect of adherence and competence on the probability of whether counselors have negative behaviors (1 versus 0), and the Poisson component models the effect of adherence and competence on the variability of negative behaviors (number of negative behaviors) for the subgroup of counselors with recorded negative behavior. For binary and categorical responses, the lowest level was set as the reference level.

All statistical tests were two sided; p values of <.05 were considered statistically significant. All statistical analyses used SAS, version 9.3.

Results

Table 1 presents results for all segments. GEE estimates (betas) show that 12 of the 16 significant findings are for ASIST trainer competence. For example, higher observed trainer competence in the asking about suicide segment was associated with greater instances of counselors asking about prior suicidal thoughts and current plans. ASIST trainer competence during the bridge simulation segment was also associated with counselors reminding callers of past survival skills and identifying personal and formal resources. Greater trainer adherence during the bridge simulation was positively associated with the counselor asking about or exploring suicidal actions with callers and exploring ways for callers to manage psychological pain. Two significant findings for adherence during the bridge simulation were not in the expected direction: the more adherent the ASIST trainer, the less likely counselors were to explore prior suicide attempts, and the rare instances of negative counselor behaviors were associated with greater trainer adherence.

Discussion

This is the first study to examine the relationship between trainer fidelity and suicide prevention program outcomes among crisis hotline counselors in the context of a TTT model. Overall, we found that trainer competence, more than trainer adherence, was associated with better performance by crisis line counselors with callers who expressed thoughts of suicide. Greater trainer adherence to program content was actually predictive of poorer outcomes in two cases.

How do we account for the finding that program content had little impact on counselor behaviors? One factor may be that these counselors were already working on the crisis line and had prior experience in suicide prevention. They, like experienced clinicians who resist learning new interventions (11), may not have been persuaded to adopt new information or practices. Moreover, consistent with previous findings that strict adherence to an intervention’s manual may be associated with poorer therapeutic outcomes (12), adherence to the ASIST program had a deleterious effect on some counselor outcomes. Future studies should study outcomes for novice counselors without prior experience or training. The reason for the positive impact of trainer competence on counselor outcomes is not known; however, it is consistent with the finding that therapist competence (but not adherence) predicted positive outcomes among adolescents in substance abuse treatment (13). Future studies on the relative importance of competence and adherence in a variety of training contexts are needed to address this gap.

Regarding limitations of the study, we did not have individual counselor or trainer data, such as the amount, type, or duration of previous crisis training. Nor did we have data on job duration or status (volunteer or paid) or center culture or climate. These factors may moderate the relationship between trainer fidelity and outcomes. We also did not have information about modifications (14) that the trainers may have made to the ASIST program.

Conclusions

Trainer competence was positively associated with counselors’ use of detailed risk assessment (asking about suicide thoughts and plans) and safety planning (identifying resources) with callers who expressed suicidal thoughts. ASIST trainer delivery is more important than program content when the “end users” are suicide crisis line counselors. Our results are consistent with findings by Gould and others (4) that ASIST-trained counselors were not significantly more likely than counselors who had not received ASIST to ask about or explore a caller’s current suicide plans, preparatory behaviors intent, or prior suicide thoughts or attempts. In this study, adherence to ASIST per se, as opposed to trainer competence, was not associated with more comprehensive risk assessments. The implication of this finding is that programs should prioritize the selection of trainers who have group facilitation skills, are comfortable directing experiential learning activities, and are willing to embrace new program content.