The Relationship Between Suicidal Behaviors and Zero Suicide Organizational Best Practices in Outpatient Mental Health Clinics

Suicide is a growing public health crisis. Since 1999, the national prevalence of death by suicide has steadily increased (1). In 2017, it was the 10th leading cause of death in the United States for all age groups and the second for ages 10–44 years (2). Most individuals (83%) who died by suicide accessed general medical or mental health care in the year before their death (3, 4), and suicide and intentional self-harm were the fastest-growing reasons for psychiatric emergency room visits between 2010 and 2014 compared with all other mental health- or substance use–related reasons (5). These facts suggest that health care systems could reduce suicide by improving identification and treatment of individuals at increased risk.

Encouraging health care systems to provide safer, more effective care of suicidal patients has become a national priority (6, 7). In 2012, the Office of the Surgeon General and the National Action Alliance for Suicide Prevention (NAASP), a public-private partnership dedicated to reducing suicide, released a broad national suicide prevention strategy (6). The NAASP Clinical Care and Intervention Taskforce focused on recommendations that were targeted specifically to health care settings (8) and based on an environmental scan of large-scale suicide prevention efforts, including the U.S. Air Force Suicide Prevention Program (9), the quality improvement initiative of the Henry Ford Health System’s Perfect Depression Care (10, 11), Central Arizona Programmatic Suicide Deterrent System, the National Suicide Prevention Lifeline, and others (12). These programs provided compelling evidence that health care systems can reduce suicide through a bundled set of interventions coupled with intentional and sustained leadership and continuous quality improvement activities. The taskforce named this approach Zero Suicide (ZS) to reflect an aspirational goal of preventing all suicides by patients in health systems and to provide a framework for suicide prevention within health care settings. A similar national effort targeting health services in the United Kingdom reduced suicide deaths and, importantly, found that mental health services with a more comprehensive approach had greater reductions, underscoring the need for developing and implementing a systematic model (13). Early adopters of the ZS framework, including Centerstone, a behavioral health system in Tennessee, and the Institute for Family Health, a large federally qualified health center primary care network in New York, observed reductions in suicide incidents of >50% within 3 years (12). The ZS Toolkit was informed by these early adopters and was developed by the Suicide Prevention Resource Center to support large-scale implementation (12, 14), and randomized controlled trials of ZS are under way (15–17).

Hogan and Goldstein Grumet (12) have defined seven core components of the ZS model to improve the identification and treatment of individuals most at risk for suicide. Three of these components address administrative best practices in managing change, including leadership, training, and measuring outcomes and conducting quality improvement. Four additional components reflect established best practices in suicide care, namely, suicide screening and risk assessment (18–21); use of systematic suicide care protocols that include safety planning and lethal means reduction (22–29); evidence-based treatment to address suicidal thoughts and behaviors directly, in addition to other mental health issues (30–34); and provision of support during care transitions, with follow-up after discharge from acute care settings such as “caring contacts” (35–37).

Further highlighting the growing national consensus around these core suicide safer care practices, two national accrediting organizations, The Joint Commission and the Commission on Accreditation of Rehabilitation Facilities, have recently amended their accreditation standards to include suicide screening, risk assessment, and follow-up care after discharge, listing ZS among suggested resources (38, 39). The ZS Toolkit provides tools to assist providers and health care systems in implementing model practices, including the ZS Organizational Self-Study, which assesses fidelity to ZS organizational best practices (40).

Growing support for the model has led to implementation projects across diverse health care systems, states, and tribal nations in the United States (16, 17). Understanding the relationship between fidelity to ZS organizational practices and suicide-related outcomes is critical to support ongoing ZS implementation and research efforts (41–43). The current study examined the relationship between fidelity to the organizational best practices promoted by the ZS model and suicidal behaviors in the year preceding a statewide implementation of ZS in mental health clinics. We expected that clinics with higher fidelity would be less likely to have had patients with suicidal behaviors during the previous year. In addition, we examined the ZS Organizational Self-Study tool’s psychometric properties to support the large-scale implementation project and to inform future use of this instrument.

Methods

Setting

The study focused on community-based mental health clinics not affiliated with hospitals in New York State (NYS) in the year before large-scale ZS implementation. The Suicide Prevention Continuous Quality Improvement (SP-CQI) project was launched in 2017 to support ZS model implementation in outpatient mental health clinics and was a collaborative effort between the NYS Office of Mental Health (NYSOMH) Bureau of Evidence-Based Services and Implementation Science and Psychiatric Services and Clinical Knowledge Enhancement System (PSYCKES) (44), the NYSOMH Suicide Prevention Office (45), and the Center for Practice Innovations at the Columbia University Department of Psychiatry, NYS Psychiatric Institute (NYSPI) (15, 46). The institutional review boards of the Nathan Kline Institute for Psychiatric Research at the NYS Office of Mental Health and NYSPI determined that the study project did not meet the definition of human subjects research.

Sample

All non–hospital-affiliated, community-based, and NYS-licensed mental health clinics (N=321) were invited to participate in the SP-CQI project. Moreover, state-run psychiatric hospital clinics were required to participate but were excluded from this study because of concerns about bias (due to a centralized governing structure mandating participation) and generalizability (NYS is one of the few states to operate a large-scale network of psychiatric hospitals). The study sample included clinics voluntarily enrolled as of April 1, 2017 (N=134 clinics, a 42% recruitment rate), that completed their baseline ZS Organizational Self-Study (N=131, 98%), and were open 6 months before and after the observation period. The final sample included 110 clinics. In preparation for the study, we compared the sizes (number of patients in a sample week) of participating and nonparticipating clinics and client characteristics, including age, race, Medicaid insurance, serious mental illness, and comorbid substance use. Chi-square analysis using the 2015 Patient Characteristics Survey (PCS) data, a biennial NYSOMH survey of mental health programs, detected no statistically significant differences.

Measures and Data Sources

Clinic and client characteristics.

All clinic and client characteristics were extracted from the 2017 PCS (47), including clinic size (number of patients served during the PCS sample week), and clinic type (whether >50% of population served at the clinic served were children).

Fidelity to ZS organizational practices.

Fidelity to ZS organizational practices (ZS fidelity) was assessed before the SP-CQI project implementation with the ZS Organizational Self-Study instrument (40), which was administered to point persons of clinical projects (clinic leadership or quality improvement staff) via SurveyMonkey between February and May 2017 to inform project planning. The tool includes 17 quantitative items (rated on a 5-point scale, with 5 indicating the highest fidelity) and six descriptive questions (excluded from this analysis). Adaptations to the instrument reflected commonly used language in NYS’s clinics and project requirements (see online supplement to this article). The 17 quantitative items were averaged to compute a total ZS fidelity score for each clinic (48, 49).

Suicidal behaviors in the previous year.

Data on suicidal behaviors—specifically, suicide attempts and deaths—were extracted from a state administrative database, the NYS Incident Management Reporting System (NIMRS) (50). NIMRS is a mandatory reporting system for all adverse incidents (e.g., medication reactions and violence) for NYSOMH-licensed programs. All suicide attempts and deaths are required to be reported within 24 hours of discovery. Because the count of suicidal behaviors was highly skewed, study clinics were dichotomized into two groups: no patients with suicidal behavior incidents (N=40) or one or more patients with suicidal behaviors (N=70). In the year of observation (April 2016–2017), most clinics (N=70, 64%) had reported at least one suicide attempt or death incident among their patients (mean±SD=3.1±4.9, median=1, range=0–27), with 22 clinics (20%) reporting one or more deaths (0.25±0.55, range=0–3).

Data Analysis

Statistical analyses were conducted with SAS, version 9.4, and SPSS, version 25; all statistical tests were two-tailed.

Relationship between ZS fidelity and suicidal behaviors among clinic patients.

A logistic regression was conducted to test whether clinic ZS fidelity was associated with patients’ suicidal behaviors in the previous year. We adjusted the model for clinic size by using 2017 PCS data on the number of patients served in the clinic during a sample week (larger clinics are more likely to have a suicide incident) and by clinic type (mostly child services vs. adult services) because of differences in the prevalence of adolescent versus adult suicide attempts (51). Exploratory analyses with analysis of covariance examined differences in fidelity for individual ZS practices among clinics with and without a suicide incident in the previous year and controlled for clinic size but not clinic type (because of parsimony considerations and lack of a finding in hypothesis testing).

Psychometric properties of the ZS fidelity assessment.

We examined the psychometric properties of the ZS Organizational Self-Study instrument because no previously published findings exist. The ZS Organizational Self-Study had high internal consistency (α=0.90). A principal component analysis (PCA) was conducted by using varimax rotation and the Kaiser criterion to identify whether the instrument had any meaningful subscales. Suitability for PCA was tested with the Kaiser-Meyer-Olkin (KMO) test for sampling adequacy, Bartlett’s test of sphericity, and a correlation matrix review. The study data were suitable for PCA; all 17 items were correlated with at least one other item in the scale (r>0.37). The KMO measure was 0.85, with individual item measures ranging from 0.60 to 0.93, and Bartlett’s test of sphericity was statistically significant (p<0.001). The item with the lowest KMO also had the lowest mean score and assessed the inclusion of suicide attempt survivors in clinic policy (item 3). Excluding this item did not change the overall findings. PCA identified four principal components (with eigenvalues >1), which together accounted for 61.4% of the total variance (26.0%, 14.0%, 11.3%, and 10.1%,). However, the grouping of items was inconsistent (e.g., related practices were not grouped together with frequent cross-loading); therefore, the interpretability criterion was not met. This lack of interpretability suggested that the ZS fidelity instrument is best interpreted as a single scale (see online supplement).

Results

Characteristics of the study clinics and their patients are presented in Table 1. Most clinics were located in urban settings (70%), and 80% predominantly served adults. The participating clinics served 30,257 patients during a typical week.

ZS Fidelity for All Clinics in the Year Before Large-Scale ZS Implementation

Clinics’ ZS fidelity assessed with the ZS Organizational Self-Study instrument ranged from very low to high (1.4–4.1 of 5.0), with a mean±SD of 3.1±0.6, reflecting moderate fidelity. Examining individual organizational practice items, we found that the highest rated item was safety planning (item 12), with a mean of 4.1±0.9, followed by screening using a validated instrument (item 8, 4.0±1.3) and routine suicide screening (item 7, mean of 3.9±0.8) (Table 2). The lowest rated items were policy input from suicide attempt survivors (item 3, 1.2±0.6), assessment of staff suicide care confidence and skills (item 4, 1.8±0.9), identifying and measuring suicide death rates (item 19, 2.5±0.9), and lethal means reduction (item 13, 2.6±1.2). Similarly, a high proportion of clinics reported high fidelity (i.e., scored 4 or 5) on safety planning (78%), but few reported high fidelity to lethal means reduction (16%).

Association Between ZS Fidelity and Suicidal Behaviors Among Patients

Results of the logistic regression model are presented in Table 3. After adjusting for patient census and population type served, we found that clinics with higher ZS fidelity had 0.31 lower odds of having a client with suicidal behavior during the previous year (adjusted odds ratio=0.31, 95% confidence interval=0.14–0.69). In other words, for each unit increase (i.e., one point on a 1–5 scale) on the ZS fidelity scale, clinics were significantly less likely to have any patients with suicidal behaviors. The model explained 33% (Nagelkerke R²) of the variance in suicidal behaviors.

Differences in ZS Organizational Practices Between Clinics With and Without Suicidal Incidents

A statistically significant difference between clinics with and without a suicide incident was observed for total average fidelity scores and for seven of the 17 organizational practice items (Table 2). Medium effect sizes were observed for two items: quality improvement activities focused on suicide prevention (item 20) and lethal means reduction (item 13) (η²=0.097 and 0.073, respectively). Examination of the proportion of clinics who achieved high fidelity (i.e., scored 4 or 5 on the ZS Organizational Self-Study instrument) on these two items identified marked differences for clinics with and without a suicide incident. Nearly half (45%, N=18) of clinics without a suicide incident reported that they met the quality improvement criteria compared with fewer than a quarter (23%, N=16) of clinics with an incident in the previous year. Small but statistically significant effects were observed for five other ZS fidelity organizational practice items: leadership commitment (item 1), assessments of confidence in suicide care and of skills among staff (item 4), suicide risk assessments (item 10), engaging hard-to-reach and no-show patients (item 16), and following up with patients who have been discharged from acute settings (item 17).

Discussion

To our knowledge, this is the first study that reports an association between greater fidelity to ZS organizational practices and lower risk for suicidal behaviors. Specifically, after adjusting for patient census and population type served (adult vs. child), we found that the results supported the hypothesis that clinics with higher fidelity to the organizational practices promoted by the ZS model were less likely to have suicidal attempts or deaths among their patients. This finding was observed before a large-scale ZS implementation and extends preliminary research suggesting that the comprehensive approach encapsulated within the ZS framework is associated with fewer suicidal behaviors among those under care (10, 11). In addition, our findings suggest that specific ZS practices may be important priorities for suicide prevention efforts, particularly initiating suicide-focused quality improvement processes and reducing lethal means.

The study results indicate that the ZS Organizational Self-Study instrument has high internal consistency and concurrent validity. Specifically, the instrument could distinguish between clinics with and without suicide incidents. The instrument is brief, accessible, and in the public domain, and it does not require special training or expert raters (40). Self-assessment is important because it allows clinical programs to use the instrument in order to support internal quality improvement processes and offers a feasible approach to examining fidelity in large-scale implementation initiatives (52, 53).

As described above, the ZS framework was based on innovative quality improvement projects that identified death by suicide as a problem within health care systems and leveraged leadership commitment to monitor and reduce such deaths among individuals under care (9, 11, 54). It is noteworthy that among the 17 ZS organizational practices, the quality improvement infrastructure item had the highest effect size. Organizational best practices for suicide-specific quality improvement were defined as having suicide care embedded in the medical chart, written clinical workflows for suicide care, and data collection and review by clinical teams (e.g., data on the quality of patient suicide care plans). Nearly half (45%) of clinics without a suicide incident reported that they met the quality improvement criteria compared with fewer than one-quarter (23%) of clinics with an incident in the year before. These findings underscore the importance of developing a sustainable data-monitoring and quality improvement infrastructure to support suicide prevention efforts.

In exploratory analyses, lethal means reduction also emerged as an important ZS practice, with the largest mean difference between clinics with and without a suicide incident. Achieving high fidelity to the lethal-means-reduction item requires documentation in safety plans as a standard practice, in addition to policies addressing clinician training, family inclusion in means reduction, and confirmation of means reduction. Interestingly, in this sample, most clinics (78%) reported high fidelity to safety planning, but few (16%) reported high fidelity to lethal means reduction, even though most safety planning interventions are supposed to include lethal means reduction. These findings suggest that staff may require additional training in lethal means reduction and safety planning to be comfortable and effective in these integrated practices. Safety planning interventions that incorporate lethal means reduction are associated with a 45% decrease in suicidal behaviors over 6 months (27). Moreover, clinic policies clarifying expectations for patients, their families, and staff to implement and confirm means reduction may be required to maximize the benefits of safety planning and means reduction counseling. These findings align with literature highlighting the role of lethal means reduction in reducing suicide (22, 23, 55). Future research may examine whether large-scale interventions, such as the SP-CQI initiative, can increase fidelity to lethal means reduction and other best practices and can decrease suicidal behaviors.

This study had several strengths and limitations. Its strengths included data from 110 mental health clinics serving a large and diverse population and the use of state-mandated reporting data as an objective measure for suicidal behaviors. Limitations included the following. First, our findings may not generalize to other treatment settings or patient populations. Second, we did not differentiate between suicide attempts and deaths. Third, the ZS Organizational Self-Study is a self-reported instrument, which may introduce bias. The role of self-reported fidelity has been debated in the literature but can offer a reliable, valid, and cost-effective method in specific contexts (41, 52, 53, 56, 57). Moreover, NIMRS, the resource we used for data indicating suicidal behaviors, is a state administrative database for monitoring serious incidents and adverse incidents and was not designed for research purposes. We could not include unreported suicidal behaviors, such as incidents of which the agency was unaware or suicidal behaviors that did not meet reporting criteria. In addition, although the size of the data set allowed for adjustment for clinic size and type, it did not have the statistical power to enable adjustment for other clinic- and patient-level characteristics that may affect organizational practices and outcomes, an important area for future study.

This cross-sectional study examined suicidal behaviors in the year before the fidelity assessment; the optimal period of such observations is unclear, given the evolution of organizational practices over time. Longitudinal study is needed to investigate fidelity over time and the relationship between gains in fidelity to changes in suicide outcomes. Finally, the exploratory analyses did not account for multiple comparisons, increasing the chance of type I errors, and ceiling effects for select organizational practice items (e.g., safety planning) may have introduced type II errors.

Conclusions

The findings of this study suggest that high fidelity to ZS organizational best practices in outpatient mental health clinics may reduce suicidal behaviors among patients. Exploratory analyses suggested that clinic engagement in suicide-specific quality improvement activities and in strategic development of effective policy- and protocol-based lethal means reduction may be particularly important for reducing suicide risk. Our findings also indicate that the ZS Organizational Self-Study instrument has high internal consistency and concurrent validity with patients’ suicidal behaviors, suggesting it is a useful tool for health care systems.