The Veterans Health Administration REACH VET Program: Suicide Predictive Modeling in Practice

Accurate risk identification is a component of high-value suicide prevention care. It can result in efficiently connecting at-risk patients to appropriate treatment. Traditionally, methods of risk identification have included screening and assessment to detect those with risk factors and warning signs. However, authors of a large meta-analysis concluded that the ability to predict suicide solely on the basis of epidemiological risk factors is only slightly better than chance (1). Given the critical public health concern suicide represents (2), researchers have developed predictive models to improve suicide prevention capabilities. An initial proof-of-concept model developed by U.S. Veterans Health Administration (VHA) and National Institute of Mental Health researchers documented that a suicide mortality risk prediction algorithm could be developed from VHA electronic health record (EHR) data (3). To facilitate implementation by the VHA, developers used machine-learning analyses to generate a model that performed similarly when compared with the original model but used 84% fewer variables (4).

Work has been done to conceptualize how to incorporate such models into clinical workflows; however, some researchers have suggested that clinical implementation may not be appropriate given potential risks (5). For example, concerns have been noted regarding health care systems’ ability to manage identified risk effectively and safely with limited resources, especially because many people identified as being at risk do not die by suicide (5). On the basis of these considerations, VHA developed and implemented a novel clinical program based on the VHA predictive model, the Recovery Engagement and Coordination for Health–Veterans Enhanced Treatment (REACH VET). VHA policy (6) requires that each veteran identified to be in the top 0.1% tier of statistical risk of suicide have their care evaluated and be contacted by a VHA provider (6). REACH VET identification was associated with a 5% reduction in the probability of a documented suicide attempt, with adjustment for patient age and sex, as well as with more outpatient encounters, increased documentation of new suicide prevention safety plans, and fewer inpatient mental health admissions and emergency department visits. Analyses did not identify an effect on suicide, but they may have been underpowered because death data were not available for the entire cohort (7). This column complements the evaluation work and describes the REACH VET clinical program, implementation strategies, and an evaluation of implementation feasibility.

Clinical Program Overview

The suicide risk prediction model was run monthly for patients who had received care in the prior 24 months at each of the 140 VHA facilities. The statistical model is composed of 61 risk variables derived from the EHR. Each patient received a risk score, and those who fell in the top 0.1% risk stratum were identified to receive intervention. Model development and validation efforts found individuals in this tier of risk to have suicide rates 39 times higher than that of the general VHA patient population (3, 4). An estimate in excess of 89 suicide deaths over 12 months is anticipated among the approximately 6,500 veterans identified each month.

Within 2 weeks of identification of a veteran at risk of suicide, the VHA provider who worked most closely with the patient was required to make contact and evaluate the patient’s clinical risk, as well as to collaboratively discuss the patient’s care needs and to offer suggestions for care enhancement. Unless contraindicated, it was recommended that providers inform patients of their identification by the predictive model and provide education regarding the REACH VET program. If the veteran had been previously identified, then the provider determined whether additional outreach was indicated.

Implementation

Rapid phased national implementation of REACH VET started in November 2016, when the predictive model identified eight veterans with the highest risk scores at each facility. By February 2017, facilities received the names of all veterans in the top 0.1% risk tier at their facility.

A wide range of strategies were used to support REACH VET implementation. Exemplars of implementation strategies (8) from a four-phase (exploration, preparation, implementation, sustainment) framework (9) are provided in Table 1. Critical program development decisions were made during the preparation phase. For example, because of the limited evidence base regarding interventions for those at statistically significantly elevated risk of suicide, it was decided that specific interventions (e.g., safety planning) should be recommended but not required.

Implementation strategies were deployed to address barriers. For example, some providers were hesitant to implement REACH VET without having program effectiveness data. In response, the implementation team identified facility-level leaders to serve as internal champions and disseminated patient feedback in support of the program. Additionally, best practices used by high-performing facilities were identified and disseminated to address barriers to integrating required tasks into existing workflows.

Feasibility Evaluation

The period of observation (April 2017–October 2020) began 2 months after full national implementation. Data were sourced from the U.S. Department of Veterans Affairs Corporate Data Warehouse. Data elements generated by completion of standardized note templates were transmitted from the EHR into the Corporate Data Warehouse. Four monthly REACH VET performance metrics were calculated with these data: “coordinator accepted” (coordinator documentation of receiving the name of the identified veteran and assigning a provider), “provider accepted” (provider documentation of receiving the name and required tasks), “care evaluation performed” (provider documentation of care reevaluation), and “outreach attempted” (provider documentation of an attempt to contact the veteran).

During the period of observation, an average of 6,593 (range 6,368–6,662) veterans were identified monthly across VHA facilities. Among the veterans identified in the final observation month (N=6,579), most were male (95.3%, N=6,268) and White (86.3%, N=5,675). More than 90% (N=6,214) of veterans had major depression or a substance use disorder, and more than 75% (N=5,055) had an anxiety disorder diagnosis. Almost half (N=3,063) had experienced homelessness in the prior 2 years.

Performance on all four metrics improved during the 3.5-year observation period. Coordinator accepted was consistently the task completed at the highest rate, with the lowest rate in October 2017 (89%) and the highest in April 2020 (100%). Outreach attempted consistently had the lowest rate, ranging from 50% to 98%. From April 2017 through October 2020, completion of the coordinator accepted task increased from 90% to 100%, provider accepted increased from 58% to 99%, care evaluation performed increased from 54% to 98%, and outreach attempted increased from 50% to 98%. All metrics met or exceeded the 95% benchmark by April 2020. The variation between facilities with respect to performance decreased over time. For example, in April 2017, performance on the outreach attempted metric ranged from 0% to 100%, but by October 2020, it ranged from 81% to 100%.

Discussion

REACH VET is a clinical intervention based on a suicide prediction statistical model. A wide range of implementation strategies were used to facilitate program uptake. Performance on all metrics related to completion of program requirements improved throughout implementation. Targets were met and sustained until the end of the observation period. That is, 98% of veterans identified were contacted by providers and had their care reevaluated. These findings suggest that it is feasible to implement a clinical program based on a suicide prediction model.

Several factors likely contributed to successful implementation of REACH VET. Suicide prevention is a top priority among senior VHA leaders. Consistent senior leadership engagement was evidenced by regular briefings between VHA national suicide prevention program staff and the Under Secretary for Health, as well as Congressional leaders. These briefings generated inquiries regarding implementation at underperforming facilities. National suicide prevention leaders also provided Veterans Integrated Service Network (VISN; i.e., regional) leaders with monthly performance metrics, which they disseminated to facilities in their region. Tailored outreach efforts, supported by VISN leaders, were provided to underperforming facilities.

Critical tasks completed during the preparation phase likely contributed to program feasibility. The implementation team made multiple ethical, clinical, and operational decisions while developing the intervention. Outer and inner contextual factors were considered throughout this process to maximize innovation-values fit, which has been shown to enhance implementation (9). For example, the value of transparency was incorporated into the decision to inform providers of patient-level clinical variables contributing to suicide risk. Additional key decisions were made in regard to identification of potential false positives. The decision not to prescribe specific interventions facilitated patient-centered care and resource allocation, because time-intensive interventions were not required for all identified patients.

The development of a diverse strategy for implementation also likely affected feasibility (8). The approach was flexibly modified during the implementation and sustainment phases, via collaboration with leadership and the field. The choice of specific implementation strategies likely enhanced feasibility. For example, REACH VET coordinators were internal, local champions who served as liaisons between the national implementation team and the local facility, enhancing local tailoring. Additionally, audit and feedback, paired with proactive technical assistance, allowed for the identification and support of underperforming facilities. The implementation team worked with stakeholders to identify and address specific barriers, provide practical solutions, and build trust.

This study did not include a formal evaluation of contextual factors, such as organizational climate, that could inform subsequent implementation efforts. Moreover, it was unknown which factors were associated with facility performance. Little is known about the patient experience of REACH VET and how coordinators and providers use the program to enhance care. One study, however, reported that veterans on an inpatient psychiatric unit found the idea of REACH VET to be generally acceptable (10). Analyses are under way to evaluate the cost-effectiveness of REACH VET.

The findings noted in this column highlight the feasibility of implementing a suicide prevention clinical program that is based on predictive modeling. Although predictive modeling is complex, it can facilitate relatively simple outreach efforts associated with decreased suicidal behavior (7). We recommend that health care systems interested in implementing a similar program utilize a multifaceted and flexible approach to implementation. Such an approach can facilitate adaptation to meet unique system needs. Additionally, this approach allows health care systems to ensure ethical implementation of such a program by empowering their leaders to consider factors such as which types of care patients at risk of suicide are connected with and how this connection occurs.