Evaluating and Improving Engagement in Care After High-Intensity Stays for Mental or Substance Use Disorders

Clinical practice guidelines emphasize continuity of care when a patient is discharged from inpatient or residential mental or substance use disorder treatment (1–3). Good continuity of care after discharge is associated with lower readmission rates, better medication adherence, and subsequent engagement in specialty outpatient treatment (4–6). Strategies to enhance discharge planning, to improve access to outpatient services, and to facilitate a patient’s transition to outpatient providers can improve continuity (7–11). However, the percentage of patients who receive a timely follow-up visit after discharge, an indicator of continuity, varies by patient factors (e.g., age, sex, race-ethnicity, and distance from care), and health care system factors (e.g., public versus private funding, catchment area, and discharge protocols) (12–14). Such variation suggests that opportunities exist to improve the quality of care after discharge from high-intensity mental or substance use disorder stays. Thus for quality improvement purposes, health care payer and provider organizations commonly evaluate rates of follow-up after discharge (15, 16).

The Veterans Health Administration (VHA) is the largest integrated health care system in the United States, with embedded specialty mental and substance use disorder treatment programs. Until October 2017, the VHA used a measure of 7-day follow-up after discharge for quality improvement purposes. The VHA stopped tracking the 7-day follow-up measure because of high levels of performance across health care systems, policy changes for suicide prevention (17), and evidence that a single follow-up visit was not associated with improved outcomes for veterans (18). In its place the VHA Office of Mental Health and Suicide Prevention developed and activated a new quality measure of postdischarge engagement (PDE) in multiple specialty outpatient mental health or substance use disorder visits in the 30 days after discharge.

From October to December 2017, VHA activated the new PDE quality measure as part of a multifaceted quality improvement program designed to incentivize focus on patient engagement, to educate providers and managers, and to enable systematic tracking of patients’ care after discharge. In October 2017, the PDE measure was weighted in VHA’s Strategic Analytics for Improvement and Learning (SAIL) model, which is a national performance management strategy that draws attention to clinical areas via top-down organizational processes—namely by putting evaluation data into executive leadership performance appraisals (19–21). In addition, education designed to support VHA’s transition from a 7-day follow-up measure of continuity to a 30-day engagement measure included three national webinars and five regional webinars with business operations and community of practice networks. Webinar recordings and supporting documents were disseminated for use in local health care systems’ education efforts.

To support change at the care delivery level, a population management tool was deployed between November and December 2017. Population management with this tool refers to care coordination using nightly updated data that allow for tracking the full panel of patients admitted to or recently discharged from qualifying stays (22). The population management tool prospectively identifies patients before they are discharged to facilitate the transition to outpatient treatment. The tool displays real-time information about suicide risk, admission and discharge, appointments, and the name of the patient’s providers (a table in an online supplement to this article lists and summarizes data elements in the tool). Information is shown to all relevant VHA clinical teams, whether they are in different treatment settings, health care systems, or geographic locations. Doing so makes explicit the shared responsibilities for care coordination across teams. An e-mail help desk system has been available for the population management tool since November 2017. The help desk expedites direct-to-field technical assistance from experts in quality improvement, program implementation, and delivery of clinical care. Approximately 1,500 help desk e-mails were answered in November and December 2017. Over 700 unique users per month across 140 VHA health care systems access the population management tool about once per business day (see online supplement).

In this cohort study, we asked the following question: was activating the PDE measure and quality improvement intervention at VHA associated with higher levels of engagement in outpatient mental or substance use disorder care after discharge from high-intensity mental or substance use disorder treatment?

Methods

We used an interrupted time-series design to examine whether activating the PDE quality measure was associated with change in engagement in mental or substance use disorder care after discharge. This evaluation study was determined to be a quality improvement project by the Stanford University Human Subjects Research and Institutional Review Board.

Calculating the PDE Measure

Information about a patient’s inpatient or residential stay, discharge dates, diagnoses, and risk of suicide was obtained from the VHA Corporate Data Warehouse, representing all 140 VHA health care systems in the United States. Discharges were analyzed from all acute care medical and psychiatric settings and from any of VHA’s specialty residential rehabilitation treatment programs or domiciliaries (23, 24). A discharge qualified for the PDE measure denominator if the stay involved either a mental or substance use disorder treatment setting (identified with VHA treating specialty codes) or a mental or substance use disorder was listed as the patient’s principal discharge diagnosis from a medical or surgical unit. Discharges to 24-hour non-VHA settings (e.g., skilled nursing) or to the justice system were excluded. Discharges were included if the criteria for engagement were met prior to a subsequent readmission within 30 days but were excluded if engagement criteria were not met before a readmission within 30 days.

To calculate whether a patient’s care met the measure numerator criteria for engagement, we stratified discharges into four groups on the basis of a patient’s risk of suicide and acuity level of treatment. To qualify as having engaged in care, a patient needed to receive at least two to four outpatient follow-up visits within 30 days of discharge, depending on the denominator group (Figure 1). High risk of suicide was determined by provider-entered flags in a patient’s medical record or by diagnosis code for suicide-related behaviors on the discharge record. For patients at high risk of suicide, four visits were required to meet the engagement definition, regardless of acuity level or treatment setting. The remaining denominator groups were based on a patient’s inpatient or residential treatment setting, as were the number of visits required to qualify as engagement (Figure 1) (see online supplement).

For a 32-month period, from October 2016 to May 2019, monthly performance on the PDE measure was calculated nationwide and for all of VHA’s 140 health care systems. To calculate each system’s denominator, we assigned patients to the health care system responsible for coordinating their mental or substance use disorder care at the time of discharge. Responsibility for coordinating care was based on one of the following three categories: where a patient’s mental health treatment coordinator (MHTC) was located; if a patient had no MHTC, where a patient’s primary care provider (PCP) was located; or if a patient had no MHTC or PCP, where the patient received the most health care visits in the past year. We note that in practice, the PDE measure assigns patients to both a discharging health care system and a coordinating health care system when these systems are different in order to expose the need for coordination across systems. We assigned patients only to coordinating systems in our analysis to avoid duplicate denominator cases. Because VHA is an integrated system, patients can receive inpatient, residential, and outpatient care from multiple health care systems. Thus outpatient care provided at any VHA health care system qualified for engagement. To calculate each health care system’s performance on the PDE measure, we divided the number of numerator cases meeting engagement criteria by the health care system’s number of qualifying discharges each month (a percentage). National performance on the PDE measure was calculated across all VHA health care systems to describe pre- and postactivation performance by denominator subgroup and by discharge diagnoses.

Analysis

A segmented mixed-effects linear regression (25) estimated average change in health care system–level performance (percentage) associated with activating the PDE measure. Separate time segments were defined for before and after the PDE measure and the quality improvement program were activated. To ensure adequate observations per time segment (25), monthly PDE performance from October 2016 through September 2017 represented the preactivation segment (12 months), and January 2018 through May 2019 represented the postactivation segment (17 months). We excluded discharges from October 2017 to December 2017 to avoid confounding from the measure activation and quality improvement rollout.

Monthly health care system performance on the PDE measure was regressed against time to estimate four fixed-effects parameters: the intercept represents average health care systems performance as of October 2016, the time parameter represents average change in performance associated with each month from October 2016 to September 2017, the intervention parameter represents change in performance during the measure activation and quality improvement intervention, and the time-after-intervention parameter represents change in monthly performance thereafter, beyond the effect of the first time parameter. Random effects for health care system intercepts and slopes were included in the model. Following reporting standards for segmented mixed-effects regression (25), we also estimated the number of patients who engaged in postdischarge care, who would not have if the measure was not activated.

While the main mixed-effects regression model takes within–health care system error into account, we took two steps to rule out the possibility that low denominator sizes confounded results. First, we visually analyzed the association between denominator size and health care system–level intervention parameter estimates by using a bubble plot. Second, we conducted a sensitivity analysis by rerunning the main analysis after we excluded health care systems with an average denominator less than 10 across the 32 months observed.

Results are reported as unstandardized estimates and are interpreted as percentage points. An alpha of 0.05 was used to determine significance.

Results

A total of 322,344 inpatient or residential discharges met the PDE denominator criteria from October 2016 to May 2019, excluding the 3-month activation period. Each month before PDE activation, the mean±SD number of discharges was 11,226±650. After PDE activation, the number of discharges nationally averaged 11,036±657. Average performance among the 140 health care systems (i.e., the percentage of qualifying discharges that met criteria for 30-day PDE) ranged from 63.9% to 67.9% each month before PDE activation and from 70.8% to 75.7% each month after, with an average standard deviation across systems of 11.0% within time segments (Figure 2).

Change in Performance Associated With Measure Activation

A segmented mixed-effects linear regression estimated change in performance. The average health care system performance in October 2016 (beginning of the preactivation period) was 65.6% (SE=0.7%, p<0.001). The time parameter in the preactivation period was not significant (estimate=0.1%, SE=0.1%, p=0.39), indicating that engagement did not change significantly month over month before PDE activation. Activating the PDE measure was associated with significantly increased performance, by 5.7% (SE=0.8%, p<0.001) (Figure 3). The month-over-month change in performance after measure activation was not significant (estimate ≤0.1%, SE=0.1%, p=0.71). On the basis of regression results, an estimated 11,464 more patients engaged from January 2018 to May 2019 than would have engaged without activation of the PDE measure and the quality improvement strategy (i.e., difference between the black and gray lines in Figure 3).

Examining the Influence of Denominator Size on Results

A sensitivity analysis excluding health care systems with denominators that averaged fewer than 10 cases yielded results similar to those in the main analysis: average performance of 65.6% in October 2016 and a significant increase in performance associated with quality measure activation (estimate 5.0%, SE=0.6%, p<0.001). Neither time parameter was significant in this sensitivity analysis. Figure 4 displays the estimated change in performance associated with activating the PDE measure, by VHA health care system and denominator size. If small denominators heavily influenced our results, then Figure 4 would contain a pattern with only small bubbles at extremes of this distribution. Variation in bubble size across the X axis of Figure 4 suggests that denominator size was not a strong determinant of change in performance associated with activation of the quality measure.

PDE by Denominator Subgroups

Performance increased after measure activation for all denominator and discharge diagnosis groups (Table 1). Engagement after discharge was lower among patients with stays in medical-surgical settings only or in inpatient mental health or substance use disorder settings, compared with those with stays in only residential treatment settings. Engagement was highest among patients with diagnoses of co-occurring mental and substance use disorders. Pre- to postactivation increases in performance were larger for patients with only mental health diagnoses than for patients with only substance use disorder diagnoses, resulting in similar engagement levels between diagnosis groups after measure activation.

Discussion

Biopsychosocial health risks are high in the month after discharge from high-intensity stays for mental or substance use disorder care, including risk for all-cause and suicide-related mortality (1–3, 26). Timely follow-up with a health care provider is associated with lower readmission rates, better medication adherence, and subsequent engagement in outpatient treatment (4–6). This national study analyzed the impact of activating a new quality measure at the VA, coupled with a quality improvement intervention, related to follow-up and engagement in outpatient mental or substance use disorder care after discharge. Activating the new PDE quality measure was associated with a 5.7% increase in engagement in mental or substance use disorder care after discharge—or an additional 11,464 patients over 17 months, a significant increase beyond the 66.1% who engaged just before activation of the PDE measure. Our results suggest that higher levels of PDE have been maintained at VHA since the measure was activated.

Clinically, activating a 30-day engagement measure shone a light on additional continuity-of-care processes beyond those typically involved with improving rates of a single follow-up visit in the 7 days after discharge. According to our results, before the PDE measure was activated, 66.1% of VA patients received at least two to four visits in 30 days after discharge, which compares favorably to public and private insurance plan performance on similarly structured follow-up continuity measures (16). Thus VHA health care systems may have had high continuity before the PDE measure was activated and may have been using best practices already (3, 4, 7–11). Nonetheless, VHA further increased engagement in postdischarge outpatient care by 5.7%, which suggests that a change in practice occurred after activation of the PDE measure. Although the PDE measure and quality improvement intervention highlighted the importance of continuity, the new measure and support tools also increased attention to care processes that promote subsequent engagement in additional specialty care by directly measuring receipt of multiple visits and by making ongoing accountability regarding care coordination explicit for clinical teams.

Three aspects of the VHA’s quality improvement strategy could have contributed to increased and sustained PDE across VHA health care systems nationally. First, the PDE measure was added to SAIL, which is tied to executive performance appraisals and might have drawn attention to engagement via top-down organizational processes (19–21). Second, the VHA Office of Mental Health and Suicide Prevention held multiple national and regional webinars and provided tools for clinical programs to educate staff about the new quality measure. Third, a population management tool was deployed to provide clinical teams with real-time data important for discharge planning and promoting engagement in care after discharge. The PDE measure denominator and population management tool made explicit shared responsibilities across clinical teams in different health care systems or settings (e.g., inpatient and outpatient teams). Data in this study do not tell us which of these factors—or other factors, such as technical assistance and suicide prevention policies (17)—might have influenced practice or organizational changes that had an impact on patient engagement.

Major quality-of-care evaluation programs in the United States, such as those at the Centers for Medicare and Medicaid Services (CMS) and National Committee for Quality Assurance (NCQA), evaluate follow-up after discharge for treatment of mental or substance use disorders as the percentage of patients who receive a single visit within 7 or 30 days (15, 16). Given widespread adoption of such measures, other organizations might also be poised to transition to a focus on PDE. Knowing concretely the differences between quality measure options might help evaluators decide whether adopting a measure, such as the PDE measure, would be worthwhile in their organizations.

An obvious difference is that a single visit is required to meet CMS and NCQA continuity measures, whereas multiple visits are required to meet VHA’s PDE measure. Visits on the same day of discharge meet the NCQA and CMS measures, whereas PDE credits same-day visits only when a patient is discharged against medical advice, given the risk of completed suicide (27, 28). The PDE measure combines all mental and substance use disorder discharges into a single measure, which aligns with the high co-occurrence of these diagnoses among veterans (29) and with VHA’s integrated care delivery model (23, 24). Discharges were combined into a single measure after consultation with subject matter experts, who expressed concern about engagement among patients with substance use disorders, particularly after discharge from inpatient medical settings. Our analyses found relatively similar levels of engagement between the group with mental disorders and the group with substance use disorders and, importantly, higher engagement in all groups after activation of the quality measure. However, a combined measure might make problems with engagement opaque for subpopulations, and a combined measure might not be appropriate for cross-system comparison with organizations that do not offer both mental and substance use disorder treatment. A final important difference between VHA’s PDE measure and follow-up measures defined elsewhere is numerator criteria that are scaled to expected clinical need.

Regarding limitations, unobserved differences between health care systems could have confounded the results, positively or negatively. Not all VHA health care systems have the same residential or inpatient services available, referral patterns vary across these systems, and we did not analyze data on care provided to veterans outside the VHA. Results could have been influenced more by some subgroups of patients, compared with others—for example, by higher volumes of discharges from inpatient specialty mental or substance use disorder settings than from medical-surgical settings. Nonetheless, engagement increased across all groups. The broad definition of PDE aligns with the VHA’s performance management agenda (20, 21), high levels of co-occurring mental and substance use disorders (29), and the VHA’s integrated care delivery model (30). Future research should identify which quality improvement strategies are effective across diagnostic groups and settings and which should be tailored for specific populations. Although the generalizability of these findings outside the VHA is not known, consensus standards and widespread evaluation of follow-up continuity after discharge increase the likelihood that other health care organizations are already focused on similar care processes around discharge and could expand to PDE. Finally, future studies should test the predictive validity of the PDE measure at the patient level—that is, whether meeting the PDE measure is associated with better patient outcomes, such as symptoms, functioning, readmissions, and suicide-related and all-cause mortality.

Conclusions

Our results offer initial support for the positive effects of implementing a multifaceted quality improvement program centered on a measure of engagement in multiple follow-up visits after discharge from residential and inpatient treatment of mental and substance use disorders.