Attributes of Provider Referrals for Digital Mental Health Applications in an Integrated Health System, 2019–2021

Anxiety and depressive disorders are the most common mental health conditions in the United States, with lifetime risks of 29% and 20%, respectively (1, 2). These conditions incur disability, high health care utilization, and low quality of life (1, 3–6). Social and financial stress experienced during the COVID-19 pandemic contributed to increases in anxiety and depressive disorders (7, 8). Increased incidence of mental disorders, pervasive provider shortages, and stigma associated with mental health treatment may have negatively affected patient access and increased unmet need (9–11).

Digital mental health applications (DMHAs) often incorporate mindfulness and meditation (MM) and cognitive-behavioral therapy (CBT) as evidence-based approaches for managing anxiety and depression and can help alleviate barriers to care (12, 13). MM strategies focus on reducing stress and anxiety through relaxation, whereas CBT strategies require active patient engagement to address the problem source.

More than 10,000 DMHAs are available online, but provider guidance may assist patients with selecting an appropriate app (14). By including DMHA referral into clinical workflows, providers can help patients select the correct app for their need, mitigate app cost, increase patient engagement, track referrals, and optimize patient app use (15). Although DMHAs should not substitute for in-person care, they may provide novel, complementary solutions that can be broadly disseminated, address subclinical mental health problems prior to clinical intervention, and supplement a care plan. Clinical trial evidence suggests that DMHAs may reduce anxiety and depression symptoms, support emotional well-being by reducing stress, and improve resiliency (14, 16–23), yet little is known about their integration and effectiveness in clinical practice.

Integrated health care systems such as the Veterans Health Administration and Kaiser Permanente (KP) have been at the forefront of integrating DMHAs into clinical care (24–27). In December 2019, the rollout of a new initiative within the KP Mid-Atlantic States (KPMAS) region (District of Columbia, Maryland, and Virginia) allowed KP’s behavioral health providers to refer patients for any of six DMHAs that focused on MM (Calm, Headspace, Whil) or CBT (myStrength, Thrive, SilverCloud) with no patient registration cost. Primary care providers in the health system were limited to referrals for Calm and myStrength.

Improved understanding of how and for which patients providers make referrals for DMHAs in clinical practice is important for identifying unmet need, informing future provider training, and establishing expectations for health systems considering implementation of DMHA referrals into clinical workflows. Experiences from integrated systems such as KPMAS can inform the implementation of DMHA referrals in other systems. Therefore, this study sought to describe trends in referrals for DMHAs over a 25-month observation period and to profile patient, clinical, and encounter attributes associated with DMHA referral clusters.

Methods

Design, Sample, and Setting

A retrospective, cross-sectional design was approved by the KPMAS Institutional Review Board. All clinical encounters for KPMAS members ≥18 years old with an initial referral for a unique DMHA and associated patient, clinical, and encounter referral attributes were extracted from the electronic health record. Inclusive dates were December 1, 2019, through December 31, 2021. As a large integrated health system, KPMAS serves a diverse group of more than 800,000 active members. The system’s coverage area includes three primary regions (Baltimore, District of Columbia and southern Maryland [DCSM], and Northern Virginia [NoVA]). By linking with a common electronic health record, the KPMAS system provides comprehensive coordination across the care continuum.

Measurements

Referrals were initially grouped into two categories (MM or CBT) to be consistent with national initiative guidance (26). Referrals were further organized into three mutually exclusive referral clusters—MM alone, CBT alone, or MM and CBT—if a DMHA from each group was ordered for the same patient during the observation period (see the online supplement to this article).

Referral clusters were described by patient, clinical, and encounter attributes by using the referral as the unit of analysis. Demographic characteristics included patients’ age at referral, gender, and self-reported race-ethnicity.

Clinical attributes associated with each referral included four mutually exclusive groups of patient-indicated reasons for visit (i.e., chief complaint): mental health (anxiety, depression, stress), learning (wellness coaching, education, counseling), annual care, and “other” (online supplement). Provider-assigned primary diagnosis for each encounter was categorized into four mutually exclusive groups: depression (single episode or recurrent) (ICD-10 codes F32 and F33), unspecified mood disorder (ICD-10 code F39), other anxiety disorders (ICD-10 code F41), or “other” (online supplement). Categories for reason for visit and primary diagnosis were chosen to be consistent with prompts from the DMHA workflow within the electronic health record and to represent patient and provider perspectives. Because Patient Health Questionnaire (PHQ) or Generalized Anxiety Disorder assessment (GAD) screening may inform DMHA referral, presence of depression or anxiety screening on the date of referral encounter was included in primary analyses.

Referrals were described by encounter type (office, telephone, or videoconferencing). Provider attributes for each referral were characterized by training (physician or nonphysician), assigned specialty affiliation (primary care, specialty care, or behavioral health care), and location within the KPMAS region, given that referral patterns may vary by these characteristics.

Analysis

Patient, clinical, and encounter attributes of referrals were aggregated and reported overall and by referral cluster. Positive PHQ-2 and GAD-2 screening scores of ≥3 on the same day of referral were stratified by DMHA referral cluster but were not included in primary multivariable analyses, because only a subset of the sample completed the screenings. Monthly referral frequency trends for the 25-month observation period were reported overall, by app category, and by individual DMHA.

Multivariable, multinomial logistic regression models were used to compare patient, clinical, and encounter attributes across DMHA referral clusters. Models included patient demographic characteristics (age, race-ethnicity, gender), clinical characteristics (presence of PHQ screening, reason for visit, primary diagnosis), and encounter attributes (visit type, provider training, and provider location within KPMAS region). Provider specialty and GAD screening were described but not included in final models because of associations with other model variables. Referrals were clustered by patient to control for within-patient correlation. Post hoc margins tests were conducted to estimate the overall probability of patients being in each referral cluster and the probabilities of various patient, clinical, and encounter attributes occurring within each referral cluster. Margins tests were calculated by keeping all variables constant at their mean to estimate the sample average. To determine attribute variations within referral clusters, the difference between the overall sample average probability and the individual probability of each attribute occurring within the referral cluster was calculated and multiplied by 100 to reflect a percentage point difference. Differences for each attribute within a referral cluster were profiled in forest plots, with a positive difference indicating a higher probability of a given attribute being in the referral cluster and a negative difference indicating a lower probability of being in the referral cluster. The unit of analysis was the referral, and statistical significance was set to α=0.05 for all analyses.

Results

Overall Referral Attributes

A total of 43,842 initial DMHA referrals were made for 25,213 unique patients; 69% of patients had ≥2 referrals (Table 1). Of patients receiving ≥2 referrals, 91% (N=27,773 of 30,342) received all referrals on the same day. The combination of MM and CBT DMHAs was most often referred (54%), followed by MM-alone DMHAs (39%) and CBT-alone DMHAs (8%). Overall, the largest proportions of patients in the DMHA referral clusters were White (39%), in the 31–50-year age range (41%), and female (72%).

Fifty-two percent of referrals had a primary psychiatric diagnosis. Same-day PHQ screening occurred for 71% of referrals, whereas GAD screening was performed concomitantly for 68% of referrals. The percentage of referrals with a positive PHQ-2 score on the same day was highest for CBT alone (47%, N=1,202 of 2,537), followed by MM and CBT (46%, N=7,848 of 17,197) and MM alone (41%, N=4,729 of 11,441) (χ²=62.88, df=2, p<0.001). In contrast, the percentage of referrals with a positive GAD-2 score on the same day was highest for MM and CBT (64%, N=10,585 of 16,494), followed by MM alone (63%, N=6,883 of 10,948) and CBT alone (60%, N=1,436 of 2,380) (χ²=15.22, df=2, p<0.001). Within the category of “other” reason for visit, MM-alone referrals were most commonly made during psychotherapy (34%, N=2,142 of 6,335), videoconferencing visits (13%, N=792 of 6,335), and medication management encounters (11%, N=719 of 6,335). CBT-alone referrals were most commonly made during psychosocial assessment (40%, N=603 of 1,499), psychotherapy (19%, N=279 of 1,499), and videoconferencing visits (6%, N=84 of 1,499). MM and CBT referrals were most commonly made during psychosocial assessment (42%, N=5,221 of 12,523), medication management encounters (21%, N=2,631 of 12,523), and psychotherapy (13%, N=1,615 of 12,523) (online supplement). Within the category of “other” primary diagnosis, MM-alone (60%, N=4,725 of 7,933), MM and CBT (60%, N=6,960 of 11,509), and CBT-alone (50%, N=778 of 1,556) referrals most commonly and consistently included a diagnosis of reaction to severe stress (online supplement).

Referrals were mostly made by nonphysicians (70%), including clinical social workers (76%, N=23,604 of 30,858), professional counselors (22%, N=6,906 of 30,858), psychiatric nurse specialists (0.9%, N=276 of 30,858), and psychologists (0.2%, N=72 of 30,858). Referrals were most common at videoconferencing visits (75%), within the DCSM location (47%), and from behavioral health care providers (88%). Mean referral rates over the 25-month study period were 84.9, 100.2, and 104.4 per referring provider for the Baltimore, DCSM, and NoVA locations, respectively.

Referral Trends

Total referrals fluctuated throughout the observation period (Figure 1A). Four months after initiative rollout, total monthly referrals accelerated and peaked at 7 months postrollout. Referrals then decelerated and reached equilibrium at 10 months postrollout, until another substantial increase was observed at 16 months postrollout. Thereafter, monthly referrals began a sustained decline until the end of the observation period at 25 months postrollout. Referral trends for CBT and MM DMHAs mirrored the total trends (Figure 1A). Calm was the most referred DMHA throughout the observation period; Headspace and myStrength alternated as the second and third most commonly referred DMHAs for most of the observation period (Figure 1B).

Predicted Probabilities of Attributes Being Within Each Referral Cluster

After adjustment for patient, clinical, and encounter attributes, overall predicted percentages of referrals were 55.2% for MM and CBT apps, 37.5% for MM alone, and 7.2% for CBT alone. Results for each type of attribute are described below.

Patient Attributes

MM and CBT cluster.

Compared with the referral cluster average, predicted percentage points of referrals were significantly higher for patients ages 18–30 years (2.45) and for Black patients (3.14). In contrast, predicted percentage points of referrals were significantly lower for patients who were ages 51–64 years (−3.27), ages ≥65 years (−9.62), Asian (−3.57), Hispanic (−4.93), and male (−1.57) (Figure 2).

MM-alone cluster.

Predicted percentage points of referrals were significantly higher for patients who were ages 51–64 years (2.84), ages ≥65 years (9.04), Asian (4.34), and Hispanic (6.79), compared with the referral cluster average. However, predicted percentage points were significantly lower for patients who were ages 18–30 years (−2.38) and Black (−3.45) (Figure 3).

CBT-alone cluster.

Compared with the referral cluster average, predicted percentage points of referrals were significantly higher for male patients (0.92) and significantly lower for Hispanic patients (−1.86) (Figure 4).

Clinical Attributes

MM and CBT cluster.

Predicted percentage points of referrals were significantly higher than the referral cluster average for patients reporting a reason for visit of “other” (6.25) and having a primary diagnosis of depression (single episode or recurrent) (3.66). Predicted percentage points of referrals were significantly lower for patients not completing a PHQ screen on the encounter date (−1.56), reporting a reason for visit of mental health (−8.89), and having a primary diagnosis of other anxiety disorders (−3.41) (Figure 2).

MM-alone cluster.

Compared with the referral cluster average, predicted percentage points of referrals were significantly higher for patients not completing a PHQ screen on the encounter date (3.25), reporting a reason for visit of mental health (9.40), and having a primary diagnosis of other anxiety disorders (4.71). In contrast, predicted percentage points of referrals were significantly lower for patients reporting a reason for visit of “other” (−6.29) and having a diagnosis of depression (single episode or recurrent; −6.34) or unspecified mood disorder (−4.98) (Figure 3).

CBT-alone cluster.

Predicted percentage points of referrals were significantly higher for patients with a primary diagnosis of depression (single episode or recurrent; 2.68) and completing a PHQ screen on the encounter date (0.74), compared with the referral cluster average. However, patients with a primary diagnosis of other anxiety disorders (−1.30), with a reason for visit of mental health (−0.51), and not completing a PHQ screen on the encounter date (−1.69) had significantly lower predicted percentage points of referrals when compared with the referral cluster average (Figure 4).

Encounter Attributes

MM and CBT cluster.

Predicted percentage points of referrals were significantly higher than the referral cluster average for patients who received care in the DCSM area (6.75) and had a videoconferencing visit (1.84) and were significantly lower for patients who received care in the Baltimore (−10.31) and NoVA (−4.16) areas and had office-based visits (−11.66) (Figure 2).

MM-alone cluster.

Compared with the referral cluster average, predicted percentage points of referrals were significantly higher for patients who received care in the Baltimore (8.30) and NoVA (3.82) areas and had office-based visits (9.79). In contrast, predicted percentage points of referrals were significantly lower for patients who received care in the DCSM area (−5.79) and had telephone (−2.29) and videoconferencing visits (−1.50) (Figure 3).

CBT-alone cluster.

Predicted percentage points of referrals were significantly higher than the referral cluster average for patients who received care in the Baltimore area (2.00) and had office-based visits (1.87) but were significantly lower for patients who received care in the DCSM area (−0.97) and from physician providers (−1.10) (Figure 4).

Discussion

Our results provide initial insight into providers’ DMHA referral patterns and describe variations that may be expected when integrating DMHA referrals into clinical workflows on a large scale. DMHA referral attributes differed from the general KPMAS (Baltimore, DCSM, and NoVA) overall membership estimates from 2020; participants in our study were more likely to be White (39% vs. 26%) and female (72% vs. 53%) and less likely to be ages ≥65 (6% vs. 15%) and Hispanic (10% vs. 14%). Providers readily engaged in DMHA referral after the initial initiative rollout. After 4 months, referrals accelerated and then reached equilibrium at 6–7 months. Referrals steadily declined from month 16 to the end of the 25-month observation period. Last, clear variations in the attributes of each DMHA referral cluster resulted in unique profiles.

Observed increases in DMHA referrals during the first 4 months of the COVID-19 pandemic in the United States, with some stabilization thereafter, were consistent with existing research (28). Given that the estimated overall number of visits during a 4-week period at KPMAS locations between May 3 and June 20, 2020, was 222,000 (29) and that approximately 2,000 DMHA referrals were made per month, on average, during the study period, referrals occurred at <1% of visits. Annual new member enrollment in the health plan that typically coincides with the beginning of the calendar year increases the number of members who are eligible for referral and likely contributed to observed increases during early periods of each calendar year. Stand-alone continuing medical education programs had mixed effects on referral patterns, suggesting the need for ongoing provider training. The COVID-19 pandemic, a seminal disruptive event, appears to have initially accelerated overall DMHA referrals. However, the decline in referrals during the later months of the observation period suggests the need for future research to assess the sustainability and clinical impact of DMHA referrals.

Providers frequently initiated multiple DMHA referrals on the same encounter date. Simultaneous DMHA referrals may be efficient and flexible and may support patient access and choice, but they also may overwhelm and confuse patients in the absence of ongoing follow-up. Multiple referrals at the same time may also reflect provider uncertainty, underscoring the need for continuing education about best practices for implementing DHMA referrals in a comprehensive care plan. Evaluation of a more stepped approach to DMHA referral that includes periodic assessments may be warranted, given the potential confusion about DMHA use in clinical practice.

Referral patterns reflected a range of patient demographic characteristics, clinical characteristics, and encounter attributes, and variations among DMHA referral clusters emerged. For example, MM-alone referrals were more likely to be given to patients who were ages 51 or older, were Hispanic or Asian, reported a reason for visit of mental health, or had a primary diagnosis of other anxiety disorders. CBT-alone referrals were more likely to be given to patients with a primary diagnosis of depression (single episode or recurrent) and less likely to be made for Hispanic patients. MM and CBT referrals were more likely to be ordered for patients who were ages 18–30, reported a reason for visit of “other,” or had a primary diagnosis of depression (single episode or recurrent), and referrals were less likely to be given to Hispanic patients or to patients ages ≥65.

Observed variations in referral clusters may be explained by perceived technological savvy (e.g., ability to easily understand and use technology), cultural receptivity, and clinical need assessed at the clinical encounter. Historically, the “digital divide” between those with and without adequate access to contemporary technology has often made the process of obtaining care more difficult for certain subgroups of the general population (30). Although organizational commitment to a digital mental health initiative (26), as observed in the KPMAS system, likely removed some of the common access barriers (e.g., registration fees, lack of formal clinical pathways), additional effort is clearly needed to ensure equitable access to and use of digital tools (15). Thought leaders have proposed provider education in five areas of competency to enhance best practices: evidence, integration, security and privacy, ethics, and cultural considerations (31). Future research should further evaluate sociodemographic and cultural differences in referral patterns.

Providers appear to consider a patient’s condition when making a DMHA referral. For example, the primary diagnosis of other anxiety disorders was more common within the MM-alone referral cluster. Moreover, the primary diagnosis of depression (single episode or recurrent) was more common in both the MM and CBT and CBT-alone referral clusters, suggesting that a diagnosis of depression may encourage the likelihood of referral for a CBT DMHA (with or without an MM DMHA). In addition, the frequencies of primary diagnoses within the category of “other” varied among referral clusters; for example, a sleep disorder diagnosis was more common in the CBT-alone referral cluster than in the MM-alone and MM and CBT referral clusters. Reaction to severe stress was less common in the CBT-alone referral cluster than in the MM and CBT and MM-alone referral clusters, which also suggests that providers are making referrals with the primary diagnosis in mind. Bivariate analyses suggest that a relationship may exist between a positive PHQ-2 or GAD-2 screen and DMHA referral cluster, although more detailed investigation of this relationship was beyond the scope of this study and requires more comprehensive analysis to understand the implications.

Although MM and CBT have some crossover in their strategies for treating depression and anxiety, our findings suggest that there may be some degree of alignment between clinical need and compatible therapeutic strategy. Consistent with existing literature, MM can address the hyperarousal that is characteristic of anxiety disorders, such as increased heart rate, panic, and restlessness. CBT can integrate active engagement and the challenging of negative cognitive ruminations to treat depression (32–34). However, when the primary diagnosis is categorized as “other” (online supplement), which may reflect a broad set of needs, providers more commonly refer MM and CBT simultaneously, underscoring potential uncertainty about which type of DMHA to recommend. Collectively, these findings suggest the need to further evaluate the use and outcomes of DMHAs in specific conditions, given that there may be shared experience of some comorbid conditions (e.g., primary anxiety that contributes to secondary depression over time).

Compared with the referral cluster average, providers at DCSM locations were more likely to refer MM and CBT apps, those at NoVA locations were more likely to refer MM apps, and providers in Baltimore were more likely to refer CBT apps. Regional variations suggest that referral patterns may differ even within one organization and may reflect diverse needs among patients within regions. Further exploration is required to understand potential sources of these variations.

Finally, within the MM and CBT cluster, there was a higher probability for referral (vs. the sample average) during videoconferencing visits. This finding is consistent with observed increases in weekly visit trends for both behavioral health and videoconferencing visits within the KPMAS system during the early phases of the COVID-19 pandemic (26). In contrast, within the MM-alone and CBT-alone clusters, the probability of referral was higher (vs. the sample average) during office-based visits. These findings may suggest that providers place more focused and targeted attention on the nuances of a particular DMHA when they see patients in person, whereas the higher likelihood of a combined MM and CBT DMHA referral during videoconferencing visits may reflect that a provider perceives the patient to have greater comfort with technology and therefore may engage in a more general referral strategy.

This exploratory observational study was limited to information in an electronic health record. Our study described only referrals to DMHAs and did not evaluate DMHA enrollment, patient experience, or the clinical consequences of DMHA use. In addition, we were unable to explore the rationale and sequencing of the referrals as well as the patient-provider dialogue that may have encouraged use of DMHAs. However, the findings raise important questions for future studies. Generalizability may be limited to settings where the cost of DMHA subscriptions can be covered by a health system or insurance plan.

Conclusions

Referrals to DMHAs included an array of patient, clinical, and encounter attributes. Monthly referrals rapidly accelerated during the early months of the COVID-19 pandemic, with a subsequent plateau and then a decline during the extended pandemic period. Observed patient, clinical, and encounter attribute variations by referral cluster support the need to further evaluate provider decision making and whether referral patterns lead to optimal patient use. Future work is needed to further explore the end-user DMHA experience, including adherence and persistence over time, and DMHAs’ subsequent clinical impact.