Evaluation of Telehealth in Child Behavioral Health Services Delivery During the COVID-19 Pandemic

The COVID-19 pandemic has disrupted models of health care delivery in the United States. In‐person office visits with health care providers were limited when stay-at-home orders were imposed, which led to policy changes that increased telehealth coverage by nearly all federal, state, and private insurers (1). Subsequently, health care providers began promoting services via telehealth, leading to a rapid and massive expansion of this technology during the COVID-19 pandemic (2). Telehealth has allowed many patients to continue accessing health care while mitigating the risk for virus exposure. Multiple reports have investigated different aspects of telehealth such as racial and ethnic disparities associated with its use (3–6) and its utilization among older adults (7), in nursing homes (8), and within large health care systems (9, 10). However, limited research exists on the associations between telehealth use and patient-level outcomes. The pandemic greatly increased the urgency and breadth with which telehealth was deployed in the United States, and the expanded use of telehealth services may endure (11, 12). Thus, additional research is needed on the clinical outcomes associated with telehealth utilization, especially among children.

Hundreds of thousands of youths and families are served by public mental and behavioral health systems, and many of these children have complex medical, psychological, and psychosocial needs. The prevalence rates of U.S. children diagnosed as having a mental, behavioral, or developmental disorder are high (approximately 17%) (13), and mental health concerns among children and adolescents have been rising (14), particularly since the onset of the COVID-19 pandemic (15). Access to mental and behavioral health services remains a significant challenge, particularly for children living in low-income families and in rural communities (14, 16). Transportation barriers affect health care access, utilization, and clinical outcomes for children. Approximately 9% of children living in families with annual incomes <$50,000 miss essential medical appointments because of transportation barriers, regardless of insurance status (17–19). Furthermore, children and youths whose caregivers have unmet needs for transportation to and from service locations have worse clinical outcomes over time compared with youths whose caregivers either never had unmet transportation needs or could resolve their transportation needs (20).

In this study, we explored how the general shift from in-person to mainly telehealth delivery during the COVID-19 pandemic affected client-level health outcomes in Idaho’s public mental and behavioral health system. Consistent with national trends, reported data suggest that most services for children in Idaho’s Department of Behavioral Health were delivered via telehealth during the first year of the pandemic (21). Our aim was to quantify the impact of the shift to telehealth on child-level outcomes during the COVID-19 pandemic (2020 cohort) by comparing outcome and other data collected during this period with the corresponding data 1 year before the pandemic (2019 cohort). We also examined differences between these two cohorts in the impact of caregivers’ unmet transportation needs on children’s clinical and functional outcomes. We hypothesized that the rapid expansion of telehealth during the pandemic may have lessened the importance of access to transportation; support for this hypothesis would be indicated by a reduction of the negative impact of unmet caregiver transportation needs on a treated child’s mental health in the 2020 cohort compared with the 2019 cohort.

Methods

We extracted data of behavioral assessments of all children and adolescents (ages 5–18 years) between April 1 and December 31, 2019 (2019 cohort), and between April 1 and December 31, 2020 (2020 cohort), from the Idaho Child and Adolescent Needs and Strengths (ICANS) database, which stores and manages the Child and Adolescent Needs and Strengths (CANS) assessments of the Idaho Department of Behavioral Health (22). The same 9-month period across the 2 years was selected to capture the key periods before and during the pandemic and to facilitate comparisons across equivalent periods. The analysis period was limited to 9 months because March 25, 2020, was the start of Idaho’s statewide stay-at-home order (23), and COVID-19 vaccines became more widely available in January 2021; the study time frame also closely corresponded to increases in nationwide telehealth delivery (24). We note that in order to participate in or receive certain state-funded programs in Idaho, a child or youth must complete a CANS assessment, which is then stored on the ICANS platform. Thus, it was highly unlikely that some children or youths were in state-funded care without having a CANS assessment. The study was conducted with the approval of the University of Kentucky Institutional Review Board.

CANS Measure

To analyze clinical outcomes, we used comprehensive behavioral assessments with the CANS instrument (25, 26). The CANS is a nondiagnostic functional and clinical behavioral assessment widely used for children and youths served in the public sector in the United States (27). The Idaho CANS comprises 82 items that cover symptom severity, functioning, and social and environmental measures across five domains: strengths (16 items), life functioning (14 items), behavioral and emotional needs (18 items), risky behaviors (15 items), and caregiver resources (19 items). The CANS instrument rates each item on a scale ranging from 0 (no need for action) to 3 (immediate action is needed). An item is rated as “actionable” if rated 2 or 3, indicating that action is warranted to address the need. Details on the Idaho CANS instrument and methods are available in previous studies (20, 28, 29).

Study Sample Selection

Both the 2019 and 2020 cohorts included children and adolescents (ages 5–18 years) who underwent at least two CANS assessments during each 9-month period. Children with prepandemic assessments were matched to those with assessments during the pandemic by using the following criteria. First, we compared the two cohorts in terms of the total number of actionable items (TAI) at the initial assessment across the four primary domains of interest (i.e., strengths, life functioning, behavioral and emotional needs, and risky behaviors). Children and adolescents who entered care in the Idaho Department of Behavioral Health before the pandemic had, on average, more complex needs and larger TAI. Second, to ensure that our findings were not due to systematic differences in characteristics of children served by the state after the initiation of statewide stay-at-home orders, improvements in the conditions of children during the pandemic were compared with those of matched children with prepandemic assessments. The matching was done by using a one-to-one propensity score and nearest-neighbor approach, with TAI across the four CANS domains as covariates (30). The second alternative analysis was performed on unmatched cohorts.

Dependent Variables

Changes in the CANS scores over time were evaluated as follows. First, for each assessment, we calculated the TAI that were “resolved up to date” across the primary domains of interest. Specifically, at the initial assessment time point t₀, all children and youths were assigned 0 items resolved in each domain. Then, if an item that was actionable at time t₀ was rated as nonactionable at the next time point, t₁, we labeled it “resolved” and counted the total number of resolved items up to that date across the four primary domains. For example, if a child or youth had items 3, 6, and 9 scored as actionable at t₀ within the behavioral and emotional needs domain, and only item 9 was still scored actionable at t₁, the number of behavioral and emotional needs at t₀ was defined as 0 and was 2 at t₁. Next, we calculated the proportion of resolved needs at a given time point by scaling the proportion to the TAI within a domain by the end of the period examined. In the example above, if a child or youth had two assessments within the study period, the child's or youth’s outcomes within the behavioral and emotional needs domain would have been y₁=0 at t₀ and y₂=2/3 at t₁.

Independent Variables

The primary independent variable of interest was the year of service, that is, 2019 or 2020. The secondary independent variable of interest was the change in the caregiver’s transportation needs during the child’s or youth’s care episode. This variable was defined on the basis of the CANS item that rates the level of transportation required to ensure that a child or youth can effectively participate in treatment. If the transportation item was never rated as actionable across all CANS assessments, we defined this child or youth as having “no issues” with transportation needs during the study time frame. If this item was actionable during one assessment but became nonactionable by the last assessment, we defined this as a resolved transportation need. If the transportation item remained actionable by the last assessment, we defined the transportation need as being unresolved. To adjust for confounders associated with socioeconomic status, financial resources, and general patterns of caregiver behavior, we calculated the first five principal components (that explained 53% of total variation) from items in the caregiver resources domain and used them as another set of covariates. Additionally, we included the individual characteristics of children and youths, such as age, gender, race-ethnicity, and length of stay in treatment, as predictors.

Statistical Analysis

First, we conducted a descriptive analysis of the study cohorts, including their demographic characteristics, number of CANS assessments, complexity of needs at the initial evaluation, and caregiver’s transportation issues. Second, we used a linear mixed-effect model to explore trends in strengths-building and needs-resolution rates during the study period. All analyses were performed with R, version 3.6.3, statistical software. Statistical significance was assessed as p<0.05.

Results

In total, 4,458 new children and youths entered Idaho state–funded mental health services programs in 2019 versus 2,794 in 2020 (Table 1). Of these patients, 201 had CANS assessments from both years but during different instances of care (i.e., these patients were discharged in 2019 and reentered care in 2020). From 2019 to 2020, measures of turnover, such as admissions and discharges, decreased. Of note, not a single new child or youth who entered care during the 9-month period in 2020 was discharged, whereas 41% were discharged among those admitted during the same period in the 2019 cohort (Table 1). The total number of CANS assessments declined almost twofold in 2020 (6,588 in 2020 vs. 10,229 in 2019). However, the age distribution remained stable (i.e., about 36% of children treated pre- and midpandemic were ages 5–10 years, about 24% were ages 11–13, and about 40% were 14–18), and the decrease in the number of assessments was uniform across all three age groups. However, the average number of per-patient CANS assessments significantly increased in 2020 (2.4 in 2020 vs. 2.3 in 2019, p<0.001).

Table 1 depicts the demographic characteristics of unmatched cohorts. (Table S1 in an online supplement to this article displays pre- and pandemic cohorts after propensity score matching.) Compared with 2019, more girls (48% vs. 50%) and fewer White children (63% vs. 60%, p<0.001) entered Idaho’s behavioral health program in 2020. Children and adolescents exhibited significantly (p<0.05) more complex needs and strengths profiles in 2019 compared with those in 2020 (strengths TAI=4.4 vs. 4.0, life-functioning TAI 2.3 vs. 2.0, and risky behaviors TAI 1.0 vs. 0.9). Finally, in 2020, fewer caregivers reported transportation need as “resolved” (1% in 2020 vs. 2% in 2019).

Table 2 shows the results of the primary matched linear mixed-model analyses. Among the matched children, 129 underwent assessments in both 2019 and 2020 (under different instances of care). The estimated coefficient, β, shows the percentage of areas addressed (i.e., needs resolved or strengths acquired) within each domain associated with every additional 90 days in the system (i.e., between two consecutive CANS assessments) among children whose caregivers never had issues with transportation to or from care. The estimated change, Δ, shows deviations from these percentages of addressed areas among children whose caregivers resolved their transportation issues, Δ_R, or did not resolve transportation issues, Δ_NR. (Table S2 in the online supplement shows the results of the same analyses but on the prepandemic sample without propensity score matching.)

Analyzing changes in rates of strengths building and needs resolution between 2019 and 2020, we observed no statistically significant changes for most children and youths (i.e., those whose caregivers never had issues with transportation access) (Table 2). Results without propensity score matching (Table S2 in the online supplement) were consistent with this finding. Unlike for the 2019 cohort, we found no statistically significant associations between caregivers’ resolution of transportation needs and children’s improvement across all four CANS domains in the 2020 cohort. The magnitudes of change in strengths and behavior-emotional domains in the 2020 cohort were nearly half those observed in the 2019 cohort for Δ_NR.

Discussion

We note several findings from our study of data in a large statewide public mental and behavioral health system for children and youths. First, new cases significantly declined during the early months of the pandemic. However, children or youths and families who did initiate care stayed in care. Second, as care shifted to telehealth models, we observed clinical outcomes of children and adolescents that were similar to those observed prepandemic. Third, associations between unmet transportation needs and worse clinical outcomes observed before the pandemic were not observed during the pandemic, when most services were delivered via telehealth.

These findings appear to be consistent with results from some other reports of benefits of telehealth among youths (31–34). A recent review of different Web-based interventions to address mental health problems among young adults during the pandemic found that all of them had effectively reduced rates of depression, anxiety, and stress symptoms (34). A nationally representative survey reported that digital interventions were mitigating the negative COVID-19–related psychosocial impact among individuals ages 16–25 years (32). Taken together, these findings suggest that telehealth services may be suitable for lessening behavioral health issues among youths, although questions remain about the wider applicability of telehealth across the whole developmental spectrum.

Importantly, the findings of our study underscore the impact of the COVID-19 pandemic on the relationship between the resolution of caregiver transportation needs and children’s strengths building or needs reduction across multiple functioning domains. We observed a significant association between resolution of caregiver transportation needs and child-level clinical outcomes prepandemic (April–December 2019), such that the resolution of caregiver transportation needs was associated with greater need reduction in the life-functioning domain and strengths-building domain. This represented approximately 5% additional needs resolved and 5% additional strengths developed with the resolution of the caregiver’s transportation needs (see Table 2). This relationship was not statistically significant, however, when the same analyses were conducted with data obtained during the pandemic (April–December 2020). These results suggest that the expansion of telehealth service delivery may have mitigated the impact of caregiver transportation needs on clinical outcomes for children with complex needs and strengths who are served in public mental and behavioral health systems.

The results of this study should be contextualized within its limitations. No precise service delivery data were available that described the exact proportion of services delivered via telehealth versus those delivered in-person during the pandemic. Data from the Idaho Office of the Governor indicate that telehealth sessions increased 40-fold from March to May 2020 (21). Moreover, because we observed similar trends in outcomes for only two of the four domains examined, strengths building and needs resolution, we wonder whether besides the shift to telehealth other changes in service delivery between 2019 and 2020 might have influenced outcomes in these two domains; we also do not know why telehealth outcomes were comparable in these, but not the other, domains. Additionally, because CANS assessments do not distinguish between unmet transportation needs for mental health and those for school, activities, and other aspects of life, it is possible that the 2020 cohort had a reduced negative impact of transportation needs on functional outcomes because the nature of those transportation needs changed because of the pandemic. However, one strength of this study included the size of the database used and the utilization of patient-level data to characterize the impact of the pandemic on mental and behavioral health outcomes of children and youths.

In terms of the pandemic’s impact on behavioral health service activities, substantially fewer children and youths were served in Idaho in 2020 than in 2019. The existing data cannot provide a definite answer as to where these “missing” children and youths went midpandemic. Nonetheless, we note that about 20% of children and youths typically enter care systems through school staff referrals, 15% are referred from social service agencies, and another 15% are directed to the system of care through the courts and correctional institutions (35, 36). We speculate that the decrease in the number of young patients seen during the early pandemic may have been due to shutdowns of several avenues through which children and youths typically enter the system of care. Furthermore, our matched prepandemic and pandemic samples were largely comparable in terms of demographic variables, but the unmatched number of young patients served and number of CANS assessments in April–December 2020 were nearly half those of the unmatched patients during the same period in 2019. This reduction in care delivery is consistent with some documented trends of reduced health care delivery in the United States during the pandemic, because telehealth utilization requires the need for connective technologies (e.g., high-speed Internet, networked devices), which may pose a barrier to both presentation for services and telehealth utilization independent of transportation need or demographic characteristics (37). The intensity and quality of care differences were unknown; however, our findings indicate that children served during the pandemic received more frequent assessments.

The difference in clinical presentation between the two unmatched cohorts, such that the 2019 cohort presented with more actionable and more complex needs, perhaps demands further investigation. Children with the most acute and complex mental and behavioral needs may not have presented to the Idaho Department of Behavioral Health for telehealth services during the pandemic. If that was the case, it would be important to understand the potential barriers to care specific to children and youths with the highest acuity needs.

Conclusions

Our findings suggest that for children with complex needs, mental health services provided primarily via telehealth were comparable to traditionally delivered services in addressing two patient‐level functional outcomes and that telehealth utilization reduced the negative impact of transportation needs on care outcomes. Our work is primarily a descriptive study of clinical and functional changes associated with a major policy shift in service delivery. As clinicians, researchers, and policy makers are advocating for permanent expansion of telehealth services, one such systems-level change suggested by our findings may include continued provision of telehealth for clients with transportation needs as a bridge to support service delivery, while administrators and providers work toward the permanent resolution of transportation issues for these clients. However, for clients without transportation needs, telehealth may be selectively effective (by type of need, level of acuity, or other) or even create new obstacles to accessibility (e.g., technological barriers).