Process Evaluation of a Treatment Program for Mood and Anxiety Disorders Among Emerging Adults: Preentry Factors, Engagement, and Outcomes

Research indicates that 15.5% of disability-adjusted life years (DALYs) worldwide arise from illnesses that begin when individuals are between the ages of 10 and 24 (1). The largest contributors to DALYs are neuropsychiatric disorders, at 45% of the total (1). However, treatment gaps of over 50% exist for such conditions (2). The World Health Organization concluded that the focus should be on emerging adulthood for prevention of chronic neuropsychiatric illnesses to decrease worldwide disability (1). Mood and anxiety disorders have a heavy symptom burden (3), with the first and second highest DALYs among psychiatric disorders, respectively (4). In North America, lifetime prevalence rates are approximately 29% for anxiety disorders and 21% for mood disorders (5), and at least 75% of these disorders begin before age 25. These disorders have a high probability of recurrence and can become chronic (6, 7). Among individuals with chronic depression, 15% die by suicide (8).

In our previous process evaluation of a program for mood and anxiety disorders among emerging adults, the First Episode Mood and Anxiety Program (FEMAP) (9), we described the outputs of the model (10, 11). The study reported here evaluated uptake in an independent sample and investigated engagement and medium-term outcomes of the program. We hypothesized that this youth-friendly, patient-focused model would provide appropriate inclusion and be successful in meeting its early goals of keeping those most in need engaged, reducing symptoms, and improving function and quality of health and that the service would be rated as satisfying by users.

Methods

The Model

This study involved a cohort of emerging adults to evaluate uptake, outputs, and medium-term outcomes of the program. The sample for this study was new and enrolled subsequently to the previous study (9). The updated logic model of the program is presented in Figure 1.

Participants

Emerging adults (ages 16–25) who contacted FEMAP between May 2013 and April 2015 and who met criteria by telephone screen were scheduled for an in-person intake interview. Research participation was essential for FEMAP enrollment but was not required for continuation in the program. Physician referral was not required for FEMAP entry (9–11). Research inclusion and exclusion criteria were identical to criteria for treatment at FEMAP. Individuals with longer than 18 months lifetime medication treatment were excluded to focus on the earlier stages of illness. Youths with developmental delays, major medical problems, and head trauma with unconsciousness lasting longer than a few moments were excluded and referred to alternative services. Individuals whose primary mental health concerns appeared to be substance use, on the basis of the timing of symptom onset, were referred to community addiction services and excluded. Individuals with substance use that appeared to be secondary to mood and anxiety symptoms were included. Willing participants provided informed consent after the study was fully explained, as approved by the Human Research Ethics Board for the University of Western Ontario, as per the Declaration of Helsinki.

Psychometric Measures

Demographic and other measures were administered in person or occasionally by e-mail at the FEMAP intake interview (baseline) and included the Montgomery-Asberg Depression Rating Scale, Self-Report (MADRS) (12); Anxiety Sensitivity Index–Revised (ASI) (13); Sheehan Disability Scale (SDS) (14); Trauma History Questionnaire (15); EuroQol Group 5-Dimension Questionnaire, thermometer only (EQ-5D) (16); and Inventory of College Students’ Recent Life Experiences (17), adjusted to ensure applicability to high school–age populations. The Adolescent Alcohol and Drug Involvement Scale screener (18, 19) was also administered.

At follow-up (see below), these questionnaires were completed again, and an adjusted Patient Satisfaction Questionnaire (PSQ) (20) was also completed. The PSQ included 20 items, after it was adapted to a single-party payer and emerging adults. The scale included questions about features not applicable to many FEMAP patients (for example, questions about medication administration did not apply because many of the emerging adults enrolled were not prescribed medication; questions about parking did not apply to the many emerging adults who used public transit). The three questions deemed most specific to any model, by face validity, were identified, and this subscale was reported along with the total PSQ score. The three items were as follows: “I liked the services I received here”; “If I had other choices, I would still get services from this agency”; and “I would recommend this agency to a friend or family member.” All referenced measures except those created by the investigators had been previously validated; several were modified as noted. All multi-item scales evaluating a single construct showed Cronbach’s alphas as reported in the tables.

Treatment Model

FEMAP provides patient-centered treatment in a youth-friendly setting by using a multidisciplinary model that is trauma informed and strength based. It utilizes standard clinician training adjusted to patients’ needs and expressed wishes and offers multiple therapeutic modalities, including psychopharmacology, cognitive-behavioral therapy, psychodynamic psychotherapy, addictions treatment, and group therapy. Treatment is determined collaboratively with the emerging adult, emphasizing patients’ individual needs and wants and clinical considerations. The team includes psychiatry, psychology, social work, addictions, and family therapy. Licensed clinicians practice according to guidelines of their colleges and licensing organizations. Identification of clinicians to work at FEMAP, including comfort with the age of the population and the practice approach, is by self-selection.

Participants completed research questionnaires at the time of in-person intake (baseline) by FEMAP’s clinical social worker. Follow-up questionnaires were completed between approximately 2 and 10 months after treatment began, based on participants’ availability and willingness to complete the questionnaires. Participants were given a $10 gift card to the movies for completion of the follow-up questionnaires.

Statistical Analysis

All statistical analyses employed SPSS, version 25. Missing items in a questionnaire were added by using multiple imputations when they were ≤10% of total items and missing at random. Questionnaire responses not meeting these criteria were removed from individual analyses. Engagement with treatment was determined by chart review. The general linear model was used for multivariate analyses.

The wait time for services after assessment for the patient group in this study was not sufficiently long for participants to serve as their own control. However, a post hoc comparison control group was created from a subsequent cohort of individuals who were on a wait list for a mean of 3 months between May 2015 and March 2017. There were no changes in recruitment criteria between these two cohorts. [A flowchart in an online supplement to this article illustrates the recruitment process for both cohorts.]

Results

Access and Pretreatment Factors

The ethnoracial backgrounds of those attending the intake interview reflected the predominantly white Canadian and European population of London, Ontario, at time of recruitment [see online supplement].

Services and number of service types sought by emerging adults prior to attending the FEMAP intake interview are shown in Table 1. Only 5% (N=21) of participants who attended the intake interview had not previously sought mental health care (an indication of the euphemistic nature of the program name). Mean number of prior service types engaged was 3.2 [see online supplement]. Fully 77% of emerging adults had received help from a school mental health program, and 73% had already been seen in primary care for their mental health concerns. A third (32%) had been to the emergency department for mental health concerns.

Of the 398 emerging adults with usable data from intake, 370 (92%) were eligible for the program and accepted and 28 (7%) were referred elsewhere (N=18) or reassured that specialized mental health care was not necessary (N=10). A total of 323 (81%) contacted the program without a physician referral. Whether or not the patient was referred by a physician was not significantly associated with acceptance.

Data on preentry factors of the 370 accepted participants are presented in Table 2. Only 14% (N=51) were not either enrolled in school or working. The group had clinically moderate levels of depression, as indicated by MADRS mean scores (21), and clinically significant levels of dysfunction, as indicated by SDS scores (22). The mean subjective rating on the EQ-5D of satisfaction with the quality of health exceeded 2 SDs below the norm for the 18–29 age group (23).

Substance use was low but present. Only 15% (N=55) of 369 entering participants with substance use data indicated no alcohol use; almost 80% (N=291) used alcohol one or two times per month or less. Fewer used drugs, including marijuana, with 46% (N=170) indicating no drug use and 72% (N=265) reporting use one or two times per month or less (including none).

Engagement

Approximately 676 emerging adults contacted FEMAP during the time of recruitment. A total of 622 were eligible for an intake, and 402 (65%) attended the intake [see flowchart in the online supplement]. Of 370 eligible participants, 322 (87%) attended a clinical assessment but 71 (19%) disengaged either before this clinical assessment or immediately thereafter. Emerging adults who disengaged did not differ significantly in age, anxiety score, or quality of health from those who were retained. In addition, neither sex nor referral method was significantly associated with disengagement. In a multivariate analysis, those who disengaged were significantly different from those who were retained (F=2.38, df=6 and 358, p<0.03). Those who disengaged had significantly higher rates of street drug use (mostly marijuana) (mean±SD days of use in the past month=2.80±2.56 versus 2.00±2.36, p=0.01). In addition, those who disengaged had lower MADRS mean scores, indicating less depression (11.31±4.38 versus 12.47±4.47, p=0.05), and lower total SDS scores, indicating less functional impairment (15.83±7.87 versus 17.69±6.97, p=0.05), compared with those who were retained. (Possible MADRS scores range from 0 to 27, with higher scores indicating greater depression. Possible SDS scores range from 0 to 30, with higher scores indicating greater functional impairment.)

Follow-up data from all 370 accepted emerging adults were pursued regardless of whether they disengaged. Mean time from baseline to follow-up for the 174 patients (47%) who completed follow-up questionnaires was 26.8±8.0 weeks (range 10.4 to 47 weeks). No individuals who disengaged completed follow-up data. Data for two individuals were lost because of technical problems, and a third person did not complete the SDS. Males were more likely than females not to complete the follow-up questionnaire (χ²=3.25, df=1, p=0.05, one-sided). Referral method was not significantly associated with follow-up completion rates. A multivariate analysis indicated a trend for emerging adults who were lost to follow-up to be different from those retained (p<0.08). Those not completing follow-up questionnaires were younger (18.70±2.20 years versus 19.54±2.78 years, p<0.01). However, no between-group differences were found in depression, anxiety, functional impairment, or quality-of-health satisfaction scores or substance use ratings.

Intermediate Outcomes

From baseline to follow-up, symptom scores indicated significant improvement (Table 3). Multivariate analysis confirmed that the MADRS, ASI, SDS total score, and EQ-5D all improved significantly (F=31.42, df=4 and 158, p<0.001). The change in MADRS score indicated a clinically significant improvement because the mean score dropped below 9 (24). ASI scores and SDS scores improved, with the latter representing a clinically meaningful change, having dropped below 12 (22). General satisfaction with quality of health also improved significantly. Noteworthy was the wide range of responses to treatment (Table 3). No significant association was found between the time between baseline and follow-up, which varied by participant from 2 to 10 months (mean=6 months), and any outcome variable.

The post hoc analysis of data for persons in the wait list control group involved 210 participants who waited 13.4±7.5 weeks (range 1–41.7) for assessment [see online supplement for scores]. Analyses of variance showed that the treatment group had significantly greater improvement in depression, as indicated by MADRS scores (F=80.64, df=1 and 383, p<0.001), as well as improvements in other areas as indicated by ASI scores (F=32.81, df=1 and 373, p<0.001), and SDS scores (F=52.61, df=1 and 376, p<0.001). Again, no significant association was found in the control group between the time from baseline to follow-up, which varied by participant, and any outcome variable.

Patient Satisfaction

Patients completing follow-up data gave the service high ratings, as indicated by the PSQ total and subscale scores (Table 3). On the full scale, all but one participant (99%) gave the program a mean score more positive than neutral. For the three-question subscale, approximately 95% (N=163) rated FEMAP more positive than neutral, and 47% (N=91) of participants gave it the highest possible score.

Discussion

The results supported our hypotheses related to the logic model of FEMAP. Age-appropriate specialty intervention provided in the relatively early stages of illness, which permitted self-referral, was successful in attracting and engaging appropriate patients without overinclusion. It was associated with symptom reduction and improved functioning as well as enhanced quality of health at a medium-term follow-up time point. Participants gave very positive ratings to the service. In contrast, individuals in the wait list control group showed significantly less improvement in symptoms and functional impairment. In addition, changes in symptoms and functioning were not correlated with the length of time between baseline and follow-up, which varied by participant, suggesting that time was not an overall factor in improvement. Although the findings were of correlation and not causation, they support the hypothesis that improvement at follow-up was a result of treatment.

Differences at baseline between males and females were consistent with the epidemiology of mood and anxiety disorders (25). Ethnoracial characteristics of the sample were consistent with the overall demographic characteristics of the community where the program is located. Research in regions with other demographic characteristics is warranted.

Pretreatment data showed that participants had sought help extensively from other sources before contacting FEMAP, which reconfirmed that including “first episode” in the program’s name does not accurately reflect the population treated (9). Because most of the emerging adults came to the program without a physician referral, the fact that they had sought help elsewhere was unexpected but reassuring. It is preferable that patients obtain care from more general health services before receiving specialty care. More than three-quarters of those who contacted FEMAP had seen a school mental health care support person before seeking help from FEMAP. Only 5% of those who contacted FEMAP reported that the FEMAP contact was their first effort at help seeking. Thus it is clear that the self-referral option did not result in a large influx of individuals who had not tried more general health care services before seeking specialty care.

Almost a third of emerging adults who contacted FEMAP had been seen in an emergency department before the FEMAP contact. Seeking help from an emergency department is not desirable; patients are best served at lower cost in specialty ambulatory settings. Several factors could contribute to seeking care in an emergency department, including a lack of knowledge about available ambulatory services; delays in help seeking resulting from stigma or a lack of information (26, 27); and restrictions in ambulatory services, including hours of operation, wait lists, and so forth. To reduce mental health care costs and use of the emergency department, efforts should be made to promote ambulatory services and their capacity for rapid response.

Data collected at baseline showed that symptom severity and functional impairment generally warranted clinical intervention (Table 2). Although most emerging adults were either enrolled in school or working, they were either not functioning or underfunctioning for approximately 4.3 days per week. This represents a high-impact deficit if allowed to continue or worsen through lack of intervention and is particularly detrimental at this life stage. Loss of school and work time and of time spent in building relationships can result in marked developmental impairment for emerging adults (28). Treatment effectiveness and cost-effectiveness of primary prevention across mental illnesses have been difficult to demonstrate (29, 30). Optimizing secondary prevention with symptomatic emerging adults, the focus of this model, may prove to be a wise use of health care resources.

Among emerging adults scheduled for an in-person FEMAP intake, 35% did not attend the intake appointment. Understanding this disengagement is challenging because the only available data for these individuals were from the initial phone screen. Others have addressed some engagement issues among emerging adults (31). Collecting more data during the FEMAP phone screen can aid understanding of early failure to engage in this setting.

Among eligible participants who attended the in-person intake, 20% dropped out before attending either the first (13%) or second (7%) appointment with a treatment provider [see online supplement]. Those who disengaged early had significantly more street drug use, compared with those retained in the program, and showed a trend for less depression and less dysfunction. Disengagement in this instance could represent self-correction of overinclusion of emerging adults in the program. At intake, those reporting moderate to high substance use were educated by the intake coordinator about their mental illness and told that a reduction in substance use would be a treatment recommendation. Individuals not interested in making such a change may have decided to disengage—or they may have accepted this advice, experienced improvements, and decided that they no longer needed treatment. Further research can differentiate these possibilities; however, in both scenarios, service uptake appeared to be refined by self-selection of those with greater motivation or need for treatment.

Individuals who disengaged early also had less severe depression and dysfunction than those who were retained in the program, although the difference was not statistically significant in multivariate analysis. Nevertheless, this suggests some self-selection by those who were not as intensively in need of tertiary mental health services. Further research should confirm this trend before it is assumed that the well-being of individuals who disengage early is of no concern.

Participants completed follow-up questionnaires between 2 months and 10 months after program entry, depending on their availability and willingness. Only 58% of those who engaged with the program completed a follow-up questionnaire. Absence of follow-up data was more common among males and younger participants. These findings may reflect level of maturity, lower prioritization of mental health care research, geographic instability, inability to find transportation, or some other factor. It is noteworthy that individuals in the age cohort of FEMAP are often transient. Emerging adults often left the area to attend school, and others were only temporarily in the area to attend school. The incentive to complete follow-up questionnaires was very minimal in this study and, although research participation was required for entry into clinical care, patients could opt out at any time thereafter. Our follow-up rate was slightly lower than that of a study of younger adolescents followed for half as long (32).

Improvement in outcomes supported our hypotheses. Depression and anxiety scores were both significantly reduced at follow-up. The reduction in depression scores was clinically significant (24). Functional impairment improved by a clinically meaningful amount, on average (22). The total SDS score fell by the largest percentage among the measures used in the study (Table 3)—35% lower than the initial score among those who completed follow-up. In the time frame under study, anxiety and quality of health were more resistant to change, but scores indicated significant improvement.

Participants appeared highly satisfied with FEMAP. This was particularly evident in responses to the PSQ subscale. Over 90% of individuals gave positive ratings on this subscale and 47% gave the highest rating. Details about what program participants found useful and challenging have been published elsewhere (27, 33).

The study had several limitations. Rating scales were self-report, presenting a limiting factor in interpreting results. Findings were correlational, and causation cannot be concluded. The post hoc control group involved a wait list, and thus the study did not demonstrate differences between FEMAP and other evidence-based care. Nevertheless, another study that matched FEMAP patients with emerging adults who were seeing physicians in Ontario but who were not involved with FEMAP indicated that two logic model short-term outcomes were met—faster access to specialty mental health services and fewer emergency department visits by FEMAP patients (34).

The wide range of changes between intake and follow-up in the symptom, functioning, and quality-of-health scores is noteworthy. Many patients improved, even though some got worse. This variability bears further investigation to determine moderators and mediators of response by subgroups of patients in this setting. Future analysis of relationships among preentry factors, treatment characteristics, and follow-up outcomes may indicate key components that are most helpful to emerging adults in their recovery.

Conclusions

This naturalistic evaluation of a program to treat mood and anxiety disorders among emerging adults in the early stages of illness examined uptake and engagement, preentry characteristics of participants, and medium-term outcomes, including symptom and functional improvement, quality of health, and satisfaction with services. Statistically and clinically meaningful improvement was associated with participation, as was high satisfaction. This study demonstrated the high potential of programs that offer personalized treatment in a multidisciplinary model and standard clinician training adjusted to patients’ needs and expressed wishes. The results of this study can provide valuable information for the development of mental health services where they are most needed—for emerging adults with the most common mental illnesses.