Evaluation of a Web-Based Training Model for Family Peer Advocates in Children’s Mental Health

To address workforce shortages in mental health and to improve parents’ engagement in their children’s mental health services, many states have funded family peer–delivered services (1). Studies demonstrate that embedding family peer advocates (FPAs) as deliverers of services can improve attendance rates and retention of families in continuing care (2). Nationally, FPAs help to address the well-documented and severe workforce shortages in children’s mental health (3). A growing number of states are developing certification standards, and a national certification process is under development. A recent survey indicated that 28 states and the District of Columbia pay for family peer support services through Medicaid (usually through a state plan amendment or a Medicaid waiver), with the training and credentialing processes varying by state (4).

The New York State Office of Mental Health (NYSOMH) has a 20-year history of supporting the FPA model in order to increase caregiver engagement in treatment through a network of FPAs embedded in children’s service provider agencies and in family-run organizations (5, 6). In New York State, FPAs receive training and credentialing through a structured process, using the Parent Empowerment Program (PEP) training that follows a health activation framework (1, 7). Furthermore, quality indicator standards are now available for use in the state in order to support clear specification of FPA roles (8). The overall goal of the FPA model has been to enhance parents’ capacity to parent, navigate care systems, and advocate for their children in part through assisting parents to become fully engaged as partners with provider teams who provide care to their children.

Educating and credentialing FPAs in New York State was a time-intensive process involving 40 hours of didactics followed by consultation calls every 2 weeks for 6 months (9). In 2010, to make training more accessible, PEP training moved to a train-the-trainer model, again showing statistically significant increases in knowledge acquisition and self-efficacy (1).

With the recent changes to children’s behavioral health services in the state, FPA services are now reimbursable with Medicaid. Thus, there is a need to increase the number of FPAs and to broaden their training so that they can support parents and caregivers across all child-serving systems (e.g., education, medical care, child welfare, substance use, juvenile justice). The NYSOMH acknowledged that 40 hours of in-person training was burdensome and costly and that FPAs needed immediate access to training to qualify for Medicaid reimbursement upon employment. Consequently, in 2014, the NYSOMH awarded New York University a contract to create a hybrid training program consisting of online learning modules, 2 days of in-person training, and 12 consultation calls over 3 months.

Web-based training programs have become increasingly popular as a cost-effective educational mechanism, particularly for training a workforce that is spread across a large geographical area. Research suggests that Web-based training is effective for health and mental health service providers (10–12). Therefore, the purposes of this study were to examine the gains in knowledge from completing the Web-based PEP level 1 training program and to compare knowledge gains from the online training with gains from the in-person train-the-trainer model (1) by using the 29 knowledge questions that assessed knowledge in both training modalities.

Methods

Data for this pre-post study were collected from individuals who enrolled in the PEP level 1 online training from February 2018 to April 2019. Of the 258 people who accessed the online modules, 144 completed the training (56%). Two participants who completed the training and three who did not were removed from the study because of missing or improperly coded data. Two were removed because their pre-post change scores on the general knowledge measures presented as extreme outliers (i.e., more than three interquartile ranges from the center of the data). Thus, data from 140 participants in the completers group were examined for knowledge gains and sociodemographic and employment factors related to knowledge gains and were compared against the results of the in-person training. The individuals completing the in-person training and those completing the online training (1) were FPAs. We also examined the sociodemographic characteristics common to both training groups and found no statistically significant age, sex, ethnicity, or educational differences (data not shown). The online training was a quality improvement effort and did not require institutional review board approval.

The traditional 5-day in-person PEP training covered the following topics: the FPA role, engagement and empowerment strategies, listening and communication skills, understanding the children’s behavioral health and education systems, partnership and negotiation skills, managing groups, boundaries, and self-care. The training approach included a mix of didactic instruction, small group discussion, and role-playing.

Because the traditional in-person training was considered effective, the new training addressed a similar set of competencies. The process of developing the online training began by reviewing the topics in the existing training, changes in the FPA scope of practice, and new competencies required by system changes. This information was gathered through discussion with current trainers, FPA supervisors, FPAs, clinical partners, and state and national leaders. The content of the existing training was then rereviewed for relevancy with any disagreements about content retention resolved through discussion. New topics for the online training were chosen through a similar process. Content experts, experienced FPAs, and a team of education technologists then reviewed each online module. Of the content in the earlier in-person training, 90% was retained in the new online level 1 modules.

The new online PEP modules are accessed through a learning management system and are completed independently by trainees. The modules are intended to introduce the role and fundamental principles of family peer support, teach basic skills, provide an overview of key child-serving systems, and explore other relevant topics. New content was added in the areas of crisis response, planning, documentation, and a structured approach to exploring family strengths and needs. A variety of activities were built into the updated training in lieu of in-person role-play and discussion. (see Table B in the online supplement for the areas addressed.)

Questions about demographic characteristics, completed prior to training, inquired about age, gender, ethnicity, language spoken, education level, and several work characteristics. The general knowledge pre-post tests consisted of 34 multiple-choice and true-false questions assessing knowledge related to the PEP curriculum content. Of note, some of the questions were designed to gauge how FPAs would respond in certain scenarios. Twenty-nine were identical to those used in the earlier evaluation (1). As in the earlier evaluation, scores for both the 34- and 29-item tests were calculated as the number of correct responses (1). Statistical analyses were performed with SPSS, version 25. Chi-square tests were conducted to examine potential differences between training completers and noncompleters.

Because data from the general knowledge test violated the assumption of normality, Wilcoxon signed-rank tests were used to evaluate change in knowledge. This test was first performed with the 34-item evaluation. Data from the 29 items that were identical across training sessions were then analyzed in the current sample and the previous sample. Because the previous study allowed participants with missing items on the pre- and posttests to be included in the comparisons, we retained participants missing no more than four item responses per test in this sample. Missing items were calculated as incorrect. A Mann-Whitney U test was then used to determine whether knowledge gains varied between the in-person and the Web-based training. Differences in knowledge gains across sociodemographic characteristics were evaluated with the Kruskal-Wallis and Mann-Whitney tests.

Results

Completers and noncompleters did not differ on demographic factors (all p>0.050). However, significant differences in employment characteristics were found between groups, including work status as an FPA (χ²=28.85, df=1, p<0.001) and whether their employer requires credentialing as an FPA (χ²=20.53, df=2, p<0.001). Fewer noncompleters were employed as FPAs or worked for settings requiring FPA credentialing (see Table A in the online supplement).

In the sample of 140 completers, scores on the 34-item posttest (mean±SD=27.82±3.89, median=29) were higher, at a statistically significant level, than on the pretest (26.39±3.66, median=27) (z=–5.24, p<0.001, r=–0.31). Scores on the common 29 items were also significantly higher on the posttest (23.65±3.45, median=24.5) than on the pretest (22.57±3.34, median=23) (z=–4.29, p<0.001, r=–0.26). The 304 participants who received the in-person training (1) also scored significantly higher on the 29 common items on the posttest (23.73±3.29, median=24) than on the pretest (22.66±3.17, median=23) (z=–6.70, p<0.001, r=–0.27). Importantly, knowledge gains across training models did not differ (Table 1). No sociodemographic characteristic was related to differences in knowledge gains (data not shown).

Discussion and Conclusions

Given the growing use of family peer support services, an efficient, effective, and accessible approach to training this emerging workforce is a critical need, particularly for states with large geographical areas. The cost and accessibility challenges of in-person didactic training prompted the NYSOMH to explore online training to teach the didactic portion of the effective in-person training models (1, 9). Data from the pre-post survey of the 140 completers of the online training showed a statistically significant gain in knowledge on both the battery of 34 items and the 29 items that were common to the pre-post tests administered in prior in-person training. Gains in knowledge were not significantly different between participants who completed the training in person and those who completed training via the Web-based modules. This finding is notable because online training is considerably more accessible, convenient, and cost-effective than in-person training. Importantly, most of the training content is not specific to New York State and, therefore, could be adopted by other states, thus potentially standardizing the educational training of FPAs nationwide.

The second important finding was that no sociodemographic characteristic was related to increases in knowledge. This is notable because data suggest that the use of technology in training is difficult for older individuals and those less comfortable with technology (13). Discomfort with technology has been identified as an important reason for dropping out of training (14).

These data have limitations. This study used a small sample of individuals from only New York State. The online program was accessed by 258 individuals, 56% of whom completed the training. Although no sociodemographic characteristics differentiated completers from noncompleters, the completers may have unmeasured characteristics that are partially responsible for the knowledge gain. However, a more likely explanation is that current FPAs for whom credentialing was not required or individuals interested in becoming credentialed as FPAs may have accessed the modules to understand the new responsibilities of an FPA. Importantly, these data are from only the online training. We have no information on whether improvements in knowledge translated into changes in the FPAs’ practice behaviors. These initial data on knowledge gains from an online training program for FPAs show promise. Having an accessible and affordable means of training this growing, important workforce is a critical step forward in improving care for vulnerable children.