Curating the Digital Mental Health Landscape With a Guide to Behavioral Health Apps: A County-Driven Resource

More than 10,000 mobile mental health apps are currently available to the public (1). Sifting through this multitude of apps often leaves consumers and clinicians overwhelmed, unsatisfied, and in many cases without a useful or effective resource (2). Although a number of frameworks exist to guide people through the process of app evaluation (3), these frameworks leave consumers and clinicians without specific app recommendations. An emerging solution to these challenges is the advent of curated lists, or app guides, that provide recommendations for a limited number of apps, usually with a specific target, such as depression (4). These guides range from relatively basic top 10 lists to expertly researched guides evaluating aspects of apps such as evidence of their effectiveness, user experience, adequacy of data security, and privacy (5). Often, these guides have been curated for specific audiences, such as veterans or employers (6).

Despite the proliferation of these guides, challenges remain. Many challenges mirror those posed by the apps themselves, including consumer and clinician awareness, unknown quality, limited relevance, and lack of direct consumer input. Mental health service organizations have an opportunity to address these challenges through a collaborative process of app guide development (4). By involving community stakeholders, subject matter experts, clinicians, and end users in this process, organizations can produce app guides that meet explicitly stated standards of quality and are tailored to local needs. Here, we present an example from a California county mental health program to illustrate the iterative process and resulting product of app guide development, and we outline opportunities and barriers that emerged.

The Help@Hand Project

We present output from a publicly funded innovation project in California, Help@Hand. In 2005, California enacted Proposition 63, the Mental Health Services Act (Assembly Bill 488), which authorized earmarked taxes for behavioral health services (7). Each county allocates the revenue generated by this tax to a combination of several specified areas, including innovation projects. Innovation projects develop or test “new, unproven mental health models” with the goal of increasing the quality of mental health services and access to those services by underserved groups (8). Help@Hand is a multiyear innovation project representing a collaboration among 14 California city and county behavioral health departments to explore the use of technology to improve the reach and impact of mental health services. A formative evaluation of this project is being conducted by a team with expertise in program evaluation, health services research, mental health app evaluation, and human-computer interaction.

Peers’ Guide to Behavioral Health Apps

As part of the Help@Hand project, Kern Behavioral Health and Recovery Services (KernBHRS) created a curated list of apps, “The Peers’ Guide to Behavioral Health Apps” (Kern App Guide), to inform consumers about various apps that could help support people’s mental health. A Spanish version, “La Guía de Compañeros Para Aplicaciones Sobre la Salud Conductual,” was also produced to reach the large proportion of Spanish speakers in the community.

The development of the Kern App Guide was an iterative process that is well suited to analysis using the replicating effective programs (REP) implementation framework (9), which has been widely used to prepare and package community-based programs for dissemination (4). The REP framework stages are as follows: preconditions (identifying local needs and implementation barriers), preimplementation (gathering stakeholder feedback and testing the intervention), implementation (intervention training, delivery, evaluation, and refinement), and maintenance and evolution (establishing organizational changes to sustain practices). We describe the Kern App Guide development process according to these four REP stages.

Preconditions.

In the context of the Kern App Guide development, the preconditions stage comprised several activities, including identification of a need for a new app guide, recruitment of a committee representing the various stakeholders affected by the project, and drafting of an initial list of apps for consideration. The idea for an app guide was first suggested by the director of KernBHRS in spring 2018 as an opportunity to create a practical product in collaboration with Help@Hand. App guide development was initiated in summer 2018 by a KernBHRS project manager, who assembled a committee of stakeholders. The App Guide Committee included 12 peers—community members with lived experience of mental health (or co-occurring) issues, experience with recovery, and training to use these experiences—working in a volunteer or employed capacity in the Recovery Services Division of KernBHRS. By November 2018, this committee had produced an initial outline of the app guide, consisting of 15 apps selected on the basis of familiarity and personal experience as a starting point for the formal evaluation process.

Preimplementation.

The preimplementation stage involved the committee meeting to determine the evaluation criteria for refinement of the list, as well as the evaluation and selection process itself. Beginning in November 2018, committee members initiated a formal review process. Several factors were chosen for app evaluation and selection on the basis of the needs of the community, including zero cost, availability in Spanish, accessibility on both iPhone and Android platforms, privacy and data protection, simplicity, and ease of use. (The rubric used for app evaluation is available as an online supplement to this column.) Using these criteria, committee members searched for new apps to add to the original list of 15 apps for formal review. Several dozen apps were identified, each of which was subsequently reviewed independently by at least three committee members. These reviews were discussed by the committee in weekly 1-hour meetings, and final inclusion was decided by majority vote. By the end of December 2018, a pilot app guide was published for internal review and distribution. The committee also solicited feedback on the pilot guide and candidate apps from the evaluation partners, who provided recommendations regarding app evaluation and selection criteria as well as design and dissemination considerations, which were incorporated before the release of the guide.

Implementation.

With the help of the marketing staff, the first edition of the Kern App Guide was published in April 2019 (all versions are available at https://www.kernbhrs.org/appguide). The 12-page first edition included 30 apps organized into six categories: behavioral health and wellness, the mind, the body, recovery, veterans, and sleep. Short descriptions, including the app’s purpose and an overview of its contents, were provided for each recommended app, as well as an indication of which apps were available in Spanish. The guide was disseminated, both digitally on the KernBHRS website and via physical handouts circulated within KernBHRS, from leadership to service delivery teams. At least 6,500 copies have been printed and distributed within Kern County in both English and Spanish. Outside of KernBHRS, the guide was shared with other county mental health service organizations, including the California Mental Health Services Authority conference of Help@Hand participants and the California Behavioral Health Directors Association.

Maintenance and evolution.

After the publication of the first edition, the committee continued meeting weekly to incorporate stakeholder feedback and select and review new apps. These meetings typically included five to nine members and involved brainstorming, discussion, and voting to add or remove apps from the guide. The evaluation partner also provided detailed recommendations regarding app evaluation and selection criteria. One example of a change made in regard to stakeholder feedback was to avoid apps that felt “classlike,” with extensive homework, reading, or lectures. A second edition was released in December 2019. To support the sustainable implementation of behavioral health apps into the existing system of care, KernBHRS created a plan to revise the guide semiannually and devoted considerable marketing efforts into distributing the guide, including promoting the guide via local media channels as an available resource during the outbreak of the COVID-19 pandemic.

Preliminary Outcomes and Learning

The Kern App Guide development process provides an example of the time, effort, and collaboration required to create a tailored app guide that meets the specific needs of local community members. A team of approximately a dozen people at KernBHRS developed the guide over a period of approximately 1 year, with each member dedicating on average several hours per week to the project. The skills and resources required to produce the Kern App Guide included knowledge of digital mental health products, community needs, and services, in addition to design and dissemination skills, and development of the guide involved community stakeholders, subject matter experts from Help@Hand, members of the marketing team, and many other KernBHRS peers and staff. The guide has been widely disseminated in Kern County, both physically and digitally, and versions of the guide were adapted for five other counties: Inyo, Mono, Modoc, San Bernardino, and Santa Barbara.

Discussion

The integration of behavioral health apps into the services provided by KernBHRS has provided lessons in both the opportunities of incorporating multiple perspectives into innovation projects and the barriers to implementing such collaboration.

Opportunities.

The inclusion of multiple perspectives, especially those of community stakeholders, was one of the key opportunities identified. The use of technology to provide mental health services often circumvents traditional delivery pathways. This may be useful for patient outreach, creating direct-to-consumer supportive care, but it can pose challenges to identifying those helped by such products, making it difficult to incorporate their feedback. This lack of feedback can be especially problematic given that low community involvement is a key weakness of many existing app guides (3). In this case, the potential disconnect that can accompany technological solutions was addressed by adopting a stakeholder-driven process of evaluation that allowed for the refinement and tailoring of recommendations (e.g., through the guide) to improve their relevance to the actual needs and expectations of the community they were meant to serve. Specifically, the involvement of community stakeholders allowed for the prioritization of features relevant to Kern County community members, such as the availability of apps in Spanish (and the production of a Spanish version of the app guide).

Barriers.

Nevertheless, this multistep process is not simple and illustrates some barriers to doing this work. Recruiting and organizing team members with a wide variety of experiences and different areas of expertise can be challenging. Once the team has been assembled, negotiating among the perspectives, preferences, and needs of different stakeholders requires care and patience. It can be difficult to arrive at a consensus regarding the many decisions required of the process, and some trade-offs may be necessary. For instance, some evaluation criteria were not weighted as heavily, such as the evidence base for app effectiveness, because much of the evidence created for these products was not cocreated with relevant communities and therefore might not be relevant (10).

Ways to address these barriers would include the use of structured and iterative processes, such as plan-do-study-act frameworks, or the creation of tool kits that identify core processes and roles and might facilitate adaptation. With this in mind, the Kern App Guide was presented to the community as just one version of the guide that could and would be updated. The challenge of conducting and updating app evaluations has been demonstrated elsewhere (11), and another solution might be increased transparency regarding the limitations present in any guide based on such evaluations.

Conclusions

Generating an app guide is a multifaceted process that requires diverse knowledge and resources. Here, we provide an example of how a publicly funded mental health service organization, in collaboration with different partners, created a tailored app guide to increase the community’s awareness of mental health apps that could serve as relevant mental health resources. The Kern App Guide, as part of the Help@Hand project, provides a concrete example of the implementation process of health app evaluation, upon which other service settings can draw. Such guides are not one-size-fits-all solutions, but the process represents a template for how app guides can be created by considering the unique needs of a community and leveraging various areas of expertise. Many organizations are considering using apps to expand mental health service delivery systems. If they choose not to develop their own apps, they must learn how to select and recommend existing apps. App guides such as the one presented here can be a useful tool to support consumers and clinicians in navigating the complexities of the digital mental health landscape.