Adoption of Clinical and Business Trainings by Child Mental Health Clinics in New York State

Scant literature exists on the adoption of state initiatives in U.S. public mental health systems. Interviews with state mental health directors and child welfare directors about adoption of evidence-based practices and quality improvements (QIs) suggest that although clinics continuously experiment with new services (1,2), they rarely implement evidence-based practices and frequently discontinue the new services (3). Common barriers to adoption of new practices include inadequate financing, pressures for organizational productivity, staff resistance to change, adaptations necessary for use in usual care, and limited access to information about the practices (1,2).

Large-scale state rollouts of evidence-based practices and QIs (3) aim to improve the quality of mental health services. They involve substantial financial investments, phasic implementation, and specific target populations (4). State departments tend to drive the focus of initiatives, which range from training in specific clinical practices or services (4,5) to organizational improvements (6). Rollout approaches vary from allocating funds to organizations as an incentive to adopt and implement the initiatives to using focused methods, such as learning collaboratives and modular learning (7).

Because of limited empirical or conceptual frameworks to guide these state rollouts, the adoption behaviors of organizations are rarely measured (8). Inconsistent definitions of “adoption” further hinder measurement efforts (9). Measures of adoption are often simplistic (for example, using “yes” or “no” to indicate adoption) and insufficient (for example, failing to indicate how many clinics were offered the initiatives), with no attention to the type and intensity of initiatives adopted. Basic outcome data related to the adoption process or information about how to use such data to design future rollouts are unavailable (10). According to the Reach Effectiveness Adoption Implementation Maintenance (RE-AIM) framework (11), these adoption behavior data not only enhance the understanding of adoption, nonadoption, and “de-adoption” (8) of training interventions but also provide important information prior to the implementation phase.

New York State’s Clinic Technical Assistance Center

Considering the limitations of state rollouts and the paucity of data about adoption outcomes, the Clinic Technical Assistance Center (CTAC) in New York State provides a unique opportunity to better understand the naturalistic, nonincentivized adoption of training in evidence-based practices and QIs. Founded in 2011 and funded by the New York State Office of Mental Health (NYS OMH), the CTAC (www.ctacny.com) offers statewide training and collects data on clinics’ attendance of its offerings. Unlike prior technical assistance efforts that focused on evidence-based clinical interventions (5), the CTAC is a training, consultation, and educational center for clinical and business needs that strengthen practitioners’ professional development and clinics’ abilities to meet the financial and regulatory challenges of health care reforms. Technical assistance is available to all 346 outpatient mental health clinics that are licensed to serve children, adolescents, and their families.

CTAC trainings are delivered in three ways––Webinar, in-person training, and learning collaborative––that represent varying levels of training intensity and clinic commitment. The trainings were developed on the basis of NYS OMH feedback and clinics’ expressed needs. Hour-long Webinars are the least intensive. In-person training requires full-day participation. Learning collaboratives are the most time intensive because they require ongoing participation in group learning sessions and consultations over a six- to 18-month period. Participation in all CTAC offerings is voluntary.[Examples of efforts in key states to roll out evidence-based practices and QIs related to children’s mental health and descriptions of specific trainings offered by the CTAC are available online as a data supplement to this article.]

Business and organizational improvement practices

Twelve trainings that target clinic and agency administrators, such as leaders, executives, and finance officers, address the changing financial drivers of clinic operations and service delivery. They range from nine Webinars on developing an effective business model by using financial modeling tools to an all-day, in-person training on open access, centralized scheduling, and concurrent documentation.

Two learning collaboratives require clinics to commit to participating in a series of Webinars, in-person trainings, and ongoing consultations. The business effectiveness and efficiencies project (BEEP) helps clinics assess and redesign their financial structures, and apply practical strategies for developing sustainable business practices over an 18-month period. The business effectiveness assessment module (BEAM) consists of a series of Webinar modules and group phone consultations on financial management that takes place over a six-month period.

Clinical evidence-informed practices

Eighteen trainings target frontline clinicians and clinical supervisors. They include 14 hour-long, lunch-and-learn Webinars about various topics, such as trauma, motivational interviewing, and cognitive-behavioral therapy, and one in-person training on family engagement. They also include three learning collaboratives that consist of a series of Webinars, in-person trainings, and phone consultations. Two learning collaboratives are based on the “4R’s and 2S’s” (rules, roles and responsibilities, respectful communication, relationships, stress, and social support) curriculum, which integrates evidence-based strategies for children with disruptive behavior disorders (12). One is a 12-month multifamily group model, and the other is a six-month model for individual families. The third learning collaborative, practitioner education and decision support (PEDS), was adapted from the “managing and adapting practice” tool (13). It offers decision support tools to improve clinical practices, accountability, and outcome monitoring throughout the year.

Hybrid practices

Three trainings that target both clinic administrators and clinicians are designed to improve business performances and practice outcomes. They include two Webinars on outcome measurement. They also include the training intervention for the engagement of families (TIES), an all-day, in-person training that addresses both business and clinical aspects of patient engagement. The TIES offers resources for overcoming organizational barriers to engaging families and trains clinicians in evidence-based family engagement interventions (14).

Study goals

This study aimed to use clinic attendance data to describe the naturalistic adoption of CTAC trainings by the population of mental health outpatient clinics that serve children, adolescents, and their families (N=346). Adoption behavior was characterized in four ways: by number, type, and intensity of trainings accessed and by adopter group. For the last category, clinics were further grouped by the highest intensity of training accessed. Examining adoption behavior in multiple ways may provide insight into factors that influence clinics’ uptake of training in evidence-based practices and QIs and could improve future state rollouts.

Methods

All 346 NYS OMH–licensed outpatient clinics that are licensed to treat children and adolescents were included in this study, whether or not they attended any CTAC trainings. Clinic attendance at CTAC trainings between September 2011 and August 2013 was examined. Institutional review board approval was waived because the study was not considered human subjects research (no interaction or intervention with individuals and no use of identifiable private information). The study classified clinic adoption of training by number, type, and intensity of trainings and by adopter group.

Number of trainings

Adoption was described by the number of trainings attended by each clinic out of a total of 33 offerings.

Type of trainings

Adoption was described by the types of trainings selected by clinics. Three types of trainings were offered: business (N=12), clinical (N=18), and hybrid business and clinical practices (N=3).

Intensity of trainings

Adoption was described by the level of training intensity selected by clinics. The three levels of training intensity are hour-long Webinars; full-day, in-person training; and learning collaboratives. To be considered a learning collaborative adopter, a clinic must have formally applied to a collaborative and have attended at least one learning collaborative session. Although these sessions are typically open only to formal learning collaborative participants, clinics occasionally attend a session as visitors; these clinics were credited with having attended the session but were not considered learning collaborative adopters for purposes of classification by adopter group.

Adopter group

Adopters were categorized by the highest level of training intensity selected by clinics. Low adopters accessed Webinars only, medium adopters accessed at least one in-person training but did not access a learning collaborative, high adopters accessed one learning collaborative, and “super-adopters” accessed at least two learning collaboratives.

Fisher’s exact tests were used to examine the association between number of trainings and adopter group to provide a better understanding of adoption behavior.

Results

Of the 346 clinics, 268 (77%) accessed at least one training, and 78 (23%) accessed none. Table 1 shows the clinics’ adoption of CTAC trainings by adoption behavior.

Number of trainings

The number of trainings accessed by adopters ranged from one to 26 (median=5, mean=6). However, the adoption distribution was positively skewed, with 48% of the adopters participating in one to four trainings, 25% in five to eight, and 27% in nine or more.

Type of trainings

Clinics participated in business (82%) and clinical (78%) trainings at comparable rates; 45% accessed hybrid trainings, likely reflecting the smaller number of hybrid trainings available. When combinations of training types were examined, we found that 100 (37%) adopters selected all three types, 64 (24%) adopted both business and clinical trainings, 47 (18%) focused exclusively on business trainings, and 37 (14%) participated only in clinical trainings.

Intensity of trainings

Participation was greatest for the most readily accessible and least time-consuming trainings: 96% of adopters participated in Webinars, 46% in in-person training, and 34% in learning collaboratives. Seven of the eight most accessed trainings (attended by 25%−34% of all adopters) were Webinars on fiscal efficiency and outcome reporting. Four of the eight least accessed trainings (attended by 4%−11% of all adopters) were learning collaboratives. Further, 109 (41%) adopters exclusively accessed Webinars, although only 56 (21%) selected all three levels of intensity and 58 (22%) selected both Webinars and in-person training.

Adopters of the most intensive type of training, learning collaboratives, were likely to have previously sampled less intensive training—22 (73%) of the 30 BEEP adopters and 46 (94%) of the 49 BEAM adopters had previously accessed a business Webinar. All 18 adopters of PEDS and all 14 adopters of 4R’s and 2S’s (individual model) had previously participated in a clinical Webinar.

Participation in learning collaboratives over time was also examined. In the two business learning collaboratives that ended, 14 (47%) BEEP adopters and 12 (25%) BEAM adopters attended at least half of the learning collaborative sessions. The BEAM adopters, however, could revisit missed training modules online. For the two clinical learning collaboratives (4R’s and 2S’s individual and group models) that ended, 21 (84%) of 25 adopters of the group model and seven (50%) of 14 adopters of the individual model achieved at least 50% participation.

Adopter group

On the basis of highest level of intensity selected, 41% of the clinics were classified as low adopters, 25% as medium adopters, 22% as high adopters, and 12% as super-adopters.

As Table 2 shows, a statistically significant association was found between the distribution of the four adopter groups and number of trainings accessed (p<.001). Low adopters not only selected low-intensity trainings but also participated in fewer trainings, with 85% adopting only one to four trainings. Not only did super-adopters sign up for at least two of the most intensive trainings, learning collaboratives, 85% also attended nine or more trainings. High adopters were the next most active group, with 37% participating in at least nine trainings.

Discussion

To our knowledge, this is the first study that prospectively examined the naturalistic adoption behavior of an entire population of state outpatient mental health clinics licensed to treat children and adolescents. Given the inconsistent operationalization of adoption in the field, this study described the uptake of training in multiple ways to better understand the nuances of clinic training adoption behavior in New York State.

By a simple count of trainings adopted, clinic behavior followed a negative binomial distribution commonly found in the utilization literature (15,16). Clinics fell into different categories depending on the adoption definition. They appeared to adopt trainings on the basis of available time and resources. Webinars, the least intensive type of training, were accessed by 96% of the adopters, nearly three times the percentage of clinics that adopted learning collaboratives, the most intensive type of training. Over 40% of adopting clinics exclusively attended Webinars.

Adopters were equally open to business and clinical training, suggesting that the clinics were cognizant of the need to improve both fiscal and clinical performances. Further, the adopter groups developed for this study, from low adopters to super-adopters, conveyed meaningful adopter profiles that reflected both the number and intensity of trainings adopted.

Implications for state systems

Our findings have important implications for efforts in New York State and other states to improve uptake of training.

Increasing the number of training offerings is unlikely to improve adoption.

Clinics accessed a median of five trainings in two years. That number may reflect the optimal balance between an investment in training and the loss of staff time or revenues associated with attendance at training. Rather than offer more training, a state should develop a reasonable number of offerings in the areas of highest demand and highest importance to the state, such as improving clinic fiscal health and capacity to report outcomes, in order to improve the depth and quality of the training offered. New or different content areas may also address the needs of nonadopters.

Business and clinical training is critical to clinics’ viability in the current health care context.

Clinics demonstrated robust and comparable adoption of business and clinical trainings, which suggests that both types of training addressed clinics’ needs. In the current health care climate of accountability and quality (17), clinics need state technical assistance to adapt to the prerequisites of health care reform. For example, the behavioral health organization (BHO) initiative in New York State (18) is designed to facilitate the public behavioral health system’s transition toward managed care. BHOs will be tasked with reviewing and monitoring the operation of clinics by using standard performance indicators, such as outpatient engagement and use of electronic medical records to communicate outcomes. Adopters of CTAC business initiatives have shown initial success in financial restructuring and use of benchmarks of productivity to improve their decision making and monitoring of fiscal health (19). Similarly, adoption of clinical training addresses the need to improve patient-centered outcomes.

Intensity and accessibility of training are important adoption criteria.

Clinics favored Webinars, the least intensive and the most accessible trainings, which are strategically delivered during the lunch hour. The convenience of remaining at one’s desk (versus traveling to an on-site training) and the minimal opportunity cost (one hour versus a full day) seemed to drive clinics’ adoption of Webinars. Sequential adoption of business and clinical trainings was evident, given that clinics were more likely to adopt learning collaboratives after having sampled lower-intensity trainings. This “small-step” preparation for more intensive adoption indicates the influence of trialability on adoption decision making, and it should be considered in introducing important future initiatives (8,9).

Although an intensive application process prepares clinics for the demands of a learning collaborative, clinics had difficulty maintaining consistent participation. In three of the four learning collaboratives that already ended, less than half of the adopters attended more than 50% of the sessions, regardless of the type and length of the learning collaborative. “De-adoption” is clearly both a reality and a risk in long-term training.

Clear adopter groups communicate meaningful and important adopter profiles.

The four adopter groups (low, medium, and high adopters and super-adopters), defined by the highest intensity of training adopted, parsimoniously captured critical information about each group’s preference for the number and intensity of trainings. Low adopters accessed low-intensity trainings (Webinars only) and a low number (one to four) of trainings; high adopters and super-adopters selected high-intensity trainings (learning collaboratives) and a high number of trainings (nine or more). These groupings can help states characterize clinic adoption behavior in a meaningful way and thus create efficiencies in targeting rollout efforts. Identifying the characteristics of clinics by adopter group could provide critical information to guide the state in the development of tailored rollout strategies, which are sorely lacking (20).

Depending on the goal, state systems should target efforts to engage clinics in training by adoption pattern.

States that aim to engage all clinics in evidence-based practices and QIs must focus their efforts on nonadopters and low adopters. Understanding the barriers to adoption by these clinics will improve adoption strategies. Alternatively, if a state’s goal is to help clinics that are low or medium adopters to become high or super-adopters, varying the timing and intensity of training efforts to build on less intensive training is likely to be successful. The low participation in six- to 18-month learning collaboratives suggests that engagement of clinics in long-term trainings will be a critical area to address in future rollout.

Ultimately, this study presents nuances of adoption behavior that may help states better understand specific clinic characteristics that may explain variations in clinic adoption behavior. Predictability of different adoption outcomes examined in this study may depend on internal characteristics, such as clinic fiscal health and client profiles, and external characteristics, such as Medicaid funding and clinic region (8,21). Future work by our group will examine these characteristics to understand the facilitators and barriers related to adoption and how the state can use this knowledge to design effective rollouts.

Limitations

New York State has a history of promoting evidence-based practices and QIs through training mechanisms besides the CTAC (5), and these mechanisms may have had an impact on clinic adoption of CTAC-specific trainings. Other rollouts unrelated to state efforts—for example, initiatives by professional organizations or conferences—that are not captured in our study may target the same population of clinics. It is unknown whether adoption of state training influences the adoption of other training and vice versa. It is also unclear to what extent our findings are generalizable to other states’ efforts to improve uptake, given the differences in relationships between clinics, counties, and the state. New York State clinics, unlike those in other states, tend to operate independently rather than under the directives of their counties. Thus, clinic adoption of training is possibly more heterogeneous in New York State than in other states. Nevertheless, our findings are consistent with existing theories of adoption behavior (8,21).

Conclusions

This study represents one of only a few attempts to characterize patterns related to adoption of mental health innovations within an entire state outpatient child mental health system. Its scope extends beyond a study of evidence-based practices in Ohio (22) to include QIs and other business improvement practices. Although this study describes baseline adoption behaviors, determining how these behaviors change over time, amid mounting pressures from state and national health care reforms, will be of paramount importance for optimizing the design of state innovations. Ideally, adoption data should be used to examine how the uptake of training translates into more effective clinical practices and, ultimately, into improved patient outcomes.