Outcomes of a Statewide Learning Collaborative to Implement Mental Health Services in Pediatric Primary Care

Pediatric primary care clinicians (PPCCs) are in a unique position to address common mental health conditions but feel unprepared to do so (1, 2). Despite increased emphasis on primary care and mental health integration, a recent survey of American Academy of Pediatrics (AAP) members found limited improvement in the care of common mental health conditions over the past decade (3). Even when PPCCs wish to provide mental health care, they can be limited by varying organizational readiness levels (4), implementation challenges (5), and financial disincentives (6). National and regional shortages in pediatric mental health providers further compound this issue (7).

Several efforts to improve mental health services in pediatric primary care have been developed, yet not fully disseminated, and vary both in modality and complexity. Some interventions, including communication skills training (8, 9) and telephone consultation models (10, 11), primarily target individual practitioners. Technology has been used to obtain parent and teacher feedback and to better standardize attention-deficit hyperactivity disorder (ADHD) care (10). Specialty mental health providers have been successfully colocated in pediatric practices, increasing access to and effectiveness of care (12). However, evaluations of some of these programs, and evidence from child welfare and child mental health service settings, have suggested that sustainable changes in clinicians’ behavior require changes in the organizational context in which they practice (13–16).

The Ohio Building Mental Wellness (BMW) learning collaborative was implemented as a statewide intervention to improve the quality of mental health care delivered by PPCCs. A primary goal of the initial program was to reduce unnecessary second-generation antipsychotic medication prescribing after observed increases in Ohio Medicaid and other pediatric populations (17). Pediatric practices involved in BMW’s first learning collaborative (referred to here as waves 1 and 2) received clinician-focused didactic training and showed modest improvements in mental health screening but no changes in second-generation antipsychotic prescribing, as assessed by using a pre-post comparison (18). BMW wave 3 used concepts from organizational-behavior theory to target improvements in care at both the individual practitioner and whole practice levels. This theory suggests that organizational context—the workplace environments and interorganizational networks in which primary care staff operate—can affect the ability of individuals in the organization to change their behavior (19–22). We previously found that practices with more positive organizational contexts for mental health at baseline achieved greater uptake of program activities (16). In this study, we evaluated intervention effects on PPCCs’ confidence with mental health communication and brief intervention skills, case identification, and prescribing practices for common mental health conditions. We aimed to increase mental health-related office visits and PPCC prescribing for anxiety, depression, and ADHD and reduce PPCC prescribing of second-generation antipsychotic medications.

Methods

BMW Wave 3 Intervention

BMW wave 3 was administered by the AAP Ohio chapter from October 2013 through May 2015. Although components of its evaluation were approved as human subjects research (Nationwide Children’s Hospital Institutional Review Board [IRB] 13–00397), the project was designed as a quality improvement program in which all practices participated in the intervention. Program activities were based upon AAP core competencies (23) and communication and brief intervention skills that can be delivered in primary care (24). The implementation team included physician leaders, AAP chapter staff, and research staff. Ohio AAP recruited practices by circulating information at meetings and through mailings. Interested practices participated in an informational call to learn more about BMW prior to enrollment. Participating practices, required to have at least 20% of their patients covered by Medicaid, enrolled in one of three groups over an 18-month period, with group start times staggered to allow the implementation team to dedicate adequate time to each practice.

The collaborative was based on the Institute for Healthcare Improvement’s modified Breakthrough Series (25) and began with a regional, daylong learning session, followed by three Plan-Do-Study-Act cycles and monthly action period calls. BMW team members conducted four interactive trainings on-site at each practice. Staff in all roles attended the first training session, which addressed the motivation for addressing mental health, the collaborative care model, and core communication skills related to engaging families (8). Three clinically oriented trainings addressed additional skills to support effective communication and management of depression, focusing also on brief interventions that could be used in primary care. PPCCs participated with a BMW staff member in informal role-playing, providing an opportunity for skills practice. Participants had access to 11 online mental health learning modules; completion of the psychotropic medication module was recommended. Additional content was reinforced and reviewed during monthly action period calls.

Staff completed a survey on organizational context and attitudes relating to mental health before the first learning session and after BMW. Results were provided in aggregate and benchmarked with other practices. Practices submitted monthly reports on program uptake, operationalized as a score representing activity completion in five categories: resources, referral tracking, mental health promotion and screening, practice-based interventions, and mental health integration. Each activity was rated on a 4-point scale, with 1 indicating no changes begun; 2, changes being tested; 3, improvements implemented; and 4, fully implemented/sustained. Activity ratings were summed, and practices were assigned a continuous score based upon their degree of implementation. When all activities in a category were completed, a practice received a “star” on the BMW star recognition system. [A description of the BMW star recognition program is available as an online supplement to this article.]

Clinician Confidence

Clinician confidence was assessed by using a survey developed by the implementation team and revised by nonparticipating PPCCs to establish face validity. Confidence was measured on a scale of 1, very confident, to 4, not at all confident, obtained by averaging ratings across 21 Likert-type, self-rated survey items. Scores were reverse-coded so that higher scores represent greater confidence. At the first site visit (baseline), PPCCs rated their confidence using communication strategies for engaging children and families and their ability to deliver brief interventions for symptoms of anxiety, disruptive behavior, or depression. Clinicians rated the same items in a separate survey one year later at postintervention.

We modeled clinician confidence as a function of time, where baseline is the reference to which postintervention is compared. The log of the total confidence score was regressed on time by using a linear (mixed-effects) model with random intercepts for practices and clinicians and random slopes for clinicians. This model addresses violation of the assumption that variation in baseline and follow-up measures are the same. We also measured the correlation between change in clinician confidence and program uptake. The relationship was expected to be positive if the intervention itself enhanced confidence. A Pearson correlation coefficient was computed to assess this relationship.

Mental Health Office Visits and Prescribing Practices

We obtained deidentified Medicaid office visit (ICD-9) and pharmacy claims data from the Ohio Colleges of Medicine Government Resource Center for each BMW clinician for the 12 months before, 18 months during, and 12 months after participation in the learning collaborative. For all children and youths having at least one visit with a BMW clinician in a given month, we assessed the proportion whose office visit was for any mental health condition and the proportion whose office visit was specifically for ADHD, anxiety or depression, or disruptive behavior disorder. Mental health visits were defined by the presence of any related ICD-9 code, not limited to primary diagnosis [see online supplement]. Eligible office visits included those for patients who met any Medicaid eligibility criteria and who were younger than 18 at the start of the reporting month. We analyzed Medicaid claims data to assess monthly prescribing practices. For all children and youths having at least one visit with a BMW clinician in a given month, we analyzed the proportion prescribed one of the following classes of medications by a BMW clinician during the month of or following their visit (to capture prescriptions not filled immediately): second-generation antipsychotic medications; stimulants, atomoxetine, and alpha agonists, which are commonly prescribed for ADHD; and selective serotonin reuptake inhibitors (SSRIs), which are commonly prescribed for anxiety or depression.

To assess changes in rates of mental health office visits and prescribing over time, we conducted interrupted time-series analyses for single group comparison. This quasi-experimental method of analysis has been recommended for quality improvement research where randomization is not feasible and where there are multiple observations of an outcome reported at an aggregate level (26, 27). Analyses were conducted using the statistical analysis software Stata (28), and we employed the itsa command for time series (29).

Results

BMW Wave 3 Participation and Uptake

Twenty-nine hospital-, school-, and other community-based pediatric primary care practices participated in the learning collaborative. Three additional practices dropped out, two prior to the start and one after the start of the collaborative, as described in greater detail by King and colleagues (16). Practices cited workload and IRB issues as reasons for dropout. Characteristics of participating practices are presented in Table 1. All but three practices reported accepting uninsured patients, and all practices accepted publicly insured patients (on average, 57% of patients were enrolled in Medicaid; range 25%−98%). Eleven of the 29 practices were school-based health centers (SBHCs) from one metropolitan area. The remaining practices were in urban (N=6), suburban (N=8), and rural (N=4) areas. Each of the 18 non-SBHCs served on average 314 patients per week (range 75–775) and had, on average, 15 staff members (range 6– 45), six of whom were pediatricians or nurse practitioners (range 2–30). Half (N=9) reported having one or more mental health specialists (psychiatrist, psychologist, counselor, or social worker) colocated at least part time. SBHCs in schools with the same range of grades pooled their data, yielding three groups. This had implications for analysis, which was conducted on 21 practice clusters.

Uptake was, on average, high but variable across practices, with an average continuous star score of 62 (range 53– 68) and an average of three stars per practice (range 1– 5). The BMW team conducted 71 site visits, and 44 of 72 clinicians at BMW-participating practices completed at least one of the 11 online modules, with a total of 122 completed modules and an average of three completed modules per participant.

Clinician Confidence

Clinician confidence surveys were completed by 52 of 72 clinicians at BMW-participating practices at baseline and again postintervention. Based on the linear mixed-effects model of total confidence scores with no covariates and a random intercept for practice, average total confidence score at the first site visit across practices was 2.92 out of 4.00 points (CI=2.8 to 3.1). Clinician confidence increased over the course of the on-site trainings by an average of 20% (CI=15% to 25%) to 3.55 postintervention. This corresponds to an average increase of 0.63 points (CI=0.47 to 0.79), moving from a mean response of “somewhat” to “very” confident in addressing mental health communication and brief intervention skills. As hypothesized, there was also a positive correlation between BMW uptake and change in practice-mean clinician confidence from baseline to postintervention (N=17; r=0.66, p=0.004).

Rates of Mental Health–Related Visits

Figure 1 displays the actual rates and trends of mental health–related visits among patients seen by BMW clinicians before, during, and after the intervention. Among unique Medicaid patients who visited a BMW clinician in the first month of the year prior to the start of the intervention, 6.65% (N=316 of 4,752 patients) did so for a mental health condition, a percentage that increased significantly every month thereafter by 0.14 points (CI=0.05 to 0.23; p=0.003). There was no significant change in this trend after the start of BMW, but there was a significant (p=0.011) leveling out at 9% that coincided with the conclusion of the four on-site training sessions. In the months following the conclusion of BMW, the proportion of patients with mental health–related visits leveled out at about 9%.

The positive trend in rates of mental health–related visits among patients seen by BMW clinicians was largely driven by growth in visits with a diagnosis of ADHD. Among patients with office visits in the first month of observation (a year prior to the start of BMW), 5.11% were seen for ADHD (N=243 of 4,752 patients). This rate increased significantly every month prior to BMW by 0.08 points (CI=0.02 to 0.14; p=.011). Increases in rates of patients with ADHD-related visits after the start of BMW were not significantly different from what would be expected given the preintervention trend. The increase continued during and postintervention, with the proportion of patients who were seen for ADHD in a month approaching 7% for 2 years following the start of BMW (Figure 2). Office visits for anxiety and depression and disruptive behavior pertained to a smaller number of children and youths (average of 60 and 45 patients per month, respectively) compared with ADHD (266 patients per month) or any mental health diagnosis (348 patients per month). While the per-month proportion of patients having office visits for anxiety/depression or disruptive behavior disorder also trended upward, from less than 1% one year prior to the start of BMW to between 1% and 2% after BMW, we found no evidence of an intervention effect.

Prescribing Practices

Rates of SSRI and ADHD medication prescribing for all patients did not change significantly over time. However, trends in ADHD medication prescribing did change following the start of BMW for those patients diagnosed as having ADHD. The proportion of patients with an ADHD office visit who were prescribed a stimulant, atomoxetine, or alpha agonist by a BMW-participating clinician was just under 60% at the start of BMW. Prescribing rates for ADHD medications to patients with ADHD visits increased significantly by 0.12 percentage points per month (CI=0.02 to 0.22; p=0.022), relative to the preintervention trend, rising to 62% in month 24—2 years following the start of BMW (Figure 3).

Trends in second-generation antipsychotic prescribing also changed significantly following the start of BMW. Among patients with an office visit in the first month of the year prior to the start of BMW, 0.40% were prescribed a second-generation antipsychotic by a BMW-participating clinician. This rate approached 0.50% in the month just prior to the start of BMW. Following the start of the intervention, there was a significant decrease in the monthly trend of second-generation antipsychotic prescribing of 0.014 percentage points per month (CI=–0.03 to –0.00; p=0.028), relative to the preintervention trend (Figure 4). In addition, there was a shift in the diagnoses associated with second-generation antipsychotic prescribing. Among patients with visits for a mental health condition in the 12 months prior to BMW, 74.1% of second-generation antipsychotic prescribing followed an office visit for ADHD and anxiety or depression. In the 12 months postintervention, this proportion decreased significantly to 61.6% (p=0.001).

Discussion

This study suggests that an intervention addressing individual staff and organizational factors in tandem may be effective in improving the implementation and quality of mental health services in pediatric primary care. PPCCs reported increased confidence in addressing mental health and conducted higher rates of mental health office visits, largely driven by increases in ADHD-related office visits. Prescribing patterns also improved: a greater proportion of children with ADHD received an indicated medication, the overall proportion of children receiving an antipsychotic decreased, and a greater proportion of these prescriptions were for potentially indicated conditions such as autism and bipolar disorder. A decrease of 0.014 percentage points per month is small but significant when one considers national data: The percentage of children and youths in the United States using antipsychotics is estimated to be between 0.14% and 0.11% for young children, 0.85% and 0.80% for older children, and 1.10% and 1.19% for adolescents (30).

Several authors have developed interventions to support identification of mental health conditions and to improve the practices of prescribing psychotropic medication in pediatric primary care. These interventions have included second opinions for ADHD prescribing outside of safety parameters (31) and telephone consultation, more generally (32). The Ohio Department of Medicaid established a collaborative targeting high prescribers (pediatricians and psychiatrists) to reduce pediatric antipsychotic prescribing (33). Participating pediatricians demonstrated improvement in prescribing two or more concomitant antipsychotics, which decreased significantly over time. Both pediatricians and psychiatrists cited difficulties linking patients with timely psychosocial services and identifying community resources as important barriers to reducing antipsychotic prescribing. BMW wave 3 differed from other interventions and from earlier waves because of its focus on effecting changes in the skills of clinical and nonclinical staff and practices’ organizational contexts (workplace environments and interorganizational networks) for mental health care. In earlier waves, a lecture format was used to teach clinician skills, and reductions in second-generation antipsychotic prescribing were not observed. Previous wave 3 qualitative analysis suggests that all-staff participation was important in facilitating uptake of the BMW intervention and enhancing the organizational context in which services are rendered (34). These findings are consistent with the adult collaborative care literature, which has demonstrated that change in organizational climate and culture is required before collaborative care can be implemented (35).

There were several important limitations to this study. It was likely that these practices represent early adopters and results may not be generalizable. The quasi-experimental design was another limitation, given that we could not control for regression to the mean or the potential existence of larger system-level influences occurring at the same time as the intervention (36), although the “leveling out” of the positive trend in mental health visits after the intervention provides some support of an intervention effect. Only data for Medicaid-enrolled patients who visited BMW practices could be used because of the study design, and patients from nonparticipating practices (a potential comparison group) could not be included in our analyses. Although our measures are clinically appropriate, we were unable to validate the accuracy of diagnoses or quality of prescribing practices.

Conclusions

The findings suggest that a complex intervention addressing individual and organizational factors in tandem may be effective in improving the quality of mental health services offered in pediatric primary care. Research is needed to identify core intervention components and how organizational context may mediate or moderate intermediate outcomes (e.g., staff attitudes toward mental health). Additional areas of exploration include clinical outcomes, cost-effectiveness, and scalability of learning collaborative models and complex interventions for primary care and mental health integration more generally.