Evaluation of a Mentoring Program for PTSD Clinic Managers in the U.S. Department of Veterans Affairs

The number of veterans diagnosed as having posttraumatic stress disorder (PTSD) who receive health care from the U.S. Department of Veterans Affairs (VA) has tripled since 1990 (1,2). With the combination of improved PTSD screening, veterans returning from conflicts in Iraq and Afghanistan, and more older Vietnam-era veterans enrolling as patients, the VA annually cares for over 300,000 veterans with PTSD diagnoses (2–4). To meet this demand, the VA has hired over 6,000 new mental health providers since 2004, including 350 providers in PTSD specialty clinics (5).

In 2008, the VA Uniform Mental Health Services Handbook set forth a vision to dramatically change mental health services in the VA and provide a more consistent quality of care (6). Criteria for improved quality included mandates to reduce wait times, make evidence-based psychotherapies for PTSD available at every VA medical center, tailor services to the needs of Iraq/Afghanistan veterans and female veterans, treat consequences of sexual trauma during military service, and improve coordination of care for co-occurring PTSD and substance use disorders. The VA supported these policy goals through national rollouts of training in evidence-based psychotherapies, appointing coordinators at each VA facility to work with Iraq/Afghanistan veterans, appointing coordinators to promote evidence-based psychotherapies, and hiring additional staff to provide care for co-occurring PTSD and substance misuse (7).

Implementing such mandates poses significant organizational challenges to clinic managers (8,9). For example, clinics are expected to provide more individual evidence-based psychotherapy and less group-based supportive counseling while continuing to serve the same number of clients or even more. Managers must maintain staff morale during a period of change. Yet clinic managers often lack formal training in management or systems thinking to help their clinics meet such challenges (10,11).

The VA PTSD Mentoring Program was launched in April 2008 at the behest of VA Mental Health Services (5). The mentoring program is intended to promote top-down and bottom-up sharing of information on new mandates and evidence-based practices, encourage peer-to-peer sharing of innovations and solutions, and facilitate mutual instrumental and emotional support.

Mental health leadership in each of the VA’s 21 regional networks invited two experienced and well-respected clinical managers of PTSD treatment programs to serve as mentors. All other heads of PTSD programs were invited to be mentees. Participation was voluntary. Mentors and mentees did not get any reduction in other job responsibilities to allow time for participation.

The 42 mentors were expected to champion best practices, offer advice from their own experience, and serve as conduits for sharing information (5). Mentors’ time involvement varied, but they typically chaired monthly regional telephone calls and provided additional individual consultation as needed. These regional efforts were supported by monthly national conference calls for the mentors, national calls open to all participants, site visits, an intranet site, a Webinar series on clinical issues, and a newsletter. Mentors received guidance and support through ongoing contact with the national mentoring program manager, who serves as an external facilitator (12).

We conducted the survey to elicit feedback from mentees to inform further program development. In addition to providing descriptive findings, the survey examined some assumptions of the program logic model (13). We anticipated that mentees who had fewer years of experience would engage more frequently in program activities. We expected that mentees who participated more frequently would derive greater personal benefit from the program. We predicted that mentees who perceived their mentor to be knowledgeable and helpful would participate more frequently in program activities and would derive more benefit from the program. Finally, we hypothesized that amount of benefit that mentees derived from the program would mediate how much they credited the program with helping them implement clinical innovations.

Methods

The Stanford Institutional Review Board (IRB) determined that this study was conducted for program evaluation and did not require IRB review. Our analysis focused on responses from mentees. All mentees received two e-mails inviting them to complete a 15-minute online survey during the month of June 2010. Participants were not compensated. Of 262 mentees contacted, 121 (46%) completed the survey. A majority of respondents were psychologists (72%, N=87) between 40 and 60 years of age (52%, N=63). Over half were managers of PTSD treatment programs (56%, N=68), and the rest were nonmanager clinicians. Mentees typically had been treating PTSD from six to 20 years (50%, N=61).

The Mentoring Program Feedback Questionnaire was developed by the authors for this evaluation. Background information included participants’ profession, age, gender, clinical role or roles, tenure with the VA, years of experience treating PTSD, and supervisory responsibilities. Participants indicated how many times they had engaged in 14 specific program activities in the past six months. Respondents also indicated to what extent lack of time, lack of leadership support, calls conflicting with other duties, and lack of benefit limited their participation in the mentoring program, using a 7-point Likert scale (from 1, “not a barrier,” to 7, “major reason for not participating more often”).

Participants indicated how much the program helped them achieve 15 specific benefits (for example, “Helping me learn about new innovations in PTSD care”) by using a Likert scale from 1, “not at all helpful,” to 7, “extremely helpful.” For descriptive purposes, scores of 5 or higher were considered endorsement of each benefit. Program benefit items formed an internally reliable scale (Cronbach’s α=.96).

Participants used a 7-point Likert scale (“strongly disagree” to “strongly agree”) to indicate how much their mentor “offers useful, practical solutions,” “is too busy to provide much help to me,” “is well-organized and uses time productively during our calls,” and “has too little power and authority to help me.” After negative items were reverse-coded, these formed a single scale (Cronbach’s α=.78).

Respondents were asked whether their clinic had made no changes, minor changes, or substantial changes in the past two years to achieve the following: better meet needs of Iraq/Afghanistan veterans, more efficiently provide support to older veterans, implement evidence-based treatments, use a new specialist to improve care for veterans with co-occurring PTSD and substance use disorders, and build staff cohesion and morale. Respondents who reported making any changes indicated how much they credited the program with facilitating that change by using a 7-point Likert scale from 1, “mentoring program did not help at all,” to 7, “mentoring program was essential.”

Descriptive results were presented as means, standard deviations, and frequencies. Bivariate associations were assessed with correlations. A structural equation model, using Amos 16.0 software, was used to test hypotheses regarding variables associated with perceived benefit from the program. Variables with significant bivariate correlations were included in the model, and paths were trimmed until the Akaike information criterion was minimized and the model chi square was nonsignificant.

Results

Participants reported engaging in program activities 12.1±6.7 times (mean±SD) in the past six months. The most commonly reported activities were attending regional conference calls (74%, N=90, participated one or more times), talking with or e-mailing their mentor (70%, N=84), and visiting the program Web site (68%, N=82). Contrary to our hypothesis, frequency of participation was not significantly correlated with years of experience treating PTSD or number of years in a supervisory role. As expected, frequency of participation was negatively associated with lack of time (r=–.25, df=120, p=.01) and positively associated with perceived mentor quality (r=.38, df=120, p=.01).

Over half of participants who responded to survey items about program benefits (N=117) reported that the mentoring program had helped them feel more connected to other PTSD program directors and clinicians (62%, N=72), better understand the goals of VA leadership (56%, N=65), learn about innovations in PTSD care (55%, N=65), and feel more supported (52%, N=61). From 40% to 50% reported that the program helped them improve implementation of VA Uniform Mental Health Services Handbook mandates in their programs (N=59), enhance quality of care (N=56), get access to training (N=54), implement VA policies (N=53), and solve practical problems (N=49). Only about one-quarter of respondents felt the program helped them achieve more distal goals, such as learning how to cut through bureaucratic red tape (N=29) or improve staff morale (N=28).

One hundred eight (89%) participants reported making substantial changes to implement evidence-based treatments (N=94, 78%), meet the needs of Afghanistan/Iraq veterans (N=78, 65%), more efficiently meet the needs of older veterans (N=38, 31%), improve coordination of care for PTSD and co-occurring substance use disorder (N=58, 48%), and improve staff morale (N=29, 24%). The program was credited with facilitating clinical improvements by about one-third (36%, N=41) of those who made changes to implement evidence-based treatments and one-quarter (25%, N=29) of those who modified services for Afghanistan/Iraq veterans. Only 14%–19% of participants whose clinics made changes to more efficiently meet the needs of older veterans (N=16), improve care for co-occurring PTSD and substance use disorders (N=12), or improve staff morale (N=13) credited the mentoring program with facilitating those changes.

The final structural equation model had a good fit to the observed data (χ²=3.6, df=5, p=.61, goodness-of-fit index=.988, root mean square error of approximation=.00) (Figure 1). The results confirmed that respondents who participated more frequently in program activities (β=.25, df=120, p<.01) and rated their mentor highly (β=.59, df=120, p<.01) reported greater benefit from the program. As hypothesized, the degree of benefit that participants perceived mediated the relationship between mentor quality, frequency of participation, and their crediting the program with facilitating clinical improvements.

Discussion

The Uniform Mental Health Services Handbook mandates dramatic changes in mental health services in the VA. Local managers and clinicians must overcome administrative challenges to carry out these mandates successfully. The PTSD Mentoring Program was initiated with the hope that we could develop and share local solutions to such problems. Results of this evaluation suggest that the mentoring program has been somewhat successful in providing the right structures and supports for administrators to identify and work with others in similar positions, for mental health leadership to directly communicate policies to the field, and for providers to share their feedback with leadership.

The mentoring program was intended to help managers address organizational challenges which in turn could facilitate implementation of new VA policy directives. However, the program was initiated alongside several other initiatives to support these VA policies. These concurrent initiatives included nationwide training in evidence-based treatments, hiring of evidence-based practice coordinators, and setting of new performance standards. It is therefore hard to assess the incremental effect of the mentoring program on clinical improvement.

Our results confirm that the mentor-mentee relationship is critical to the program’s success. Program effectiveness can be enhanced by further strengthening the mentor-mentee relationship. First, it is important to select mentors who not only have the requisite expertise but can also dedicate sufficient time to mentoring. Second, mentors’ roles are being redefined to more explicitly support quality improvement. Each participant seeking mentoring is now expected to specify a quality improvement goal; mentors are tasked with helping mentees clarify the changes needed to achieve their goal, providing problem-solving support, linking them with peers who successfully instituted similar changes, and giving feedback. A potential future enhancement to improve effectiveness would be providing mentors with more formalized training in best coaching practices (12,14).

This study had important limitations. Although the 46% response rate is not unusual for a survey of this type, our sample likely overrepresents people most engaged with the program. Data were obtained by self-report. However, we obtained widely varying responses, suggesting that some participants were willing to report negative perceptions. The cross-sectional design allowed us only to show patterns of association, not prove causality.

Conclusions

The VA PTSD Mentoring Program is one of the first large-scale efforts to help mental health clinic managers address organizational challenges they face in implementing new policy mandates and local innovations to improve care. This evaluation provides initial indications that mentees who were actively engaged with the program derived important benefits. The quality of the mentoring relationship is critical to mentees’ experience. Efforts are under way to further enhance the program by strengthening mentor selection and training.