An Approach to Evaluating Competence in Assessing and Managing Violence Risk

The growing national emphasis on competence as the outcome of education of health care professionals has created a need for methods to assess competency in specific clinical skills (1,2). Competence in evaluation and management of patients' risk of violence is expected of psychiatrists, psychologists, and other mental health professionals (3,4). For example, the American Board of Psychiatry and Neurology (ABPN) and the Accreditation Council of Graduate Medical Education have identified the assessment and management of risk of violence as core competencies for psychiatrists (3,5). Major credentialing organizations, such as the ABPN, have begun to phase out oral examinations and have placed more responsibility on residency training programs to assess the competency of individual practitioners (6). Although these trends increase the need for training programs to assess the competency of psychiatry residents, clinical psychology interns, and other clinical trainees in evaluation and management of risk of violence, our review of the literature identified no published methodology for assessing this competency.

This report describes development of a tool for evaluating the competency of trainees in psychiatry and psychology in violence risk assessment and management. We evaluated the reliability, validity, and acceptability of the measure using an objective structured clinical examination (OSCE) (7), in which learners performed a violence risk assessment of a standardized patient while being observed by faculty members. We addressed these questions: What are the internal consistency reliability and interrater reliability of the instrument? Do senior learners perform better on the measure than junior learners? To what extent are learners and faculty satisfied with this approach?

Methods

This study was approved by the Committee on Human Research of the University of California, San Francisco.

We developed the Competency Assessment Instrument for Violence (CAI-V) on the basis of the literature on violence risk assessment and management (8–10), adaptation of criteria for evaluating the quality of violence risk assessment from a previous study of the impact of training on violence risk assessment (11), and the literature on measurement of competencies in medical education (12). To increase content validity, we obtained feedback on drafts of the CAI-V from faculty focus groups at three sites of a large academic psychiatry department: a county hospital, a university hospital, and a veterans hospital. The focus groups discussed what they considered important for trainees to master in performing a competent risk assessment for interpersonal violence, including content-related items (such as asking about violent ideation and knowledge of civil commitment laws) and process-related information (including reviewing the medical record and obtaining information from collateral sources, such as other clinicians and family members).

The CAI-V includes a checklist of 31 components of violence risk assessment: interviewing and data collection (six items each for sources of information and types of information), case formulation and presentation (six items), treatment planning (12 items), and documentation (one item). [A copy of the CAI-V is available as an online supplement to this brief report at ps.psychiatryonline.org.] Each checklist item is rated on a scale from 1, task not done, to 4, advanced. In addition to the checklist, the CAI-V includes a global rating of the overall quality of the risk assessment for violence, ranging from 1, unacceptable, to 8, advanced.

Learners were 31 trainees who attended a five-hour workshop on risk assessment for violence and suicide in July 2008. Twenty-six learners were residents in psychiatry (12 in the first postgraduate year and 14 in the second postgraduate year), and five were clinical psychology interns. The workshop included a pretest, lectures on violence and suicide, an OSCE, and a posttest course evaluation. This report concerns the components relevant to violence. The workshop presented a model for assessment and management of violence risk based on the Historical, Clinical, and Risk Management-20 (HCR-20) instrument (13), as well as review of medical and legal aspects of clinical documentation (8,11). The lectures outlined an approach for gathering information about risk and protective factors supported by the research literature on violence, rationally weighing the significance of these variables to form a professional judgment about the level of risk of violence, developing and implementing a plan to reduce the risk, and documenting this process. After the didactic portion of the workshop, the learners gave informed consent to participate in the OSCE. We explained what the OSCE involved, that participation in the OSCE was voluntary, and that the results would be used to refine education methods and would not become part of learners' individual training files.

Thirty-one faculty members (26 psychiatrists and five psychologists) served as observers in the OSCE. After receiving an overview of the content of the lectures provided to learners, faculty observers received two hours of training in how to use the CAI-V in an OSCE setting. As part of the training, faculty members watched mock OSCEs in which learners interviewed standardized patients and presented the violence risk assessment findings to observers. Faculty members rated the quality of each violence risk assessment with the CAI-V.

Thirty-one senior trainees (27 third- or fourth-year psychiatry residents and four clinical psychology postdoctoral fellows) received two hours of training in how to serve as standardized (simulated) patients. They learned to follow a script based on a clinical vignette about a young adult patient presenting to an emergency department. The script included the patient's chief complaint; history of present illness; psychiatric, medical, and psychosocial histories; and mental status examination findings.

Each OSCE team included a learner, a standardized patient, and a faculty observer. The entire OSCE procedure took approximately one hour. After receiving a brief description of the presenting problem, the learner was asked to perform a violence risk assessment of the standardized patient. The learner interviewed the standardized patient, discussed additional information he or she would obtain if this were a real situation, wrote a progress note, and gave an oral summary of the assessment and plan regarding the patient's risk of violence. The faculty observer rated the CAI-V and then gave feedback to the learner.

To assess the interrater reliability of the CAI-V, we conducted a second study in July 2009 in which six faculty observers (four psychiatrists and two psychologists) rated videos of three mock OSCEs. In each video, a separate learner role-played interviewing a standardized patient and wrote a progress note summarizing the assessment and plan regarding the patient's risk of violence.

We used Cronbach's alpha to assess the internal consistency reliability of the CAI-V. To evaluate the hypothesis that senior learners would perform better on the CAI-V than junior learners, we used one-tailed t tests for continuous variables and chi square analyses for categorical variables. In the subsidiary study of the interrater reliability of the CAI-V, we used the intraclass correlation coefficient (ICC). Because we expect that in practice the CAI-V will usually be used by one faculty observer, we calculated the single-measure ICC (ICC₁) to characterize its interrater reliability.

Results

The internal consistency reliability of the 31-item CAI-V checklist was high (α=.93), suggesting that the checklist items measure a common domain. We categorized learners as senior (second-year psychiatry residents, who had six months of formally supervised inpatient experience that included frequent risk assessments) and junior (first-year psychiatry residents and predoctoral psychology interns). Mean±SD scores on the checklist of components of violence risk assessment were significantly higher for senior learners (90.6±12.9) than junior learners (81.9±14.2) (t=−1.78, df=29, p<.05). Similarly, the global ratings of the overall quality of the violence risk assessment were significantly higher for senior learners (5.6±.9) than junior learners (4.6±1.1) (t=−2.67, df=29, p<.01).

Summative judgments of whether performance in the OSCE demonstrated competency were estimated by dichotomizing the global ratings of overall quality of the violence risk assessment as either competent (rated as 5, competent, to 8, advanced) or not competent (rated as 1, unacceptable, to 4, working toward competency). The risk assessments by senior learners were significantly more likely than those by junior learners to be rated as competent (χ²=4.18, df=1, p<.05). Ninety-three percent (13 of 14) of the violence risk assessments by senior learners were rated as competent, compared with 47% (eight of 17) of risk assessments by junior learners.

At the end of the workshop, when learners were asked to rate, on a 7-point scale ranging from 1 (not at all) to 7 (extremely), whether the CAI-V would be helpful for getting feedback from supervisors in real encounters with potentially violent patients, the mean±SD rating was 5.9±1.0. When learners were asked whether the OSCE was helpful for learning about working with potentially violent patients, the mean rating was 6.0±1.0. Similarly, the mean rating by faculty observers on the same 7-point scale was 5.4±1.0 for whether the CAI-V was helpful for rating competency in working with potentially violent patients and 5.5±.9 regarding whether the OSCE was helpful for assessing competency in working with violent patients. In summary, learners and faculty expressed considerable satisfaction with the CAI-V and OSCE for evaluating competency in risk assessment for violence.

In the subsidiary study of interrater agreement, the ICC for the 31-item CAI-V checklist was .93. The ICC for the global rating of the overall quality of the risk assessment was .93.

Discussion and conclusions

Historically, assessment of the competency of individual clinicians has been left to summative evaluations such as examinations for licensure or board certification. In view of the national movement toward expectations that clinical training programs in psychiatry, psychology, and other mental health professions will document that graduates have attained key clinical competencies, we expect increased demand for objective approaches to assess these competencies. The results of this study support the promise of a new approach to evaluating competency in violence risk assessment and management. The CAI-V showed high internal consistency reliability and interrater reliability. Content validity was addressed by basing the measure on the literature on violence risk assessment, which was complemented by feedback from faculty focus groups at multiple sites of a large academic psychiatry department. Evidence of concurrent validity is that senior learners in the mental health professions performed better than junior learners on the CAI-V in the context of an OSCE. Moreover, both learners and faculty expressed satisfaction with the method, reporting that the CAI-V and OSCE provided a helpful structure for feedback and supervision concerning violence risk assessment and management.

In the field of violence risk assessment, research in recent years has yielded multiple instruments with the potential to structure the assessment process in ways that can enhance decision making about patients' risk of violence (14). The learners in this project were trained in the HCR-20, an instrument with empirical support as an aid to violence risk assessment. Although such tools may assist decision making about potentially violent patients, clinical competency in risk assessment and management includes skill in deciding what information needs to be obtained, facility in obtaining it, ability to formulate the clinical problem and develop a relevant treatment plan, and ability to document this process. To our knowledge, this is the first published description of an objective, structured approach to evaluation of competency in violence risk assessment and management.

Future research is needed to explore the generalizability of the CAI-V across multiple cases within the OSCE format. It would also be helpful to evaluate utility of the CAI-V for supervision of learners working with actual patients. We expect that the initial application of the method will be in formative evaluations that assist faculty in giving structured feedback to learners. The method has potential for informing summative evaluations of development of competency in risk assessment, if applied to learners' performance with multiple cases over time. Finally, future research, with larger samples, is needed to determine the generalizability of the results to other institutions and training programs.

In summary, the findings of this study suggest the promise of a new approach to evaluating skills in violence risk assessment and management and a format for providing structured and specific feedback to clinicians who are acquiring competency in this domain.