0
Get Alert
Please Wait... Processing your request... Please Wait.
You must sign in to sign-up for alerts.

Please confirm that your email address is correct, so you can successfully receive this alert.

1
Brief Reports   |    
An Approach to Evaluating Competence in Assessing and Managing Violence Risk
Dale E. McNiel, Ph.D.; Erick K. Hung, M.D.; Robert J. Cramer, Ph.D.; Stephen E. Hall, M.D.; Renee L. Binder, M.D.
Psychiatric Services 2011; doi: 10.1176/appi.ps.62.1.90
View Author and Article Information

Portions of this report were presented at the annual convention of the American Psychological Association in San Diego, California, August 12–15, 2010.

Copyright © 2011 by the American Psychiatric Association.

Objective:  The authors developed and evaluated a tool for assessing competency in violence risk assessment and management.

Methods:  The Competency Assessment Instrument for Violence Risk (CAI-V) was based on the literature on violence risk assessment, which was complemented by feedback from faculty focus groups. In an objective structured clinical examination, 31 faculty observers used the CAI-V to rate the performance of 31 learners' (26 psychiatry residents and five psychology interns) risk assessments of standardized (simulated) patients. In an interrater reliability study, six faculty members rated video-recorded risk assessments.

Results:  The CAI-V had good internal consistency reliability (α=.93). Senior learners performed better on the CAI-V than junior learners, supporting the instrument's concurrent validity. Interrater reliability was good (intraclass correlation coefficient=.93). Participants reported that the CAI-V provided a helpful structure for feedback and supervision.

Conclusions:  The results supported the potential of this new approach for appraising competency in violence risk assessment and management. (Psychiatric Services 62:90–92, 2011)

Abstract Teaser
Figures in this Article

The growing national emphasis on competence as the outcome of education of health care professionals has created a need for methods to assess competency in specific clinical skills (1,2). Competence in evaluation and management of patients' risk of violence is expected of psychiatrists, psychologists, and other mental health professionals (3,4). For example, the American Board of Psychiatry and Neurology (ABPN) and the Accreditation Council of Graduate Medical Education have identified the assessment and management of risk of violence as core competencies for psychiatrists (3,5). Major credentialing organizations, such as the ABPN, have begun to phase out oral examinations and have placed more responsibility on residency training programs to assess the competency of individual practitioners (6). Although these trends increase the need for training programs to assess the competency of psychiatry residents, clinical psychology interns, and other clinical trainees in evaluation and management of risk of violence, our review of the literature identified no published methodology for assessing this competency.

This report describes development of a tool for evaluating the competency of trainees in psychiatry and psychology in violence risk assessment and management. We evaluated the reliability, validity, and acceptability of the measure using an objective structured clinical examination (OSCE) (7), in which learners performed a violence risk assessment of a standardized patient while being observed by faculty members. We addressed these questions: What are the internal consistency reliability and interrater reliability of the instrument? Do senior learners perform better on the measure than junior learners? To what extent are learners and faculty satisfied with this approach?

This study was approved by the Committee on Human Research of the University of California, San Francisco.

We developed the Competency Assessment Instrument for Violence (CAI-V) on the basis of the literature on violence risk assessment and management (810), adaptation of criteria for evaluating the quality of violence risk assessment from a previous study of the impact of training on violence risk assessment (11), and the literature on measurement of competencies in medical education (12). To increase content validity, we obtained feedback on drafts of the CAI-V from faculty focus groups at three sites of a large academic psychiatry department: a county hospital, a university hospital, and a veterans hospital. The focus groups discussed what they considered important for trainees to master in performing a competent risk assessment for interpersonal violence, including content-related items (such as asking about violent ideation and knowledge of civil commitment laws) and process-related information (including reviewing the medical record and obtaining information from collateral sources, such as other clinicians and family members).

The CAI-V includes a checklist of 31 components of violence risk assessment: interviewing and data collection (six items each for sources of information and types of information), case formulation and presentation (six items), treatment planning (12 items), and documentation (one item). [A copy of the CAI-V is available as an online supplement to this brief report at ps.psychiatryonline.org.] Each checklist item is rated on a scale from 1, task not done, to 4, advanced. In addition to the checklist, the CAI-V includes a global rating of the overall quality of the risk assessment for violence, ranging from 1, unacceptable, to 8, advanced.

Learners were 31 trainees who attended a five-hour workshop on risk assessment for violence and suicide in July 2008. Twenty-six learners were residents in psychiatry (12 in the first postgraduate year and 14 in the second postgraduate year), and five were clinical psychology interns. The workshop included a pretest, lectures on violence and suicide, an OSCE, and a posttest course evaluation. This report concerns the components relevant to violence. The workshop presented a model for assessment and management of violence risk based on the Historical, Clinical, and Risk Management-20 (HCR-20) instrument (13), as well as review of medical and legal aspects of clinical documentation (8,11). The lectures outlined an approach for gathering information about risk and protective factors supported by the research literature on violence, rationally weighing the significance of these variables to form a professional judgment about the level of risk of violence, developing and implementing a plan to reduce the risk, and documenting this process. After the didactic portion of the workshop, the learners gave informed consent to participate in the OSCE. We explained what the OSCE involved, that participation in the OSCE was voluntary, and that the results would be used to refine education methods and would not become part of learners' individual training files.

Thirty-one faculty members (26 psychiatrists and five psychologists) served as observers in the OSCE. After receiving an overview of the content of the lectures provided to learners, faculty observers received two hours of training in how to use the CAI-V in an OSCE setting. As part of the training, faculty members watched mock OSCEs in which learners interviewed standardized patients and presented the violence risk assessment findings to observers. Faculty members rated the quality of each violence risk assessment with the CAI-V.

Thirty-one senior trainees (27 third- or fourth-year psychiatry residents and four clinical psychology postdoctoral fellows) received two hours of training in how to serve as standardized (simulated) patients. They learned to follow a script based on a clinical vignette about a young adult patient presenting to an emergency department. The script included the patient's chief complaint; history of present illness; psychiatric, medical, and psychosocial histories; and mental status examination findings.

Each OSCE team included a learner, a standardized patient, and a faculty observer. The entire OSCE procedure took approximately one hour. After receiving a brief description of the presenting problem, the learner was asked to perform a violence risk assessment of the standardized patient. The learner interviewed the standardized patient, discussed additional information he or she would obtain if this were a real situation, wrote a progress note, and gave an oral summary of the assessment and plan regarding the patient's risk of violence. The faculty observer rated the CAI-V and then gave feedback to the learner.

To assess the interrater reliability of the CAI-V, we conducted a second study in July 2009 in which six faculty observers (four psychiatrists and two psychologists) rated videos of three mock OSCEs. In each video, a separate learner role-played interviewing a standardized patient and wrote a progress note summarizing the assessment and plan regarding the patient's risk of violence.

We used Cronbach's alpha to assess the internal consistency reliability of the CAI-V. To evaluate the hypothesis that senior learners would perform better on the CAI-V than junior learners, we used one-tailed t tests for continuous variables and chi square analyses for categorical variables. In the subsidiary study of the interrater reliability of the CAI-V, we used the intraclass correlation coefficient (ICC). Because we expect that in practice the CAI-V will usually be used by one faculty observer, we calculated the single-measure ICC (ICC1) to characterize its interrater reliability.

The internal consistency reliability of the 31-item CAI-V checklist was high (α=.93), suggesting that the checklist items measure a common domain. We categorized learners as senior (second-year psychiatry residents, who had six months of formally supervised inpatient experience that included frequent risk assessments) and junior (first-year psychiatry residents and predoctoral psychology interns). Mean±SD scores on the checklist of components of violence risk assessment were significantly higher for senior learners (90.6±12.9) than junior learners (81.9±14.2) (t=−1.78, df=29, p<.05). Similarly, the global ratings of the overall quality of the violence risk assessment were significantly higher for senior learners (5.6±.9) than junior learners (4.6±1.1) (t=−2.67, df=29, p<.01).

Summative judgments of whether performance in the OSCE demonstrated competency were estimated by dichotomizing the global ratings of overall quality of the violence risk assessment as either competent (rated as 5, competent, to 8, advanced) or not competent (rated as 1, unacceptable, to 4, working toward competency). The risk assessments by senior learners were significantly more likely than those by junior learners to be rated as competent (χ2=4.18, df=1, p<.05). Ninety-three percent (13 of 14) of the violence risk assessments by senior learners were rated as competent, compared with 47% (eight of 17) of risk assessments by junior learners.

At the end of the workshop, when learners were asked to rate, on a 7-point scale ranging from 1 (not at all) to 7 (extremely), whether the CAI-V would be helpful for getting feedback from supervisors in real encounters with potentially violent patients, the mean±SD rating was 5.9±1.0. When learners were asked whether the OSCE was helpful for learning about working with potentially violent patients, the mean rating was 6.0±1.0. Similarly, the mean rating by faculty observers on the same 7-point scale was 5.4±1.0 for whether the CAI-V was helpful for rating competency in working with potentially violent patients and 5.5±.9 regarding whether the OSCE was helpful for assessing competency in working with violent patients. In summary, learners and faculty expressed considerable satisfaction with the CAI-V and OSCE for evaluating competency in risk assessment for violence.

In the subsidiary study of interrater agreement, the ICC for the 31-item CAI-V checklist was .93. The ICC for the global rating of the overall quality of the risk assessment was .93.

Historically, assessment of the competency of individual clinicians has been left to summative evaluations such as examinations for licensure or board certification. In view of the national movement toward expectations that clinical training programs in psychiatry, psychology, and other mental health professions will document that graduates have attained key clinical competencies, we expect increased demand for objective approaches to assess these competencies. The results of this study support the promise of a new approach to evaluating competency in violence risk assessment and management. The CAI-V showed high internal consistency reliability and interrater reliability. Content validity was addressed by basing the measure on the literature on violence risk assessment, which was complemented by feedback from faculty focus groups at multiple sites of a large academic psychiatry department. Evidence of concurrent validity is that senior learners in the mental health professions performed better than junior learners on the CAI-V in the context of an OSCE. Moreover, both learners and faculty expressed satisfaction with the method, reporting that the CAI-V and OSCE provided a helpful structure for feedback and supervision concerning violence risk assessment and management.

In the field of violence risk assessment, research in recent years has yielded multiple instruments with the potential to structure the assessment process in ways that can enhance decision making about patients' risk of violence (14). The learners in this project were trained in the HCR-20, an instrument with empirical support as an aid to violence risk assessment. Although such tools may assist decision making about potentially violent patients, clinical competency in risk assessment and management includes skill in deciding what information needs to be obtained, facility in obtaining it, ability to formulate the clinical problem and develop a relevant treatment plan, and ability to document this process. To our knowledge, this is the first published description of an objective, structured approach to evaluation of competency in violence risk assessment and management.

Future research is needed to explore the generalizability of the CAI-V across multiple cases within the OSCE format. It would also be helpful to evaluate utility of the CAI-V for supervision of learners working with actual patients. We expect that the initial application of the method will be in formative evaluations that assist faculty in giving structured feedback to learners. The method has potential for informing summative evaluations of development of competency in risk assessment, if applied to learners' performance with multiple cases over time. Finally, future research, with larger samples, is needed to determine the generalizability of the results to other institutions and training programs.

In summary, the findings of this study suggest the promise of a new approach to evaluating skills in violence risk assessment and management and a format for providing structured and specific feedback to clinicians who are acquiring competency in this domain.

The authors thank Lowell Tong, M.D., and John Q. Young, M.D., M.P.P., for their help in coleading the workshop.

The authors report no competing interests.

Lurie  SJ;  Mooney  CJ;  Lyness  JM:  Measurement of the general competencies of the Accreditation Council for Graduate Medical Education: a systematic review.  Academic Medicine 84:301–309, 2009
 
 Report of the APA Task Force on the Assessment of Competence in Professional Psychology .  Washington, DC,  American Psychological Association, 2006
 
Scheiber  SC;  Kramer  TA;  Adamowski  SE (eds):  Core Competencies for Psychiatric Practice: What Clinicians Need to Know: A Report of the American Board of Psychiatry and Neurology, Inc.   Washington, DC,  American Psychiatric Publishing, 2003
 
 Specific Licensure Requirements by State/Province/Territory (Handbook) .  Peachtree, Ga,  Association of State and Provincial Psychology Boards, 2010. Available at asppb.net.  Accessed Jan 26, 2010
 
 ACGME Psychiatry Residency Requirements .  Chicago,  Accreditation Council for Graduate Medical Education. Available at www.acgme.org/acWebsite/downloads/RRC_progReq/400pr1104.pdf
 
Moran  M:  ABPN announces timetable for phasing out oral exam.  Psychiatric News ,  Feb 6, 2009, p 4
 
Newble  D:  Techniques for measuring clinical competence: objective structured clinical examinations.  Medical Education 38:199–203, 2004
 
Appelbaum  PD;  Gutheil  TG:  Clinical Handbook of Psychiatry and the Law , 4th ed.  Philadelphia, Lippincott,  Williams and Wilkins, 2006
 
Monahan  J:  Limiting therapist exposure to Tarasoff liability: guidelines for risk containment.  American Psychologist 48:242–250, 1993
 
Simon  RI:  Clinically-based risk management of potentially violent patients; in Textbook of Violence Assessment and Management . Edited by Simon  RI;  Tardiff  K.  Washington, DC,  American Psychiatric Publishing, 2008
 
McNiel  DE;  Chamberlain  JR;  Weaver  CM  et al:  Impact of clinical training on violence risk assessment.  American Journal of Psychiatry 165:199–200, 2008
 
Andrews  LB;  Burruss  JW:  Core Competencies for Psychiatric Education: Defining, Teaching, and Assessing Resident Competence ,  Washington, DC,  American Psychiatric Publishing, 2004
 
Webster  CD;  Douglas  KS;  Eaves  D  et al:  HCR-20: Assessing Risk for Violence (Version 2) .  Burnaby, British Columbia, Canada,  Simon Fraser University, Mental Health, Law, and Policy Institute, 1997
 
Otto  RK;  Douglas  KS (eds):  Handbook of Violence Risk Assessment .  New York,  Routledge, 2009
 
References Container
+

References

Lurie  SJ;  Mooney  CJ;  Lyness  JM:  Measurement of the general competencies of the Accreditation Council for Graduate Medical Education: a systematic review.  Academic Medicine 84:301–309, 2009
 
 Report of the APA Task Force on the Assessment of Competence in Professional Psychology .  Washington, DC,  American Psychological Association, 2006
 
Scheiber  SC;  Kramer  TA;  Adamowski  SE (eds):  Core Competencies for Psychiatric Practice: What Clinicians Need to Know: A Report of the American Board of Psychiatry and Neurology, Inc.   Washington, DC,  American Psychiatric Publishing, 2003
 
 Specific Licensure Requirements by State/Province/Territory (Handbook) .  Peachtree, Ga,  Association of State and Provincial Psychology Boards, 2010. Available at asppb.net.  Accessed Jan 26, 2010
 
 ACGME Psychiatry Residency Requirements .  Chicago,  Accreditation Council for Graduate Medical Education. Available at www.acgme.org/acWebsite/downloads/RRC_progReq/400pr1104.pdf
 
Moran  M:  ABPN announces timetable for phasing out oral exam.  Psychiatric News ,  Feb 6, 2009, p 4
 
Newble  D:  Techniques for measuring clinical competence: objective structured clinical examinations.  Medical Education 38:199–203, 2004
 
Appelbaum  PD;  Gutheil  TG:  Clinical Handbook of Psychiatry and the Law , 4th ed.  Philadelphia, Lippincott,  Williams and Wilkins, 2006
 
Monahan  J:  Limiting therapist exposure to Tarasoff liability: guidelines for risk containment.  American Psychologist 48:242–250, 1993
 
Simon  RI:  Clinically-based risk management of potentially violent patients; in Textbook of Violence Assessment and Management . Edited by Simon  RI;  Tardiff  K.  Washington, DC,  American Psychiatric Publishing, 2008
 
McNiel  DE;  Chamberlain  JR;  Weaver  CM  et al:  Impact of clinical training on violence risk assessment.  American Journal of Psychiatry 165:199–200, 2008
 
Andrews  LB;  Burruss  JW:  Core Competencies for Psychiatric Education: Defining, Teaching, and Assessing Resident Competence ,  Washington, DC,  American Psychiatric Publishing, 2004
 
Webster  CD;  Douglas  KS;  Eaves  D  et al:  HCR-20: Assessing Risk for Violence (Version 2) .  Burnaby, British Columbia, Canada,  Simon Fraser University, Mental Health, Law, and Policy Institute, 1997
 
Otto  RK;  Douglas  KS (eds):  Handbook of Violence Risk Assessment .  New York,  Routledge, 2009
 
References Container
+
+

CME Activity

There is currently no quiz available for this resource. Please click here to go to the CME page to find another.
Submit a Comments
Please read the other comments before you post yours. Contributors must reveal any conflict of interest.
Comments are moderated and will appear on the site at the discertion of APA editorial staff.

* = Required Field
(if multiple authors, separate names by comma)
Example: John Doe



Web of Science® Times Cited: 6

Related Content
Articles
Books
The American Psychiatric Publishing Textbook of Psychiatry, 5th Edition > Chapter 41.  >
The American Psychiatric Publishing Textbook of Psychiatry, 5th Edition > Chapter 41.  >
The American Psychiatric Publishing Textbook of Psychiatry, 5th Edition > Chapter 44.  >
Gabbard's Treatments of Psychiatric Disorders, 4th Edition > Chapter 5.  >
The American Psychiatric Publishing Textbook of Substance Abuse Treatment, 4th Edition > Chapter 24.  >
Topic Collections
Psychiatric News
PubMed Articles