Validation of Brief Screening Tools for Mental Disorders Among New Zealand Prisoners

New Zealand, like other countries in which relevant surveys have been conducted, has a high prevalence of serious mental illness in remanded and sentenced prisoner populations ( 1 , 2 ) but has no validated screening method for identifying prisoners with mental illness with the greatest need for treatment.

Two brief, empirically validated screening tools were identified for testing: the Brief Jail Mental Health Screen (BJMHS), developed by Steadman and colleagues ( 3 ), in the United States and the English Mental Health Screen (EMHS), a brief, four-item screen developed by Grubin (Grubin A, unpublished manuscript, 2003) in the United Kingdom that is based largely on historical variables and is embedded within a general health screen given to prisoners upon admission.

The acceptability and accuracy of these tools needed to be determined within New Zealand, because there is evidence that New Zealand Maori, a group with comparatively less frequent service use, might be disadvantaged by the use of a screening tool that is predominantly based on historical engagement of services ( 4 ).

The aim of this study was to validate two brief screening tools to detect high-risk cases of serious mental illness among New Zealand male prisoners at the point of admission. These two screening tools could be delivered along with a screen for imminent suicide risk.

Methods

Design

A prospective survey was conducted of consecutively admitted male remanded and sentenced prisoners. All participants answered the BJMHS and EMHS questions upon admission. The performance of the screening tools was validated with a standard psychiatric diagnostic rating scale, the Mini International Neuropsychiatric Interview (MINI). The study was approved by the Multi-Region Ethics Committee.

Participants

All male prisoners admitted to Christchurch Men's Prison (August 2006 to December 2006) and sentenced to Auckland Central Remand Prison and Mount Eden Prison (July 2007 to August 2007) were given the screens.

Measures

The BJMHS ( 3 ) and the EMHS (Grubin A, unpublished manuscript, 2003) were used individually and in composite as screening tools to allow for comparison with each other and against previous reported performance.

BJMHS. The BJMHS comprises questions organized into two sections. The first section includes six questions about thought interference by others, weight loss or gain, psychomotor retardation, increased energy or activity, and inappropriate guilt. The second section includes two items that ask whether the inmate was ever hospitalized for emotional or mental health problems and whether the inmate is taking psychotropic medication. The BJMHS is rated as a positive screen if any two of the six symptom questions are endorsed or if either one of the mental health history questions are endorsed. The BJMHS has been validated against the Structured Clinical Interview for DSM-IV-Lifetime Version (SCID-L) and deemed suitable for use in male jail settings ( 3 , 5 ).

EMHS. The EMHS involves four questions: whether the inmate had been charged with homicide, ever visited a psychiatrist outside prison, received medication (antidepressant or antipsychotic) for any mental health problems, or tried to harm himself or herself ( 6 ; Grubin A, unpublished manuscript, 2003). A screen was considered to be positive if there was a positive response to any of the questions. It has been validated against the Schedule for Affective Disorders and Schizophrenia-Lifetime Version (SADS-L) ( 7 ).

The EMHS question "Have you ever seen a psychiatrist outside prison?" was adapted to read, "Have you ever seen a psychiatrist outside prison for emotional or mental health problems?" The BJMHS item asking about weight loss or gain of "two pounds" was adjusted to the metric system to read, "one kilogram" (about 2.2 pounds).

MINI. The MINI was used to validate the performance of the screening tools in detecting five specific DSM-IV axis I diagnoses previously regarded as indicative of severe mental illness (major depressive disorder, depression with melancholia, hypomania or mania, current psychosis, and psychotic mood disorder) ( 3 , Grubin A, unpublished manuscript, 2003). The MINI is regarded as a gold-standard short diagnostic structured interview, with strong performance in terms of its validity versus the Structured Clinical Interview for DSM-III-R-Patient Version (SCID-P) ( 8 ).

Training

The Department of Corrections nursing staff (comprehensively trained) at the Christchurch and Auckland prison sites received formal training on the administration of the composite pilot screen, information on the background to the development of the screen, and practical training on the use of the composite screen. Staff who used the MINI screen received standardized formal training on how to rate prisoners for specific psychiatric diagnoses.

Procedure

Male inmates who were consecutively admitted were given the composite screening interview by trained prison nursing staff. The duration of the composite screen (approximately five minutes) was comparable to nonvalidated screening procedures used elsewhere in New Zealand. Each prisoner with a positive screen was referred for specialist mental health assessment and was also referred for a MINI screen, as was a randomly generated 20% sample of persons who screened negative. Participation in the MINI screen was voluntary, and no incentive was offered. Written informed consent was obtained from participants (92% of those approached).

Statistical analysis

The performance of each screening tool was compared with the MINI to assess for overall accuracy (the proportion of cases correctly identified as either positive or negative for mental illness), sensitivity, specificity, and false-positive and false-negative rates, and the overall referral rate was examined for those with a positive screen.

Results

A total of 1,292 screens (761 in Christchurch and 531 in Auckland) were completed. A total of 530 MINI screens were completed (312 in Christchurch and 218 in Auckland). The dominant ethnic groups were New Zealand European (N=546, 42%) and Maori (N=440, 34%).

Table 1 shows that of the sample of 530, 51% of participants met MINI criteria for one of the five targeted major mental disorders, including major depressive disorder (40%), depression with melancholia (36%), and hypomania or mania (22%). Compared with New Zealand European prisoners, Maori prisoners had higher rates of all five diagnoses. Prisoners who were remanded had higher rates than convicted prisoners for all five diagnoses, particularly for psychosis.

Enlarge table

Performance of EMHS and BJMHS

Table 2 shows high levels of endorsement for EMHS questions pertaining to psychiatric history, although rates were lower for Maori prisoners. Similarly, the BJMHS revealed significant rates of past psychiatric hospitalization (10%) and current treatment with psychotropic medication (9%), with greatest endorsement for the "feeling useless or sinful" item (13%) and the weight loss item (12%). Maori prisoners had higher rates of endorsement of the thought insertion and mind reading items, compared with New Zealand European prisoners.

Enlarge table

Of the 212 prisoners who met MINI criteria for major depressive disorder, 79 (37% of the group of 212 prisoners, or 6% of the whole sample) endorsed suicidal ideation. Of these 79 prisoners, 63 (80%) screened positive on the EMHS and 54 (68%) screened positive on the BJMHS.

As shown in Table 3 , compared with the U.S. study, our study showed that in New Zealand the BJMHS performed against the standard with lower sensitivity, higher specificity, a lower false-positive rate, and a significantly higher false-negative rate. The BJMHS generated a much higher overall referral rate for male prisoners in New Zealand than for male prisoners in jail in the United States (23% versus 10%). Compared with the U.K. study, our study showed that the EMHS performed against the standard with markedly lower sensitivity, less specificity, much higher false-positive rates (40% versus 10%), and higher false-negative rates and generated a moderately higher overall referral rate (33% versus 26%). The highest overall accuracy was obtained by the BJMHS among U.S. jail detainees.

Enlarge table

A composite screen in which participants were required to screen positive on either the BJMHS or the EMHS performed with low to moderate sensitivity, good specificity, a high false-positive rate and a high false-negative rate, and generated a high referral rate (38%). A composite tool that required participants to screen positive on both the BJMHS and the EMHS performed with very low sensitivity, very good specificity, high false-positive and false-negative rates, and generated a relatively low overall referral rate (19%). The various screens performed similarly for the remanded and convicted populations.

Analysis of false negatives

For both the BJMHS and EMHS, the majority of persons who were categorized as having a false negative according to the MINI criteria had a mood disorder; few had a diagnosis of psychotic disorder ( Table 4 ). The lowest false-negative rates for each diagnosis, particularly for the psychotic diagnosis, were achieved by the composite screen, which required a positive screen on either the BJMHS or the EMHS.

Enlarge table

Discussion

Overall rates of mental disorder

Using the MINI as the validating standard, we identified very high rates of serious mental illness in the New Zealand prison population, in line with previous international research ( 2 ) and New Zealand research ( 1 ).

However, the rates of mental disorder in New Zealand prisons were significantly higher than the corresponding rates identified in the validation studies for the EMHS and BJMHS, which raises questions about the reliability and validity of the MINI as a standard diagnostic instrument in this context. There are several reasons that support the use of the MINI as a validation instrument. First, interrater reliability testing in Christchurch did not identify any systematic rater bias. Second, the prevalence rate of mental disorders (N=90, 20%) identified using the SADS-L in the United Kingdom was about half the level expected by the authors ( 6 ), reducing the apparent discrepancy between their study and ours. Third, several clinical indicators, including high rates of previous hospitalization, consultation with a psychiatrist, or use of psychiatric medication, supported the presence of high psychiatric morbidity in our sample. Fourth, the MINI has been validated against other, more lengthy, gold standards, including the SCID ( 8 ) and the Composite International Diagnostic Interview (CIDI) ( 8 , 9 , 10 ). Finally, the MINI has an interview structure similar to the other gold standards, and there is no reason to suspect that the longer standard interviews would produce significantly different results for the New Zealand population.

It has also been argued that any epidemiological research that relies on structured diagnostic instruments runs the risk of finding arbitrarily high levels of mental illness because of the tendency to categorize general, common symptoms as a mental disorder. In the study presented here, very high rates of endorsement of depressive symptoms emerged, which may reflect an individual's transient response to adjusting to the stressful prison environment, with symptoms diminishing as adjustment occurs, rather than reflecting the true presence of a depressive disorder. It is possible that a clinical assessment would have greater ability to differentiate between an adjustment response with depressive symptoms and a true depressive disorder.

If the MINI is overinclusive in the manner described, our data are likely to underestimate the performance of the screening tools. Artificially high rates of mental disorders would lead to lower sensitivity ratings, because the screens would need to detect potential mental illness among those for whom a diagnosis was dubious.

Performance of the screening tools

The basic performance data suggest that neither the BJMHS nor the EMHS was particularly effective as a diagnostic screening tool for mental disorders in New Zealand prisons. However, secondary analysis of false-negative cases suggested that the screens were effective in identifying individuals with psychosis, although the screens were not as specific in detecting mood disturbance. It is arguable that depressive symptoms are such a common finding in the prison setting that it is not desirable for the screening tool to screen individuals as being positive on the basis of these symptoms in the absence of, for example, psychosis or suicidality.

The implications of a false-negative and a false-positive finding should be interpreted with the prison context in mind. If a prisoner with low-grade but persistent depressive symptoms screens negative, the implication is that the prisoner would continue to experience these symptoms without further mental health intervention, although the prisoner can still be identified after admission—for example, by officers on the prison wing who can observe the prisoner's longitudinal presentation. A false positive means that the prisoner with either current symptoms or a relevant psychiatric history will undergo further brief questioning by a mental health professional. Although service-level resource implications must be considered, the outcome for the individual is not harmful and could be beneficial.

Both the BJMHS and the EMHS had comparatively low levels of sensitivity in the New Zealand study. The pilot study of the EMHS in the United Kingdom appears to be the outlier in terms of an unusually good performance by a screening tool—that is, a zero rate of false negatives in the male prison population and extremely high sensitivity rates for depression, a disorder with a high prevalence, arising from a small number of general, historically based, and widely endorsed screening questions. Although we must not minimize the importance of detecting mood disturbance in the prison setting, it is somewhat reassuring that the lower levels of sensitivity of the screens in the New Zealand setting were predominantly due to the detection by the MINI of very high rates of mood disturbance; few cases involving psychosis went undetected by either screen.

The pattern and rate of mental disorders in the false-negative cases were remarkably similar for both the BJMHS and EMHS. Also, no single historical question in either screen appeared to be especially powerful in identifying prisoners with mental illness, and all questions appeared to contribute to the overall performance of the EMHS in detecting cases of psychosis.

Because of the prevalence of depressive symptoms in the prison setting and the relatively benign meaning of a false-positive screen at the individual level, it is arguable that the clinical priority for the study presented here should be the minimization of false-negative cases of psychosis. The specific diagnoses may be less relevant in this context than the detection of an at-risk population, specifically, those who are currently psychotic and those who are actively suicidal (which would be addressed by a supplementary screen). These cases require at least semi-urgent psychiatric assessment, and sometimes acute care is required.

In this sense, both the EMHS and the BJMHS perform adequately: although they have low sensitivity in terms of diagnoses, both the EMHS and the BJMHS were good at detecting psychosis. Although it would be simpler to use either the EMHS or the BJMHS, neither performed as well on its own as they did in combination, and the time that it takes to complete both screens is in line with current, nonvalidated approaches to screening in New Zealand. Overall, the most effective screening approach from a clinical perspective, particularly in terms of minimization of false-negative rates, was achieved by a screen where referral for mental health nursing assessment was triggered by a positive screen on either tool.

A two-tiered strategy for screening

Adopting a combination of both screens leads to the challenge of how to manage the high false-positive rate in terms of the demand placed on forensic mental health services. However, it should be noted that the developers of the BJMHS had a higher false-positive rate than that found in the study presented here, yet they felt able to endorse the screen for the male detainees in jail ( 3 ). It is unlikely that a brief screening instrument will be able to be developed that can be administered easily by prison nursing staff and that does not produce significant levels of false-positive cases, making it unrealistic to pursue an ideal screening tool with a much lower referral rate.

The data presented here suggest that an effective strategy might be to introduce a second stage to the screening process, entailing a brief triage interview with a mental health professional for those who screened positive, in order to refer prisoners with symptoms of psychosis, mania, severe depression, and suicidality for specialist psychiatric assessment, as opposed to those with uncomplicated depressive symptoms, who could be reviewed by primary health care services. This approach would address the national cultural context by providing an opportunity for a focus on potential cultural considerations—for example, to identify cultural beliefs that may mimic psychosis. This two-stage screening strategy would resemble the three-stage model proposed in the American Psychiatric Association's guidelines for psychiatric services for jails and prisons ( 11 ).

Impact on regional forensic psychiatry services

In Christchurch, 54 (8%) of the 656 prisoners who were screened and who underwent triage assessments by a psychiatric nurse and subsequent consultant psychiatrist assessment were maintained on the caseload. In Auckland, referrals to the prison team increased from an average of eight per week to a maximum of 43 per week during the screening study, and the caseload of the prison team rose from 104 to a maximum of 141.

Limitations

An important limitation of the study presented here was the restriction of the sample to male prisoners. The performance of the BJMHS with female prisoners cannot be assumed to be the same as it is for male prisoners, although there is recent evidence that additional questions targeting psychiatric morbidity in the female population do not necessarily improve the performance of the screen ( 12 ). As indicated above, the use of a structured diagnostic instrument as the gold standard as opposed to interviews with an experienced clinician leaves open the question of the validity of such instruments. Our study may have had a slightly higher false positive rate than the U.K. study, because the U.K. study rated a screen as being positive only if the individual had received treatment from a psychiatrist. We categorized a screen as positive if an individual had used psychiatric medication or had seen a psychiatrist (although it should be noted that rates of psychiatric medication use and seeing a psychiatrist were similar in our study). Similarly, our study also did not differentiate between a history of self-harm outside as opposed to inside of prison, which might have led to further false-positive cases, but which still would have identified a relevant clinical history.

Conclusions

In conclusion, although the BJMHS and EMHS did not perform well in terms of screening for MINI diagnoses, they appeared to be good at identifying a core group of prisoners at risk because of their mental disorders and who may require urgent or semi-urgent intervention by mental health services—that is, prisoners who are psychotic. A pragmatic way forward would be to accept the higher referral and false-positive rates and to focus on further development of the existing second-tier screening interview process by mental health professionals.

Acknowledgments and disclosures

Funding was provided by the Department of Corrections and the Ministry of Health. The authors thank the representatives from the Regional Forensic Psychiatry Services and those involved in project management and data collection. The opinions expressed in this article are those of the authors and do not necessarily represent the official positions or policies of the New Zealand Department of Corrections or Ministry of Health.

The authors report no competing interests.