Revalidating the Brief Jail Mental Health Screen to Increase Accuracy for Women

The most recent data from the Federal Bureau of Investigation indicate that approximately 14 million people were arrested in 2005 ( 1 ). The best clinical estimates suggest that about 1.1 million of these detainees met criteria for having mental illness in the year before incarceration ( 2 , 3 ). The importance of gender considerations in responding to the needs of these 1.1 million individuals is illustrated by data showing that rates of current serious mental illness are about two times higher for female detainees than for male detainees (12% versus 6.4%), largely because of higher rates of depression among female detainees ( 4 ).

To respond to the vast needs of detainees with serious mental illness, the American Psychiatric Association's Guidelines for Psychiatric Services in Jails and Prisons ( 5 ) recommends that all detainees should be screened immediately upon booking and that the initial screening should be done by correction officers. This recommendation is extremely challenging to implement because of the high volume of jail bookings and limited medical and mental health staff members. Practically, this means that there must be some type of standard screening tool that can be administered quickly during the usual booking process for which minimal training is required. Although the screening tool must be sensitive enough to detect those with a serious mental illness, the rate of detainees identified for referral for subsequent, fuller mental health assessment must not be so high as to overwhelm a resource-poor organization.

The need for such a tool has been recognized for some time, but a workable, valid instrument has been elusive. Relying on data from the Diagnostic Interview Schedule from 728 jail detainees with and without mental illness, Teplin and Swartz ( 6 ) created the Referral Decision Scale (RDS).

A number of follow-up studies of the RDS found it problematic on a number of psychometric issues ( 7 , 8 ), on face validity issues that related to the items' lack of fit with jail circumstances ( 9 ), and on excessive numbers of false positives ( 9 ).

In this context, two parallel projects were funded by the National Institute of Justice that took similar approaches to creating such a tool ( 10 ). A group at the University of Connecticut developed the Correctional Mental Health Screen for Men and the Correctional Mental Health Screen for Women ( 10 ). The team in Maryland and New York created the Brief Jail Mental Health Screen (BJMHS).

This brief report focuses on attempts to further refine the BJMHS. As originally reported ( 11 ), the accuracy rate for the BJMHS was 74% for men and 62% for women (N=357). However, compared with men, women had a substantially higher false-negative rate (35% versus 15%)—that is, detainees scored by the BJMHS as not having a current mental illness although they actually had one.

Given the importance of developing a more gender-sensitive instrument, we set out to refine the BJMHS by adding questions for depression and posttraumatic stress disorder (PTSD). We hypothesized that adding these questions would reduce the false-negative rates of the screen for women while retaining the high overall accuracy rates of the original study, maintain a low false-positive rate for both men and women, and maintain a rate of referrals that jails could actually be expected to handle (11% in the prior study).

Methods

The BJMHS was revised by reanalyzing the data from the previous study and conducting an extensive review of existing mental health screens. In the previous study ( 11 ) it was found that the modal Structured Clinical Interview for DSM-IV (SCID) diagnosis for missed cases (both male and female) was major depression. The existing SCID interview data were analyzed using the all-subsets-regression method to find the best predictive model for a diagnosis of major depression. Three symptoms of depression were identified that did not have a corresponding question in the original BJMHS or in its parent screen, the Referral Decision Scale ( 6 ). With slightly modified format consistent with prior questions, these were added to the BJMHS (the first two were adapted from the RAND Depression Screener and one was adapted from the Mini-International Neuropsychiatric Interview).

It was felt that part of the reason that the original BJMHS performed relatively poorly with females was that the BJMHS did not measure the symptoms of anxiety associated with the high incidence of PTSD experienced by female detainees. Thus a fourth question, relating to PTSD, was added to the screen. These four new questions were placed immediately before the original eight questions. The revised instrument is referred to herein as the BJMHS-R. As with the previous study, the SCID was used as the gold standard for clinical validation of the BJMHS-R.

Participants included jail detainees admitted to one of four county jails—two in Maryland and two in New York. Data were collected between November 2005 and June 2006. At three of the jails (Rennselaer County, New York; Montgomery County, Maryland; and Prince George's County, Maryland) all the participants were screened upon admission to the jail. At the fourth jail (Monroe County, Rochester, New York), because of the heavy volume of detainees, participants were screened after their initial court hearing, which was generally within 24 hours of their initial booking.

The screening data were used to identify an equal size subsample of detainees who scored positive and negative for referral on the BJMHS-R for the SCID validation.

The BJMHS-R is organized into two sections. The first section includes the ten symptom items that ask whether someone is currently experiencing the occurrence of mental health symptoms. The original six questions can be found in the prior report ( 11 ). The new questions asked whether one is feeling sad, empty, or depressed nearly every day; whether one has lost interest in things like work, hobbies, and other things usually enjoyed; whether one is currently having trouble sleeping nearly every night; or whether one has ever had any experience that was so frightening, horrible, or upsetting that it led to recent nightmares or being easily startled.

The second section of the BJMHS-R includes two items addressing whether a detainee was ever hospitalized for emotional or mental health problems and whether he or she is currently taking psychotropic medication.

For the purpose of the validation study, serious mental illness was defined as the presence of one or more of the following SCID diagnoses: major depressive disorder, depressive disorder not otherwise specified, bipolar disorder (I and II and not otherwise specified), schizophrenia disorder, schizoaffective disorder, schizophreniform disorder, brief psychotic disorder, delusional disorder, and psychotic disorder not otherwise specified.

Correctional classification officers in all four jails received training on administration of the BJMHS-R. This training, which took place in the jails, included a brief video that provided a description of the research project and instructions on completing the BJMHS-R during the booking process.

Sixteen clinical research interviewers participated in a two-day training session on administration of the SCID by a clinically trained SCID instructor. In addition, all of the clinical research interviewers conducted practice interviews with acquaintances and with psychiatric patients who volunteered. In order to ensure interrater reliability, all the interviewers rated two SCID reliability tapes and were observed conducting interviews at the jail by the SCID trainer and other project staff.

As soon as detainees were classified into the referral and nonreferral groups, the clinical research interviewers, who were blind to the detainees' sampling group statuses, approached the detainees on their list of potential participants and completed SCID interviews with those who consented to participate in the study. All interviews occurred within 72 hours of a detainee's admission to the jail.

Participation in this study was voluntary. Informed consent forms approved by the institutional review board were required and obtained for all SCID interview participants. Participants were informed that their decision to participate would not affect their stay in the jail and were administered a brief quiz to assess their competency to consent. The overall refusal rate was 31%. This refusal rate, while high, is identical to the 31% in our prior research ( 11 ) and is most likely due to the lack of compensation and scheduling difficulty within the jail constraints.

Results

Participants included 10,562 male and female jail detainees admitted to one of four county jails from November 2005 to June 2006. The participants were predominately male (8,864 detainees, or 84%), and the mean±SD age was 32±11 years. Data were available for race and ethnicity for 9,495 detainees. Over half were African American (5,857 detainees, or 62%), 1,965 (21%) were white, 1,321 (14%) were Hispanic, and 352 (4%) were of another race or ethnicity.

Detainees were considered to be referred for further evaluation if they endorsed at least two of the ten symptoms from section 1 of the BJMHS-R or either of the two medical history questions from section 2 of the BJMHS-R. Of the 10,255 detainees with valid BJMHS-R screening data, 2,235 (22%) were classified as needing a referral for further mental health assessment. Female detainees (655 of 1,614 women, or 41%) were more than twice as likely as men (1,580 of 8,607 men, or 18%) to be classified as needing a referral.

There was a clear gender difference among correction officers doing the screens. Detainees who were screened by a female officer were twice as likely to be referred as those who were screened by a male officer (1,098 of 3,501 detainees screened by a female officer, or 31%, compared with 960 of 6,071 detainees, or 16%). The pattern was similar for both male and female detainees. Of the male detainees screened by male officers, 15% (816 of 5,589 detainees) were referred using the screen, compared with 26% (636 of 2,445 detainees) of the males screened by a female officer. For female detainees, the proportions were 31% (142 of 460 detainees) and 44% (461 of 1,046 detainees), respectively. At the Monroe County jail, nearly all the female detainees were screened by female officers. If this jail is excluded from the analysis, the discrepancy for females is considerably less (136 of 446 detainees screened by a male officer, or 30%, compared with 254 of 698 detainees screened by a female officer, or 36%), but the general pattern holds.

The screening data were used to identify a subsample of 945 detainees who were selected for a detailed clinical assessment. Approximately 26% of those selected (N=248) were released before the interview or were unavailable for a variety of reasons, such as illnesses. An additional 219 detainees (23%) refused to participate in the study, and 14 (1%) had incomplete interviews. Participants in the final SCID subsample were 464 detainees that included 175 referrals (68 males and 107 females) and 289 nonreferrals (138 males and 151 females).

Table 1 presents the clinical cross-validation results of the 12-item BJMHS-R and the eight-item BJMHS for the subsample of 464 detainees who received the SCID. For comparison purposes, the table also shows the cross-validation results of the eight-item BJMHS from the original study (N=357) ( 11 ). Clearly, the four added questions in the BJMHS-R did not improve the performance of the original instrument. Also, as measured by difference of means test for the eight-item BJMHS, the study presented here showed a significant improvement in the percentage of females correctly classified, compared with the original study (z=–2.13, p=.033). For sensitivity, specificity, false-negative rates, and false-positive rates there is no gold standard for this type of instrument. In practice, it is a policy call as to what are acceptable rates given real-world practices, procedures, and potential liability. Nonetheless, the key point here is that the original eight-item BJMHS seems preferable in balancing the competing interests of accurately identifying people in need of services while controlling system burden from the number of referrals for more intensive mental illness assessment.

Enlarge table

Discussion

Despite making a number of what seemed to be clinically informed changes to the BJMHS, when new data were collected at three of the four original study sites plus a fourth new site, the original BJMHS turned out to be more powerful. These results reinforce the value of this screen for men and suggest that the earlier reservations about its use for women were overstated.

It's unclear why the 35% false-negative rate in the original study was reduced to 14% in the study presented here for the eight-item BJMHS. It would seem that, in effect, the confidence interval for false negatives for womenfor the eight-item BJMHS is between 14% and 37%, depending on the particular jails in which it is being used. It is quite possible that the results for this screening tool vary not only by sample, but they also may be influenced by other factors, such as the base rate of mental illness in the jail.

In the study presented here, female booking officers were slightly more likely than male booking officers to elicit responses that resulted in referral when using the 12-item BJMHS-R, compared with the eight-item BJMHS; however, these gender differences were statistically significant for both versions of the BJMHS. It is important to note that the overall accuracy results from the eight-item BJMHS do not significantly differ from female booking staff to male booking staff (76% versus 78%, respectively). This is explained by the fact that although female officers tended to refer more individuals and had a lower false-negative rate, they also had a higher false-positive rate, resulting in an overall equivalency in accuracy. We do not believe that these observations mean that the BJMHS should be done only by female booking staff because it is simply impractical in most jails and the overall accuracy rate is almost identical with female and male correction officers.

Conclusions

At this time, the BJMHS, at very low expense, provides very accurate identification of detainees with current symptoms of mental illness. It produces a reasonable proportion of those screened (between 11% and 16%) who should be referred for more intensive assessment by medical staff. Within the constraints of time, staff, and money in U.S. jails, this instrument can be administered by regular corrections staff in two to three minutes and offers the strongest empirical basis for regular use as a practical tool of any instruments currently available. With these new data, there is even stronger support for the conclusion of the recent National Institute of Justice report on the two gender-specific Correctional Mental Health Screens and the BJMHS, "Based on their successful validation results, it is anticipated that these tools will be disseminated nationwide for use in all correctional facilities" ( 10 ). To this we would add that such instruments should be used for universal screening in all U.S. jails in keeping with the American Psychiatric Association's Guidelines for Psychiatric Services in Jails and Prisons ( 5 ).

Acknowledgments and disclosures

This project was supported by grant 2005-IJ-CX-0004 from the U.S. Department of Justice, National Institute of Justice, Office of Justice Programs. The authors are grateful to their recently deceased colleague Steven M. Banks, Ph.D., for his statistical advice and interpretations on this and so many other projects. They are indebted to Tara Agnese, M.A., who coordinated the two Maryland sites. They are also grateful to the jails and their administrators who participated in the research: Col. Robert R. Loveridge at Rensselaer County (New York), Maj. John Caceci at Monroe County (New York), Arthur Wallenstein at Montgomery County (Maryland), and Barry Stanton at Prince George's County (Maryland). The authors also acknowledge the contributions of the New York data collectors: Elizabeth Stewart, M.S., Robert Hawkes, M.A., Corrine Kindzierski, Ph.D., and Christine Sergent, M.S. (Monroe County); Janet Poole, B.S., Lucas Jacobs, M.A., Steven Hallock, B.A., and Beth Kellerman, Ph.D. (Rensselaer County); and Jim Blucher, M.A., Michael Sklar, M.A., Alana Henninger, M.A., Jenny Daffin, M.A., and Tiffanie Sim, B.S. (Maryland). The project would not have been successful without the commitment to field interviewing exhibited by these interviewers and the SCID training and observation provided by Jennifer Norbeck, Ph.D. Points of view in this article are those of the authors and do not necessarily represent the official position or policies of the U.S. Department of Justice.

The authors report no competing interests.