Computer-Administered Versus Paper-and-Pencil Mental Health Surveys

To the Editor: Computers are increasingly used to assess mental health and substance abuse treatment outcomes (1). Computerized assessment has the potential to reduce burden on clinical staff, increase the efficiency of data collection and management, reduce the rate of missing data, increase interrater reliability of clinician-administered assessments, eliminate inapplicable items by using branching logic, provide immediate feedback of results, and help identify individuals at risk of poor outcomes in a timely manner that could allow for clinical intervention (2,3). However, it is important to determine whether responses to computerized assessments are comparable to those obtained by paper-and-pencil methods (4).

We conducted a small study to compare feasibility, respondent acceptance, reliability, and mean scores for computerized assessment versus a paper-and-pencil survey. We used a repeated-measures design in which psychiatric outpatients completed two self-report mental health questionnaires by using paper and pencil and by computer. The two survey instruments used were the Polaris Strength Scales, a new 29-item scale focused on goal setting, resiliency, development of emotional and relational skills, and social supports, and the revised Behavior and Symptom Identification Scale (BASIS-24), a 24-item scale that assesses depression and functioning, interpersonal relationships, psychotic symptoms, substance abuse, self-harm, and emotional lability (5). Both instruments are designed for use among persons with serious and persistent mental illness.

The study was conducted over a two-month period in 2002. The sample consisted of 52 English-speaking adults, evenly divided by gender. The age of the participants ranged from 19 to 64 years. A total of 41 participants (84 percent) were Caucasian, 33 (67 percent) were unemployed, and 28 (57 percent) had been in treatment for more than ten years. Two respondents refused to complete the computerized survey. All those who agreed to complete it were able to use the computer after receiving brief (less than five minutes) instructions. More respondents needed help completing the survey by computer than by paper and pencil (eight respondents, or 17 percent, compared with one respondent, or 2 percent; χ²=4.28, df=1, p<.05), although respondents preferred the computer by a ratio of 3:1. No significant differences were noted in respondents' evaluation of the time to complete the instruments, understanding of items, clarity of instructions, or any other concerns.

The reliability of ratings made by computer versus by paper and pencil was high, with intraclass correlation coefficients ranging from .83 to .9 for the Polaris Strength Scales and from .82 to .95 for the BASIS-24 domains. Differences in mean scores on the domains assessed were not statistically significant, with one exception: participants reported significantly more frequent psychotic symptoms by paper and pencil than by computer (F=7.19, df=1,44, p=.01).

Limitations of this study include its small sample and possible memory or order effects (that is, whether respondents used the computer first or paper and pencil first), which were not assessed. Self-report of mental health status by computer may enhance quality improvement efforts by improving the capacity to monitor treatment outcomes with greater efficiency than can be achieved with paper-and-pencil methods. Further research is needed to test computerized administration of standardized instruments in larger samples from multiple sites.

Dr. Eisen is affiliated with the health services department of the Boston University School of Public Health and the Center for Health Quality, Outcomes, and Economics Research at the E. N. Rogers Memorial Hospital in Bedford, Massachusetts. Dr. Toche-Manley and Dr. Grissom are with Polaris Health Directions in Fairless Hills, Pennsylvania.