A Flawed Study Design?

To the Editor: In the article entitled "Adverse Effects of Poor Management of an Inpatient's Difficult Behaviors" in the July 1999 issue, Hilton and Simmons (1) report the results of an A-B-A-B study design that purports to analyze the consequences of a "management plan" on the behavior of a psychotic man. The authors conclude that "the case reported here illustrates that poor behavioral management can have adverse effects" and that "these interventions provided what might be a prototypical example of poor behavior management in institutions."

This man was held on a male ward in Ontario's maximum-security hospital. Patients on his ward are very aggressive, psychotic, developmentally challenged, and brain injured. They do not take assault on their genitals (the target behavior) kindly, and the man was the victim of physical retaliation on several occasions.

Apart from the pejorative language and the statement, unsupported by any evidence, that front-line staff had a malevolent motivation to punish the patient, there are serious flaws in this study.

The first A component of the study occurred in the 42 days between November 7 and December 18, 1996. During this period, the man was secluded for a total of 22 days. The authors calculate that the man committed 13 assaults, for an average of .31 assaults per day. This is a gross underestimate of the assault rate. Using the authors' criteria to examine the data, I counted 16 assaults, for an average rate of .38 per day. However, the man could not assault others while in seclusion, although he often acknowledged that he would do so if he had the opportunity. Given the days the man spent in seclusion, a more realistic estimate of the assault rate was the "opportunity" rate—16 assaults in 20 days, or an average of .80 assaults per day.

At this point, the nursing staff developed a management plan to end the use of seclusion and return the man to the community of the ward. In maximum-security environments, management plans, in contrast to treatment plans, are designed to ensure a safe, secure environment for the patient, peers, and staff. The distinction is not "arbitrary" or "made to circumvent administrative barriers that would have prevented treatment," as the authors state.

The first B component of the study occurred over an eight-week period between December 19, 1996, and February 13, 1997. The man was not secluded during this period. The authors took the first six weeks of the period and counted 39 assaults, including two occasions when the man told staff he had "the urge to grab." I excluded these two incidents because they were outside the criteria used in the first A period and computed an average assault rate of .88 per day. Thus the data do not demonstrate that the assaults were "significantly more frequent during the time periods when the plan was in effect."

The second A component of the study occurred over a six-week period between February 14 and March 27, 1997, when no seclusion was used. During this period, the man committed five assaults (not eight, as the authors report), for an average rate of .12 assaults per day.

From examination of the records for the entire period of the study, it is apparent that the man had three discrete psychotic episodes between November 7, 1996, and September 29, 1997. The second A component of the study occurred following remission of the first episode. The second shorter episode of 19 days' duration occurred 21 days before the third. In it, the man was severely psychotic and aggressively assaultive and required seclusion.

The second B component of the study occurred in the six weeks between August 19 and September 29, 1997, almost six months following the termination of the second A component and three weeks into the third psychotic episode. During this period, my review of the data shows that the man committed 31 assaults (not 42, as the authors report), for an average of .74 assaults per day. However, the long time interval between the second A and B components, and the many things that transpired during that interval, do not allow for the comparison that the authors seem determined to make.

In short, the only supports for their conclusions are a flawed research design, biased data collection, and inappropriate use of statistical analysis. The front-line nursing staff of this hospital, of whose work I am proud, do not deserve to be villified by such research.

Dr. Hector is clinical director of the Oak Ridge Division of the Mental Health Centre in Penetanguishene, Ontario.