Dr. Rollinson is affiliated with Psychological Therapies and Primary Care, Norfolk and Waveney Mental Health Partnership NHS Trust, Community Resource Centre, Northgate Hospital, Great Yarmouth NR30 1BU, United Kingdom (e-mail: rebecca.rollinson@nwmhp.nhs.uk). Dr. Haig and Professor Fowler are with the Faculty of Health, University of East Anglia, Norwich, United Kingdom. Professor Warner is with the Department of Anthropology, University of Colorado, Boulder. Professor Garety, Professor Kuipers, and Dr. Freeman are with the Institute of Psychiatry, Kings College, London. Professor Bebbington is with the Department of Mental Health Sciences, Royal Free and University College Medical School, London. Professor Dunn is with the Division of Epidemiology and Health Sciences, University of Manchester, Manchester, United Kingdom. Data from this study were presented at the British Association of Behavioural and Cognitive Psychotherapies annual conference, Canterbury, United Kingdom, July 20—23, 2005.

There is an increasing demand for cognitive-behavioral therapy (CBT) to be made available to people with treatment-resistant positive symptoms of psychosis. The increasing body of randomized controlled trials indicating evidence of effectiveness (1,2,3,4,5,6,7,8) has resulted in the inclusion of CBT in the United Kingdom and U.S. national guidelines for best practice in the treatment of schizophrenia (9,10). Nonetheless, recent reviews have emphasized both the scarcity of trained professionals able to deliver CBT for psychosis and the lack of available evidence regarding its effectiveness in routine practice (11).

Many contextual differences between research and routine clinical settings may influence the way an intervention is applied. These include the professional discipline of the mental health practitioner, waiting list pressures, case management responsibilities, primary outcome measures, size of caseload, crisis prevention, and the availability of training and supervision. Although efficacy studies in controlled research settings are vital for the development and evaluation of therapeutic interventions, the presence of these contextual differences means that they cannot be assumed to translate directly into routine practice (12). In order to answer the call for increased dissemination of this therapeutic approach, we need to understand more about the process of applying this therapy in routine settings.

When examining therapy practice, it is useful to clarify two key concepts, adherence and competence. Therapist adherence refers to "the extent to which a therapist used interventions and approaches prescribed by the treatment manual and avoided the use of interventions proscribed by the manual" (13). Therapist competence refers to, "level of skill [that is] the extent to which the therapists took the relevant aspects of the therapeutic context into account and responded to these contextual variables appropriately" (13).

Four trials to date have evaluated the effectiveness of CBT for psychosis in routine settings. One study used a single-case experimental design (N=18) and found positive responses for one-third of participants (14). Another compared the outcomes of clients receiving CBT for psychosis with those in a waiting list control group and reported positive results (15). However, expert therapists provided the therapy. Two studies of the efficacy of CBT for psychosis provided by nonexpert practitioners within a clinical setting have been reported (16,17). Both reported significant outcomes, but these were more modest than those obtained in research settings.

Of interest, the therapy provided in both of these studies was modified to some extent from that of expert research trials (1,3). One trial provided six sessions of CBT for psychosis that focused on psychoeducation and problem solving and included caregivers, when they were available (16). The other (17) was shown to be competent cognitive therapy on the Cognitive Therapy Scale (Young J, Beck AT, unpublished manuscript, 1980), but only four of 13 CBT tapes were rated as adherent on the Cognitive Therapy for Psychosis Adherence Scale (CTPAS) (18), a scale designed specifically for CBT for psychosis. These modifications may of course reflect the difficulties involved in transferring therapy techniques from the research setting to clinical practice.

In order to further investigate the process of disseminating CBT for psychosis into routine clinical practice, this study compared the content of therapy sessions provided in research and clinical settings. The therapy model used in both settings is described in the Fowler and colleagues therapy manual (19). The authors propose six nonlinear stages of therapy: engagement and assessment, techniques to promote self-regulation of psychotic symptoms, the development of a new model of psychosis with the patient, techniques to address delusions and beliefs about voices, strategies to address dysfunctional assumptions, and strategies to address social disability and risk of relapse. The manual emphasizes the need for collaborative, matched, and individualized therapeutic work. It encourages a focus on the development of a therapeutic relationship and emphasizes the value of developing a shared formulation in order to promote belief flexibility and to consider alternative explanations and coping responses.

The purpose of this study was to use the revised version of the CTPAS (R-CTPAS) (20) to compare the content of CBT for psychosis sessions provided by therapists in a specialist research setting with the content in routine clinical settings by using a between-group experimental design.

Design

A between-group experimental design was used to compare the content of CBT for psychosis sessions across two groups of therapists. The first group was providing CBT for psychosis as part of a research trial, the second as part of their clinical caseload. The groups were matched as closely as possible according to therapist characteristics (gender and age) and client characteristics (gender, age, and diagnosis).

Participants

Both groups consisted of 20 therapist-client dyads. After we gave a complete description of the study, all participants provided informed consent. The appropriate institutional review board provided ethical approval in each setting.

Research setting. The participants in the research setting were patients receiving CBT in the Wellcome Trust-funded Psychological Prevention of Relapse in Psychosis Trial who consented to the audiotaping of their therapy sessions.

The Prevention of Relapse in Psychosis Trial was a large multicenter trial conducted between October 2001 and June 2006 in the National Health Service in four regions of the United Kingdom: South London, North London, Essex, and Norfolk. The trial aimed to assess the efficacy of CBT and family work in preventing relapse in psychosis. Patients were eligible for recruitment into the trial at the time of a relapse in positive symptoms from either a previously recovered state or a state of persisting symptoms. The data used in this study were collected between October 2001 and May 2004.

Clinical setting. Three clinical settings contributed audiotapes of therapy sessions: the Mental Health Center of Boulder County, Colorado, and the North Essex Community Mental Health Team and the Norwich Early Intervention Team in the United Kingdom. Participants in all three locations were patients actively receiving CBT for psychosis from the team therapists.

Clients in both the research and clinical settings were considered eligible for inclusion if they were between ages 18 and 65 and presented with a history of, and current difficulties with, positive psychotic symptoms. They were excluded if the psychotic symptoms were secondary to a primary diagnosis of affective disorder. Clinical diagnosis was given as recorded in the participant's medical notes.

Data were collected between October 2004 and May 2005.

Therapy provision

Research setting. Therapy was provided primarily by five research clinical psychologists employed full-time on the trial (seeing 72% of patients) and by ten local therapists within the research centers who worked part-time on the trial. All trial therapists received initial training by experts, followed by ongoing intensive expert and peer supervision. CBT was offered for approximately 20 sessions over nine to 12 months.

Clinical setting. In all three clinical settings, therapy was provided as part of a case management service. All therapists had attended specialist training from the authors of the therapy manual (19). The therapists working with the Boulder and Norwich samples had established ongoing group or individual supervision. There were no formal restrictions on the length of therapy contact within the clinical settings. It was not possible to estimate the total number of eligible therapy providers within each service because this was determined in part by the suitability of each practitioner's caseload.

Sampling

Research setting. In order to monitor levels of therapy adherence within the Prevention of Relapse in Psychosis Trial, each therapist provided tapes for rating by a research clinical psychologist outside his or her own research center. To ensure representative sampling across the course of therapy, the 20 planned therapy sessions were stratified into four blocks of five sessions. Each therapist provided one tape from each block of five sessions for adherence rating. At the time of analysis (two years into recruitment) session ratings were available for 52 clients receiving CBT.

Clinical setting. Therapists recorded three therapy sessions per dyad. The researchers then randomly selected one tape per dyad for rating by a reliable index rater separate from the clinical service. All ratings were confidential and made available to only the therapist and rater.

Matching procedure. To make the groups more comparable, matches on demographic variables were attempted. The participant dyads within the combined clinical samples were stratified by therapist age, therapist gender, client age, client gender, and client diagnosis. The same stratification procedure was applied to the Prevention of Relapse in Psychosis sample. Dyads from the research group were then selected to match the routine sample as closely as possible according to the stratified demographic variables. An exact one-to-one match was not possible, however. Only one tape was selected per client. Where multiple tapes were available for a client in the research setting, one was selected at random.

Measures

Startup and colleagues (18) developed a 12-item adherence measure specifically to reflect the therapy manual of Fowler and colleagues (19). A revised, 21-item, version of the scale was used in this study (19). The scale was used as an observer-rated measure on the basis of an audiotape and loose transcription of a complete therapy session. The observer first rates each of the 21 items for their frequency within the session on a scale of 1 to 7, where 1 is "sufficiently present to be considered a dose" and 7 is "present throughout the whole session." The observer then determines whether the activities considered present were carried out in a manner that met the scale's minimal competence criteria (broadly, to be individualized, collaborative, and matched to the client's presentation). If an item does not meet these criteria, the frequency score is translated to a negative scale ranging from -1 to -7 that has the same anchor points as described above. Scores can theoretically range from 0 to 147, although a maximum score is unlikely because it would indicate that all 21 therapy activities were being applied throughout the entire session. A recent psychometric analysis of the scale reported good levels of interrater reliability (20).

Tape ratings

Research setting. Observer R-CTPAS ratings of therapy tapes were collected throughout the trial. Therapists were asked to submit at least one tape in five for rating by a research clinical psychologist in one of the other research centers as part of the internal supervision processes within the trial. The trial psychologists rating the tapes were all doctoral-level clinical psychologists with an average of three years' experience of providing CBT for psychosis under expert supervision. They had received training on the use of the R-CTPAS from the scale authors and carried out regular (bimonthly) calibration checks with each other.

Clinical setting. Tapes were rated by a final-year trainee who was taking a doctorate-level course in clinical psychology. Interrater reliability had been established on 12 tapes with an index rater, a research clinical psychologist from the Prevention of Relapse in Psychosis trial and joint author of the revised scale (intraclass correlation coefficient of .94, p<.05).

Analysis plan

Major differences in the type of therapy provided among the three clinical centers were first explored through visual inspection of descriptive data. Reports of the demographic variables and descriptive data in the research and clinical groups were then provided. Because the research and clinical samples were not matched on an exact one-to-one basis, a between-group comparison was used. Because the R-CTPAS item scores were not normally distributed, a Mann-Whitney U test was used to compare the overall level of adherence and individual R-CTPAS item scores across the research and clinical groups. A t test was used to compare R-CTPAS total scores because these were normally distributed.

Four therapist-client dyads were recruited in Essex, four in Norwich, and 12 in Boulder. The focus of the therapy content was similar across the three clinical settings, with agreement across the centers regarding the R-CTPAS items rated most and least frequently. The data from the three clinical settings were subsequently analyzed as one clinical group.

Table 1 shows the demographic characteristics of clients in the research and clinical samples. Most clients were male and had a diagnosis of schizophrenia. There appeared to be a greater variety of diagnoses in the clinical group, and those in the research sample were somewhat older.

Table 2 shows the therapist demographic characteristics for both samples. The therapists in the clinical sample were slightly older than those in the research sample and were drawn from a broader range of professional backgrounds.

Table 3 illustrates the mean item scores and item frequency (number of sessions for which each item was rated as present) for the clinical and research groups. Because this is a new measure, normative data are not yet available with which to compare these means. All item mean scores do, however, reflect an adherent rating (that is, the therapy activity met the minimal competence criteria), suggesting no difference between the groups regarding overall level of therapy adherence.

Because a Kolmogorov-Smirnov test of normality suggested that the sample was not normally distributed, we used a Mann-Whitney U test to explore the difference between the clinical and research samples on R-CTPAS items.

The clinical sample received a significantly higher mean rating than the research sample on the R-CTPAS item of assessing psychotic experiences (z=-2.01, p<.05). The research sample achieved significantly higher ratings on three items: schema work (z=-1.98, p<.05), formulating a personal model of relapse (z=-2.61, p<.01), and relapse prevention interventions (z=-2.08, p<.05). For the research group we found a trend of scoring more highly on developing a model of psychosis (z=-1.90, p=.057).

The total R-CTPAS scores for the two groups were normally distributed, and there was no significant difference in variance. No significant difference was found between the clinical and research sample total scores.

Summary of findings

Both the research and clinical groups met the minimal competency levels defined within the R-CTPAS. Low-frequency R-CTPAS items in both groups were recognizing problems, verbal challenge of delusions, and validity testing, whereas higher-frequency items included assessing psychotic experiences, enhancing self-regulatory strategies, and engagement. The research group scored significantly higher on schema work, relapse prevention interventions, and formulating a model of relapse. A trend was found for the research group to score more highly on developing a model of psychosis. The clinical group scored significantly higher on assessment of psychotic symptoms. Although the research sample had a higher mean score for most items and a higher mean total score, this difference was not statistically significant.

Interpretation of findings

In terms of Fowler and colleagues' therapy manual (20), the therapy provided in the clinical settings appeared to focus on engagement and assessment activity. Therapists then seemed to use assessment information to develop strategies to manage current symptoms. Within the research setting, therapeutic activity appeared to follow a different profile. In addition to the engagement and the assessment work, the sessions focused more on relapse formulation, relapse prevention interventions, schema work, and formulation work. These activities could be considered inherently more challenging for therapists to apply. They require an extension and adaptation of traditional CBT skills used with other types of disorder, and they involve a greater potential challenge to the client's belief system and therefore to the therapeutic alliance. They also extend discussion beyond the traditional case management role and may consequently feel riskier for case managers to carry out.

We were not able to draw any conclusions from the data regarding the clinical effectiveness of these different therapy profiles. Formulation work, however, is often cited as a factor contributing to the ongoing improvement reported in follow-up studies of CBT compared with supportive counseling interventions (3,21). Fowler and colleagues (19) particularly emphasized a shared formulation as an important vehicle for promoting belief flexibility in CBT for psychosis. Further research is needed to examine more directly whether differences in therapy profile entail a significant difference in therapy outcome.

Implications for the dissemination of research-based interventions

These findings suggest that therapists working in routine clinical settings are able to establish good therapeutic relationships with people with psychosis and to work on assessing and coping with their psychotic symptoms. They also suggest, however, that in a routine clinical setting it may be more difficult to use therapeutic approaches that may be more challenging to the client's belief system or the therapist's case management role.

The different professional background of the therapists may have contributed to these different therapy profiles; Table 2 indicates a greater proportion of clinical psychologists within the research setting. Whereas other therapist variables are likely to influence therapy provision (such as waiting list pressures, case management responsibilities, size of caseload, amount of training, and supervision available), these were not controlled for in this study because they were considered an important aspect of the variability between the clinical and research settings being examined.

The wider organizational context in which therapists were working may be an important influence on the provision of therapy. The two settings differed in the amount of organizational support, role clarity, and supervision available for therapists. This difference could indicate a need for ongoing case supervision, clear organizational support, and skills training in order to assist therapists in this potentially more challenging therapy role (22,23).

Methodological issues

This study was limited by the small sample and the use of composite samples. The small sample meant that sources of variability, including the fact that patient data were nested (analytically) within therapist, could not be accounted for in the analysis. There are likely to have been other sources of variability that could not be controlled for that influenced the type of therapy provided. However, this study is one of the first to provide a preliminary estimate of effect size differences between expert therapists in clinical trial settings and clinicians in routine clinical practice. Future studies of this kind would need to provide greater control over variables such as client symptom severity and therapist experience working in CBT.

Adherence ratings on the R-CTPAS indicated that the CBT for psychosis provided by therapists in a routine clinical setting was characterized by more assessment of current symptoms, with less focus on schema issues and relapse prevention.

This research received funding from program grant 062452 from the Wellcome Trust.

The authors report no competing interests.