Using Feedback From Patient-Reported Outcome Measures in Mental Health Services: A Scoping Study and Typology

Routine formal evaluation of the outcomes of patient care has become increasingly widespread and plays an important role in mental health service provision (1). Over time, an immense array of patient-reported outcome measures (PROMs) has been developed, with the aim of including patients’ perspectives in the process of health service provision. PROMs have typically been defined as patient-rated standardized measures of health or functional status, disability, participation, quality of life, well-being, or other specific and relevant outcomes of treatment, such as depression or anxiety (2,3).

Systematic reviews of the literature on the use of PROMs in clinical practice have typically found that use of PROMs is associated with improvements in some aspects of care outcomes and quality of care. However, clear conclusions are difficult to derive because of methodological limitations and lack of clarity regarding the goals and mechanisms of using PROMs (4,5). Greenhalgh (6) provided an overview of the various ways and purposes of PROM use in clinical practice and presented the following categories: screening tools, monitoring tools, promotion of patient-centered care, decision aids, methods to facilitate communication among multidisciplinary teams, and evaluation of the effectiveness of routine care and assessing quality of care. The first three uses involve individual-level data, and the last three involve group-level data.

Boyce and Browne (7) systematically reviewed studies that investigated the effects of providing PROM feedback to health care professionals. Only one of the 16 studies in their review reported an overall positive effect. This study reported on the results of an intervention at a hospital-based psychotherapy clinic (8). That study used as the PROM the 45-item Outcome Questionnaire (OQ-45) (9), which assesses client progress in therapy. Patients in the patient-therapist feedback group, in which results from repeated PROM administration were discussed between patient and therapist, later showed significantly greater improvements in PROM scores than patients in the group that received treatment as usual and those in another group in which only the therapist received PROM feedback.

Using PROM feedback with patients is consistent with the principles of recovery, which focuses on the transformative aspects of overcoming mental health issues and thus emphasizes self-determination and an individual’s sense of agency (10,11). PROM feedback not only allows the clinician to provide information on the patient’s progress but also attempts to capture patients’ view about whether they are progressing and helps patients appraise themselves and reflect on their own recovery journey.

Previous reviews of the effects of using PROMs have focused on various aspects of PROMs, such as the purpose and nature of applications (6), or the usefulness of PROM feedback at the patient and group level (7). However, a systematic description of the range of procedures by which patient feedback is obtained in mental health services is lacking. In particular, the various levels of provider-patient communication associated with these procedures have not been systematically explored. The purpose of this scoping study was to provide a typology of the ways in which studies reporting on PROM use in mental health services have administered PROMs. Understanding the scope of the literature and categorizing studies by levels of intensity of PROM feedback highlights new ways of analysis that can help explain the diversity of outcomes in investigations of the effects of PROM use (6,7) and provide clarity about whether the provision of PROM feedback is indeed associated with positive outcomes.

Methods

Scoping Study

Scoping studies are particularly suitable when the goal is to determine the scope and nature of a field that includes studies with a large range of methodologies (12,13). The procedures used are similar to those used in systematic literature reviews but tend to focus more on breadth rather than depth of the literature and thus do not exclude studies on the basis of quality criteria. Because of the diversity of the study methods examined, the common analytical framework is a descriptive-analytical method within the narrative tradition (12). Scoping studies chart the evidence and procedures of studies to increase conceptual clarity and to map the conceptual boundaries of a specific topic area (14).

Note that various terms are used to describe people receiving mental health services, including patients, clients, consumers, and service users. In order to be consistent with the established term PROM, we generally refer to this group as patients, while acknowledging the sensitivity of this term, particularly to those who regard themselves as service users in recovery.

Search Strategy

This scoping study was guided by an iterative search strategy (12). After we became familiar with the literature on PROMs, we conducted structured searches on the database Scopus for peer-reviewed journal articles, with no restrictions on year of publication or language. Given the variety of terms used to describe this broad topic, search strategies were initially based on a related systematic review in palliative care (15), as well as on other recommendations in regard to the most sensitive and specific combination of terms with mental health content (16). Our review focused on PROM use in mental health settings, although the focus was initially broadly defined to capture a wide range of articles. The database search retrieved 59 articles, of which 13 were retained for more detailed review (3,17–28). Manual searches yielded two further review articles (7,29). After iterative searches of reference lists, citation searches, and specific searches of articles from prominent researchers in the area, a total of 166 articles were identified. Of these, 109 were excluded because they used measures that were not standardized PROMs or because they were not about research in mental health settings but about mental health aspects of other fields, such as oncology, rehabilitation, general clinical practice, or substance abuse. Studies were also excluded if they merely reported on psychometric properties of PROMs, were surveys on the uptake of PROMs, or were opinion pieces. Of the remaining 57 articles, 28 were review articles, and 29 empirical articles were categorized as outlined below. During the peer-review process, the anonymous peer reviewers identified another six studies that were also included.

Categorization of Articles

Scoping studies follow an iterative process (12) that continually refines mapping criteria as new evidence is identified and analyzed. Therefore, we met regularly for discussion to agree on adequate ways to categorize articles into levels of intensity of PROM feedback used. The final typology is presented in the box on the next page. It included five categories, ranging from category 1 (PROM scores not fed back to clinician or patient) to category 5 (PROM scores fed back to clinician and patient, with a formalized structure to guide clinician-patient discussions).

Although studies in category 1 by definition cannot provide any information on the effects of PROM feedback, retaining this category was useful for the purpose of establishing a typology of PROM feedback provision. Category 2 studies provide PROM feedback to clinicians, and studies in category 3 provide feedback to both clinicians and patients. In studies of categories 2 and 3, discussion of PROM results may take place, although entirely at the discretion of the clinician. Any such discussion would therefore be incidental only. In studies included in category 4, clinician-patient PROM discussion is actively encouraged, but no formal structure guides this process. And finally, category 5 includes studies in which clinician-patient PROM discussion on the basis of available formal guidelines was actively encouraged.

For any study to be allocated to one of the five categories, group consensus was required. Two of the authors (CUK and KJC) carefully read and categorized the articles independently and iteratively. Disagreement was resolved by discussion, which at times resulted in further refinement of the category wording. The other authors assisted with categorization of a selection of articles.

Most studies included control groups (typically category 1), but our categorization was based on the procedure of the intervention group. Some studies included two interventions that belonged to different categories, in which case the study was allocated to the highest category (8,30–32). Of the 35 reviewed studies, four could not be assessed because of incomplete information (24,33–35). Three additional studies were removed because they reported on a data set that was used in another study included in the review (36–38). Two studies reported on different subgroups of the same data set and were treated as one study (39,40).

Results

Table 1 lists the 27 studies included in this review and provides a description of each study’s sample, PROM feedback procedure, and results. Two studies belonged to category 1 (41,42), eight to category 2 (39,43–49), four to category 3 (32,50–52), two to category 4 (53,54), and 11 to category 5 (8,30,31,55–62). Almost half the studies involved samples from the United States (8,30,31,41,42,44,46,47,49,58,59,61,62). One article reported on a study conducted in six European countries (57). Apart from an Australian study (53), the remaining studies were from European countries: Germany (39,43,48,54), the United Kingdom (50,52,60), the Netherlands (32,45), Ireland (56), Norway (55), and Sweden (51). The study populations were diverse, including hospital inpatients (39,43,53,54,62), hospital or institution outpatients (8,32,45,51,61), and clients of a variety of community-based services (41,44,48,50,52,55,57,60). Eight studies reported data from clients of university counseling services (30,31,42,46,47,49,56,58,59), all of which, except for three (56,58,59), were from the same university.

Lambert and colleagues authored ten of the articles listed in Table 1 (8,30,31,39,42,46,47,49,61,62), and all of these used the OQ-45 (9). The OQ-45 was used in four additional studies (32,45,51,54), making it the most frequently used PROM. The second most frequently used PROM was the four-item Outcome Rating Scale (ORS) (63). This measure, derived from the OQ-45, was used in four of the studies listed in Table 1 (55,56,58,59).

Category 1 functions as a baseline in the typology presented in the box on this page. Only two articles belonged to this category (41,42), largely because the scoping strategy outlined above searched for articles that reported on the use of PROM feedback. Although articles in this category by definition cannot provide any information on the effectiveness of PROM feedback, these two articles serve as exemplars of procedures in which PROMs are used but no feedback is provided to the clinician or client.

All category 2 studies purported to investigate the effects on patient outcomes of PROM feedback to clinicians. Six of these were randomized controlled trials (39,43–46,48), and the remaining two were quasi-experimental designs with close resemblance to the design of the other six studies (47,49). Table 2 summarizes information about studies that reported a significant effect of PROM feedback on PROM scores and on treatment duration. Two studies reported significant positive effects (43,44), and the remaining studies reported significantly larger improvements only for clients considered “not on track” or “at risk” (39,45–47,49) or no effect (48). Effect sizes were generally small or medium. In four of the studies that reported data on treatment duration (46–49), feedback was associated with significantly longer treatment for not-on-track clients; in three of these studies (46,48,49), feedback was also associated with significantly shorter treatment duration for on-track clients. One study reported no effect on treatment duration (39).

All four category 3 studies (32,50–52) were randomized controlled trials, and none reported a significant effect of PROM feedback to clinicians and patients compared with category 1 control conditions. One of the two category 4 studies reported a significant effect for only a subgroup of the sample and on some measures only (53), and the other category 4 study did not obtain a significant effect (54). However, although discussion of feedback was encouraged in that study (54), the authors reported that actual clinician-patient conversations about PROM feedback were rare.

Of the 11 studies in category 5, nine reported a significant effect of structured PROM feedback discussions (8,30,31,55,57–59,61,62). Two studies obtained partial effects (56,60)—namely, significant results for only a subgroup in the sample or for only some of the outcome measures. Effect sizes were generally either small or medium.

Category 5 generally contained studies with more complex designs, such as multiple experimental groups. Three studies compared the effects of category 5 feedback with the effects of category 2 and category 1 feedback (8,30,31). In all three studies, feedback resulted in significantly greater improvements in PROM scores compared with category 1. However, two studies did not find a significant difference between the effect of category 2 and category 5 feedback (30,31), whereas one did (8).

Harmon and colleagues (30) reported significantly longer treatment duration for not-on-track clients, and Slade and colleagues (31) found that clients in the control condition required significantly more treatment sessions than clients in the feedback conditions. These two studies were also the only ones that used quasi-experimental designs in category 5. The other nine studies were randomized controlled trials, and of the six that reported on treatment duration data (8,56,58,60–62), none found a significant effect of PROM feedback on treatment duration.

Discussion

This scoping study mapped previous research studies in mental health according to levels of intensity of PROM feedback use: no feedback (category 1), clinician-only feedback (category 2), feedback to clinicians and patients (category 3), encouragement of mutual PROM discussion (category 4), and availability of formalized mechanisms to guide such discussion (category 5). Previous systematic reviews concluded that evidence is lacking about whether PROM feedback to health care professionals improves outcomes, as illustrated by Boyce and Browne’s (7) review of systematic reviews. In their own systematic review, Boyce and Browne reported that only one of the 16 studies found a positive effect of PROM feedback, and six other studies found partial effects. Our review of the mental health literature indicated that of the 25 studies that provided information on the effectiveness of PROM feedback (categories 2 to 5), 11 reported significant effects with generally small to medium effect sizes, eight reported partial effects, and six reported no effects. Of the 11 studies in category 5, nine found significant effects and two found partial effects, indicating that formalized clinician-patient PROM feedback was most strongly associated with improved outcomes. Compared with categories 2 to 4, category 5 had a significantly higher proportion of studies reporting a statistically significant partial or full effect of feedback versus no effect (χ²=6.20, df=1, p<.05) and a significantly higher proportion of studies reporting a statistically significant full effect versus only a partial or no significant effect (χ²=11.40, df=1, p<.01).

The likelihood of reporting significant effects, however, did not increase in a linear fashion with feedback levels. Two of the category 2 studies found a significant effect, and five of the category 2 studies found a partial effect, whereas none of the category 3 studies and only one of the two category 4 studies reported a partial effect. Two studies that examined both category 2 and category 5 experimental conditions did not find a significant difference between outcomes of these two conditions in a sample of clients at a university counseling center (30,31). In a study of hospital outpatients, Hawkins and colleagues (8), in contrast, reported improved outcomes for patients in category 5 compared with those in category 2, which could indicate that clinician-patient feedback may be more effective than clinician-only feedback in specific settings only.

With the exception of one category 1 study (42), the studies associated with the research program of Lambert and colleagues were either in category 2 or 5, and all these studies used the OQ-45. The OQ-45 can be used in conjunction with its associated clinical support tools. Previous studies applied clinical support tools with not-on-track patients, which resulted in better treatment outcomes compared with use of patient progress feedback with the OQ-45 only (64). Only one study applied clinical support tools also for patients on track to recovery and found that this did not substantially enhance outcomes (40). Our typology presents a unidimensional outline of intensity of PROM feedback use with clients. Within each category, additional variables were associated with positive therapeutic outcomes, thus creating variability of results within each category of feedback intensity. A formalized structure maximizes the likelihood that feedback is discussed with clients, which appears to drive the beneficial results of PROM use in studies included in category 5. Other aspects of procedural formalization may also be relevant, such as presence of computerized support tools (64), frequency of feedback (44), and whether PROMs are discussed among clinicians (65).

The lack of a feedback effect in studies included in categories 3 and 4 is somewhat surprising but could be related to procedural variations. Newnham and colleagues (53) speculated about whether their delivery of feedback during group therapy may have been qualitatively different from feedback delivered during individual client-clinician interactions. Therapists’ commitment to using PROMs was also found to be related to effectiveness of feedback (45), and the lack of a feedback effect in the other category 4 study (54) may thus be linked to the reportedly low frequency of therapist-initiated PROM discussions in that study. Finally, the feedback effects in the studies by Lambert and colleagues included in category 2 were largely related to clients considered “not on track” (64). With the exception of two studies (32,53), none of the other studies in categories 3 and 4 reported analyses by subgroups, which may have revealed some partial feedback effects.

The ORS questionnaire was the second most frequently used PROM. Three studies reported significant effects of category 5 feedback (55,58,59), and one study reported partial effects (56). The ORS is rarely offered on its own but is typically offered with the Session Rating Scale (SRS) (66), which assesses the therapeutic alliance between client and clinician. Of the four studies that used the ORS, only one did so without also using the SRS (56). The fact that the latter study noted only a partial effect may thus indicate that elements in addition to PROM feedback may be responsible for positive therapeutic outcomes.

Feedback is an integral part of meta-therapeutic dialogue, which, in addition to PROMs, often includes assessment of client needs and preferences and assessment of the therapeutic alliance (67). Although the effects of PROM feedback might be difficult to disentangle from the effects of other aspects of such dialogue-directed approaches, qualitative reports explicitly point to positive experience of PROM feedback. Cheyne and Kinn (50) did not obtain a significant effect in their study involving category 3 PROM feedback, which may have resulted from their small sample. In another article, however, these authors extensively reported on the positive observations of counselors in discussions of PROM scores (36). Counselors found that the Schedule for the Evaluation of Quality of Life (68) functioned well as an aid to client reflection and to enhance therapeutic alliance. Similarly, Sundet (35) reported that completing items on the ORS may trigger very specific reactions, enhancing client-therapist dialogue by initiating, directing, or focusing conversations.

Because of the lack of uniform terms to describe the approach of providing or discussing PROM feedback, the scoping method was chosen to map the field and inform our typology. Most of the articles were not obtained through database searches but through extensive iterative searches of citations and reference lists, manual searches, and searches for specific authors. However, because of the tendency of the scoping method to focus on breadth rather than depth, some relevant articles may have been missed. Unlike previous reviews (7), our search was not limited to articles published in English. Two German-language articles were included (43,48); however, articles in languages other than English and German may have been missed.

Allocation of some articles to the five categories was difficult because of unclear or incomplete information provided. In addition, category allocation was based on reported procedures and not on how PROM feedback may have actually occurred. Studies in the lower categories may have been de facto studies of higher categories if therapists frequently discussed PROM feedback with their clients. Similarly, studies of higher categories may have been de facto studies of lower categories, such as in the case of Puschner and colleagues (54), who reported that clinician-patient discussions rarely occurred despite being planned.

This literature search identified a number of studies that used the Clinical Outcomes in Routine Evaluation instruments (69). These category 1 articles were not included because the inclusion criteria did not extend to articles that reported results from primary care settings. Future reviews may analyze the extensive literature on primary care by using the typology presented here.

Conclusions

This scoping study reviewed studies that reported on the effects of PROM feedback in mental health settings. We provide a procedural typology of intensity of PROM feedback. Unlike previous reviews that reported minimal effects of PROM feedback, this review, which synthesized results on the basis of a procedural typology, showed that the availability of formalized guidelines for clinician-patient discussion of PROM feedback was most highly associated with improved therapeutic outcomes. Certainly, other variables, such as the presence of computerized support tool software (64) or the frequency of feedback (44), are also related to positive therapeutic outcomes, and these can be integrated into the typology presented here as factors that affect variability of results within each category of feedback intensity.

Use of PROMs supports patient-centered care (6) because it recognizes patients as participant consumers who should be active in plans and decisions about treatment options. Qualitative reports favor the use of PROM discussion, such as by enhancing clinician-patient communication and providing clients with mechanisms for reflective practice (36,70). However, because therapeutic approaches to discussing PROM feedback with clients tend to occur in conjunction with a general emphasis on the therapeutic alliance and meta-therapeutic dialogue, future work is required to isolate the effects of PROM feedback from such procedural correlates.