In recent years, there have been increasing calls for the routine collection and use of patient-reported outcomes data in order to improve the real-world treatment of individuals with mental disorders (1). Many evidence-based practice guidelines call for gathering information about an individual's progress at specific time points, as a move toward measurement-based care (2,3). Routinely gathering information about progress in treatment can also make important contributions to the clinical process by allowing clinicians and patients to make decisions that are more informed regarding treatment options (4,5,6), it has been associated with greater clinical improvement of individuals in treatment (7,8,9,10), and when information about progress is discussed with patients, it can enhance their involvement in treatment and treatment decisions (11,12,13,14,15). Routinely gathered outcomes data may also be used to document the clinical benefits of a treatment program (16,17).

The calls for increased collection and use of patient-reported outcomes data and the potential benefits from use of such data have resulted in a substantial increase in the number of organizations collecting such information. A recent survey of large community mental health organizations serving children found that 74% attempt to collect some type of standardized outcomes information about the families they serve (18). Such efforts, however, face a range of challenges. Many measures are relatively long, making them burdensome and time consuming for families and staff to complete, and the completion of such measures is not commonly reimbursable as a separate service (16). After measures are completed, many organizations lack the resources to score the measures in a timely manner and make results available to clinicians or families (19,20,21). As a result, clinicians are commonly hesitant about or resistant to routinely collecting outcomes data (22,23), and even when information is collected, the vast majority of clinicians report limited value and minimal routine use of such information in their ongoing care of individuals (16).

The challenges to using outcomes information to improve care involve both the routine collection and use of information. To enhance the understanding of these processes, we report on an academic-community collaborative effort to routinely and systematically collect information from parents of children receiving mental health treatment and the use of this information in treatment sessions. Below we describe the academic-community collaborative process to develop an outcomes tool, examine the use of the resulting tool by families and clinicians in treatment sessions, and explore factors associated with variations in its use.

Collaborative development of a child outcomes survey

The collaboration—involving clinicians and administrators of a child-serving mental health provider organization, clinician-researchers, and representatives from a large nonprofit academically affiliated Medicaid managed behavioral health organization (MBHO)—grew out of a desire of clinical providers to more uniformly and consistently gather and use information about children in treatment and to be able to use such information to better assess the effectiveness of treatment programs. During monthly meetings, the group identified many important issues and needs in prior efforts to collect and use patient-reported outcomes data. These included having a brief assessment to minimize burden on already busy clinicians and families; having a strength-based measure with sufficient face validity appropriate for children of different ages with various diagnoses and emotional and behavioral problems; and having a measure simple enough that the results are available to clinicians and families in a timely manner (24). After receiving input from parents of children in treatment and discussing the relative priorities and benefits of different approaches to gathering and using child outcomes information, the group chose to emphasize the brevity of the measure in order to minimize burden and facilitate broad participation; a single measure that is useful across child populations; and a measure that is easy to score, facilitating incorporation of resulting information into clinical sessions without requiring substantial administrative support.

After review of existing diagnosis-specific and general child measures, a decision was made to develop a brief survey that included a strength-based assessment of successful functioning in key domains (for example, home, family, friends, and school), comparable to other widely used psychometrically established measures (25); a brief assessment of the parent perception of the therapeutic relationship between the family and clinician; and a single item assessing utilization of the information by parents and clinicians. To assess child functioning, four parent-completed items asked about how successful the child was in the past week at completing household tasks, getting along with family, having a social life and getting along with friends, and doing well at school. Responses on a 10-point Likert scale ranged from "not at all successful" to "extremely successful." Four parent-completed items regarding the therapeutic relationship included feeling understood and respected in what parents were trying to accomplish for the child, working on goals that are most important to the child's needs, establishing an approach that is best suited for the child, and being confident that the work they are doing (with the therapist) will help the child. Parent responses were scored on a 10-point Likert scale ranging from "not at all" to "very much." An additional parent-completed item assessed the extent to which parents discussed or used the information from the previous Child Outcomes Survey with their clinician. Parent responses were on a 10-point Likert scale ranging from "not at all" to "very much." The tool development process also involved feedback from volunteer clinicians and families from six agencies who had experience using an earlier version of the tool in their sessions.

Following the development process, use of the Child Outcomes Survey was initiated by ten organizations providing intensive community-based and school-based services to children and their families. Clinicians asked families of children in treatment to routinely complete the Child Outcomes Survey. Anecdotal reports indicate that it was most common for clinicians to have parents complete the survey at the beginning of a treatment session, but some clinicians chose to have parents complete the survey later in the session, and in some situations clinicians chose to have parents complete the survey before the session (for example, while in a waiting area). The timing of completion of the survey with respect to the session was at the discretion of the clinician. The MBHO provided timely longitudinal results to clinicians and families in a graphical format to facilitate interpretation and use, consistent with recommendations from prior efforts (11). [A figure showing an example of Child Outcomes Survey feedback for clinicians and families is available as an online supplement at ps.psychiatryonline.org.] These graphs were faxed to the provider organization within several days of receiving the completed Child Outcomes Survey, allowing clinicians to share the graph with parents within a week of survey completion.

Sample

To examine the extent to which parents discussed or used the information from the previous Child Outcomes Survey with their clinician, we utilized data from ten organizations providing intensive community-based and school-based services to children and their families. These provider organizations implemented the Child Outcomes Survey process for gathering and using data on a routine basis from June 2008 to November 2009. The University of Pittsburgh Institutional Review Board approved the study.

Measures

The item from the Child Outcomes Survey that assesses parent-reported discussion of the information from the previously completed survey (described above) was used to examine the use of survey information with clinicians. Child functioning and therapeutic relationship were assessed using items from the Child Outcomes Survey (described above). Information about the provider organization and child gender, race-ethnicity, age, and status as a new or existing client was obtained from the child mental health provider organization.

Analysis

For purposes of analysis, parent-reported levels of discussion with the clinicians were categorized as low (scale scores 1—6), moderate (scale scores 7—8), and high (scale scores 9—10). The mean child functioning score was calculated across completed items and was categorized as high functioning (7.1—10.0), moderate functioning (5.0—7.0), and low functioning (.0—4.9). A therapeutic relationship score was calculated across completed items and was categorized as good (9.0—10.0), fair (7.1—8.9) and poor (1.0—7.0). Child age was categorized as youngest (four to ten years), middle (11 to 14 years), or oldest (15 to 18 years).

We calculated descriptive statistics for each of the scales described above and for sociodemographic and clinical variables. We then used Wald chi square tests to examine the difference in proportions of parents reporting low, medium, and high rates of discussion with their clinician about the information gathered from the previous Child Outcomes Survey and the child's gender, age, race-ethnicity, therapeutic relationship, successful functioning, and status as a new or existing client. We also assessed the marginal effects of improving the relative level of discussion between parents and clinicians about information gathered from the previous survey from a low score (score of 6) to a high score (score of 9). First, we modeled the level of functioning as a function of the level of discussion about the information gathered from the survey while the analysis controlled for the child's sex, race, and age. Then we simulated the improvement by fixing the value of the conversation score at both the low and high level and generated predicted values for each child while leaving the other covariates with their original values. Finally, we compared the averaged predicted marginal effects. The same approach was used for assessing the impact of improving the level of discussion about the information gathered from the survey on the therapeutic relationship score.

Across the ten provider organizations, there were 1,215 completed Child Outcomes Surveys from parents of children aged four to 18 years who were receiving mental health treatment. The children were predominantly male (N=847, 70%), four to ten years old (N=438, 36%), Caucasian (N=623, 51%), and initiating as a new client (N=729, 60%) (Table 1). In the measure that assessed the extent to which parents discussed with their clinician or used the information about their child's functioning from the previous Child Outcomes Survey, the majority of parents (N=738, 61%) reported high levels of discussion, 25% reported moderate levels of discussion (N=302), and 14% (N=175) reported low levels of discussion.

Although there were no differences across age groups, parents of boys were significantly more likely than parents of girls to report discussing with their clinician the information about their child's functioning from the previous Child Outcomes Survey (65% versus 51%) (p<.001). Parents of Latino children (68%) and parents of children in the "other" racial-ethnic group category (68%) were significantly more likely than parents of Caucasian children (57%) and African-American children (53%) to report discussing with their clinician the information about their child's functioning from the previous survey (p<.01).

We also found significant associations between the amount of parent-clinician discussion of information about the child's functioning from the previous survey and the child's current level of functioning and the family-clinician therapeutic relationship. Parents reporting a high degree of discussion about children's functioning had substantially better therapeutic relationships with clinicians and reported higher levels of successful child functioning. In contrast, parents reporting lesser levels of discussion of results of the previous Child Outcomes Survey commonly had lower ratings of their therapeutic relationships with clinicians and had substantially lower rates of successful functioning among their children (Table 1).

Potential impact of enhancing conversations about functioning

To better understand the potential impact of enhancing parent-clinician discussions on the therapeutic relationship and child functioning through the use of a tool such as the Child Outcomes Survey, we modeled the impact of improving the parent rating of the amount of discussion from low (score of 6) to high (score of 9) while the analyses controlled for the other clinical and sociodemographic information available. We found that such an improvement in the amount of parent-clinician discussions would potentially be associated with a 16% improvement in child functioning (child functioning score increasing from 5.5 to 6.4) and a 17% improvement in family-clinician therapeutic relationship (therapeutic relationship score increasing from 7.8 to 9.1).

In this examination of clinician-parent use of a brief, parent-reported child outcomes measure in therapeutic sessions, we found that overall, parents reported high levels of discussion with clinicians about outcomes information obtained in the measure. Because of past reports of clinician reluctance and difficulties in trying to incorporate such client-reported outcomes measures into clinical sessions (26), the reported levels of utilization are encouraging. The brevity, simplicity, and face validity of the measure and the one-page feedback graph are in contrast to the measures and information obtained in a number of other patient-reported outcomes information pilot projects (5,8,11) and may have contributed to the increased use of the information by clinicians in conversations with families, compared with longer reports (11). Such conversations can enhance the goal of making services more patient centered (27,28), may increase families' engagement in treatment planning for their child (29), and may enhance the likelihood that clinicians are addressing the child's emotional and behavioral problems that are of greatest concern to parents.

Prior research has found that clinician-reported use of patient-reported outcomes measures is associated with better clinical outcomes (8,10,30). This is consistent with the finding in this population that parents reporting the most discussion with clinicians also reported the highest level of overall successful current functioning of their child. Using a brief measure to structure clinician-parent conversations during sessions about a child's progress while in treatment may enhance the routine gathering and discussion of information about the child's functioning, potentially enhancing the ability of clinicians to adjust the treatment approach more quickly and easily when appropriate (31). However, the cross-sectional design does not allow us to determine causality, and it may be that families or clinicians are more reluctant to discuss outcomes when things are not going well or that sessions of children who are not doing well are more likely to be characterized by discussions about current crises than discussions about the child's progress, or lack thereof, in treatment. Future longitudinal research is needed to better understand to what extent discussing child outcome information with parents has a positive impact on the quality and effectiveness of mental health services for children (16).

We also found a strong association between discussing the information in the Child Outcomes Survey and the family-clinician therapeutic relationship. Clinician-client conversations about important treatment outcomes and treatment options to achieve those outcomes are a hallmark of shared decision making (32) and have been associated with improved therapeutic relationships and outcomes among adults (33). Empirical studies are needed to examine whether such an association is observed when a child is the identified patient.

The results exploring the potential impact of enhancing routine discussions between parents and clinicians regarding child outcomes suggest the importance of such efforts, particularly for clinicians and families for whom such discussions are relatively uncommon. The successful treatment of children commonly requires an ongoing and effective partnership between parents and clinicians, and the results suggest how important routine conversations about the progress of children in treatment can be for such a partnership. Many factors, however, may affect the relationship between parents and clinicians of a child receiving mental health treatment, including the child's age, perspectives of the child and parents on the need for treatment, and ongoing access to mental health care. A tool to enhance parent-clinician conversations about a child's functioning addresses only one small aspect of this complex and important relationship.

The findings of this study must be examined within the context of a number of limitations. The brief measure was developed to maximize feasibility and acceptability of use by parents of children in treatment and their clinician by facilitating discussions about treatment and child progress. Although many clinicians and parents who were involved in the development and use of the measure have commented on its face validity, the psychometric properties of the measure have not yet been established. An adequate assessment of the psychometric properties of the Child Outcomes Survey is necessary before there is confidence in the reliability and validity of the measure in assessing child functioning and the therapeutic relationship. We note, however, that other widely used measures, such as the CANS (Child and Adolescent Needs and Strengths), also first examined the way in which a measure was used in a session and only subsequently examined the measure's psychometric attributes (34,35). The relative importance of a measure's attributes are a function of how it is to be used, and while attributes such as validity and reliability are always valuable, other attributes such as strong face validity, simplicity, brevity, and transparency play a critical role if a measure is to routinely and consistently be used in clinical settings (36).

The Child Outcomes Survey has been used primarily with parents whose children are receiving community-based treatment or school-based treatment, and we do not know whether use would be the same by clinicians and parents of children being treated in more traditional outpatient settings. All clinicians in participating clinical programs were asked to routinely complete the Child Outcomes Survey with families. Although discussions with participating providers indicate that clinicians and families regularly completed the survey, completion was not monitored, and we do not know how many families did not complete the measure. We do not know how their participation may have influenced the findings, nor to what extent the findings would generalize to parents who did not complete Child Outcomes Surveys. The analysis is cross-sectional, as noted above, which limits our ability to draw conclusions about causality of using the results or information on child outcomes in sessions.

The universal call to use measurement in mental health treatment has led many clinicians to attempt to do so (19). However, the process of gathering information from families can be burdensome, and resources are often limited for making results available for use in treatment sessions (18). This pilot study examined the use of a more efficient and streamlined process for gathering and using outcomes data in child mental health treatment. The results demonstrate that many clinicians and parents will routinely use a brief parent-reported outcomes measure to gather information, and in turn, such a routine and structured process may enhance parents and clinicians' discussions during treatment sessions about progress or lack of progress in key domains. The successful treatment of children and families requires an ongoing and effective partnership between parents and clinicians, and the results suggest how important these routine conversations about the progress of children in treatment can be. Results from this study contribute to the limited empirical data regarding the actual use of outcomes measurement information in mental health care (37). Further longitudinal research is needed to understand the impact of gathering and using such data on the process and outcomes of mental health treatment for children and families.

Support for this work was provided by Community Care Behavioral Health Organization.

The authors are indebted to the Community Care Performance Management Committee for useful feedback and comments on prior versions of this article, to Amelia Haviland, Ph.D., for statistical advice, and to Laura Greenberg, M.A., for research assistance.

The authors report no competing interests.