Use of the Pediatric Symptom Checklist in Strategies to Improve Preventive Behavioral Health Care

Epidemiological studies indicate that 17 to 27 percent of children in the United States have a psychiatric disorder (1,2,3,4,5,6,7,8,9). Poor and minority children—a growing proportion of the U.S. population—are particularly at risk (10,11,12). However, it is estimated that only one-sixth to one-half of children who have psychiatric disorders are identified, and of these, less than half receive mental health services (13,14). Numerous studies have shown that untreated mental health problems can develop into more severe psychosocial impairment and lead to increased use of health care services (15,16,17).

Pediatricians have traditionally been seen as a primary resource for recognizing psychosocial problems in children and adolescents (18). Although pediatricians, as part of their professional mission, are concerned with early identification and prevention of such problems, they face many barriers in identifying at-risk children and referring them to behavioral health programs. These barriers include inadequate training, lack of a practical screening tool, hesitancy to attach stigmatizing labels to children, and lack of time during the office visit (18,19).

The pediatric preventive behavioral health program described in this report is the product of several years of collaboration between Neighborhood Health Plan, the Child Psychiatry Services at Massachusetts General Hospital, and Neighborhood Health Plan's behavioral health program contractor, Beacon Health Strategies.

Neighborhood Health Plan is a not-for-profit health maintenance organization that currently serves a Massachusetts membership of 120,000 individuals, 85 percent of whom are enrolled in Mass-Health, the state's Medicaid program. Although 61 percent of Neighborhood Health Plan's members are children and adolescents—20 to 25 percent of whom are at risk of behavioral health problems (20)—use of behavioral health services by members under the age of 19 is relatively low compared with that of older members. Surveys of network practitioners revealed the need for a proven method of behavioral health screening and referral to assist them in their clinical work.

To address these needs, Neighborhood Health Plan developed a pediatric preventive behavioral health clinical guideline that would provide pediatricians in its primary care network with an efficient and reliable clinical method of early identification and intervention for children and adolescents at risk of behavioral health problems. The guideline was also designed to meet federal Early and Periodic Screening, Diagnosis, and Treatment mandates (21) and National Committee for Quality Assurance standards. This paper describes the use of the Pediatric Symptom Checklist (PSC) (22) as a screening tool and as the basis for the clinical guideline that Neighborhood Health Plan developed.

Methods

Initial screening

The PSC was used to screen a total of 570 pediatric patients for behavioral health problems. The PSC is a one-page, 35-item questionnaire designed to reflect a parent's impression of his or her child's psychosocial functioning. It is typically filled out in a pediatrician's waiting room, and it can be completed and scored in less than five minutes. Use of the PSC has been shown to improve recognition rates for psychosocial problems in pediatric primary care settings (23,24).

The screenings were conducted at three clinics—the Whittier Street Neighborhood Health Center in Roxbury, the Chelsea Memorial Health Center, and the East Boston Neighborhood Health Center—between April and August 1996. The population served at these health centers has a high Neighborhood Health Plan membership rate and is culturally diverse. The screenings took place two to three times a week at each site during pediatric clinic visits. All patients in the waiting room were considered potential study participants, regardless of their insurance status. The subcommittee on human studies at the Massachusetts General Hospital approved the study.

A bilingual (English-Spanish) research assistant approached the parent of each child aged two to 18 years and explained that the clinic was including questions about children's emotions and behavior as part of their pediatric visit. The parents were then asked if they would be willing to complete the PSC; about 90 percent of those asked agreed to do so.

The research assistant gave the parent the questionnaire on a clipboard and retrieved it when the parent finished. The form explained that completion of the checklist was completely voluntary, that the information would be included in a research project, and that the form would be inserted in the child's chart. The form also sought consent for researchers to contact the parent for a follow-up interview.

After the checklist was completed, the original was clipped to the front of the chart for the pediatrician to review during the visit, and the project staff retained a copy. Pediatricians were asked to use the screening as an adjunct to their clinical judgment, viewing it as an indicator of the need for further services. Pediatricians could nevertheless refer children for behavioral health services if they saw the need, regardless of the results of the screening.

The research assistant scored the completed questionnaires. Each item was assigned a score of 2, often; 1, sometimes; and 0, never. A positive score, that is, a score indicating the presence of or potential for behavioral health problems, was defined as 24 or higher for children under four years of age, and 28 or higher for children aged four and older.

Follow-up interviews

To test the reliability and validity of the PSC and to gather further information about the children and their parents, the parents of 95 children were selected by random sampling to be interviewed by a research assistant four to six weeks after the initial screening. In the 60-minute, face-to-face interview, parents were asked in-depth questions about the child's functioning in the home, at school, and in the community; his or her emotions and thinking; and, for older children, substance use. The interview elicited detailed information about the child's overall functioning, history, behavioral health and medical service use, and symptoms. The interviewer rated the child's overall level of functioning using the Child and Adolescent Functional Assessment Scale (CAFAS) for children aged seven to 18 years and the Preschool and Early Childhood Functional Assessment Scale (PECFAS) for children aged two to six years (25). The interviewer was blind to the child's PSC scores when conducting the interviews and scoring CAFAS or PECFAS ratings.

Case conferences

During clinical meetings at the three health centers, multidisciplinary teams discussed the findings for 25 children who had received positive scores on the PSC and for whom follow-up interviews had been conducted. The teams included each center's pediatric staff and representatives from Massachusetts General Hospital, Neighborhood Health Plan, and the center's mental health and substance abuse department. Cases were discussed to determine appropriate disposition and to identify children in need of behavioral health services. In addition, treatment options and clinical and programmatic issues were discussed more broadly for children who were considered to be at high risk.

Results

Checklist validity and reliability

The PSC was found to be valid when case classifications made as a result of the screening were compared with classifications based on ratings derived from the follow-up interviews. The checklist's sensitivity—the percentage of children who were found upon interview to be truly impaired and who had screened positive on the PSC—was 91 percent. Specificity—the percentage of children who were found upon interview to be healthy and who had screened as healthy—was 65 percent.

The validity of the PSC was good for school-aged children (age six and older) from both English- and Spanish-speaking families (sensitivity, 94 percent; specificity, 79 percent). For pre-school-aged children (younger than six years) from English- and Spanish-speaking families combined, sensitivity and specificity were somewhat lower (85 percent and 61 percent, respectively). The lowest levels of accuracy were found among pre-school-aged children from Spanish-speaking families, where sensitivity was 75 percent and specificity was 53 percent.

The PSC was also found to be reliable. The parents of 78 children completed the checklist a second time during the follow-up interview. The correlation between the initial screening score at the clinic and the second screening score at the interview was .8.

Rate of behavioral health problems

Of the 570 children in the screening sample, 144 (25 percent) received a positive score on the PSC (Table 1). Of the 297 pre-school-aged children, 67 (23 percent) received a positive screening score on the PSC. Of the 283 older school-aged children from both English- and Spanish-speaking families, 77 (27 percent) received a positive score.

Table 2 shows the relationship between positive screening scores and positive scores on the CAFAS for the 95 children who received follow-up interviews. Because the PSC was found to be valid after correction for false negatives and false positives on interview, it can be used to estimate the overall prevalence of behavioral health dysfunction in the community. Using a CAFAS score of 40 or higher—which is equivalent to a DSM-IV Global Assessment of Functioning (GAF) score of 60 or lower—as indicative of dysfunction, the overall adjusted rate of behavioral health problems among the children in our sample was 26 percent (29 percent among school-aged children and 19 percent among pre-school-aged children).

The accuracy of the screen was excellent with both pre-school-aged and school-aged children with serious behavioral health problems, missing very few of the children with very severe or severe problems and only a small fraction of those with moderate problems. Most of the children who were found upon interview to be truly well had negative screens on the PSC.

About one-quarter of the children met the criterion for serious emotional disturbance (that is, a CAFAS or PECFAS score of 40 or above); 2 percent met the criterion for very severe emotional disturbance (a CAFAS or PECFAS score of 80 or above), which is often considered the criterion for eligibility for public mental health services and for the consideration of out-of-home placement.

Treatment services

Of the 43 children in the follow-up interview sample (total N=95) who had received scores indicating moderate to severe problems (a CAFAS or PECFAS score higher than 30), only 19 (45 percent) were currently receiving mental health treatment. Thirty-three (77 percent) of the parents of the untreated children with moderate to severe problems and all of the parents of untreated children with severe or very severe problems stated that they wanted additional services for their children. These rates did not differ between Spanish-speaking and English-speaking families. Even among children with very severe problems (CAFAS score higher than 70), only 78 percent were reported by parents to be receiving treatment.

Of the 43 interview cases in which the children's scores indicated moderate to severe problems, 25 were reviewed by the multidisciplinary teams. Five of these children were found on interview to be functioning adequately, and the team members agreed with this initial assessment in all five cases. The remaining 20 children were confirmed by the multidisciplinary team to have moderate to severe problems. In this group, a much wider range of statuses was found than had been expected.

Four of the 20 children who had a serious emotional disturbance had not been so identified by their pediatrician. These children were referred for appropriate services. In 16 cases (80 percent), the child's pediatrician knew that the child had behavioral health problems; however, the team members judged treatment to be adequate for only four (20 percent) of these children.

For three of the 20 children (15 percent), it was found that referrals for additional services had lapsed or gone unfulfilled; these referrals were subsequently reinitiated. In four other cases (20 percent), the positive screening prompted clinicians to recommend additional services such as individual counseling and medication for a child and parent counseling. In five cases, no adequate services were reported by the pediatrician to be available to meet the needs of the child and family. Steps were taken to address this need.

Neighborhood Health Plan reviewed the lifetime medical utilization records of the 75 screened children who were enrolled in the plan. This review showed that children who received a positive score on the PSC had an average of 25 percent more outpatient medical visits than children who received a negative score (26.3 visits and 21.1 visits, respectively).

Discussion and conclusions

The PSC proved to be a valid, reliable method for increasing awareness of behavioral health problems in pediatric primary care settings. It was also shown to be a quick, easy, and effective way to screen large numbers of pediatric patients in busy, resource-challenged, and ethnically diverse health centers.

Twenty-five percent of the children in our sample received a positive score on the PSC—a rate comparable to those found in other studies at low-income pediatric clinics in Boston (20,21). Using overall ratings for level of functioning derived from in-depth clinical interviews with parents as the gold standard, the English and Spanish versions of the PSC demonstrated high rates of sensitivity and specificity among school-aged children (93 percent), evidence of their validity with a low-income urban pediatric population. The Spanish version of the checklist was subsequently revised in response to its relatively weaker sensitivity and specificity among pre-school-aged children from Spanish-speaking families.

The estimated level-of-functioning ratings for the whole sample suggest a high rate—20 to 25 percent—of moderate to severe behavioral health impairment among the pediatric population, confirming the need for pediatricians to include an assessment of behavioral health problems within the scope of their work.

The multidisciplinary team meetings provided an important means by which to improve linkages between the health centers' primary care services, behavioral health providers, and health maintenance organizations. Through these case reviews, it was readily apparent that several of the children were seriously dysfunctional and had already been referred multiple times; yet there was a lack of follow-through. Many of the families were so chaotic that traditional models of clinic-based psychotherapy were judged inadequate. In some families, protective or parenting issues were noted that required a significantly different approach.

Perhaps the most compelling evidence in this study for increased attention to pediatric behavioral health problems is the fact that only about one-third of the children with significant behavioral health disorders whose cases were reviewed by the multidisciplinary team were receiving needed services. Even among the most impaired children—those classified as severely to very severely emotionally disturbed—only about one-half were receiving any behavioral health services at all. A reported shortage of existing services due to staff turnover and long waiting lists may account in part for this deficiency. However, for many cases, services deemed to be effective, such as parenting classes and outreach teams to engage resistant families in treatment, were either unavailable locally or unknown.

The results of our study also imply a basic systemic health care problem characterized by inadequate follow-through on referrals by families of children with significant behavioral health disorders and a lack of awareness about, or availability of, needed behavioral health care resources in the primary care setting. This observation suggests the need for enhanced care referral and tracking systems as well as for ongoing education of the pediatric staff about available behavioral health resources.

Although behavioral health programs traditionally focus on the identification and treatment of behavioral health disorders such as depression, anxiety, substance abuse, and conduct disorders, Neighborhood Health Plan is explicitly aware that myriad psychosocial stressors experienced by its members often contribute to the advent or exacerbation of such disorders. Poverty, inadequate housing, domestic abuse, gang violence, readily available illicit drugs, child abuse and neglect, poor nutrition, and inadequate education, among other problems, clearly have a deleterious impact on the overall health of this population, including behavioral health.

Although the amelioration of socially based stressors is not customarily within the purview of health maintenance organizations, Neighborhood Health Plan and its subcontractor, Beacon Health Strategies, collaborate actively with the Massachusetts Division of Medical Assistance (Medicaid), the Department of Mental Health, child welfare agencies, community programs, and other resources in an effort to meet the broader psychosocial needs of its members. In addition, Neighborhood Health Plan pays the health centers with which it has contracts an enhanced rate to support the work of their social services departments in helping members address these psychosocial stressors.

As the next step in improving the early identification and treatment of children at risk, the researchers developed a pediatric preventive behavioral health clinical guideline (26) that incorporates the PSC. This guideline has been adopted by Neighborhood Health Plan. It features a highly specialized intensive case management component for the most impaired children, offering outreach and other nontraditional behavioral health services to meet the needs of the client and the family. Neighborhood Health Plan and Beacon are currently working with Massachusetts General Hospital and the health centers to implement the guideline, with the goal of improving access to appropriate behavioral health services.

An outcome of this study was the development of a pediatric preventive behavioral health care clinical guideline designed to more effectively meet the needs of children at risk We hope the information in the study will be of value to other behavioral health organizations and pediatric practices that are striving to improve the overall health status of children and their families.

Acknowledgments

This research was supported by a grant from Neighborhood Health Plan. The authors thank Kathleen Bennett, M.D., Julie Bertolini, Michael Jellinek, M.D., Jill Lack, John Colburne, Anup Vidyarthy, Alison Hauser, Michele Navon, and Deborah Reiter for their contributions.

Mr. Navon was formerly affiliated with Neighborhood Health Plan in Boston and is currently a senior consultant at Lightbridge, Inc., in Burlington, Massachusetts. Dr. Nelson is vice-president of quality management at Beacon Health Strategies in Woburn, Massachusetts, and lecturer on health services in the department of psychiatry at Harvard Medical School in Boston. Dr. Pagano is instructor of psychology and Dr. Murphy is assistant professor of psychology at the Harvard Medical School. Dr. Pagano is also a research associate and Dr. Murphy is also a staff psychologist at Massachusetts General Hospital in Boston. Address correspondence to Mr. Navon, 8 Jackson Road, Marblehead, Massachusetts 01945 (e-mail, [email protected]).

Enlarge table

Enlarge table