Reliability of the Services Assessment for Children and Adolescents

An estimated 14 to 25 percent of children in the United States meet criteria for a psychiatric diagnosis and are functionally impaired, yet many do not receive mental health services (1,2,3,4,5,6,7). Furthermore, little information is available about the few services that they do receive (8,9,10,11).

The literature on mental health services utilization by children has identified several issues that have yet to be thoroughly investigated. First, in order to examine the predictors of onset and cessation of service use, data must be collected in meaningful units (12). Episodes of care have been used for this purpose, and they are considered to be the boundaries for summing all inputs to the care process by specifying the beginning and ending points of treatment and the course of services used during treatment (13). The Health Insurance Experiment identified the potential importance of using episodes of care as the unit of measure because the investigators noted that different features were related to first visits for service than to subsequent visits (14,15,16). However, few studies have examined episodes of care or their applicability to children's mental health problems.

Second, despite the documented need for additional or more effective mental health services (17,18), exploration of the pathways into, between, and out of mental health services has been inadequate (9,11,19,20,21,22). Only the study by Farmer and colleagues (23) has directly assessed pathways.

Third, instruments used to measure service use need to address the intensity, content, and coordination of care. A number of studies have concluded that coordination of services is necessary but not sufficient for better outcomes (22,24,25,26,27,28). The results of these studies, along with those of studies that examined the effectiveness of protocol-driven services, point to the need to understand both the intensity and the content of care received (29,30).

Finally, few studies have examined how racial identity, acculturation, and cultural mistrust affect help-seeking patterns, yet these factors are important for the development of any type of measurement (31). Programs for children who have serious emotional disturbances need to address the unique cultural characteristics of these children, their families, and their communities. Several studies have found that ethnic minorities underuse particular services, even when service use barriers related to income and availability are reduced or eliminated (32,33,34,35,36,37,38).

We evaluated currently available instruments to determine whether they could be used to obtain key information about children's use of mental health services. We found that the service-use questions used in the Methods for the Epidemiology of Child and Adolescent Mental Disorders study, which was sponsored by the National Institute of Mental Health, could not identify episodes, provide any information on pathways, or differentiate the provider of care from the service setting (8).

The Child and Adolescent Service Assessment (CASA) has demonstrated good to excellent parent and child three-month test-retest reliability (39,40). For parent data, the kappa coefficient was .82 for any inpatient service use, .82 for any outpatient service use, and .62 for use of any school services. Its validity was similar, with a kappa of .84 for any service use. However, in the form that we reviewed, it did not examine the most recent episode of care, it could not distinguish the type of provider from the service setting, and it did not examine pathways into care.

The Services for Children and Adolescents-Parent Interview, which was developed for use in the Multisite Treatment Study for Attention Deficit Hyperactivity Disorder, has no youth version, and it has a three-month time frame. Therefore, it cannot address either lifetime or one-year use.

The Referral Sequence and Problems Interview details the process of entry into mental health services (41); however, it is not designed to distinguish the type of service from the provider and setting or to assess the duration, intensity, or content of services received.

Because we found no instrument that could assess all of the underinvestigated issues identified in the literature review, we constructed a new one by modifying the CASA and adding elements from other instruments. Our instrument, the Services Assessment for Children and Adolescents (SACA), was constructed in modules by service setting and structured such that, with longitudinal assessment over defined, consistent time periods, episodes of care could be identified.

The analyses reported here describe the formal reliability testing of the SACA. Data on its validity, parent-child agreement, and content of care have been published elsewhere (42,43).

Methods

In preparation for a large national study of children's mental health, we examined the reliability and validity of the SACA. Data were collected in St. Louis, Missouri, by investigators from the department of psychiatry at Washington University and in Ventura County, California, by investigators from the University of California, Los Angeles (UCLA), and Rand. The study took place in 1996.

Sample

St. Louis site. The clinical subjects for the study were recruited from a list of children aged 4 to 17 years who had been treated at the inpatient and outpatient clinics (N=88 and 443, respectively) of a local hospital within the previous 12 months. Children who had a diagnosis of mental retardation were not included in the study. Additional subjects, who were not service users, were recruited from a list of children who attended public schools and a list of children who attended a local day care facility (N=320).

A letter was sent to the parents of all children in both the clinical and nonclinical groups. The parents were invited to participate in the study by returning an enclosed response card that would allow a research assistant to contact them to explain the study. To meet our goal of approximately half of the study participants being African American, 26 families were recruited by asking African-American participants to identify friends and neighbors to contact.

A total of 146 parent-child dyads completed the test and retest interviews. Most of the parents were female (91 percent), and most were the child's biological mother (85 percent). Seventy-five parents (51 percent) were European American, and 70 (47 percent) were African American. Ninety-two parents (63 percent) had more than a high school education. By design, the children ranged in age from nine to 17 years.

UCLA site. A list of addresses of households with children aged 4 to 17 years who had used mental health services within the previous 12 months was obtained from the Ventura County Mental Health management information system. Members of the households were invited to participate and were screened for language and ethnic eligibility. For each of the eligible households, an attempt was made to randomly recruit from the surrounding neighborhood a household matched for ethnicity and children's age group in which no child had used mental health services during the previous 12 months.

A total of 470 households in which children had used services were invited to participate. Of these, 159 (34 percent) were eligible, and 94 (59 percent) agreed to participate. A total of 527 households in which children had not used services were invited to participate; of these, 108 (20 percent) were eligible, and 51 (47 percent) agreed to participate. A total of 106 parent-child dyads completed the test interview, and 91 dyads completed the retest interview. Like their St. Louis counterparts, participants were mostly female (90 percent) and were typically the child's biological mother (82 percent). Fifty-five (60 percent) were European American, and 25 (27 percent) were Hispanic; two were in the category "other," and seven were biracial. Just over half (47 participants, or 52 percent) had more than a high school education.

Procedure

At test and retest the SACA was administered separately to parents and to children in their homes by trained lay persons. Retests were conducted within one week, with a range of four to 14 days. Randomly selected interviews were audited during the interview process. Interviewers were blinded by having each test and retest conducted by a different interviewer who was not informed of any previous responses. The instrument took between two minutes and 45 minutes to administer. The initial module (see below) took two to 15 minutes.

Instrument

The SACA-Parent Version, in the form tested, begins with a module that gathers data on a child's lifetime and past-year use of 30 service settings that are grouped into three broad areas: inpatient, outpatient, and school. Following the initial module are 30 individual modules that ask more specific questions about each of the settings. The initial module can be used independently or in conjunction with the individual modules. Because the instrument was developed for use across the United States, general service setting terminology is used. To avoid an excessively long instrument, the amount of information requested varies; only specialty mental health settings and school counseling are questioned in detail.

Queries about reasons for seeking treatment for the child are designed to pinpoint specific problems rather than to elicit global descriptions (for example, depression) or life events (for example, a divorce). Questions about treatment are also specific and ask respondents about child-related interventions, such as therapy or counseling, medications, case management, and evaluation or testing, as well as family-oriented services. The temporal order in which the different service types were used in the past 12 months is then ascertained, and the modules are arranged in that order to facilitate the unfolding of the respondent's story.

The SACA also contains two specialized modules. The final services module is asked of everyone, regardless of whether they reported using services. For respondents who report the use of a service, a series of questions ascertains the most and least beneficial service, whether additional services were felt to be needed, and, if so, why any additional services were not used. If no service use is reported, respondents are asked whether they thought services were needed, the reason for needing services, and why services weren't used.

The other specialized module is the medication module, which was designed for parent report only. It elicits information about prescriptions for psychoactive medications, including dosage and duration of use, and insurance coverage for any psychoactive medication.

Analysis

All statistical analyses were conducted with SAS (version 6.12). After the data were carefully reviewed to identify any out-of-range values or inconsistencies, the test-retest analyses were undertaken. As is customary in epidemiologic investigations of mental health issues, kappa statistics were calculated to evaluate the agreement between categorical data and to correct for chance associations (44,45,46). Kappa values over .75 represent excellent agreement, values between .40 and .75 indicate good agreement, and values less than .4 indicate poor agreement (47,48). The reliability of continuous data was evaluated with the intraclass correlation coefficient with the same agreement categories (49,50).

Results

Of the 237 adult respondents (all test and retest pairs), 164 (69 percent) reported at least one occasion of service use at some time during the child's lifetime; 73 (31 percent) reported no service use. The majority of reported service contacts were in an outpatient setting. In the first test, 360 of the reported 591 service contacts (61 percent) were in outpatient settings, and of these, 240 were in a mental health clinic or a private practitioner's office (67 percent). In the retest, 342 of the reported 570 contacts were in outpatient settings (60 percent), with 231 (68 percent) taking place in a mental health clinic or a private practitioner's office. School-based services accounted for only 165 and 160 (28 percent) of the reported service settings in both tests.

The same pattern was seen in service use by setting, with outpatient settings accounting for the majority of service settings—232 of the 381 contacts reported during the first test (61 percent), and 226 of the 382 contacts reported during the retest (59 percent).

Reports of the type of service use settings did not vary significantly between the first test and the retest for either lifetime or 12 month. Furthermore, with the exception of four types of outpatient settings—mental health clinics, in-home services, private mental health practitioner's office, and medical doctor's office—and four school settings, the use of individual service settings for the previous 12 months was reported too infrequently for reliable calculation of the kappa statistic (51). Therefore, all analyses are reported combined across the two sites and in six superordinate categories: any use, any inpatient use, any specialty mental health inpatient use (psychiatric hospital, psychiatric unit in a general hospital, drug or alcohol unit, or residential treatment center), any outpatient use, any specialty mental health outpatient use (mental health clinic, day treatment, drug or alcohol clinic or psychologist, social worker, or counselor), and any school use.

Module reliability

Initial module. As shown in Table 1, for parents, both lifetime and 12-month test-retest reliability figures are excellent, with kappas for data combined across two sites ranging from .82 to .94 for services ever used and .75 to .86 for service use within the previous 12 months. Information about specific items in the initial module showed that the reliability of the question about overall pathways was moderate to excellent when the item asked about the person whom the respondent spoke to before using services but poor when the item asked about the advice given by that person (data not shown).

The children's test-retest reliability increased with age, with those aged 11 years and older demonstrating moderate to excellent reliability. For these children, the kappas for reported services ever used ranged from .64 to .96, and for self-reported services used within 12 months the kappas ranged from .63 to .77 (Table 2). The results for the younger children—those in our sample aged nine and ten years—were adequate but less encouraging, with kappas for lifetime use ranging from .41 to .64, and for 12-month use, a kappa of .64 (data not shown). These children had difficulty distinguishing between closely related service settings such as psychiatric hospitals and psychiatric units in general hospitals.

To determine whether memory for the SACA questions contributed to the excellent test-retest results, we compared the results of respondents who were retested within seven days with the results of respondents who were retested after more than seven days. For all categories, there were minimal or no differences in kappa scores between the two groups, with no consistent trend favoring scores obtained within seven days. In the ever-using-services category, for example, kappas ranged from .94 for any use to .84 for a school use for respondents who were retested within seven days and .94 for any use to .80 for a school use for those who were retested after more than seven days.

Individual service setting module. Parents' and older children's reports of reasons for use of services were highly reliable between initial testing and retesting, indicating that people who use services do so for well-developed reasons. Kappa values for parents' reliability were 1 for psychiatric hospitals, .89 for mental health clinics, and .87 for mental health professionals. Kappa values for older children were .70 for psychiatric hospitals, .80 for mental health clinics, and .75 for mental health professionals. Younger children reported reasons for seeking care at psychiatric hospitals or mental health clinics too infrequently for reliability figures to be obtained; their kappa value for seeking care from mental health professionals indicated only moderate reliability (kappa=.5).

Reliability for parents' reports of intensity of services, as measured by the number of inpatient admissions and the number of visits for the most recent outpatient mental health clinic use, was uneven. The number of hospital admissions showed moderate reliability (intraclass correlation coefficient=.44), but the number of clinic visits was unreliable (ICC=.03). The reliability of older children's reports was poor for both inpatient psychiatric hospital admissions (ICC=.27) and number of visits to mental health clinics (ICC=.24).

The reliability of parents' reports of the types of treatments received in any one setting were generally moderate to excellent for treatments offered by mental health clinics (kappas ranging from .57 to .8) and poor to moderate for treatments offered by mental health professionals (kappas ranging from .3 to .55).

The number of children reporting specific types of treatments in any one setting was generally too low to be used as an indicator of reliability, with the exception of treatment from a mental health professional. In this case, the reliability of older children's reports of child therapy, medications, evaluation or testing, and family therapy were low to moderate (kappas ranging from .38 to .54) (data not shown).

The only benefit question for which the sample was large enough for analysis was parents' assessments of the benefits of mental health clinic use. The reliability of this item was good (kappa=.77 for mental health clinic use). The number of reports of follow-up services and cultural sensitivity were too small for reliable analysis. The financial questions showed very poor reliability (ICC ranging from 0 to .25 for out-of-pocket costs) (data not shown).

Medications module. Reliability for questions about any use of medications within the past six months was generally very good among adult respondents (kappa=.96) (Table 3). Methylphenidate was the single most prevalent medication named—41 of 149 respondents (28 percent). Central nervous system stimulants (38 percent) and antidepressants (41 percent) were about equally represented. During retesting, parents were able to name the same drugs about 87 percent of the time.

The reliability of reported dosage and number of days that medications were used were analyzed separately for parents who reported the use of one medication (56 percent) and those who reported the use of more than one medication (30 percent). The percent agreement as to whether the child was still taking the medication was excellent. Agreement was also excellent for knowledge of whether health insurance paid for the medication (kappa was 1 for parents who reported use of one medication and .73 for parents who reported use of more than one medication), amount paid out of pocket (ICC=.96 and .96, respectively), the dose of medication given (ICC=.88 and .99), and the number of times per day the child took medication (ICC=.86 and .85). The reliability of the number of days in the previous month that the child took medication was poor (ICC=.41 and .06) (data not shown). Questions about dosage schedules, particularly the questions that attempted to capture "drug holidays" or varying weekly or daily schedules, proved problematic.

Discussion

This test-retest reliability study showed that the SACA-Parent Version had uniformly excellent reliability for both lifetime and 12-month service use. When we compared these data with the parent test-retest reliability data reported for the CASA, we found that 12-month reliability data for the SACA indicate reasonable recall over a longer period. We thought it was important to assess a 12-month period because one year is a common time frame for gathering diagnostic and payment information, both in provider settings and in services research.

The structure of the SACA—with its initial module—appears to prevent the common problem of attenuation of responses in instruments that have a repetitive format (52). Furthermore, an examination of test-retest results stratified by the length of recall period—one week or less compared with more than one week—indicated consistently high agreement over the two time spans. Although these findings do not definitively rule out memory as a reason for the high reliability, the finding that a longer time between test and retest did not affect the SACA's reliability provides evidence that the respondents were remembering the services used rather than just the questions asked.

The data for the children presented a more varied picture. For children aged 11 years and older, the kappas for lifetime and 12-month use were in the good to excellent range. These results are consistent with the findings from child reports for the CASA (39).

The reliability figures for children aged nine and ten years were in the moderate range and were considerably lower than those for the older children. Our examination of the younger children's responses suggested considerable confusion about several of the service use settings; thus, we question whether children who are younger than 11 are reliable reporters of their own use of mental health services. These results are similar to those found when using the Diagnostic Interview Schedule for Children with children aged nine to 11 years (53).

Types of treatments—with the exception of case management and other support services, such as help with rent and food—had excellent test-retest reliability for adults and children, which suggests that services that are clearly demarcated are reliably reported. The experience with the SACA suggests that researchers who are gathering information about case management should focus their inquiries on specific activities.

Similarly, assessments of benefit and follow-up were usually reported quite reliably by both parents and children. Parents and children answered questions about cultural sensitivity too infrequently to enable us to draw firm conclusions. This lack of information is unfortunate, because perceptions of the cultural responsiveness of services may predict how well clients maintain contact with their service providers (8).

Results for the medications module demonstrate that information about dosage, form, and frequency of use as well as insurance coverage can be reliably reported. Detailed information about dosages on different days was difficult to report reliably and time-consuming to collect. The question about the start date of medications appeared to have caused some confusion. Better information might be elicited if the instrument were revised to ask how long the child had been on medication in terms of months and years.

Conclusions

The SACA reliability study demonstrates that parents and children aged 11 years and older can report both lifetime and 12-month use of mental health services with good to excellent reliability. The SACA can be used to collect some information about mental health services that has been missing in previous research and thus greatly expand the knowledge base in children's mental health services.

Acknowledgments

This work is a product of the multisite study of Mental Health Services Use, Needs, Outcomes, and Costs in Child and Adolescent Populations supported by the National Institute of Mental Health; the Administration for Children, Youth, and Families; the Center for Mental Health Services; the National Institute of Child Health and Human Development; and the Department of Education.

Dr. Horwitz is with the departments of epidemiology and public health and the Child Study Center at Yale University School of Medicine and the Institute for Social and Policy at Yale. Dr. Hoagwood is associate director for child and adolescent mental health services at the National Institute of Mental Health in Rockville, Maryland. Dr. Stiffman is with the George Warren Brown School of Social Work at Washington University in St. Louis. Dr. Summerfeld is with the department of psychology at Michigan State University in East Lansing. Dr. Weisz is with the department of psychology at the University of California, Los Angeles (UCLA). Ms. Bean is with Rand in Santa Monica, California, and with UCLA. Dr. Costello is with the department of psychiatry and behavioral sciences at Duke University Medical Center in Chapel Hill, North Carolina. Dr. Rost is with the department of family medicine at the University of Colorado in Denver. Dr. Cottler is with the department of psychiatry at Washington University School of Medicine. Dr. Leaf is with the department of mental hygiene at Johns Hopkins University in Baltimore. Ms. Roper is with the division of services and intervention research at the National Institute of Mental Health in Bethesda, Maryland, where Dr. Norquist is division director. Address correspondence to Dr. Horwitz at Yale University School of Medicine, Department of Epidemiology and Public Health 60 College Street, Room 310, New Haven, Connecticut 06520 (e-mail, [email protected]).

Enlarge table

Enlarge table

Enlarge table