0
Get Alert
Please Wait... Processing your request... Please Wait.
You must sign in to sign-up for alerts.

Please confirm that your email address is correct, so you can successfully receive this alert.

1
Article   |    
Use of Peer Ratings to Assess Sociability Among Inpatients With Severe Psychiatric Disorders
David L. Penn, Ph.D.; Dorie Reed, Ph.D.; Mary Sullivan, M.S.W.; Will Spaulding, Ph.D.
Psychiatric Services 1998; doi:

OBJECTIVE: A critical component of inpatient treatment for persons with severe psychiatric disorders is an evaluation of their social impairments. Most existing methods for such evaluations involve staff input and can be both time-consuming and expensive. This paper reports on the use of peer ratings as a method for assessing sociability, an aspect of social functioning in this clinical population. METHODS: Thirty-two inpatients with severe psychiatric disorders who had spent an average of 18 months on an inpatient unit in a state facility rated the popularity of their inpatient peers by completing a 7-point scale measuring how much they enjoyed visiting with each patient. The reliability of peer ratings and their association with staff ratings of patients' behavior on the unit were assessed. Results and conclusions: The peer ratings had excellent test-retest reliability and were highly associated with staff members' independent evaluations of patients' behavior. Peer ratings appear to have promise as a measure of social functioning among inpatients with severe psychiatric disorders.

Abstract Teaser
Figures in this Article

Persons with schizophrenia and other severe psychiatric disorders often have impairments in social functioning that may be independent of positive and negative symptoms (1). These impairments have been shown to predict relapse rate and community functioning among outpatients (2,3,4). Given the prognostic importance of social functioning among patients with severe psychiatric disorders, procedures for identifying persons at risk for social failure and rehospitalization are needed. Once these individuals are identified, they may be able to benefit from family interventions and social skills training focused on teaching the skills necessary for maintaining community tenure and improving quality of life (5,6).

An important aspect of inpatient treatment planning is evaluation of patients' social skills and deficits. Numerous methods for evaluating social competence of inpatients with severe psychiatric disorders exist, including role play, semi-structured interviews, self-report measures, and naturalistic observation (7,8). These techniques have different strengths and weaknesses that must be considered in selecting an assessment tool. For example, self-report measures are generally less time intensive for both staff and patients, but their psychometric properties may be questionable, especially when they are used with inpatients with schizophrenia (7). Conversely, role play allows direct assessment of overt behavior and has sound psychometric properties (2), but considerable staff training may be needed to administer the intervention and to code the results. Furthermore, if the role play does not match significant problem areas in the patient's life, it may be viewed as contrived or not externally valid.

Naturalistic observation affords advantages not provided by role play or self-report. As described by Mueser and Sayers (7), naturalistic observation tends to be an ecologically valid procedure for evaluating a wide range of social behaviors. Furthermore, information may be obtained about how a number of others, not just a single individual in a role play, respond to the target patient. Therefore, the reciprocal relationship between the person and the social environment may be assessed. Some examples of naturalistic rating scales for use with inpatients include the Time-Sample Behavioral Checklist (9), the Nurse's Observation Scale for Inpatient Evaluation (NOSIE-30) (10), the rehabilitation evaluation scale of Baker and Hall (REHAB) (11), and the recently developed Social-Adaptive Functioning Evaluation (12).

Staff-rated observation of patients' naturalistic behavior has several disadvantages. First, the person's behavior may be more reflective than naturalistic because the process of being observed by staff could produce behaviors that are not typically demonstrated by the person in most contexts. For example, a male patient who tends to act sexually inappropriately with female patients may act appropriately in the presence of staff. Second, a certain amount of resources must be devoted to initiating and maintaining a staff-rated observation system. Such resources include staff training and periodic evaluation of observer reliability. Unfortunately, some treatment facilities may not have the financial support or the commitment from upper-level administration to implement a staff-rated observation system.

Peer assessment is an alternative procedure for evaluating naturalistic behavior that circumvents many of the problems listed above. Peer assessment refers to a set of techniques in which the peer group is the primary source of information about the group members' behavior. The two most popular peer assessment techniques are peer nominations and peer ratings, which have been used to measure the social competence of children and adolescents (13,14,15). Peer nomination procedures ask children to identify, or nominate, a certain number of peers who meet some interpersonal standard. For example, a child might be asked to nominate his or her three best friends (positive nominations) and three least liked friends (negative nominations). Peer ratings typically require the children to rate all other children in a group using Likert-type scales. For example, a child may be asked to rate how much he or she likes to play with each of the other children in a group on a scale from "very much" to "not at all."

Both techniques have good psychometric properties, although peer ratings tend to have better reliability than nomination procedures (14,15). Whether these two techniques measure similar constructs is controversial (13). However, it appears that peer nominations assess the level of popularity, and peer ratings measure overall acceptability among the peer group. Bukowski and Hoza (13) have argued that peer ratings, relative to nomination procedures, represent a composite measure of popularity, by virtue of incorporating information from both the "the acceptance and rejection dimensions of the popularity construct." Therefore, peer rating strategies may provide the most comprehensive data concerning an individual's popularity within his or her peer group.

This paper reports the results of an evaluation of peer ratings as a method for assessing the social functioning of inpatients with chronic schizophrenia and other severe psychiatric disorders. The findings focus on two issues. First, the reliability of peer ratings for use in this clinical population was investigated. Second, the construct validity of peer ratings was evaluated by assessing their association with staff ratings on a standard observational measure, the NOSIE-30. We hypothesized that peer ratings would be a reliable method for assessing the sociability of persons with severe psychiatric disorders and that they would be positively associated with adaptive behavior on the unit and negatively associated with maladaptive behavior.

+

Participants

The study participants were 32 patients treated on an inpatient psychiatric unit at the community transition program of the Lincoln Regional Center, a state facility in Lincoln, Nebraska, during the period from June 1992 to July 1992. The sample included 19 male patients and 13 female patients. Twenty-eight patients were Caucasian, three were African American, and one was Hispanic. Their mean±SD age was 34.9±7.8 years.

The majority of the patients had an axis I diagnosis of major psychiatric disorder made by a research psychiatrist. Twenty-six patients had a diagnosis of schizophrenia, four had schizoaffective disorder, one had delusional disorder, and one had personality disorder not otherwise specified. A more detailed discussion of the diagnostic procedures is presented elsewhere (16). The subjects had spent an average of 18±25.8 months on the unit. Their average daily dosage of medication was 1,780± 1,894.6 mg of chlorpromazine equivalents.

Patients in the sample had given their consent to participate in the study as part of a larger research project on the cognitive rehabilitation of patients with schizophrenia. Patients were informed that the peer ratings were optional and that refusal to participate in that aspect of the study would not affect their treatment on the unit.

+

Measures

+

Staff ratings.

The study participants' behavior on the unit was rated by staff members using the NOSIE-30 (10). The NOSIE-30 is checklist of 30 behavioral items completed by staff based on observation of patients over the previous 72 hours. The frequency of each behavior is rated on a 5-point Likert-type scale from "never" to "always."

In this study, NOSIE-30 ratings were made by staff psychiatric technicians who were blind to the study hypotheses. The ratings were collected routinely on the unit as part of the rehabilitation assessment and treatment battery. To control for minor fluctuations in patients' functioning, the weekly NOSIE-30 ratings were averaged over a four-week period.

A total of 14 staff members—eight men and four women, all of whom were Caucasian—were involved in rating patients' behavior. In general, each patient was rated for at least three of the four weeks by the psychiatric technician who was assigned to provide routine care for the patient. The second rater came from the remaining pool of day or evening shift psychiatric technicians.

NOSIE-30 data are summarized by six subscales—social competence, social interest, personal neatness, irritability, psychoticism, and psychomotor retardation. Periodic analyses of interrater reliability revealed Pearson correlations of .68 to .72 for all scales.

+

Peer popularity ratings.

Peer popularity ratings were obtained through a survey that listed the names of all patients who remained on the unit for at least two months. Patients were instructed to rate how much they enjoyed visiting with each patient using a 7-point Likert-type scale anchored by the statements "not at all" and "very much." Patients were instructed to leave items blank for any patients they did not know.

Thirty-six staff members, including psychiatric technicians, nurses, social workers, therapists, and other support staff in daily contact with the patients, also rated each patient's popularity using the 7-point Likert-type scale. Twenty female staff members and 16 male staff members, all of whom were Caucasian, completed the ratings.

The two indexes of patients' popularity—peer popularity and staff popularity—were derived by summing the ratings across raters and then dividing by the total number of raters. This procedure resulted in a fixed range of ratings from 1 to 7.

Using another 7-point scale, patients rated how much they believed the staff members and other patients enjoyed visiting with them. These ratings constituted patients' prediction of their own popularity as rated by staff members and other patients. Finally, a measure of each patient's overall interest in visiting with others was derived by averaging the patient's ratings of how much he or she enjoyed visiting each of the other patients on the unit.

To determine reliability, patients were asked to complete the peer ratings a second time one week after the first administration. Test-retest reliabilities were obtained by calculating Pearson r correlations for the patients' ratings at time 1 and time 2. A second administration of the survey was not conducted with the 36 staff members. After test-retest reliability was established, the patient's ratings were calculated as an average of the two scores. All analyses were conducted using peer and behavior ratings that were done during the same month.

+

Reliability analyses and descriptive statistics

Analyses of test-retest reliability of patients' ratings showed that all ratings were highly reliable. Correlation coefficients were .93 for patients' ratings of the popularity of their peers, .79 for their predictions of staff members' rating of their popularity, .92 for their predictions of other patients' ratings of their popularity, .91 for their level of interest in others, and .95 for items left blank, indicating that the respondent did not know the patient listed on the survey form. All correlations were significant (p<.001).

Mean ratings of patients' popularity by peers and staff members and patients' mean scores on the NOSIE-30 staff-rated measure of behavior are summarized in T1. In general, patients were rated by peers and staff in the average range of popularity. Furthermore, staff ratings of patients' behavior were above average, compared with other long-term-stay patients with severe psychiatric disorders (17).

+

Correlational analyses

Peer and staff popularity ratings were significantly correlated (r=.75, p< .001), indicating that patients and staff members tended to perceive popularity similarly. T2 summarizes the relationships between peer and staff ratings of popularity with NOSIE-30 ratings of behavior on the unit. Most of the peer and staff popularity ratings were significantly associated with staff ratings of patients' behavior. After the Bonferroni correction was applied to reduce type I error, peer and staff popularity ratings were significantly associated with ratings on the NOSIE-30 subscales measuring social interest and neatness, and staff popularity ratings were negatively associated with the NOSIE-30 subscale measuring irritability.

+

Correlates of patients' popularity

Two factors that could influence patients' popularity are the number of patients not known by the respondent and the length of time an individual has been on the unit. Correlational analyses revealed a significant negative association between the number of patients the respondent did not know and popularity (r=-.39, p<.05); greater popularity was associated with knowing more patients by name. However, the relationship between patients' popularity and time on the unit was not significant.

Finally, correlational analyses were conducted to determine whether patients' popularity was associated with their overall interest in visiting with others and their ability to predict others' interest in visiting with them. The only significant correlation was between patients' popularity and the prediction of patients' ratings (r=.39, p<.05); greater popularity was associated with higher expectations that other patients would want to spend time with the respondent. The association between popularity and interest in visiting with others was not significant.

The study had two major findings. First, popularity of inpatients with severe psychiatric disorders can be reliably assessed using a survey form. Second, peer popularity ratings were highly correlated with staff popularity ratings, as well as with staff members' independent evaluations of patients' behavior on the unit. These findings suggest that peer ratings of popularity may be a useful measure of a component of social functioning among inpatients with severe psychiatric disorders.

The consistently high correlations between peer popularity ratings and staff ratings of behavior suggest that staff ratings may partly reflect how much staff like the patients. This finding may have clinical significance, as previous research has demonstrated that the "likability" of patients is associated with outcome for individuals with psychiatric disorders (18). However, the role of likability in staff ratings of patients' behavior should not be overstated, as peer popularity ratings accounted for no more than 31 percent of the variance in any one staff-rated index of behavior. The remainder of the variance in behavior is likely to be accounted for by factors such as patients' specific behaviors, for example, prosocial behaviors, and by perceived physical attractiveness, which is related to social skill in this population (19).

What are the clinical implications of the findings presented in this paper? From an economic perspective, the use of peer ratings appears to be both cost- and time-effective. Patients completed the survey in a timely manner with little burden on staff resources. The patients did not object to completing the survey, nor were they overly concerned about how others would rate them.

A second implication for clinical work, which has been discussed by Hansen and colleagues (20), is that peer ratings may be useful for determining which types of behaviors are associated with popularity among particular patient subgroups—those identified by diagnostic group or gender, for example—and thus peer ratings could be used as outcome measures for social skills interventions. However, Hansen and associates (20) have shown that peer ratings, at the nomothetic or group level, may be too stable to detect subtle changes in social skills. For example, a summary score based on peer ratings of numerous individuals may not be sensitive to a change in a few ratings as a function of improved social skills. Thus an idiographic approach should be adopted in addition to a nomothetic approach when using peer ratings as an outcome measure for a psychosocial treatment.

The utility of an idiographic approach to peer ratings for treatment planning may be illustrated by the following case vignette. After patients completed the peer ratings, the researchers noticed that one of the respondents knew almost none of the other patients by name, despite having been on the unit for more than a year. This finding led to some concern that the patient's social isolation would interfere with psychiatric rehabilitation.

One of the authors (DR) developed an intervention to help this patient learn the names of others on the unit. Photographs of patients and direct-care staff members were taken with their permission, and a set of five photographs, each accompanied by the person's name, were presented to the patient once a week. After three months, the patient learned the names of all the patients and staff members on the unit, as well as the roles of various staff members. After this intervention, the patient began to greet other patients and staff members by name and to show signs of knowing which staff members to approach for particular kinds of assistance or information. The patient's tendency to become isolated from others may never have been brought to staff attention if not for the peer ratings.

Two unexpected findings emerged from the study. First, the association between peer ratings and social competence was significant only at the conventional significance level, which was somewhat surprising because peer ratings were considered a component of social functioning. However, the items that make up the NOSIE-30 social competence subscale—for example, "has to be reminded what to do," "has to be told to follow hospital routine"—suggest that this scale may more readily measure ability to independently follow a schedule rather than social performance, per se.

Second, the association between patients' interest in visiting with others and popularity ratings by peers was not statistically significant. This finding suggests that patients who state that they enjoy spending time with others may not necessarily visit with others frequently on the unit. These individuals may particularly benefit from social skills training, which could teach them methods to convert their interest in others to actual behavioral contacts.

The study had a number of limitations. First, although the NOSIE-30, the measure of behavior on the unit, has established validity and reliability in use with persons with severe mental illness, it provides a better assessment of general social behavior than of specific social skills. Therefore, the construct validity of peer ratings may be enhanced by including assessments of actual social skills through role plays or other formats and comparing the results to ratings by peers. Second, although the study provides evidence of a cross-sectional association between peer ratings and behavior on the unit, it does not speak to the potential predictive validity of peer ratings. Future research should examine whether peer ratings obtained during an inpatient stay predict patients' subsequent relapse rate. Finally, the generalizability of the findings to other treatment settings, such as a day program, is unknown. However, participants who have sustained contact with one another should be able to reliably complete peer ratings.

The study results suggest that peer ratings may provide reliable, and perhaps clinically significant, information about the social functioning of inpatients with severe psychiatric disorders. In addition to providing an alternative method for assessing a component of social behavior—sociability—the findings indicate an additional method for involving persons with severe mental illness in the treatment process. Finally, as illustrated by our case vignette, peer ratings may be useful in identifying gaps in service delivery. The ratings may help staff discern individual patients' sense of social isolation, which is often difficult to detect by traditional assessment procedures and which may not be addressed by routine interventions.

Change of Address

Authors of papers under peer review or being prepared for publication in Psychiatric Services are reminded to notify the editorial office of any changes in address. Please call the editorial office at 202-682-6070, or send updated information by fax to 202-682-6189 or by e-mail to psjournal@psych.org

The authors thank Diane Quartarolo.

Dr. Penn is assistant professor in the department of psychology at Louisiana State University, 236 Audubon Hall, Baton Rouge, Louisiana 70803-5501 (e-mail, dlpenn@unix1.sncc.lsu.edu). Dr. Reed and Ms. Sullivan are affiliated with the community transition program of Lincoln Regional Center in Lincoln, Nebraska. Dr. Spaulding is with the department of psychology at the University of Nebraska in Lincoln.

 
Anchor for JumpAnchor for JumpAnchor for Jump
Table 1.

Mean ratings of patients' popularity by patients (N=32) and staff members and staff members' mean ratings of patients' behavior

 
Anchor for JumpAnchor for JumpAnchor for Jump
Table 2.

Correlations between staff members' ratings of patients' behavior and patients' and staff members' ratings of patients' popularity

Lenzenweger MF, Dworkin RH: The dimensions of schizophrenia phenomenology: not one or two, at least three, perhaps four. British Journal of Psychiatry 168:432-440,  1996
 
Bellack AS, Morrison RL, Wixted JT, et al: An analysis of social competence in schizophrenia. British Journal of Psychiatry 156:809-818,  1990
 
Perlick D, Stastny P, Mattis S, et al: Contribution of family, cognitive, and clinical dimensions to long-term outcome in schizophrenia. Schizophrenia Research 6:257-265,  1992
 
Sullivan G, Marder SR, Liberman RP, et al: Social skills and relapse history in outpatient schizophrenia. Psychiatry 53:340-345,  1990
 
Bellack AS, Mueser KT: Psychosocial treatment for schizophrenia. Schizophrenia Bulletin 19:317-336,  1993
 
Penn DL, Mueser KT: Research update on the psychosocial treatment of schizophrenia. American Journal of Psychiatry 15:607-617,  1996
 
Mueser KT, Sayers MD: Social skills assessment, in Schizophrenia: An Overview and Practical Handbook. Edited by Kavanagh DJ. London, Chapman & Hall, 1992
 
Phelan M, Wykes T, Goldman H: Global function scales. Social Psychiatry and Psychiatric Epidemiology 29:205-211,  1994
 
Paul G, Menditto AA: Effectiveness of inpatient treatment programs for mentally ill adults in public psychiatric facilities. Applied and Preventive Psychology 1:41-63,  1992
 
Honigfeld G, Gillis RD, Klett CJ: NOSIE-30: a treatment ward behavior scale. Psychological Reports 19:180-182,  1966
 
Baker R, Hall JN: REHAB: a new assessment instrument for chronic psychiatric patients. Schizophrenia Bulletin 14:95-113,  1988
 
Harvey PD, Davidson M, Mueser KT, et al: The Social-Adaptive Functioning Scale (SAFE): a rating scale for geriatric psychiatric inpatients. Schizophrenia Bulletin 23:131-145,  1997
 
Bukowski WM, Hoza B: Popularity and friendship, in Peer Relationships in Child Development. Edited by Berndt TJ, Ladd GW. New York, Wiley, 1989
 
Dygdon JA: Peer-based assessment in the study of children's social competence and social skills, in Progress in Behavior Modification, vol 23. Edited by Hersen M, Eisler RM, Miller PM. Newbury Park, Calif, Sage, 1988
 
Gresham FM, Little SG: Peer-referenced assessment strategies, in Handbook of Child and Adolescent Assessment. Edited by Ollendick TH, Hersen M. General psychology series vol 167. Boston, Allyn & Bacon, 1993
 
Penn DL, Mueser KT, Spaulding WD, et al: Information processing and social competence in chronic schizophrenia. Schizophrenia Bulletin 21:269-281,  1995
 
Philip AE: A note on the Nurse's Observation Scale for In-patient Evaluation (NOSIE). British Journal of Psychiatry 122:595-596,  1973
 
Sussman S, Mueser KT, Rosen AJ, et al: The relation of interpersonal attraction to clinical adjustment in psychiatric patients. International Journal of Social Psychiatry 29:134-139,  1983
 
Penn DL, Mueser KT, Doonan R: Physical attractiveness in schizophrenia: the mediating role of social skill. Behavior Modification 21:78-85,  1997
 
Hansen DJ, Nangle DW, Ellis JT: Reconsideration of the use of peer sociometrics for evaluating social skills training: implications of an idiographic assessment of temporal stability. Behavior Modification 20:281-299,  1996
 
Anchor for JumpAnchor for JumpAnchor for Jump
Table 1.

Mean ratings of patients' popularity by patients (N=32) and staff members and staff members' mean ratings of patients' behavior

Anchor for JumpAnchor for JumpAnchor for Jump
Table 2.

Correlations between staff members' ratings of patients' behavior and patients' and staff members' ratings of patients' popularity

+

References

Lenzenweger MF, Dworkin RH: The dimensions of schizophrenia phenomenology: not one or two, at least three, perhaps four. British Journal of Psychiatry 168:432-440,  1996
 
Bellack AS, Morrison RL, Wixted JT, et al: An analysis of social competence in schizophrenia. British Journal of Psychiatry 156:809-818,  1990
 
Perlick D, Stastny P, Mattis S, et al: Contribution of family, cognitive, and clinical dimensions to long-term outcome in schizophrenia. Schizophrenia Research 6:257-265,  1992
 
Sullivan G, Marder SR, Liberman RP, et al: Social skills and relapse history in outpatient schizophrenia. Psychiatry 53:340-345,  1990
 
Bellack AS, Mueser KT: Psychosocial treatment for schizophrenia. Schizophrenia Bulletin 19:317-336,  1993
 
Penn DL, Mueser KT: Research update on the psychosocial treatment of schizophrenia. American Journal of Psychiatry 15:607-617,  1996
 
Mueser KT, Sayers MD: Social skills assessment, in Schizophrenia: An Overview and Practical Handbook. Edited by Kavanagh DJ. London, Chapman & Hall, 1992
 
Phelan M, Wykes T, Goldman H: Global function scales. Social Psychiatry and Psychiatric Epidemiology 29:205-211,  1994
 
Paul G, Menditto AA: Effectiveness of inpatient treatment programs for mentally ill adults in public psychiatric facilities. Applied and Preventive Psychology 1:41-63,  1992
 
Honigfeld G, Gillis RD, Klett CJ: NOSIE-30: a treatment ward behavior scale. Psychological Reports 19:180-182,  1966
 
Baker R, Hall JN: REHAB: a new assessment instrument for chronic psychiatric patients. Schizophrenia Bulletin 14:95-113,  1988
 
Harvey PD, Davidson M, Mueser KT, et al: The Social-Adaptive Functioning Scale (SAFE): a rating scale for geriatric psychiatric inpatients. Schizophrenia Bulletin 23:131-145,  1997
 
Bukowski WM, Hoza B: Popularity and friendship, in Peer Relationships in Child Development. Edited by Berndt TJ, Ladd GW. New York, Wiley, 1989
 
Dygdon JA: Peer-based assessment in the study of children's social competence and social skills, in Progress in Behavior Modification, vol 23. Edited by Hersen M, Eisler RM, Miller PM. Newbury Park, Calif, Sage, 1988
 
Gresham FM, Little SG: Peer-referenced assessment strategies, in Handbook of Child and Adolescent Assessment. Edited by Ollendick TH, Hersen M. General psychology series vol 167. Boston, Allyn & Bacon, 1993
 
Penn DL, Mueser KT, Spaulding WD, et al: Information processing and social competence in chronic schizophrenia. Schizophrenia Bulletin 21:269-281,  1995
 
Philip AE: A note on the Nurse's Observation Scale for In-patient Evaluation (NOSIE). British Journal of Psychiatry 122:595-596,  1973
 
Sussman S, Mueser KT, Rosen AJ, et al: The relation of interpersonal attraction to clinical adjustment in psychiatric patients. International Journal of Social Psychiatry 29:134-139,  1983
 
Penn DL, Mueser KT, Doonan R: Physical attractiveness in schizophrenia: the mediating role of social skill. Behavior Modification 21:78-85,  1997
 
Hansen DJ, Nangle DW, Ellis JT: Reconsideration of the use of peer sociometrics for evaluating social skills training: implications of an idiographic assessment of temporal stability. Behavior Modification 20:281-299,  1996
 
+
+

CME Activity

There is currently no quiz available for this resource. Please click here to go to the CME page to find another.
Submit a Comments
Please read the other comments before you post yours. Contributors must reveal any conflict of interest.
Comments are moderated and will appear on the site at the discertion of APA editorial staff.

* = Required Field
(if multiple authors, separate names by comma)
Example: John Doe



Web of Science® Times Cited: 1

Related Content
Articles
Books
Gabbard's Treatments of Psychiatric Disorders, 4th Edition > Chapter 5.  >
Textbook of Psychotherapeutic Treatments > Chapter 18.  >
Gabbard's Treatments of Psychiatric Disorders, 4th Edition > Chapter 53.  >
Gabbard's Treatments of Psychiatric Disorders, 4th Edition > Chapter 24.  >
Textbook of Traumatic Brain Injury, 2nd Edition > Chapter 38.  >
Topic Collections
Psychiatric News
APA Guidelines
PubMed Articles