Clients' Satisfaction With Project Liberty Counseling Services

Immediately after the September 11, 2001, attacks on the World Trade Center, the New York State Office of Mental Health (NYOMH) applied for funding from the Federal Emergency Management Agency (FEMA) to provide short-term public education, outreach, and crisis counseling to residents within greater New York City ( 1 ). Called Project Liberty, these services continued for 27 months, were expanded during 2003 to include more intensive mental health services, and were evaluated through clients' responses to telephone surveys and brief paper-and-pencil questionnaires and via analysis of service encounter logs.

Clients' ratings of satisfaction with and helpfulness of their counseling experiences provide valuable information about the real-world effectiveness of such interventions ( 2 ). Available research suggests consistently that relationships between client satisfaction and counseling outcomes are statistically reliable, although the size of these relationships and their meaning have varied widely ( 3 , 4 ). This study evaluated satisfaction reported by Project Liberty counseling recipients along multiple dimensions of services and daily functioning.

Methods

Between 16 and 27 months after the attacks on the World Trade Center, Project Liberty counselors at selected sites were instructed to offer individuals at least 18 years of age who spoke either English or Spanish the opportunity to evaluate the services they had been receiving ( 5 ). NYOMH identified eight sites that had been submitting a high volume of claims for individual crisis counseling to participate in the evaluation. These sites varied widely in terms of location, cultural and ethnic surroundings, organizational structure, and methods of service provision. NYOMH chose four sites to offer only telephone interviews and four to offer both telephone interviews and brief questionnaires. The institutional review boards of both the Mount Sinai School of Medicine and NYOMH approved the research protocol for this investigation.

We combined data from three samples—brief questionnaires completed anonymously by 463 crisis counseling recipients (collected in two waves, January through April 2003 from four pilot sites and July through November 2003 as a programwide distribution to 101 sites), structured telephone interviews conducted with 153 individuals who had received crisis counseling (February through June 2003), and telephone interviews conducted with 93 enhanced services recipients (October 2003 through May 2004). To avoid analyzing duplicate data, we excluded 61 and 41 respondents from the two respective telephone samples who reported that they also had completed questionnaires.

The respondents from the resulting sample of 607 service recipients had a mean±SD age of 45±14 years; 379 (62 percent) were women; 364 (65 percent) were Caucasian, 77 (14 percent) were African American, and 118 (21 percent) were identified as "other" or of mixed race. In addition, across these racial categories, 130 (22 percent) reported that they also were of Hispanic origin. A majority (80 percent, or 470 respondents) said English was their primary language, and 426 (70 percent) received Project Liberty services at a New York City site. In terms of exposure risk related to the attacks on the World Trade Center, 223 service recipients (41 percent) reported thinking that they might be injured or killed during the attacks, 229 (38 percent) reported losing a family member or friend, 134 (24 percent) reported losing their job, and 126 (21 percent) reported being involved in rescue efforts.

Although this was a sample of convenience, individuals who participated in the interviews and brief questionnaires generally resembled the full population of Project Liberty service recipients. Table 1 shows the demographic breakdown for those who participated in each wave of the brief questionnaire, as well as in the crisis counseling and enhanced services interviews. To show how these proportions compared with the larger population of people receiving services from Project Liberty during the same period (calendar year 2003), we also show demographic information from the service encounter logs for the entire population of service recipients.

Enlarge table

To show how each subsample compares with the total sample of service encounter logs, we show in Table 1 the breakdown for each subsample. Because individuals may be in more than one subsample, the total of the subsamples exceeds the total for the entire sample. The brief questionnaires and interviews each had some missing data; hence, the demographic (that is, gender, language, race, and ethnicity) categories within each subsample may not add up to the total for the entire subsample. The percentages of the demographic categories are based on nonmissing data. Proportions for Caucasians must be interpreted with the understanding that the service encounter logs had Hispanic and Caucasian as mutually exclusive racial categories, whereas the interviews and brief questionnaires asked about race and ethnicity separately. Hence, to compare like categories, one can sum the Caucasian and Hispanic categories from the logs and compare those sums to the Caucasian groups for the interviews and brief questionnaires.

We queried participants about their risk exposure during the attacks, demographic characteristics, and satisfaction with Project Liberty services. We structured questions to reflect content that had been used in prior surveys ( 6 , 7 ) and worded these questions to a sixth grade reading level. Respondents rated their satisfaction with the quality of services in 11 areas: counselor respect, willingness of the counselor to listen, cultural sensitivity, ability of the counselor to speak the person's language, amount of time spent counseling, convenience of meeting time, convenience of meeting location, value of information received about reactions people frequently have after a disaster, likelihood of using Project Liberty services again, whether respondents would recommend Project Liberty to a friend or to a family member, and a final summary quality item. In addition, respondents rated how well their counselor helped them in four domains of life functioning: carrying out daily responsibilities at home, work, or school; maintaining relationships with family and friends; finding ways to take care of oneself physically; and staying involved in community activities. Respondents rated each of the 15 items using a 4-point Likert scale (1, poor; 2, fair; 3, good; and 4, excellent). We report descriptive statistics of the ratings provided by service recipients.

Results

Across all 11 service quality items, more than 90 percent of the respondents rated Project Liberty as either good or excellent in terms of their satisfaction with services ( Table 2 ). Counselor respect (85 percent) followed by the counselor's willingness to listen (83 percent), received the highest satisfaction ratings, followed by the counselor's ability to speak the service recipient's language (81 percent). Over 80 percent of respondents rated their counselors as excellent in these three domains, and 77 percent indicated that their counselors showed excellent sensitivity to their cultural background. Convenience of meeting times and locations were less highly rated. In addition, at least 90 percent said that Project Liberty did either a good or an excellent job of helping them in each of three functional domains: maintaining relationships (93 percent reported either good or excellent), taking care of physical health (90 percent), and carrying out daily responsibilities (93 percent). In addition, 89 percent said that Project Liberty did either a good or an excellent job of helping them maintain their involvement in community activities. These very favorable evaluations were found to be equivalent across both gender and racial and ethnic groupings.

Enlarge table

Discussion

A substantial majority of respondents evaluated multiple dimensions of Project Liberty services as good or excellent. This endorsement provides support for the ability of crisis counselors to provide accessible and acceptable disaster-related mental health services even while working under extremely difficult conditions. Exceptionally high ratings for counselor respect were especially noteworthy in that this element has been shown to be one of the most robust and powerful predictors of favorable counseling outcomes ( 8 , 9 ), although we have no data that explicitly connect ratings and outcomes for these samples. A substantial number of counselors also were rated as showing excellent sensitivity to the specific cultural needs of service recipients, a factor that enhances outcome effectiveness ( 10 ). Also, the ability of Project Liberty counselors to satisfactorily provide information about the effects of trauma helps "normalize" the consequences of traumatic symptomatology and mute their effects ( 11 ).

One limitation of this evaluation was our inability to identify significant predictors of satisfaction ratings because so few recipients rated services as less than good. However, we also viewed this result as a strength of the service providers, who were able to provide under difficult circumstances services that most recipients found both of high quality and helpful. We also acknowledge that those who reported good services may have been different in many ways from those who reported that services were excellent. However, for this reason, we have taken care to show these numbers separately in Table 2 . Finally, given the nature of the convenience sample, it is possible that those who saw the services as less effective were less likely to continue treatment or less likely to complete the evaluation.

Even though these largely positive endorsements from survey respondents concerning the quality and usefulness of Project Liberty are encouraging, they cannot, because of the small sample sizes obtained, be safely generalized as indicative of the opinions and experiences of all individuals who received services through this program. Evaluations of future disaster mental health response programs will need to find ways to boost response rates if we are to truly leverage the potential of recipient assessments of care as a means to document the personal impact of response efforts—for example, to return to predisaster levels of functioning—as well as the fidelity of these efforts to programmatic goals and principles, such as providing demographically diverse outreach and easy access to services.

Project Liberty was extremely successful in obtaining service encounter data from crisis counselors with simple paper-based forms. This success reflects not only compliance with program requirements but also the development of the necessary infrastructure to capture counselor-reported encounter data on a large scale at the inception of Project Liberty as a core administrative component. Specifically, all counselors were required to collect and submit encounter data as part of their agencies' contract to provide services under Project Liberty. They used standard forms and followed procedures established for submitting completed forms to comply with a contract with a data entry vendor and NYOMH database structures for the analysis and reporting of encounter log data. By contrast, recipient feedback surveys were designed and tested later in the program, initially as an experiment to assess their feasibility of use in other disasters. Our results suggest that such surveys are in fact feasible and can provide useful information.

Thus future disaster mental health efforts would likely benefit from the design and deployment of consumer surveys to solicit feedback, as well as processes to capture data from counselor service encounters, as part of program implementation and ongoing administration. Making such surveys a required component of administering disaster mental health programs, if accompanied with the development of enabling infrastructure, would likely substantially increase response rates as well as encourage further experimentation with methodologies not tried in this instance—Web-based surveys, for example—that might boost consumer feedback.

Conclusions

Evaluation of counseling by service recipients is important in identifying especially helpful therapeutic aspects of the process. Such evaluations can be conducted in a cost-effective manner and integrated with evidence-based practice to ultimately ensure that recipients of counseling receive the most efficient and effective interventions. The evaluation of Project Liberty tested on an experimental basis a number of methods for obtaining recipient feedback ( 5 ). The findings presented in this article suggest that broader use of recipient surveys in future disaster response programs could yield useful data concerning the outcomes of disaster mental health services.

Acknowledgments

This evaluation was funded by grant FEMA-1391-DR-NY (titled "Project Liberty: Crisis Counseling Assistance and Training Program") to New York State from the Federal Emergency Management Agency. The Center for Mental Health Services of the Substance Abuse and Mental Health Services Administration administered the grant.

^a Some data are missing for each of the pilot samples, and percentages are based on only existing data.