Going to the Source: Creating a Citizenship Outcome Measure by Community-Based Participatory Research Methods

In previous qualitative research on mental health outreach to persons who were homeless, we found that even multidisciplinary and resource-rich approaches for those with mental illness had limited capacity to facilitate their transition from the marginalization of homelessness to full community membership (1). As we have argued, individuals' relationships, in both instrumental and expressive terms, to their community and society are the source of their sense of belonging and of their access to participation in society and the status of first-class citizens (2). On the basis of these findings and our review of relevant social science literature, we developed a theoretical framework of citizenship with particular attention to the social integration needs of persons with mental illnesses. We conceptualized citizenship as a measure of the strength of an individual's connection to “five Rs”—the rights, responsibilities, roles, and resources that society offers to people through public and social institutions and relationships involving close ties, supportive social networks, and associational life in one's community (1–4).

Ware and colleagues (5) have employed the concept in qualitative research on social integration, which they define as a process through which persons with mental illnesses develop their capacities for connectedness and citizenship. McCabe and colleagues (6) have called for efforts to enhance the “cultural citizenship” of persons with mental illness and others who are socially excluded. Prior (7) has argued that theories of citizenship and human rights in regard to people with mental illnesses are responses to the stigma of mental illness and the deprivation of rights to housing, employment, and family life. There is a dearth of empirical research on the concept of citizenship, however, and we have not been able to identify any validated measures of citizenship that encompass a broad range of experiences, beliefs, and actions, including a sense of belonging and participation in society.

The theories of “life disruption” and “off-timedness” have proved useful to us in our thinking about citizenship. Life disruption, a criterion for diagnosis of mental illness (8), is a shared element across a variety of experiences, including major general medical illnesses and incarceration. The intrusion of mental illness often delays or diverts people from achieving important markers, such as obtaining a high school diploma or beginning a career (9). Life disruption contributes to a sense of “off-timedness,” whereby people may not meet culturally and socially defined developmental milestones in the expected time frame (10,11).

We have previously applied our theoretical framework of citizenship in a randomized controlled trial that compared clinical and community outcomes of persons with serious mental illness and criminal justice charges who received standard clinical services that could include jail diversion (control group) or who received standard clinical services that could include jail diversion plus a citizenship-based intervention (intervention group) (12). The citizenship intervention included a class component that addressed topics such as negotiating the criminal justice system, problem solving, relationship building, and social integration; projects involving community-valued roles, such as teaching police cadets about consumers' encounters with police; and wraparound peer mentor support for participants.

Employing mixed-models analysis, we found that participants in the intervention group had overall significantly lower levels of alcohol use across six- and 12- month follow-up periods compared with those in the control group (12). There was also a significant interaction in which intervention participants showed decreasing levels of alcohol use and control group participants showed increasing levels of alcohol use across follow-up periods. Qualitative findings included the program's development into a supportive subcommunity over time and that “being a student” was a valued role in and of itself for participants (13).

After completion of the study, two concerns led us to undertake a measure development study. First, application of our theoretical framework left a gap between theory and practice. We had taken a “reasonable and sensible” approach to linking the five Rs of citizenship to program elements, but we lacked empirical data on the elements of citizenship on which to build and improve further interventions. Second, although citizenship theory encompasses both political-legal realms (13) and civic participation realms (14), discourse generally addresses only the former. We wondered whether persons with mental illness who had not participated in the citizenship study and others would regard citizenship as a multifaceted concept or more narrowly as a political-legal concept. We concluded that we needed to develop an instrument to administer to an individual to measure citizenship and that the concepts of off-timedness and life disruptions would help guide us in comparing the perceptions of citizenship among persons with mental illness and among those who had or had not experienced other significant life disruptions.

Methods

Participants

We received approval for this study from the Yale University School Of Medicine Institutional Review Board. Because no personal health information was collected, the board waived written consent and permitted verbal consent for participation. To be eligible for focus group participation, individuals had to meet one of the following criteria: currently enrolled in public mental health services, currently receiving medical treatment for a general medical illness that interfered with their daily life, or currently on probation or parole. We also enrolled individuals who had experienced more than one type of these life disruptions (mental illness, general medical illness, or probation or parole) and individuals who had not experienced any of these life disruptions.

We recruited focus group participants through newspaper advertisements and flyers. A research team member conducted a brief phone screening to determine eligibility and category and invited eligible persons to a focus group based on the category of their life disruption or nondisruption. Participants for concept-mapping sessions were recruited from among interested focus group participants and through phone screening of persons who were eligible for focus groups but did not participate because of scheduling conflicts or because the focus group for which they were eligible was at capacity.

Because we did not collect personal health information, we treated focus groups and concept-mapping sessions as separate and nonduplicative substudies of the larger study. Thus focus group participants could also participate in concept-mapping sessions, but concept-mapping participants need not have participated in a focus group. Focus groups and concept-mapping sessions were held at the authors' office. We enrolled a total of 141 persons in seven focus groups (N=75) and three concept-mapping sessions (N=66). The total sample included 82 women (58%) and 53 men (38%); six participants (4%) did not report a gender. A total of 92 participants (65%) were African American, 42 (30%) were white; and one (>1%) was American Indian; six (5%) did not report a race. Twelve participants (9%) of the sample identified as Hispanic or Latino. The mean±SD age of participants was 42.35±10.77. No significant differences in these demographic characteristics were found between the five groups.

For both focus groups and concept-mapping sessions, participants were read a consent statement and raised their hands if they agreed to participate in the study. Paper copies of the consent form were handed out to those who requested a copy, and researchers were available to talk to persons who had questions or concerns about the process.

One focus group was conducted with each of the following sample groups: persons who currently were receiving public services for a mental illness; persons who were on probation or parole; and persons who had a general medical illness that interfered with their lives. Two focus groups were conducted with each of the following sample groups: persons who had experienced more than one type of these life disruptions, and persons who had not experienced any of the three types of life disruptions (no mental illness, no general medical illness, and not on probation or parole). Of the 75 persons who participated in the focus groups, 14 (19%) were receiving mental health services, four (5%) were currently on probation or parole, six (8%) had a general medical illness, 27 (36%) had experienced two or more types of life disruptions, and 24 (32%) had not experienced any of the life disruptions. We oversampled the last group—the “nondisrupted' group—in order to have an adequate sample size to conduct between-group comparisons for the combined “disrupted” groups versus the nondisrupted group.

There were 66 concept-mapping participants (some of whom also participated in the focus groups). Of these 66 persons (with overlap for those with more than one disruption) 35 persons (53%) currently were receiving mental health services, 19 (29%) were currently on probation or parole, 18 (27%) had a current general medical illness, and 18 (27%) did not have one of the above-named disruptions. Because concept mapping is an individual task, these sessions were not structured by life disruption group but were instead organized to provide adequate space for participants to complete this task.

Data collection—from the first focus group through the final concept-mapping session—took place between October 11, 2010, and November 10, 2010.

Procedures

For this study, we employed an approach using community-based participatory research (CBPR) and concept-mapping methodology. In CBPR, target population members participate in all aspects of the research process, from conceptualization through dissemination of findings (15–17). This method seemed particularly well suited to research on citizenship. We recruited an eight-member co-researcher team (CRT) of persons with mental illnesses, including four men and four women of African-American (N=3), Latino (N= 2), and Caucasian (N=3) race or ethnicity. Some CRT members also had a serious general medical illness or a criminal justice history. CRT members were trained in protection of human subjects and Health Insurance Portability and Accountability Act (HIPAA) requirements and in conducting focus groups and concept-mapping procedures. (Two authors of this article are self-disclosed persons in recovery from mental illness; they were not CRT members.)

Concept mapping employs multiple strategies to capture conceptual data on a topic of interest (18), integrating input from multiple stakeholders by using sophisticated multivariate data analyses to create a series of “maps” or visual representations of the data that can be used to guide action planning, program development, and measurement development (19). We incorporated CRT participation at each of four research phases, as shown in Figure 1 and described below.

Phase 1: focus groups.

Because this study involved development of a measure of citizenship, it was appropriate to use that term in our focus group questions and statements. In addition, the research team discussed the possibility of drawing on terms such as “sense of belonging,” “social inclusion,” and “community integration” for questions or follow-up probes. CRT members, however, noted their strong preference for “citizenship” over “social inclusion,” “community integration,” and other terms, because “citizenship” seemed broader to them, whereas they associated the other terms with the marginalized status of people with mental illnesses. For our goal of generating items for the outcome instrument, the research team agreed that questions and statements should be simple and few, with as little follow-up as possible, and consensus was reached on a single statement to be used by the focus group leader: “To me, being a citizen means … ”

CRT members and two of the authors together conducted the focus groups, with at least one CRT member and one of the two authors for each group. Participants were asked to speak freely and generate as many ideas and items as possible. Examples of some items are “Being included,” “Feeling safe in my neighborhood,” and “Having equal opportunities.” Participants responded to the “To me, being a citizen means …” statement without requests for clarification, and the only follow-up probes were for researcher clarification (for example, “Can you say more about that?”).

Participants in each focus group, which lasted for about two hours, were paid $50. At the end of each focus group, researchers handed out information to persons interested in participating in concept-mapping sessions.

Phase 2: focus group analyses.

Research team members (CRT members and the authors) reviewed all statements generated in the focus groups, compiled a list of citizenship statements, reviewed these statements to remove duplication or to clarify meaning, and separated multithemed statements into single-themed statements. A final list of 100 unique items was generated.

Phase 3: concept-mapping sessions.

Stakeholders from each group participated in concept-mapping sessions. Individuals were given lists of the 100 citizenship statements and asked to sort them based on similarity. They were given the following ground rules: create more than one group (that is, one group with 100 items was not acceptable), have more than one item in any given group (that is, for the 100 items, no more than 50 categories could be generated, although categories generated ranged from two to 22 items), and do not have a miscellaneous pile (that is, each item had to be placed in a category).

Participants were then asked to rate each statement on a Likert scale of 1 to 5. The first rating was based on the importance of the item to them individually in regard to citizenship. The second rating was based on their perception of their status regarding—or their degree of achievement of—that item. To help prevent ordering effects, several randomly ordered sets of statements were used. Participants provided basic demographic information and identified the stakeholder group they represented. They were paid $40 for each concept-mapping session.

Phase 4: statistical analyses.

Statement groupings and importance ratings were entered into a Microsoft Access database and analyzed using the categories module of SPSS, version 20.0 (20). Concept-mapping analysis uses categorical data to construct a matrix (100×100) in which any two items are scored 1 if they are sorted into the same pile and 0 if they are sorted into different piles by an individual. A total 100×100 matrix was obtained by summing the values within each individual's matrix. A high sum for a pair of items indicates that many people sorted the items into the same grouping, which reflects a strong perception of interrelatedness for these items.

The total matrix was then analyzed by using multidimensional scaling analysis (21). This analysis yields a two-dimensional representation of the data: statements that are piled together more often are located closer together in the clusters than those piled together less often. Data from the multidimensional scaling is then input into hierarchical cluster analysis by using Ward's algorithm (22). This forces the cluster analysis to place the multidimensional scaling into nonoverlapping clusters.

Item analysis was also conducted to identify items rated most and least important and most and least accessed or achieved by individuals in each stakeholder group—mental illness, probation or parole, general medical illness, mixed disruptions, and no disruptions. For this analysis, dummy variables were created to identify group membership in the following eight stakeholder groups (with mutual exclusion maintained within each dummy variable but not across stakeholder groups): receiving mental health services or not, on probation or parole or not, serious general medical illness or none, some type of life interruption or none. For each of the eight stakeholder groups, scatterplots were created for items within each cluster with importance ratings on the X axis and possibility ratings on the Y axis. Four quadrants were assessed: highly important and highly possible, highly important and less possible, less important and more possible, and less important and less achievable.

Results

Hierarchical cluster analysis sorted citizenship items into a dendrogram (a diagram used to depict the arrangement of clusters) based on concept-mapping participants' individual categorizations of statements. The dendrogram yielded several optimal cluster arrangements ranging from one to nine domains. The CRT reviewed maps based on the dendrogram's clustering of items and then examined the individual items within clusters to determine which grouping made the most theoretical and practical sense—that is, which cluster arrangement yielded the most conceptually coherent and distinct groups of items. The CRT and authors decided that seven clusters was the optimal solution for the conceptual model of citizenship and named them (for the purpose of measure development, not for use in piloting the instrument): personal responsibilities, government and infrastructure, caring for self and others, civil rights, legal rights, choices, and world stewardship. Figure 2 shows the clusters and some key themes within each.

The team then examined the results from item analyses and determined the most appropriate items to include in the citizenship measure and the ranking procedures to be used. Decisions were made to keep items that were overall most important to each of the eight stakeholder groups and that had discrepancies between one or more stakeholder groups in terms of quadrant location denoting importance and possibility.

Items that had no discrepancies between stakeholder groups in terms of quadrant location and that were not rated highly important were discarded. Forty-six items were selected from the seven domains already determined by the cluster analysis. For each item, participants are asked, “Thinking about your life in general now, rate how much the item applies to you on a 1 to 5 Likert Scale.” The instrument was piloted with CRT members, and minor adjustments to wording and item selection were made. Initial validation of the measure has been conducted with persons receiving public mental health services at the Connecticut Mental Health Center and is ongoing.

Discussion

Our purpose in developing this instrument was to help practitioners and researchers develop more targeted and effective citizenship interventions and approaches for persons with mental illnesses and to identify, with them, areas of desired change and action to fully achieve specified aspects of citizenship. We also hope that use of the validated instrument will help us improve our previously tested citizenship intervention—for example, by asking intervention participants to complete the instrument to prepare for discussions of individual goals in peer-mentor counseling sessions.

In future studies we will continue to compare the convergent and discriminant validity of our instrument with instruments that measure social capital, social inclusion, and other relevant domains. Additional psychometric properties to be evaluated include criterion and predictive validity. We then hope to conduct a randomized controlled trial of the citizenship intervention enhanced by our findings from measure development and from administration of the instrument to persons with mental illnesses.

This study had several limitations. First, because participants self-selected and were not chosen at random, they may have had stronger feelings about citizenship than others in our target groups. Second, our categories of life disruptions were broadly defined. For example, they did not distinguish between type or severity of mental illness, and they excluded persons with mental illness who were not in treatment. Third, participants who had not experienced life disruptions were given this classification only in relation to three types of life disruptions, and they may have experienced other life disruptions. Finally, this study did not include measurement validation, which we see as a critical next step in our research and which, as noted above, is ongoing.

Conclusions

Our approach to the measurement of social integration holds promise as a component in the larger project of enhancing the social integration of persons with mental illnesses. In particular, “going to the source” with community-based participatory research and factor analytic methods, such as concept mapping, holds promise for helping populations of interest identify their social integration goals and needs.

Figures and Tables