Development and Validation of a Housing First Fidelity Survey

The Housing First model of supported housing provides homeless individuals immediate access to permanent housing and access to a treatment team that provides flexible, client-driven services (1). Studies of Housing First have found it to be effective at improving residential outcomes among homeless persons with serious mental illness (2,3). Major supported-housing initiatives, which share operational and philosophical similarities with the Housing First model, have also demonstrated success in achieving residential stability for persons with serious mental illness (4,5).

Key elements of the Housing First model include access to affordable, permanent, scattered-site housing with tenancy rights and to recovery-oriented, team-based services with an emphasis on consumer choice, self-determination, and independence; active use of harm reduction, assertive engagement, and person-centered planning by program staff; and the absence of coercive practices. Service teams support participants by helping them pursue goals related to mental and general medical health, family and social integration, employment, education, and the pursuit of other meaningful activities. In contrast to traditional approaches, Housing First does not require participants to demonstrate sobriety, psychiatric symptom stability, or completion of a period of time in treatment or transitional housing to have access to permanent, independent housing (2).

Although all supported-housing programs provide access to housing and intensive services, there is substantial variation in their approaches to housing and treatment and in their levels of client choice and client involvement. Examining this variation can indicate the degree to which program philosophy and practices correspond to a Housing First model. Pathways to Housing, Inc., has developed a Housing First fidelity scale that utilizes data from site visits to measure the degree to which supported-housing programs are aligned with the Housing First model (6). The goal of this research is to develop and validate a program-based, self-administered survey that can be used to describe fidelity to the Housing First model across a large number of programs.

In California, the Mental Health Services Act (MHSA) applied a voter-approved tax of 1% on incomes over $1 million to fund new public mental health services. The MHSA was designed to fill gaps in the traditional mental health service delivery system, specifically with respect to underserved populations, including homeless persons with serious mental illness (7). The cornerstone of the MHSA is the implementation of full services partnerships (FSPs)—integrated supportive housing and team-based treatment models that do “whatever it takes” to improve housing and mental health outcomes among persons with serious mental illness who are homeless or at risk of homelessness (8). FSPs are also tailored to serve transition-age youths, adults, criminal justice populations, and older adults.

Consistent with prior efforts to reform the delivery of mental health care in California, the MHSA emphasizes concepts of services integration, recovery orientation, permanent housing, and flexible funding as well as stakeholder engagement and community involvement (9). The emphasis on developing integrated, recovery-oriented care to do “whatever it takes,” the flexibility in funding, and the influence of stakeholders, combined with a lack of specificity and oversight regarding expected FSP practices, led to the implementation of a diverse set of FSP programs (9). This large-scale implementation provides an opportunity to develop and examine the construct validity of a survey that could serve as a self-report measure of fidelity to Housing First.

Methods

The Housing First fidelity scale was developed for use during a site visit by multiple independent raters, which was not feasible to conduct across a large number of programs. Therefore, it was necessary to adapt the fidelity scale to a survey format. Like the fidelity scale, the fidelity survey assesses program implementation across five domains (6), including housing process and structure, separation of housing and services, service philosophy, service array, and team structure.

Fidelity items that were scored by trained evaluators after direct observation were reconfigured to allow responses by program staff. A task force reviewed the survey questions for relevance to fidelity, relevance to the FSPs, objectivity, and ability of FSPs to respond. The task force included county administrators, providers, health service researchers, and client advocates. On the basis of their feedback, the proposed survey was revised through an iterative process and finalized by the developers of the scale.

Ultimately, 46 questions were approved to measure fidelity across the five domains. The items ask respondents either to report the percentage of participants in certain categories or to select a response option that qualitatively describes the program’s operation. For some questions, respondents choose one response from a range of options representing a spectrum of fidelity, for example, from lowest to highest fidelity, and other questions provide a checklist that permits multiple choices. To minimize response bias, questions allowing multiple responses often include options that are in line with Housing First as well as those that are antithetical to the model. Questions that allow multiple responses are scored either by applying a hierarchy or by summing responses.

We recruited county mental health directors to participate in the study through an FSP advisory work group that was sponsored by the California Institute of Mental Health. Participating directors then recruited the FSPs in their respective counties. The survey instructions requested that FSP staff review the survey and respond as a team, including the program manager, several staff members, and at least one, but preferably two, client representatives. These instructions were aimed at increasing the range of staff input and including clients’ voices in survey responses.

Exploratory and confirmatory factor analyses were employed in an iterative fashion to arrive at a set of items and a model structure that best fit the data provided by the FSP surveys (10). A principal factor analysis of the survey items was conducted. We limited the factors to those that most meaningfully represent the five fidelity domains. We limited the retained items to those with an absolute-value factor loading of .3 or higher. We verified the communality of the items by examining their squared multiple correlations with other variables and their sampling adequacy by using the Kaiser-Meyer-Olkin method (11).

After limiting the set of factors and excluding nonassociated items, we conducted a second factor analysis by using an orthogonal varimax rotation (12) and predicted regression scores on the basis of the varimax-rotated factors. We used the following measures to evaluate goodness of fit: the Comparative Fit Index (CFI) (13), with values greater than .9 indicating reasonable model fit; and the Root Mean Square Error of Approximation (RMSEA) (14), with values less than .085 indicating reasonable model fit. Internal consistency was examined by calculating Cronbach’s alpha (15).

The survey was developed between January and May 2010. The University of California, San Diego, Human Research Protections Program approved the use of these data for the purpose of this study in accordance with HIPAA.

Results

The Housing First fidelity survey was administered to 93 FSPs statewide. The overall internal consistency of the survey was estimated to be α=.81. Two factors were reasonably associated with the conceptual model of fidelity as defined in the Housing First fidelity scale. After eliminating the other factors and excluding items with factor loadings below .3, we conducted the second factor analysis.

The resulting model included 16 items and two factors—one factor related to housing process and structure, separation of housing and services, and service philosophy and one factor related to service array and team structure (Table 1). [A list of survey items, response sets, and results is available online as a data supplement to this report.] The 16 items and two factors provided a reasonable model fit (CFI=.95 and RMSEA=.044). The two factors had an acceptable level of internal consistency (Cronbach’s α=.72 and .78, respectively). There was virtually zero correlation between the two factors (ρ=.016, p=.878). Among the programs in the top five with respect to fidelity on factor 1, only two were among the top five programs with respect to factor 2. Similarly, two programs fell among the five lowest-scoring programs for both factors.

The survey results revealed that FSPs exhibited higher fidelity with respect to service array and team structure than housing process and structure, separation of housing and services, and service philosophy. Only 13 (14%) programs indicated that at least 85% of participants were living in scattered-site housing (item 2), while 53 (57%) programs reported at least one of the following housing-readiness requirements (item 3): completion of time in transitional housing, outpatient, inpatient, or residential treatment; sobriety or abstinence from alcohol or drug use; medication compliance; psychiatric symptom stability; and willingness to comply with a treatment plan that addresses these issues. In contrast, 69 programs (74%) used their team meetings to address four core functions of the team approach to service delivery (item 16), and teams implemented a wide spectrum of services to address individuals’ employment, education, community participation, social integration, general medical health and substance use treatment needs, with a minimum of 65 (70%) programs offering at least three required service components in each of these areas (items 9–14).

Discussion

We developed a Housing First fidelity survey and examined its construct validity among a sample of supported-housing programs in California. The resulting model provided a reasonable fit and an acceptable level of internal consistency. The self-report survey provides a framework to assess fidelity on a continuum, from high to low, to the Housing First model and holds potential as an alternative to on-site fidelity measurement.

Core components of the Housing First model were retained in the factor analysis. Key items such as permanent scattered-site housing; no requirements for housing readiness, substance use treatment, and formal psychiatric treatment; harm reduction; and a wide array of services that is responsive to consumer choice suitably represent some of the most salient features of the Housing First model. On the other hand, the factor analysis excluded two items that can be essential to the model from a programmatic perspective—participant choice in housing, which can be key to housing satisfaction and long-term housing retention, and access to psychiatric services. Overall, FSPs demonstrated greater fidelity to service array and team structure than to housing and service philosophy.

Items loading on the housing and service philosophy domains were generally those components that are more unique to Housing First. As communities seek to implement Housing First nationwide, increased attention and support may be needed to assist programs with implementing those critical features in particular. They represent items pertaining to whether housing is integrated into the community (not reserved solely for individuals with psychiatric disabilities), the absence of “housing readiness” or formal treatment requirements, and the role of consumer choice in services. Items loading on the service array and team structure domains reflected the range of services provided by programs—gauging their ability to successfully address a spectrum of aspects of the participants’ lives—and assessed basic team functioning.

A survey offers an expeditious way of obtaining information on a critical array of practices across a wide range of programs. However, a combination of social desirability, limitations of self-assessments, and the need for brevity may make some items more reliable than others. For example, programs may know that they “should” endorse client choice in theory, but they may not have an adequate understanding of how to fully accommodate and support choice in a wide range of areas of the client’s life, particularly when choices are problematic. Similarly, programs may indicate that they are able to provide a wide range of employment services and thus score high on a checklist survey; however, a more in-depth assessment would factor in how often the services are provided, how they are structured, and whether there are exclusion criteria, for example, active substance use or failure to complete prevocational assessments. Assessments of items that are more difficult to capture in surveys can benefit from site visits, which additionally yield insight into contextual factors that facilitate or impede fidelity.

The data presented here are part of a larger, statewide, mixed-methods study of FSPs that included 20 Housing First fidelity site visits that were conducted in a subsample of purposefully selected programs. Future work will assess the relationship of fidelity survey results to outcomes. It will also compare the results of the Housing First fidelity survey to the fidelity site visits in order to determine the extent to which fidelity scores are consistent across the survey and site visits. This information will illuminate the relative merits of the breadth of data that can be gathered from a survey compared with the depth of information that is obtained from a site visit.

Conclusions

Our Housing First fidelity survey successfully mapped program practices by using a common metric that was able to capture variation in fidelity across a large-scale implementation of programs. A brief self-administered survey is a useful tool for obtaining a program “snapshot” of several basic practice ingredients across a large number of programs, both providing a rough metric of fidelity and spotlighting areas for technical assistance.