0
Get Alert
Please Wait... Processing your request... Please Wait.
You must sign in to sign-up for alerts.

Please confirm that your email address is correct, so you can successfully receive this alert.

1
Brief Reports   |    
Development and Validation of a Housing First Fidelity Survey
Todd P. Gilmer, Ph.D.; Ana Stefancic, M.A.; Marisa Sklar, M.S.; Sam Tsemberis, Ph.D.
Psychiatric Services 2013; doi: 10.1176/appi.ps.201200500
View Author and Article Information

Dr. Gilmer is affiliated with the Department of Family and Preventive Medicine, University of California, San Diego, 9500 Gilman Dr., La Jolla, CA 92093 (e-mail: tgilmer@ucsd.edu). Ms. Stefancic and Dr. Tsemberis are with Pathways to Housing, Inc., and Ms. Stefancic is also with the Department of Sociomedical Sciences, Mailman School of Public Health, Columbia University, both in New York City. Ms. Sklar is with the San Diego State University–University of California, San Diego, Joint Doctoral Program in Clinical Psychology. The preliminary results of this study were presented at the Housing First Partners’ research conference, March 21–23, 2012, in New Orleans; the Academy Health annual research meeting, June 24–26, 2012, in Orlando, Florida; and the Agency for Healthcare Research and Quality annual conference, September 9–12, 2012, in Bethesda, Maryland.

Copyright © 2013 by the American Psychiatric Association

Objectives  Programs that use the Housing First model are being implemented throughout the United States and internationally. The authors describe the development and validation of a Housing First fidelity survey.

Methods  A 46-item survey was developed to measure fidelity across five domains: housing process and structure, separation of housing and services, service philosophy, service array, and team structure. The survey was administered to staff and clients of 93 supported-housing programs in California. Exploratory and confirmatory factor analyses were used to identify the items and model structure that best fit the data.

Results  Sixteen items were retained in a two-factor model, one related to approach to housing, separation of housing and services, and service philosophy and one related to service array and team structure.

Conclusions  Our survey mapped program practices by using a common metric that captured variation in fidelity to Housing First across a large-scale implementation of supported-housing programs.

Abstract Teaser
Figures in this Article

The Housing First model of supported housing provides homeless individuals immediate access to permanent housing and access to a treatment team that provides flexible, client-driven services (1). Studies of Housing First have found it to be effective at improving residential outcomes among homeless persons with serious mental illness (2,3). Major supported-housing initiatives, which share operational and philosophical similarities with the Housing First model, have also demonstrated success in achieving residential stability for persons with serious mental illness (4,5).

Key elements of the Housing First model include access to affordable, permanent, scattered-site housing with tenancy rights and to recovery-oriented, team-based services with an emphasis on consumer choice, self-determination, and independence; active use of harm reduction, assertive engagement, and person-centered planning by program staff; and the absence of coercive practices. Service teams support participants by helping them pursue goals related to mental and general medical health, family and social integration, employment, education, and the pursuit of other meaningful activities. In contrast to traditional approaches, Housing First does not require participants to demonstrate sobriety, psychiatric symptom stability, or completion of a period of time in treatment or transitional housing to have access to permanent, independent housing (2).

Although all supported-housing programs provide access to housing and intensive services, there is substantial variation in their approaches to housing and treatment and in their levels of client choice and client involvement. Examining this variation can indicate the degree to which program philosophy and practices correspond to a Housing First model. Pathways to Housing, Inc., has developed a Housing First fidelity scale that utilizes data from site visits to measure the degree to which supported-housing programs are aligned with the Housing First model (6). The goal of this research is to develop and validate a program-based, self-administered survey that can be used to describe fidelity to the Housing First model across a large number of programs.

In California, the Mental Health Services Act (MHSA) applied a voter-approved tax of 1% on incomes over $1 million to fund new public mental health services. The MHSA was designed to fill gaps in the traditional mental health service delivery system, specifically with respect to underserved populations, including homeless persons with serious mental illness (7). The cornerstone of the MHSA is the implementation of full services partnerships (FSPs)—integrated supportive housing and team-based treatment models that do “whatever it takes” to improve housing and mental health outcomes among persons with serious mental illness who are homeless or at risk of homelessness (8). FSPs are also tailored to serve transition-age youths, adults, criminal justice populations, and older adults.

Consistent with prior efforts to reform the delivery of mental health care in California, the MHSA emphasizes concepts of services integration, recovery orientation, permanent housing, and flexible funding as well as stakeholder engagement and community involvement (9). The emphasis on developing integrated, recovery-oriented care to do “whatever it takes,” the flexibility in funding, and the influence of stakeholders, combined with a lack of specificity and oversight regarding expected FSP practices, led to the implementation of a diverse set of FSP programs (9). This large-scale implementation provides an opportunity to develop and examine the construct validity of a survey that could serve as a self-report measure of fidelity to Housing First.

The Housing First fidelity scale was developed for use during a site visit by multiple independent raters, which was not feasible to conduct across a large number of programs. Therefore, it was necessary to adapt the fidelity scale to a survey format. Like the fidelity scale, the fidelity survey assesses program implementation across five domains (6), including housing process and structure, separation of housing and services, service philosophy, service array, and team structure.

Fidelity items that were scored by trained evaluators after direct observation were reconfigured to allow responses by program staff. A task force reviewed the survey questions for relevance to fidelity, relevance to the FSPs, objectivity, and ability of FSPs to respond. The task force included county administrators, providers, health service researchers, and client advocates. On the basis of their feedback, the proposed survey was revised through an iterative process and finalized by the developers of the scale.

Ultimately, 46 questions were approved to measure fidelity across the five domains. The items ask respondents either to report the percentage of participants in certain categories or to select a response option that qualitatively describes the program’s operation. For some questions, respondents choose one response from a range of options representing a spectrum of fidelity, for example, from lowest to highest fidelity, and other questions provide a checklist that permits multiple choices. To minimize response bias, questions allowing multiple responses often include options that are in line with Housing First as well as those that are antithetical to the model. Questions that allow multiple responses are scored either by applying a hierarchy or by summing responses.

We recruited county mental health directors to participate in the study through an FSP advisory work group that was sponsored by the California Institute of Mental Health. Participating directors then recruited the FSPs in their respective counties. The survey instructions requested that FSP staff review the survey and respond as a team, including the program manager, several staff members, and at least one, but preferably two, client representatives. These instructions were aimed at increasing the range of staff input and including clients’ voices in survey responses.

Exploratory and confirmatory factor analyses were employed in an iterative fashion to arrive at a set of items and a model structure that best fit the data provided by the FSP surveys (10). A principal factor analysis of the survey items was conducted. We limited the factors to those that most meaningfully represent the five fidelity domains. We limited the retained items to those with an absolute-value factor loading of .3 or higher. We verified the communality of the items by examining their squared multiple correlations with other variables and their sampling adequacy by using the Kaiser-Meyer-Olkin method (11).

After limiting the set of factors and excluding nonassociated items, we conducted a second factor analysis by using an orthogonal varimax rotation (12) and predicted regression scores on the basis of the varimax-rotated factors. We used the following measures to evaluate goodness of fit: the Comparative Fit Index (CFI) (13), with values greater than .9 indicating reasonable model fit; and the Root Mean Square Error of Approximation (RMSEA) (14), with values less than .085 indicating reasonable model fit. Internal consistency was examined by calculating Cronbach’s alpha (15).

The survey was developed between January and May 2010. The University of California, San Diego, Human Research Protections Program approved the use of these data for the purpose of this study in accordance with HIPAA.

The Housing First fidelity survey was administered to 93 FSPs statewide. The overall internal consistency of the survey was estimated to be α=.81. Two factors were reasonably associated with the conceptual model of fidelity as defined in the Housing First fidelity scale. After eliminating the other factors and excluding items with factor loadings below .3, we conducted the second factor analysis.

The resulting model included 16 items and two factors—one factor related to housing process and structure, separation of housing and services, and service philosophy and one factor related to service array and team structure (Table 1). [A list of survey items, response sets, and results is available online as a data supplement to this report.] The 16 items and two factors provided a reasonable model fit (CFI=.95 and RMSEA=.044). The two factors had an acceptable level of internal consistency (Cronbach’s α=.72 and .78, respectively). There was virtually zero correlation between the two factors (ρ=.016, p=.878). Among the programs in the top five with respect to fidelity on factor 1, only two were among the top five programs with respect to factor 2. Similarly, two programs fell among the five lowest-scoring programs for both factors.

 
Anchor for Jump
Table 1Factor analysis of survey items associated with fidelity to Housing First, by program domain
Table Footer Note

a Cronbach’s α=.72

Table Footer Note

b Cronbach’s α=.78

Table Footer Note

c Squared multiple correlation

Table Footer Note

d Kaiser-Meyer-Olkin statistic

The survey results revealed that FSPs exhibited higher fidelity with respect to service array and team structure than housing process and structure, separation of housing and services, and service philosophy. Only 13 (14%) programs indicated that at least 85% of participants were living in scattered-site housing (item 2), while 53 (57%) programs reported at least one of the following housing-readiness requirements (item 3): completion of time in transitional housing, outpatient, inpatient, or residential treatment; sobriety or abstinence from alcohol or drug use; medication compliance; psychiatric symptom stability; and willingness to comply with a treatment plan that addresses these issues. In contrast, 69 programs (74%) used their team meetings to address four core functions of the team approach to service delivery (item 16), and teams implemented a wide spectrum of services to address individuals’ employment, education, community participation, social integration, general medical health and substance use treatment needs, with a minimum of 65 (70%) programs offering at least three required service components in each of these areas (items 9–14).

We developed a Housing First fidelity survey and examined its construct validity among a sample of supported-housing programs in California. The resulting model provided a reasonable fit and an acceptable level of internal consistency. The self-report survey provides a framework to assess fidelity on a continuum, from high to low, to the Housing First model and holds potential as an alternative to on-site fidelity measurement.

Core components of the Housing First model were retained in the factor analysis. Key items such as permanent scattered-site housing; no requirements for housing readiness, substance use treatment, and formal psychiatric treatment; harm reduction; and a wide array of services that is responsive to consumer choice suitably represent some of the most salient features of the Housing First model. On the other hand, the factor analysis excluded two items that can be essential to the model from a programmatic perspective—participant choice in housing, which can be key to housing satisfaction and long-term housing retention, and access to psychiatric services. Overall, FSPs demonstrated greater fidelity to service array and team structure than to housing and service philosophy.

Items loading on the housing and service philosophy domains were generally those components that are more unique to Housing First. As communities seek to implement Housing First nationwide, increased attention and support may be needed to assist programs with implementing those critical features in particular. They represent items pertaining to whether housing is integrated into the community (not reserved solely for individuals with psychiatric disabilities), the absence of “housing readiness” or formal treatment requirements, and the role of consumer choice in services. Items loading on the service array and team structure domains reflected the range of services provided by programs—gauging their ability to successfully address a spectrum of aspects of the participants’ lives—and assessed basic team functioning.

A survey offers an expeditious way of obtaining information on a critical array of practices across a wide range of programs. However, a combination of social desirability, limitations of self-assessments, and the need for brevity may make some items more reliable than others. For example, programs may know that they “should” endorse client choice in theory, but they may not have an adequate understanding of how to fully accommodate and support choice in a wide range of areas of the client’s life, particularly when choices are problematic. Similarly, programs may indicate that they are able to provide a wide range of employment services and thus score high on a checklist survey; however, a more in-depth assessment would factor in how often the services are provided, how they are structured, and whether there are exclusion criteria, for example, active substance use or failure to complete prevocational assessments. Assessments of items that are more difficult to capture in surveys can benefit from site visits, which additionally yield insight into contextual factors that facilitate or impede fidelity.

The data presented here are part of a larger, statewide, mixed-methods study of FSPs that included 20 Housing First fidelity site visits that were conducted in a subsample of purposefully selected programs. Future work will assess the relationship of fidelity survey results to outcomes. It will also compare the results of the Housing First fidelity survey to the fidelity site visits in order to determine the extent to which fidelity scores are consistent across the survey and site visits. This information will illuminate the relative merits of the breadth of data that can be gathered from a survey compared with the depth of information that is obtained from a site visit.

Our Housing First fidelity survey successfully mapped program practices by using a common metric that was able to capture variation in fidelity across a large-scale implementation of programs. A brief self-administered survey is a useful tool for obtaining a program “snapshot” of several basic practice ingredients across a large number of programs, both providing a rough metric of fidelity and spotlighting areas for technical assistance.

This work was funded through the American Recovery and Reinvestment Act of 2009 by an award from the Agency for Health Care Research and Quality for Healthcare Delivery Systems Research (1R01HS019986).

The authors report no competing interests.

O’Hara  A:  Housing for people with mental illness: update of a report to the President’s New Freedom Commission.  Psychiatric Services 58:907–913, 2007
[CrossRef] | [PubMed]
 
Tsemberis  S;  Eisenberg  RF:  Pathways to housing: supported housing for street-dwelling homeless individuals with psychiatric disabilities.  Psychiatric Services 51:487–493, 2000
[CrossRef] | [PubMed]
 
Pearson  C;  Locke  G;  Montgomery  A  et al:  The Applicability of Housing First Models to Homeless Persons With Serious Mental Illness.   Washington, DC,  US Department of Housing and Urban Development, Office of Policy Development and Research, 2007
 
Metraux  S;  Marcus  SC;  Culhane  DP:  The New York-New York housing initiative and use of public shelters by persons with severe mental illness.  Psychiatric Services 54:67–71, 2003
[CrossRef] | [PubMed]
 
Shern  DL;  Felton  CJ;  Hough  RL  et al:  Housing outcomes for homeless adults with mental illness: results from the second-round McKinney program.  Psychiatric Services 48:239–241, 1997
[PubMed]
 
Tsemberis  S:  The Pathways Model to End Homelessness for People With Mental Illness and Addiction Manual .  Center City, Minn,  Hazelden, 2010
 
Scheffler  RM;  Adams  N:  Millionaires and mental health: Proposition 63 in California.  Health Affairs  (suppl Web exclusives):W5-212–W5-224, 2005
 
Gilmer  TP;  Stefancic  A;  Ettner  SL  et al:  Effect of full-service partnerships on homelessness, use and costs of mental health services, and quality of life among adults with serious mental illness.  Archives of General Psychiatry 67:645–652, 2010
[CrossRef] | [PubMed]
 
Cashin  C;  Scheffler  R;  Felton  M  et al:  Transformation of the California mental health system: stakeholder-driven planning as a transformational activity.  Psychiatric Services 59:1107–1114, 2008
[CrossRef] | [PubMed]
 
Nunnally  J;  Bernstein  I:  Psychometric Theory.   New York,  McGraw-Hill, 1994
 
Kaiser  H:  An index of factorial simplicity.  Psychometrika 39:31–36, 1974
[CrossRef]
 
Kaiser  H:  The varimax criterion for analytic rotation in factor analysis.  Psychometrika 23:187–200, 1958
[CrossRef]
 
Bentler  PM:  Comparative fit indexes in structural models.  Psychological Bulletin 107:238–246, 1990
[CrossRef] | [PubMed]
 
Steiger  J:  Structural model valuation and modification: an interval estimation approach.  Multivariate Behavioral Research 25:173–180, 1990
[CrossRef]
 
Cronbach  L:  Coefficient alpha and the internal structure of tests.  Psychometrika 16:297–334, 1951
[CrossRef]
 
References Container
Anchor for Jump
Table 1Factor analysis of survey items associated with fidelity to Housing First, by program domain
Table Footer Note

a Cronbach’s α=.72

Table Footer Note

b Cronbach’s α=.78

Table Footer Note

c Squared multiple correlation

Table Footer Note

d Kaiser-Meyer-Olkin statistic

+

References

O’Hara  A:  Housing for people with mental illness: update of a report to the President’s New Freedom Commission.  Psychiatric Services 58:907–913, 2007
[CrossRef] | [PubMed]
 
Tsemberis  S;  Eisenberg  RF:  Pathways to housing: supported housing for street-dwelling homeless individuals with psychiatric disabilities.  Psychiatric Services 51:487–493, 2000
[CrossRef] | [PubMed]
 
Pearson  C;  Locke  G;  Montgomery  A  et al:  The Applicability of Housing First Models to Homeless Persons With Serious Mental Illness.   Washington, DC,  US Department of Housing and Urban Development, Office of Policy Development and Research, 2007
 
Metraux  S;  Marcus  SC;  Culhane  DP:  The New York-New York housing initiative and use of public shelters by persons with severe mental illness.  Psychiatric Services 54:67–71, 2003
[CrossRef] | [PubMed]
 
Shern  DL;  Felton  CJ;  Hough  RL  et al:  Housing outcomes for homeless adults with mental illness: results from the second-round McKinney program.  Psychiatric Services 48:239–241, 1997
[PubMed]
 
Tsemberis  S:  The Pathways Model to End Homelessness for People With Mental Illness and Addiction Manual .  Center City, Minn,  Hazelden, 2010
 
Scheffler  RM;  Adams  N:  Millionaires and mental health: Proposition 63 in California.  Health Affairs  (suppl Web exclusives):W5-212–W5-224, 2005
 
Gilmer  TP;  Stefancic  A;  Ettner  SL  et al:  Effect of full-service partnerships on homelessness, use and costs of mental health services, and quality of life among adults with serious mental illness.  Archives of General Psychiatry 67:645–652, 2010
[CrossRef] | [PubMed]
 
Cashin  C;  Scheffler  R;  Felton  M  et al:  Transformation of the California mental health system: stakeholder-driven planning as a transformational activity.  Psychiatric Services 59:1107–1114, 2008
[CrossRef] | [PubMed]
 
Nunnally  J;  Bernstein  I:  Psychometric Theory.   New York,  McGraw-Hill, 1994
 
Kaiser  H:  An index of factorial simplicity.  Psychometrika 39:31–36, 1974
[CrossRef]
 
Kaiser  H:  The varimax criterion for analytic rotation in factor analysis.  Psychometrika 23:187–200, 1958
[CrossRef]
 
Bentler  PM:  Comparative fit indexes in structural models.  Psychological Bulletin 107:238–246, 1990
[CrossRef] | [PubMed]
 
Steiger  J:  Structural model valuation and modification: an interval estimation approach.  Multivariate Behavioral Research 25:173–180, 1990
[CrossRef]
 
Cronbach  L:  Coefficient alpha and the internal structure of tests.  Psychometrika 16:297–334, 1951
[CrossRef]
 
References Container
+
+

CME Activity

There is currently no quiz available for this resource. Please click here to go to the CME page to find another.
Submit a Comments
Please read the other comments before you post yours. Contributors must reveal any conflict of interest.
Comments are moderated and will appear on the site at the discertion of APA editorial staff.

* = Required Field
(if multiple authors, separate names by comma)
Example: John Doe



Web of Science® Times Cited: 2

Related Content
Articles
Books
The American Psychiatric Publishing Textbook of Geriatric Psychiatry, 4th Edition > Chapter 32.  >
The American Psychiatric Publishing Textbook of Geriatric Psychiatry, 4th Edition > Chapter 32.  >
APA Practice Guidelines > Chapter 4.  >
The American Psychiatric Publishing Textbook of Geriatric Psychiatry, 4th Edition > Chapter 32.  >
The American Psychiatric Publishing Textbook of Geriatric Psychiatry, 4th Edition > Chapter 32.  >
Topic Collections
Psychiatric News
PubMed Articles
What makes housing work? Ment Health Today 2014 Jul-Aug;():24-7.