In recent years there has been steady growth in the availability of quality indicators that are designed to inform patients, providers, and policy makers about the quality of health care (1,2). In addition to traditional clinical measures of outcome, questionnaires are increasingly used to assess health professionals' and patients' perceptions about health care quality (3). The Ministry of Health for Norway has decided that satisfaction with cooperation among health care providers shall form part of the Norwegian quality indicator system (4). The first national evaluation relates to the interface between primary and secondary care, specifically family physician and general practitioner experiences with community mental health centers. The centers are professionally independent units responsible for a substantial part of general adult psychiatric services in specific geographical areas. Community mental health centers are part of specialized mental health care and are a key component in the national plan for improving psychiatric services in Norway (5). General practitioners refer patients to the centers, have contact with patients while they are treated there, and continue treatment and monitoring of patients after discharge from the centers.

The results from the first national evaluation should produce quality indicators at the institutional level, as part of the Norwegian national quality indicator system. Therefore, the indicator should have relatively large variation among providers that is not due to random variation or patient characteristics (6). Several international studies have included general practitioners' or family physicians' evaluations of mental health services (7,8,9,10,11,12,13,14,15,16,17,18,19). However, none of these studies have been used as a basis for quality indicators at the institutional level. Studies of patient satisfaction, on the other hand, have been used in a variety of settings as the basis for quality indicators.

After such surveys are conducted, the usual approach for institutional benchmarking is to develop an appropriate case-mix model and correct for multiple comparisons in statistical testing (20,21). Another emerging approach is to use multilevel analysis to estimate the amount of variation in experiences or satisfaction scores that can be explained by levels above the individual level (22,23,24,25,26,27). These studies have found that only a small part of the variation is attributable to the organizational level. In contrast to patient satisfaction studies, none of the studies relating to general practitioners' or family physicians' evaluation of mental health services have been used to compare institutions. We found neither a comprehensive approach to adjust for differences in general practitioner composition nor any multilevel analysis to assess variation at different levels. However, the question of the contribution from the organizational level compared with the individual level is highly relevant in comparisons of centers based on evaluations by general practitioners. First, the use of center assessment scores as quality indicators in the national quality indicator system raises the question of the appropriateness of benchmarking centers by these indicators; that is, how much of the variation in the general practitioner evaluations can be explained by center-level factors? Second, patients in psychiatric and somatic health care have relatively little in common apart from their being ill and their encounters with health care institutions, whereas general practitioners have similar professional training and work experience, often sharing offices with each other, and other factors contribute to making general practitioners a more homogeneous group than patients. Therefore, the effect of clustering can be expected to be larger among general practitioners than among patients.

We conducted a national survey in Norway in 2006, where all general practitioners were asked to evaluate the community mental health center responsible for general adult psychiatric services in their area (28). The national survey achieved an acceptable response rate (29), and the survey instrument, the General Practitioners' Experiences Questionnaire (GPEQ), has good psychometric properties (30). In this study we assessed the amount of variation in the five GPEQ scale scores that was explained by variables at the individual level and the community mental health center level. In line with findings from other multilevel analyses of patient satisfaction or experiences (22,23,24,25,26,27), we hypothesized that individual variables would explain more of the variation than organizational variables. We also hypothesized that compared with the findings from patient satisfaction studies, where only a small part of the variation in experiences or satisfaction scores were attributed to the organizational level, our study with general practitioners would show that the above-mentioned clustering effect would produce a higher intraclass correlation coefficient, with larger variation at the organizational level.

Sample

The study sample included all registered general practitioners in Norway for January 2006, with the exception of those in the two counties included in the pilot study. The questionnaire was mailed to 3,463 general practitioners in January 2006. The physicians were asked to evaluate the community mental health center responsible for general adult psychiatric services in their locality. Nonrespondents were sent three postal reminders, and there was an additional telephone follow-up with nonrespondents after four postal contacts. The data collection took place from January 20 to March 23, 2006. Telephone respondents were excluded from the analysis that follows to avoid mixed-mode effects. The procedure regarding informed consent, study design, and data collection was approved by the Norwegian Social Science Data Services.

Materials

The GPEQ questionnaire comprises five scales with good evidence of data quality, internal consistency, test-retest reliability, and validity (30). The competence scale includes four items relating to assessment and treatment skills. The discharge letter scale includes three items relating to quality of discharge letter, further plans for general practitioners' follow-up of patients, and assessment of the time it takes to receive the discharge letter. The emergency situations scale includes two items relating to contact with and help from the center in emergency situations. The guidance scale includes three items relating to cooperation meetings between practitioner and center, organized training for general practitioners, and receipt of necessary professional support. The workforce situation scale includes four items relating to stability in key positions and physician coverage at the center. All items have a 5-point response format that ranges from "not at all" to "very large extent" and produce five scale scores from 0 to 100, where 100 indicates the best possible experience with the centers.

The questionnaire also includes questions about knowledge of the center, experiences of negative episodes with the center, and general impression of psychiatric services. The questions about negative episodes with the center and knowledge of the center concern one specific center, and the question of general impression relates to all experiences with psychiatric services. Hence the latter also includes assessment of other centers in addition to the one practitioners were asked to assess within the study, in addition to past and present experiences with other mental health services in primary and specialized mental health care. These variables are at the ordinal level but were treated as continuous independent variables because they have a significant linear association with all five dependent variables.

Demographic and practice-related data about general practitioners came from the Norwegian Medical Association and the Norwegian Labour and Welfare organization and included gender, age, number of years as a general practitioner, type of practice (group or solo), additional employment, available positions on the general practitioner list, and address of the practice.

The size of the catchment area and the number of consultations in 2005 for each center were extracted from the SAMDATA reports in Norway for mental health services (31). The number of inhabitants in the center catchment area was grouped as follows: 15,000—29,999; 30,000—74,999; and 75,000—150,000 (32). Center size was grouped into less than 5,000 consultations, 5,000—9,999, 10,000—14,999, 15,000—19,999, and more than 20,000 consultations in 2005. Information about center location came from the Norwegian Directorate for Health and Social Affairs. We constructed a variable for center location, which refers to proximity to urban settlements and their size (33). We categorized designations into least central municipalities (<5,000 inhabitants), somewhat central municipalities (5,000—14,999), quite central municipalities (15,000—50,000), and most central municipalities (<50,000).

Analysis

We present descriptive statistics for the independent and the dependent variables. The association between the independent variables and the five scales was first assessed by bivariate linear regression in SPSS 15.0. To assess the variation at the individual and center levels and the effects of individual and organizational variables on the five scales, we used multilevel regression in MLwiN 2.02 (34). Multilevel modeling is recommended when testing the effects of group-level variables (35), because the analysis specifically divides the total variance in the GPEQ scales into variance at the center (macro) level versus the general practitioner (micro) level. The empty model included the centers as random intercepts. This model was used to calculate the original amount of variance explained at the center level for each of the five GPEQ scales using the intraclass correlation coefficient.

The full model included all variables at the individual and organizational levels for each of the five scales. Individual-level variables for general practitioners were gender, age, years of service as a general practitioner, knowledge of center, group versus solo practice, general impression of mental health care, and experience of negative episodes that the center could have prevented. Center-level variables were center size, number of inhabitants in the catchment area, and center location. Fixed effects of the explanatory variables were tested by the t ratio, defined as the estimated coefficient divided by its standard error. The proportional reduction in variance was assessed separately for the two levels according to the procedure described by Snijders and Bosker (35).

A total of 2,120 general practitioners (61.2%) evaluated 71 community mental health centers (Table 1). There were no gender or age differences between respondents and the total sample. There were statistically significant differences for other background variables, but these had little association with the main outcome variables (29). Ten centers were located in the southern health region, 21 in the eastern health region, 16 in the western region, ten in the middle region, and 14 in the northern region.

The sample size per center varied from four to 71, with a mean of 30 (Table 1). The number of consultations per center in 2005 varied from 1,815 to 32,545, with a mean of 10,153. The mean number of inhabitants in the catchment area per center was 47,397, and varied from 9,597 to 116,810. Close to half (46.5%) of the centers were located in the most central municipalities, and 16.9% of the centers were located in the least central municipalities.

The mean age of the general practitioners was 48.6 (Table 1), 67.8% were male, and 79.0% were part of a group practice. A majority of general practitioners reported very good (12.7%) or rather good (54.4%) knowledge of the center, with a small proportion reporting rather poor (6.8%) or very poor (.6%) knowledge of the center. A total of 27.6% of the general practitioners had experienced negative episodes with the center a few times, 2.1% many times, and 56.3% had never experienced negative episodes that the center could have prevented. When asked about having a good general impression of psychiatric services, 28.6% of the general practitioners rather disagreed and 16.2% totally disagreed. The national mean scores for the GPEQ scales were competence, 54.9; discharge letter, 51.8; emergency situations, 52.3; guidance, 30.9; and workforce situation, 44.8 (Table 2).

The bivariate regression models showed that all independent variables had one or more significant associations with the scales (Table 3). The multilevel regression models showed that female general practitioners had significantly poorer experiences with the centers than did males for three of five scales (discharge letter, guidance, and workforce situation), after adjusting for the other individual- and center-level variables (Table 4). Years as a general practitioner had no significant association with the scales, whereas age and type of practice had one each. Age was positively associated with discharge letter, whereas general practitioners in solo practices had significantly higher scores for the workforce scale than general practitioners in group practices. Knowledge of the center, negative episodes with the center, and general impression of psychiatric services were highly significantly associated with all scales.

At the center level, size of catchment area had no significant association with the five scales. Center size had only one significant association with the scales; centers with 15,000—19,999 consultations had significantly higher scores for the workforce situation scale than centers with more than 20,000 consultations. Center location had several significant associations with the scales. Centers in the most central municipalities had significantly lower scores for two GPEQ scales (competence and guidance) than centers in the least central municipalities. Centers in quite central municipalities had significantly lower scores than centers in the least central municipalities on three scales (competence, guidance, and workforce situation). Health region was not significantly related to any of the scales (results not reported here).

Total explained variance at the individual level varied from 24.4% for emergency situations to 38.2% for competence (Table 5). Total explained variance at the center level varied from 40.3% for emergency situations to 69.9% for competence. The intraclass coefficient varied from 10.3% for discharge letter to 23.2% for workforce situation.

The study found that general practitioners' experiences with community mental health centers have substantial variation at the center level. The intraclass correlation coefficients ranged from 10% for discharge letter to 23% for workforce situation. At the individual level, practitioners' general impression of psychiatric services, knowledge of the center, and negative episodes with the center were significantly associated with the five GPEQ scale scores. Female general practitioners had significantly lower scores than males on three scales. At the center level, the center's location was significantly related to three scales. The final multilevel regression models explained a substantial part of the variation at the individual and center levels for all scales.

In line with findings of other multilevel analyses of patient satisfaction or experiences data (22,23,24,25,26,27), we found that individual variables explained more of the variation than organizational variables. We also found support for our hypothesis of a higher intraclass correlation coefficient in this study than in surveys of patients' experiences. Both patients and general practitioners are subject to a center or institutional clustering effect. Moreover, general practitioners have common professional experiences, which make them a more homogeneous group than patients. Female general practitioners had significantly poorer experiences than males had for three of the GPEQ scale scores. The differences were relatively small but indicate the need to include gender in the adjustment model when comparing results between centers. One possible explanation for the gender difference might be that female general practitioners had slightly higher expectations regarding the centers; another explanation is that female general practitioners may have more severely ill patients (36), therefore having greater need for services from the centers.

Poor knowledge of the center, negative episodes with the center, and poor general impression of psychiatric services had by far the strongest associations with GPEQ scores. Each of the factors might have a different set of explanations, but it is possible to interpret these findings in relation to the large amount of literature that has identified the need for improving communication and cooperation between general practitioners and specialists (11,16,37,38,39,40). By establishing good communication and cooperation with the local general practitioners, including mutual respect, community mental health centers might improve the knowledge and image of their centers and psychiatric services in general and reduce the occurrence of negative episodes.

The work in Norway has focused on baseline measurement at a national level. Local initiatives for quality improvement and their relationship with changes in general practitioner experiences over time will be assessed after the next national survey, proposed for 2009. The scales and concrete questions in the questionnaire might be a means for identifying problem areas and form a basis for quality improvement. For instance, dialogue with local family physicians about the frequency and quality of cooperation meetings and guidance could identify relevant quality improvement initiatives. This must be supplemented with cost-effective initiatives to improve collaboration between primary care and specialized mental health care identified within the scientific literature, including Cochrane reviews (41,42).

The substantial contribution from the center level, as measured by the intraclass correlation coefficients, supports the appropriateness of using GPEQ scores as quality indicators in the national quality indicator system. However, our study indicated that GPEQ scores should be adjusted for general practitioners' gender and general impression of psychiatry when comparing centers. It is important to adjust for important factors outside the centers' control, but the consequences and possible implications of adjusting GPEQ scores should be further explored. Future studies should explore the association between GPEQ scores and other quality indicators and include additional information about the centers to provide further understanding of the contribution from the center level. The latter consideration is of particular importance for the GPEQ scales of guidance and workforce situation, where the center residual variance is largest.

The development and validation of the questionnaire have followed a rigorous scientific process to test and document the reliability and validity in a Norwegian context. The GPEQ is acceptable to general practitioners and has good evidence for data quality, internal consistency, test-retest reliability, and validity (30). However, the GPEQ cannot automatically be assumed to have reliability and validity in other countries. We recommend that the GPEQ be considered by other countries for assessing the quality of community mental health centers after consideration of organizational aspects of care and necessary cross-cultural validation work.

Our study is relevant for a large part of psychiatric services in Norway and relevant for many Western countries. Within Norway, community mental health centers offer a variety of services and are the largest part of specialized mental health services. Almost all outpatient and ambulant services are provided by community mental health centers, and nearly half of the inpatient services are at the centers. Therefore, the results of our study of family physicians' assessments of community mental health centers are both representative and include a substantial proportion of specialized mental health services in Norway. Primary health care in Norway is organized much the same way as in Canada, the United Kingdom, and several other Western countries, with the exception of the United States. This means that specialized mental health services in Norway have mostly the same responsibilities as in other Western countries. Consequently, the results of our study have large international relevance.

Our study demonstrates the appropriateness of comparing general practitioners' evaluations of community mental health centers. The national GPEQ scores show a large potential for quality improvement, with mean scores well below the maximum possible. The multilevel regression models explained a substantial part of the variation at the individual and the center levels for all scales, with general impression of psychiatric services, knowledge of the center, and negative episodes with the center being the strongest factors associated with GPEQ scores. Discussion, implementation, and evaluation of initiatives to improve communication and cooperation with general practitioners may be necessary to improve their assessments of community mental health centers.

The collection of data and writing of the report were funded by the Norwegian Knowledge Centre for the Health Services. The views expressed are those of the authors.

The authors report no competing interests.