WHO's Assessment Instrument for Mental Health Systems: Collecting Essential Information for Policy and Service Delivery

The global burden of neuropsychiatric disorders is substantial. When measured by years lived with disability and years lost as a result of premature death in disability-adjusted life years, psychiatric and neurological conditions accounted for 13% of the global burden of disease in 2002 ( 1 ), because many of these conditions are chronic, recurring, and quite disabling. These figures, however, do not capture other types of burden associated with mental disorders, including the burden of caregiving for family members, financial costs, stigma, and human rights violations ( 2 , 3 , 4 , 5 ). There is a large body of research that shows that there is a range of effective interventions to help people with severe mental disorders in both Western ( 6 ), and non-Western countries ( 7 ). Cost-effectiveness research using economic modeling has shown that effective treatment of common mental disorders appears in the same range of cost-effectiveness as antiretroviral drugs for HIV-AIDS, secondary prevention of hypertension, and glycemic control for diabetes, with a cost that is below the average per capita income ( 8 ).

Despite the huge burden of mental illness and the availability of effective interventions, few resources are directed toward mental health care. Mental health spending in many countries in the world is less than 1% of health expenditures (which are already very low in most middle- and low-income countries), 31% percent of countries have no separate mental health budget, and the number of mental health professionals is grossly deficient ( 9 , 10 ). Resources for mental health are particularly sparse in low-income countries ( 9 , 10 , 11 , 12 ). For example, the median number of psychiatrists per 100,000 population in low-income countries is .05 (compared with 10.5 in high-income countries), and the number of psychiatric nurses is .16 (compared with 32.9 in high-income countries) ( 9 ). Approximately, one-third of all countries in the world have no mental health policy or plan. Moreover one in three people in the world live in countries without mental health legislation and thus without specific legal protection for people with mental illness ( 9 ). Because of the low level of resources devoted to mental health, it is not surprising that a majority of people with mental illness in the world go untreated despite the fact that effective treatments exist. Global estimates of untreated cases vary from 32.2% for schizophrenia (including other nonaffective psychosis) to 56.3% for depression to 78.1% for alcohol and drug use disorders ( 13 ). Estimates for untreated serious mental disorders in developing countries are as high as 85.4% ( 14 )

In order to reduce the burden of neuropsychiatric disorders, it is important to obtain valid and reliable information about a country's mental health system ( 15 ). Unfortunately, a number of countries are handicapped by the lack of information available on their mental health systems. Indeed, the World Health Organization (WHO) Atlas Study ( 9 ) reported that more than 24% of 202 WHO member states and associated members, territories, and areas do not have a system for collecting and reporting even basic mental health information. Other countries have information systems, but these systems are typically neither comprehensive nor appropriate for mental health planning. Problems caused by a lack of information include a deficient planning process, impeded accountability, incapacity to monitor the change promoted by mental health reforms, and the potential for developing ad hoc solutions before understanding the situation. In this article we present an instrument designed for middle- and low-income countries that assesses key components of a mental health system and thereby generates essential information that can be used to strengthen mental health policy and service delivery.

The WHO-AIMS

The WHO Assessment Instrument for Mental Health Systems (WHO-AIMS) ( 16 ) differs from other widely used monitoring and assessment instruments—for example, Performance Indicators for Mental Health Trusts, which assesses the performance of local mental health services in the United Kingdom ( 17 ); the Mental Health Report Card, which was developed to monitor mental health plans in the United States from a mostly consumer perspective ( 18 ); the mental health indicator scheme of the European Community Health Indicators, covering population-level measures of negative and positive mental health ( 19 ); and the mental health indicators included in Healthy People 2010, the U.S. health targets to be achieved between 2000 and 2010 ( 20 , 21 ). In contrast to the aforementioned schemes, WHO-AIMS was specifically designed for the needs of developing mental health systems in low- and middle-income countries. Consequently the scope, objectives, structure, contents, and data collection methods of the WHO-AIMS instrument differ significantly from existing indicator schemes. For example, WHO-AIMS was designed to provide a comprehensive assessment of the mental health system, as well as services and supports for people with mental disorders that are provided outside the psychiatric services sector—for example, mental health in primary care. In contrast, most existing monitoring instruments focus more narrowly on the psychiatric services sector.

There are also differences in objective. Whereas WHO-AIMS was designed to map all of the essential existing formal mental health resources in a country, the primary focus of most other instruments is to measure the quality or performance of mental health services or a particular service facility. There are also structural differences between WHO-AIMS and other common indicator schemes. For example, many indicator schemes contain a large number of outcome measures, primarily because these instruments are performance focused. Because many low- and middle-income countries lack the basic infrastructure of a mental health system (for example, community-based services) or a functioning information system, outcome data—essential for assessing effectiveness—would be extremely difficult to collect. Thus the WHO-AIMS instrument primarily consists of input indicators (resources that are used to develop or modify system and services) and process indicators (assessment of utilization of services, as well as aspects of quality of services and programs).

WHO-AIMS also contains a number of ordinal rating scales, again because precise data are difficult to collect in some settings. For example, very few low-income countries record the number of times that patients are restrained or secluded in mental hospitals, making it difficult to obtain precise figures for this important issue. In WHO-AIMS the issue of restraint and seclusion of patients has been addressed through an ordinal rating scale that asks for estimates on the percentage of patients who were restrained or secluded at least once in the past year. Estimates are based on a 5-point scale in which A=over 20%, B=11%–20%, C=6%–10%, D=2%–5%, and E=0%–1%. Respondents are instructed to provide precise data if these are available, and if data are not available, to provide their best estimate based on other information or a data source—for example, consulting key informants in mental hospitals and reports from nongovernmental organizations. In addition, in terms of content, WHO-AIMS contains a number of items that measure important mental health resources in low- and middle-income countries that may not be as critical in high-income countries. For example, there are items addressing traditional healers and paraprofessional primary health care workers. Finally, in terms of data collection, WHO-AIMS collects information from all relevant organizations at all levels of the health system. (For details of this process see pages 6–8 of WHO-AIMS 2.2 available at www.who.int/mentalhealth/evidence/AIMSWHO22.pdf. This section describes a process of the systematic data collection, which is conducted in consultation with and with regular support from the WHO-AIMS team in Geneva. After the data are collected, they are systematically checked by the team to identify potential errors.)

Development of WHO-AIMS 2.2

The ten recommendations of the World Health Report 2001 ( 6 ) served as the foundation of the instrument, because they represent WHO's vision for the improvement of mental health systems to reduce the burden of mental disorders. These recommendations are providing treatment in primary care; making psychotropic drugs available; providing care in the community; educating the public; involving communities, families, and consumers; establishing national policies, programs, and legislation; developing human resources; linking with other sectors; monitoring community mental health; and supporting more research. These general recommendations provide a vision for mental health planning.

In order to operationalize the recommendations (each recommendation was considered a domain of interest), a large number of items were generated and grouped together into a number of facets (subdomains). For example, for the World Health Report recommendation to "establish national policies, programmers, and legislation," a large number of items were developed pertaining to the policy and legislative framework. These items were then grouped into subdomains, including mental health policy, mental health plan, mental health legislation, monitoring of human rights, and financing of mental health services. Experts and key focal points from resource-poor countries provided input through two consultations to ensure the clarity, content validity, and feasibility of the generated items. On the basis of this feedback, a pilot version of the instrument was released and tested in 12 low- and middle-income countries—Albania ( 22 ), Barbados, Ecuador, India, Kenya, Latvia, Moldova, Pakistan, Senegal, Sri Lanka, Tunisia, and Viet Nam. These countries were selected on the basis of regional diversity and size. The aim of the pilot study was to assess both the clarity and feasibility of the WHO-AIMS items and the meaningfulness and usefulness of the information collected. The results suggested that the set of indicators was useful to assess the mental health services and system in a comprehensive manner. However, the number of items needed to be reduced to improve the feasibility of the instrument.

The instrument was substantially revised and shortened on the basis of data from the pilot study, including information on the number of countries that were able to collect data for each item, information on which items showed too little variability across countries, and the number of countries reporting the presence of some mental health activity for each item. Items were dropped that were problematic—for example, items that had a very low response rate. For the remaining items, 12 WHO technical staff members ranked all items in terms of their importance for planning public mental health action in low- and middle-income countries. Finally, five WHO staff members rated each item on a 3-point scale (low, medium, and high) on the extent to which it was perceived to be meaningful, feasible, and actionable. The research literature on mental health indicators suggests that good indicators possess these attributes ( 23 , 24 ). Other considerations included to what extent each item added value over other items, how sensitive each item was to change, and whether the items together were comprehensive enough to cover the whole mental health system.

All of the information obtained through the procedures described above was utilized in producing a revised version of the instrument. The instrument was subsequently reviewed by 14 representatives of low- and middle-income countries. A revised and edited version, WHO-AIMS 2.2, was published in 2005.

WHO-AIMS 2.2 consists of six domains: policy and legislative framework, mental health services, mental health in primary care, human resources, public information and links with other sectors, and monitoring and research. These domains address the ten recommendations of the World Health Report 2001 ( 6 ) through 28 facets and 155 items. An overview of the domains and facets of WHO-AIMS 2.2, along with sample items, is provided in Table 1 . All six domains need to be assessed to form a basic, yet broad, picture of a mental health system, with a focus on health sector activities. The current version of the instrument includes supporting documentation (answers to frequently asked questions, guidance on data collection in WHO-AIMS, and definitions of frequently used terms), a data entry program, and a template for writing country reports. In addition, WHO-AIMS 2.2 contains a list of items that comprise a brief version of the instrument (WHO-AIMS-Brief), which may be used for a rapid assessment of mental health systems. However, the use of the full instrument is strongly recommended for obtaining a comprehensive picture for planning relevant mental health action.

Enlarge table

WHO-AIMS is primarily intended for assessing mental health systems in low- and middle-income countries. It can be used for an entire country or for a region, state, or province within a large country, such as India, Brazil, or China. In addition, most items on the instrument are relevant and applicable to resource-poor settings within high-income countries. The instrument may also be useful for high-income countries where there is a desire to achieve a comprehensive picture of the mental health system outside of psychiatric services. Although duration of data collection will vary from country to country, the data for WHO-AIMS can be collected in most countries in two to three months.

Use and benefits of WHO-AIMS

WHO-AIMS provides essential information for mental health policy and service delivery. Countries will be able to develop information-based mental health policy and plans with clear baseline information and targets. Moreover, they will be able to monitor progress in implementing reform policies, providing community services, and involving consumers, families, and other stakeholders in mental health promotion, prevention, care and rehabilitation.

Through WHO-AIMS, countries will have a fairly comprehensive picture of the main weaknesses in their mental health system, and this knowledge should facilitate improvements over time. Moreover, it is hoped that the process of data collection will stimulate system-level thinking by governments and managers of health systems and prompt them to build a data infrastructure, implement data system improvements, and build a network for mental health action. In addition, WHO-AIMS also provides a template for assessment and action by agencies other than WHO. For example, the Department of Health of a low-income country using the WHO-AIMS instruments finds that psychotropic drugs are available in only 50% of the primary health clinics. Policy makers and health managers decide that they need to improve access to psychiatric medications in primary care, and they agree on two goals for the next five years: increasing by 10% every year the percentage of clinics in which the drugs are available and training primary care staff on psychotropic drugs in the clinics in which the psychotropic drugs become available. The Department of Health can carry out an analysis of the needed resources, mobilize funds in partnership with a nongovernmental organization, and use the relevant WHO-AIMS items to follow the improvement and provide quick feedback.

Discussion and conclusions

Although WHO-AIMS provides a useful planning tool for low- and middle-income countries, limitations exist. One of the most notable limitations involves the lack of psychometric approaches employed during the development process. Although the development of the instrument was systematic and involved mechanisms to assess face and content validity, it was not possible to conduct traditional quantitative psychometric analyses (for example, item analysis and factor analysis) to establish other types of validity and reliability of the instrument because of the small sample in the pilot study. The difficulty of obtaining a sample large enough to conduct traditional psychometric analyses is not unusual in the program evaluation field, where the unit of assessment is programs rather than individuals. Consequently, the establishment of content and face validity through expert rankings on a number of dimensions is a common and accepted practice in the field ( 25 , 26 , 27 ). In addition to the measures taken to establish content and face validity for the instrument, a systematic evaluation of the domain, facet, and item structure will be undertaken on newly collected data. The feasibility and usefulness will also be evaluated at this time.

In addition, there are certain models of care implicit in WHO-AIMS—for example, ICD mental and behavioral disorders, psychiatric and psychosocial services, and policies on the need to reduce the burden of disease. This is consistent with the World Health Report 2001 ( 6 ), the model of care that is used by almost all Ministries of Health in the world. However, alternative models of care exist outside the formal health sector. This instrument does not assess these alternative models of mental health care.

Other weaknesses of the instrument include the difficulty of operationalizing some key constructs. For example, the issue of equity of access to mental health services for minority groups is a significant one in most countries of the world, because most mental health resources in countries are used by the urban, affluent members of the ethnic (or religious) majority population ( 28 , 29 , 30 ). Measuring the extent of this inequity is important in promoting more equitable access to mental health care. However, it is very difficult to operationalize quantitative items that assess this issue. Consequently, many of the items in WHO-AIMS that address issues of equity employ ordinal rating scales. Although it would be preferable to include only items that utilize objective data from reliable sources, these data are often not available. Potential use of surveys (complementary to use of WHO-AIMS 2.2) may help improve assessment of complex issues, such as measuring equity in access to care and within-country differences.

A related issue concerns constructs that are important to include but were not able to be operationalized even in a qualitative fashion. For example, we were interested in whether any restrictions are placed on the coverage of mental health treatments by social insurance schemes. However, it was difficult to create an item to measure this issue, because there are many different ways in which restrictions could be imposed—for example, only hospital services are provided, only hospital services plus a few ambulatory interventions are provided, or restrictions are placed on the length of treatment. Consequently, this issue is not covered in WHO-AIMS 2.2. However, this item and others like it are on a "developmental list" for possible inclusion in a future edition. WHO-AIMS is also limited in assessing mental health promotion activities, including the measurement of community supports for those in distress. Finally, despite the comprehensiveness of WHO-AIMS in describing mental health systems to assist people with mental disorders, it does not assess the quality or performance of mental health services or a particular service facility.

To our knowledge, WHO-AIMS is the first comprehensive mental health system assessment designed for low- and middle-income countries, giving an overview of the nature and extent of mental health services in a country. More specifically, it is unique in that both the conceptual foundation for the instrument (the World Health Report 2001 ) as well as the development process took into account knowledge of unique mental health system needs of low-income countries. For example, the instrument does not rely on sophisticated technologies for the data collection. More important, in-country experts from low- and middle-income countries were collaborators in the instrument's development by piloting the instrument and providing feedback. Their input was sought at every stage of the instrument's development, and we believe that the active involvement of in-country collaborators helped to raise the relevance, feasibility, and usefulness of the instrument to low-resource settings.

The instrument will continue to be developed and refined. Particular attention will need to be given to assessing and improving response bias, interrater reliability, and other threats to the validity of the collected information. For example, Delphi methods may potentially be used to obtain consensus estimates of data that are particularly difficult to collect. Future revisions based on ongoing data collection should serve to strengthen the instrument. As most items in WHO-AIMS describe aggregate information (usually at the country level), further development of WHO-AIMS may involve linking collected data with geographical information systems to map within-country differences. This process should not only improve the instrument but also provide the necessary information to improve policy and service delivery for people with mental disorders around the world.

Acknowledgments and disclosures

This project received financial assistance or personnel from the National Institute of Mental Health; the Center for Mental Health Services; the Health Authority of Regione, Lombardia, Italy; the Ministry of Public Health of Belgium, and the Canadian Institutes of Health Research. WHO Regional Office staff and numerous expert reviewers contributed to the work reported in this article. A complete list of these persons is available on page 5 of the published version of WHO-AIMS 2.2 (www.who.int/mentalhealth/evidence/AIMSWHO22.pdf).

The authors report no competing interests.