A Critical Review of Internet Information About Depression

Since its inception in the late 1960s as a small network of academic and military computers, the Internet has grown into a vast commercial enterprise that is poised to transform modern commerce, entertainment, news, and publishing. Dot-com companies, online shopping, and teleconferencing are changing the way millions of Americans conduct their everyday business. In the open marketplace of the Internet, anyone who has a computer and a modem has access to information and ideas from around the world.

As Internet connection speeds and consumer interest have grown, so has the online presence of businesses, institutions, and individuals. Over the past decade, the number of U.S. households with access to the Internet increased from 2,000 to 31.3 million (1), and the number of Web pages exploded to more than a billion (2). Online spending increased from $4.9 billion in 1998 (3) to $66 billion in 1999 (4).

The number of U.S. Internet users aged 16 years and older is estimated to be 92 million (5)—a sizable proportion of the total population. Within a decade, high-speed Internet access may become as ubiquitous as television. With the increasing presence of the Internet in daily life, concern about the quality and character of the information it contains has grown.

Initial governmental attempts to regulate Internet content were struck down in 1997, when the U.S. Supreme Court unanimously ruled that the Communications Decency Act was unconstitutional. Although this legislation addressed primarily pornography and content deemed unsuitable for children, the Court's unwillingness to uphold any limitations set a precedent that gives anyone wide scope to post virtually any type of information. U.S. business and trade regulations prevent outright fraud regardless of the medium, but many nations have neither the resources nor the inclination to enforce similar rules. Web pages can come from anywhere—the house next door, the former Soviet Union, or an atoll in the South Pacific.

A number of articles in the major medical journals have raised concerns about the quality of Internet sites that provide health care information (6,7,8,9,10,11). In 1999, "Operation Cure.all" was launched by the U.S. Federal Trade Commission to stop bogus health claims and fraudulent products from being offered online. At that time, more than 800 sites were offering miracle cures—from shark cartilage to magnetotherapy. Even after they were informed that they were in violation of U.S. federal law, more than three-quarters of site owners failed to remove unsubstantiated claims (12).

The fundamental problem in monitoring health care information on the Internet is one of scope. More than a billion Web sites are accessible at any given time, and they can appear and disappear almost instantly. They relocate, change format, and, increasingly, provide content tailored to the individual viewer. Anyone—with or without medical credentials—can create a professional-appearing Web site and offer medical advice and treatments.

The Internet is in many ways the ultimate vanity press—there is no peer review, no way to distinguish authentic information from biased or unqualified sources, and little governmental oversight to prevent fraud. There are few consequences outside of the civil realm for publishing misleading, unsubstantiated, or grossly inaccurate medical information. With relative ease, anyone can offer their expertise for sale, worldwide, in the form of books, advice, miracle cures, or clinical services. Despite these concerns, few formal studies of the quality and quantity of health-related information on the Internet have been conducted.

Internet statistics and usage patterns are derived mainly from surveys and Nielson/NetRatings (13,14,15). These sources uniformly document Internet users' short attention span, their tendency to rely on the Internet for a wide variety of personal and professional information, and their increasing use of the Internet to purchase goods and services.

Most consumers locate information on the Internet through the use of search engines such as Netscape, InfoSeek, AltaVista, and Yahoo (14). Keywords are entered and a list of relevant Web sites is displayed. The order in which the sites are displayed depends on a number of factors, including popularity, size, and the placement fee paid to the owner of the search engine.

A review of the medical literature revealed only two studies that evaluated the quality of health care documents generated by Internet searches using medical keywords. In 1998 Rose and colleagues (16) surveyed 100 patients at an outpatient orthopedic knee clinic about their Internet access and use. Seventy-nine percent of the patients had access to a computer at home or at work, and 56 percent used a computer regularly. The patients were asked to identify the keywords they themselves would use to search the Internet for knee-related information. The authors then conducted their own search using 25 of these words—which included, for example, "knee," "arthroscopy," and "cruciate."

Of the 5,946 Web pages Rose and colleagues reviewed, about 20 percent contained patient information about orthopedics, and 19 percent contained information aimed at professionals. However, of the sites that contained patient information, only 7 percent contained actual knee-related information. Thus a search for patient information on knee-related orthopedic topics generated nearly 6,000 pages, of which only 395 contained the information being sought. The authors concluded that there was a clear need for reliable and valid Web sites that would be useful for patients.

McClung and colleagues (17) conducted a keyword search in 1998 on three popular Internet search engines using the terms "diarrhea" and "treatment" together. The information in the first 300 matches was evaluated for compliance with current American Academy of Pediatrics (AAP) guidelines for treating diarrhea. Of the 70 sites that contained information relevant to the topic, 86 percent were from traditional medical sources, such as teaching centers, physicians, and professional societies, and 14 percent were from alternative sources, such as homeopathic and chiropractic sources. Of the sites from traditional medical sources, only 46 percent discussed treatment of diarrhea and only 20 percent made recommendations that approximated AAP guidelines. None of the recommendations made on the alternative sites were in compliance with the guidelines.

Of particular concern to the authors of that study was the abundance of inaccurate information and deviations from accepted scientific fact, even at sites provided by traditional medical sources. Examples of grossly erroneous information included statements that diarrhea is the body's method for eliminating undesirable elements, that it is caused by junk food, and that it is the cause of Reye's syndrome. Potentially harmful treatment suggestions included incorrect recipes for rehydration solutions, instructions to limit oral intake, recommendations for antibiotics that have not been approved for children, and recommendations for unproven oatmeal, fat, and protein treatments. These authors also concluded that there was a need for reliable and accurate medical information on the Internet.

To our knowledge, no studies have formally evaluated the quality of mental health information on the Internet. The purpose of this study was to locate and analyze information about a common mental health problem—depression.

Methods

We conducted a prospective, observational literature review by performing keyword searches with ten of the most popular search engines: Excite, AltaVista, Infoseek, Netscape, Microsoft Network, Snap, GoTo, Hotbot, Yahoo, and Lycos (14). To simulate an Internet search that was likely to be undertaken by clinicians or patients looking for information about depression, the phrase "depression and treatment" was entered. The search was conducted in April 2000.

The first 20 sites generated by each search engine were reviewed, for a total of 200 sites. The sites were first categorized by whether the source of information was a for-profit or a not-for-profit entity. Any site that directly advertised products for sale, including those that displayed banner advertisements, was categorized as for-profit, as were sites that were affiliated with major pharmaceutical manufacturers, manufacturers of health care equipment, or corporations known to function in a for-profit capacity, even if they were not directly offering a product for sale.

Not-for-profit sites were defined as those not affiliated with a profit-oriented entity and not offering products for sale. The majority of these sites contained postings by academic and government agencies, individual health care consumers, and consumer advocacy groups.

Sites were further categorized by source. Sources included online health information services, academic institutions, pharmaceutical manufacturers, hospitals, online pharmacies, booksellers, publishers, companies that sold other products (for example, light boxes and medical devices), professional organizations, consumer advocacy groups, private health care providers (for example, counselors, physicians, and naturopaths), government agencies, and the personal pages of individual patients.

Each Web site was scored for content on the basis of how well the information it provided complied with accepted medical knowledge about the diagnosis and treatment of depression. The diagnostic standard by which all the sites were judged was DSM-IV criteria for major depressive episode. Each site was assigned a score of 0 to 14 on the basis of its description of the cardinal features of depression. One point was given for each of the nine symptoms or signs of depression, and an additional point was awarded for each major criterion listed (A, B, C, D, and E). When a site included diagnostic information that diverged significantly from DSM-IV criteria, one point was deducted for each erroneous statement. Although point deductions were rarely necessary, they were taken when symptoms of depression included "drinking too much" or "itching sensations."

We had intended to use American Psychiatric Association guidelines as the standard for judging treatment recommendations; however, the paucity and vagueness of treatment information contained on the Web sites made this standard inapplicable. Instead, a more basic system was devised in which each site was assigned a score of 0 to 3 on the basis of the mere mention of a generally accepted modality for treating depression. One point was awarded for mentioning consultation with a physician or a mental health care professional of any sort, another for the use of medications as an option, and a third for listing psychotherapy as a possible treatment. Points were given regardless of where in the Web site these treatments were mentioned—that is, they did not have to appear in a section devoted exclusively to treatment.

Points were not deducted for explicit or implicit suggestions to purchase or seek additional resources or to explore alternative treatments unless a suggestion was so erroneous and baseless as to represent fraud or constitute a potential danger. For example, the suggestion to purchase a "super-mood formula," which consisted of a blend of vitamins, minerals, and plant extracts of no proven benefit, resulted in a score of 0.

Results

All Internet search strategies yield some inactive links and extraneous documents. Of our initial 200 sites, 22 (11 percent) were inactive or contained information completely unrelated to depression in the psychiatric sense—one, for instance, was devoted to the economic depression of the 1930s. Many of the remaining 178 sites appeared multiple times, regardless of which search engine was used. When the 178 Web sites were pooled, 43 percent of them emerged from the same 24 sites. Among the sites that appeared multiple times were hypericum.com, which is owned by a for-profit St. John's wort vendor; depression.com, which is owned by PlanetRx.com, an online pharmacy and supplier of herbal remedies; mentalhealth.com, a Canadian public information resource; mentalhelpnet.com, which is owned by a publisher of psychology and self-help materials; and psycom.net, a not-for-profit guide to physician treatment of refractory depression, created by a biological psychiatrist.

Table 1 summarizes the sites' scores for diagnostic criteria and treatment recommendations. The majority of sites contained little reliable information about the diagnosis of depression. Half of the sites listed no DSM-IV symptoms or criteria at all, and only one-quarter listed 11 to 14. Most of the sites failed to make even basic treatment recommendations. More than half made no mention at all of medications, psychotherapy, or any type of professional help as possible treatments for depression, or they mentioned only one option. Seventy-five sites (42 percent) mentioned all three options.

As Table 1 shows, 109 of the 178 sites (61 percent) were classified as for-profit. Most of the not-for-profit sites were owned by online health information services, academic centers, or consumer activist or support groups. Although nearly three-quarters of all the sites targeted health care consumers, the majority of for-profit sites (96, or 88 percent) targeted health care consumers or both consumers and clinicians; not-for-profit sites targeted consumers to a lesser degree and clinicians and both groups to a greater degree.

The for-profit and not-for-profit sites differed in the quality of information they provided. For-profit sites provided a smaller amount of accurate information: 67 sites (61 percent) mentioned no accepted treatment options for depression, and only 30 (28 percent) mentioned all three (medications, psychotherapy, and any type of professional help). Many sites implicitly recommended treatments that have not been reliably established as efficacious, including self-medication with herbal remedies, high-dose vitamin supplements, self-help books and cassettes, homeopathy, aromatherapy, and massage.

Not-for-profit sites fared better on our measure of treatment recommendations. Although 17 sites (25 percent) failed to mention any of the treatment options, 45 (65 percent) mentioned all three. Only a minority of these sites suggested nonapproved treatments for depression—and all of these sites were consumer pages that described personal experiences with St. John's wort.

For-profit and not-for-profit sites also had different scores for providing DSM-IV criteria. Sixty-seven of the for-profit sites (61 percent) mentioned none of the diagnostic symptoms or criteria for depression; only 17 (16 percent) mentioned 11 to 14. Of the not-for-profit sites, 28 (41 percent) listed 11 to 14 criteria, and 23 (33 percent) failed to mention any.

Discussion and conclusions

Reliable information about depression and its treatment was low across the spectrum of Web sites, and it was substantially lower on for-profit sites. A significant number of sites failed to mention any of the cardinal features of depression or even the most basic options for treatment—consultation with a professional, medications, and psychotherapy.

Of particular concern were recommendations by a number of for-profit sites that depression should be treated on a "self-help" basis that would involve purchasing herbal remedies, "super-pills," books, audiocassettes, scented candles and soaps, or other unproven and unsubstantiated treatments. Several of the herbal remedies contained extracts from a variety of biologically active plants—for example, kava, hypericum, and nutmeg—that could be expected to interact with other pharmaceuticals and have unpredictable effects.

With the exception of one document, which contained information on homeopathic medicine, none of the sites explicitly discouraged seeing a psychiatrist for the treatment of depression. However, a subtle antiallopathic bias was apparent on a number of for-profit sites, either directly or by implication. Herbal medicines were offered as "safe, powerful, and effective." Self-help books were promoted as "natural, safe, and a way to avoid undesirable 'drugs.'" Alternative devices or treatments were billed as "safe," "powerful," and "proven."

Sites created by professional organizations or providers tended to recommend the use of one specific type of provider—for example, a psychologist, a social worker, or a psychiatrist. Of particular interest was the fact that sites created by professional psychiatric organizations and academic health care centers were poorly represented among the sites found by the search engines. The American Psychiatric Association, departments of psychiatry at universities, and state psychiatric associations were completely unrepresented among the first 20 sites generated by the search engines, despite the fact that many offer online information about depression and its treatment. The only high-quality, not-for-profit sites that we found were the National Institute of Mental Health's site and mentalhealth.com.

Clearly, high-quality and accurate medical information on the Internet is needed. We would argue that the stigma of mental illness makes this especially true, as the Internet provides a completely anonymous method of accessing information about socially "unacceptable" illnesses. The majority of Internet users rely on search engines to locate information online, and the results of this study suggest that this strategy predominantly yields advertisements for products of no proven benefit.

The lack of representation of high-quality sites in this search may stem from a number of factors. To be listed and ranked highly by a search engine, a Web site owner must pay for placement, register individually with each search engine, or use meta-tags—keywords in crucial places—so as to be noticed by search engine programs. Because many search engines rank sites according to the number of times they are visited and the number of links they have to other sites, it is conceivable that some site owners repeatedly access their own sites in order to attain a higher placement. This technique is likely employed by for-profit corporations with sufficient human and technological resources to continually "hype" their own Web sites by creating the illusion that they are popular and frequently visited.

For-profit companies, such as vitamin manufacturers and publishers, have a clear incentive to make certain that their sites are ranked highly when consumers search the Internet for terms related to their products. Physicians, on the other hand, have a duty to provide information to the public but lack the resources available to major commercial enterprises.

Placement among the first several sites listed by a search engine dramatically increases the likelihood that the site will be visited, which creates the potential for large amounts of misinformation to reach a large number of people. Behavioral health care organizations continue to spend less than .5 percent of their operating budgets on information technology (18), so their lack of online representation is not surprising.

A Web site that is not listed by the major Internet search engines exists in a virtually inaccessible vacuum of cyberspace. Physicians and health care providers who have created educational sites related to depression should examine how highly their sites are ranked by the major search engines and determine whether these sites are easily accessible to professionals and the public. By keeping their sites updated, registering the sites with major search engines, and making sure that keywords in the meta-tags used by search engines to identify content are comprehensive, behavioral health care providers can ensure that their views will not be lost on the information superhighway.

Dr. Lissman is a psychiatric resident and Dr. Boehnlein is professor of psychiatry at Oregon Health Sciences University in Portland, Oregon. Dr. Boehnlein is also associate director of the Veterans Administration Northwest Network at the Mental Illness Research, Education, and Clinical Center in Portland. Send correspondence to Dr. Lissman, Department of Psychiatry, Mailcode UHN 80, 3181 S.W. Sam Jackson Park Road, Portland, Oregon 97201 (e-mail, [email protected]).

Enlarge table