Communications of the ACM

Communications of the ACM

"of Course It's True; I Saw It on the Internet!": critical Thinking in the Internet Era

By Leah Graham, Panagiotis Takis Metaxas
Communications of the ACM, May 2003, Vol. 46 No. 5, Pages 70-75
10.1145/769800.769804
Comments

The Internet is revolutionizing research methods at colleges and universities around the world. Though it can be extremely useful to researchers, the Net presents a significant challenge in that it is quite different from traditional sources. The lack of uniform standards and the ease of access have made the Internet a powerful but uncertain medium. Substantial effort is required to adequately evaluate its information, and this may not always be apparent to users [5]. This is particularly challenging for students, as many have come to rely on the Net as a primary source of information without formal instruction about the difficulties involved. The Internet has gained a primary place in research methods, and it is vital that students become able to critically evaluate the information it provides.

Several solutions have been suggested to determine accuracy in Internet research. In [1], Jerry Campbell supports the Association of Research Libraries' plan to develop an Internet portal to "trustworthy" information. This portal would "promote the development of and provide access to the highest quality content on the Web." Many colleges have also adopted this approach by providing lists of approved online sources to students. While it appears to provide a practical alternative to information.coms that focus more on advertising than accuracy, this approach suffers from several drawbacks. First, it is impossible to continually monitor all the content found using these portals. Web sites change overnight and expand at exponential rates, and attempting to continuously verify every page of each linked site would be an incredibly time-consuming task. Clearly, this is not feasible, but it would be necessary to ensure the accuracy and timeliness expected of information found using a "scholar's portal." Additionally, this approach places the responsibility of evaluation on the Web masters of these portals. A more interactive approach that encourages users to develop critical-thinking skills would provide lasting value, while preventing them from becoming dependent on these portals for the correct answers.

Developing other approaches requires a firm understanding of how students currently use the Internet for research. Consider the results of an informal questionnaire distributed by Angela Weiler in 1999 at SUNY College of Agriculture and Technology, Morrisville, NY. In response to a question asking how students would ascertain if online sources were accurate enough to be considered "a good source of information," 29% said they accepted Internet information regardless, with only 34% considering additional verification important [5]. These startling results confirm the importance of further study to provide specific information about students' online research practices. To address this, we developed a six-question survey administered to 180 Wellesley College students during the 2000–2001 academic year. Students' responses to this survey helped explain how college students, from different backgrounds, class years, and majors, react to information on the Internet.

Back to Top

Research Methods

Participants in this study were students from the "Computers and the Internet" class; it was, in fact, their first assignment. Students were told the purpose of the survey was to understand how students conduct searches. The survey was divided into seven email messages. The first explained the process of responding to the survey and included a personal information questionnaire. The following six email messages each contained one question and asked students to report their answer and search strategies.

The survey was designed to answer three research questions:

• How strongly do students rely on the Internet for information?
• What claims are students more likely to believe?
• Who is most susceptible to misleading claims?

To identify students' reliance on the Internet, they were told to answer the questions in whatever way they wished. They were free to use any resource available, including visiting the library, and they were asked to report which search methods were used for each question.

The six survey questions were used to determine students' ability to evaluate information, as well as their inclination to verify their responses. Four questions tested particular areas of misinformation: advertising claims, government misinformation, lobby group propaganda, and scams. Preliminary research indicated these areas could present a significant challenge to students. Two additional questions were used to determine if students were more diligent about accuracy and verification when the information was easy to find.

Each response was given a score from 0–3, with 3 being the highest score. The scoring system placed equal weight on accuracy and the students' efforts to double-check responses (see Table 1). An optimal answer was therefore defined as a correct response confirmed in at least two sources. Other scores were categorized as follows:

A 0 indicates no response, a 1 an incorrect response that was not double-checked, and a 2 either a correct answer that was not double-checked or an incorrect response that was double-checked. The 2 category contains both types of responses, as dividing the category would require placing more importance on accuracy or verification. Neither of these attributes, when considered individually, wholly constitutes adequate research practices. As such, the 2 category remains the middle category for responses not entirely acceptable due to a lack of accuracy or verification.

Finally, to evaluate which groups of students are in greater need of assistance, students were asked to fill out a questionnaire asking for age, class year, and other factors. This data was matched with their responses to the survey questions.

Back to Top

Results

The findings were remarkable. Regarding students' reliance on the Internet, it became apparent that students are very eager to use the Internet—and only the Internet—in conducting research. Though the survey was not in any way limited to Internet resources, less than 2% of students' responses to all questions included non-Internet sources. Many of these responses also quoted online sources at some point. This finding emphasizes the importance of teaching good Internet research skills, as students rely so heavily on the Internet.

The survey also revealed the extraordinary confidence students have in search engines. If the survey question did not mention a particular Web site, almost all students immediately turned to a search engine. Many remained faithful to one search engine throughout the survey, even if it did not immediately provide the answer sought. This is particularly interesting since experts believe no single search engine captures more than 16% of the entire Internet. With all search engines combined, this only increases to 42% [2]. Additionally, students were asked a question to determine the extent of their understanding of search engines. Few students responded with any degree of awareness of the process by which search engines post results. This is distressing, as the reliability of search engines to faithfully and selflessly guide users to appropriate materials has often been questioned.¹

The second research question about the types of information most problematic to students yielded disheartening results. Students were overwhelmingly susceptible to three types of misinformation—advertising claims, government misinformation, and propaganda—and somewhat susceptible to scam sites.

The two most successful misleading claims were advertising and government misinformation. To study the impact of advertising claims, students were asked: "List three major innovations developed by Microsoft over the past 10 years." The term "major innovation" was left vague, as Microsoft's innovative history is a widely debated issue. There are many opinions on the topic, and we expected students overall to discuss at least several.

However, 63% of students responded that Microsoft was responsible for many major innovations based on information from only one source. Almost all of these students immediately went to the Microsoft Web site and used the Microsoft Museum Timeline that details Microsoft's achievements—or at least, what Microsoft claims to be its achievements. Only 12% checked several sources and made more complete argument. Some 22% fell in between these two groups, receiving a score of 2. These results are intriguing in view of recent litigation against Microsoft that drew worldwide attention to its business practices and innovation efforts. Yet almost two-thirds of students responded without a shadow of a doubt that Microsoft was completely honest about its claims.

Government misinformation followed closely behind advertising claims. Students were asked: "Did the 1999 Rambouillet Accords allow NATO to operate in all of Yugoslavia or only in Kosovo?" The correct answer—all of Yugoslavia—can be found in the actual document, though it is difficult to wade through its 82 pages. The complete text can be found online, but summaries and reviews are much more common. A frequently found summary is the U.S. Department of State Bureau of European Affairs fact sheet released on March 1, 1999, which implies that NATO presence is limited to Kosovo.²

---

The survey revealed the extraordinary confidence students have in search engines. If the question did not mention a Web site, almost all students immediately turned to a search engine. Many remained faithful to one search engine throughout the survey, even if it did not immediately provide the answer sought.

---

A total of 62% of students said that NATO is limited to acting within Kosovo based on one source, and many listed the State Department memo as their only source. And 26% said the same thing but made some effort to double-check the information or happened to find the correct answer on the first attempt. Many students in this category stumbled on anti-NATO Web sites and reported that information without checking another less-biased source. Only 10% found the correct answer and verified it in two places.

Political lobbying groups are another common source of misinformation or half-truths. Students were asked to evaluate a claim made by getoutraged.com. This Web site is the work of an anti-smoking lobby, though it is officially copyrighted by the Massachusetts Department of Public Health. Students were asked: "Getoutraged.com says that tobacco is responsible for 30% of all deaths in the 35–69 age group. Would you cite this information in a research paper?" This statistic, taken from a pamphlet entitled "Growing Up Tobacco Free," is actually a projection made in 1992 on how many deaths tobacco will probably cause in the 1990s, but getoutraged.com lists this as if it were a proven fact [3]. The number of deaths was actually estimated to be closer to 20% by organizations such as the American Cancer Society and the U.S. Center for Disease Control and Prevention.³

Despite this, 48% of students said they not only believed the statistic from getoutraged.com, but they would confidently cite it in a research paper. They did not attempt to find a corroborating source. Only 21% expressed reluctance to use this information after checking with additional sources, with 30% falling in between. What is most disturbing is that many of the students who readily believed this statistic realized the site was probably the product of an anti-smoking lobby, but the fact it was sponsored by the Massachusetts Department of Public Health reassured them. Students seemed to believe that because a source was cited and the Massachusetts government copyrighted the Web site, the statistic would naturally be accurate.

Fortunately, the results are not entirely dim. Students were much less susceptible to the scam Web site. They were asked to evaluate vespro.com's "revolutionary" product Vespro GHS containing Human Growth Hormone (hGH), an emerging medical treatment to combat the effects of aging. According to the Web site, this product will decrease body fat, reduce wrinkles, restore lost hair, and normalize blood pressure, among a variety of other benefits—an absolute miracle drug. This Web site provides quotes from medical journals that are generally taken out of context to support its claims. For instance, there is a quote from a 1989 article in the New England Journal of Medicine that seems to support the beneficial effects of hGH, though its conclusion simply states further research is necessary [4].

Students were asked: "Would you recommend Vespro Life Science's hGH product to a friend concerned about getting older?" Only 13% of the students immediately agreed to recommend this product without consulting another source while 35% conducted further research and reported they would not recommend the product without more information. And 52% of students received a score of 2. Though these results are not overly encouraging, they demonstrate that students remain skeptical of this type of information on the Internet.

The remaining two questions were used to determine students' inclination to verify information. The first question asked students to report the creator of Linux. The answer is easily found quickly online. The second question asked students to find the land area of Lisbon, Portugal. While this sounds elementary, it can take a tremendous amount of time to locate any answer on the Internet, and even longer to find a second source. For the easy Linux question, 78% of students reported the first answer they found, without verifying it from another source. For the more difficult Lisbon question, 75% of students reported the first answer they found without double-checking. It appears that students are just as likely to avoid verifying an answer, regardless of the time or effort needed to do so.

Finally, to determine which groups of students are more susceptible to misleading claims, responses to the personal information questionnaire were matched with answers to the six survey questions. Using class year, we hoped to see if students became better Internet researchers over the course of their years at Wellesley. The results indicate there was no significant difference in performance based on class year (see Table 2).

We then looked at self-reported confidence in their Internet searching abilities to determine if students who were more "Internet-savvy" were better able to critically evaluate information on the Internet. The categories available were very confident, fairly confident, slightly confident, and not very confident. Table 3 indicates the total number of scores (0–3) given to students in each confidence level. The distribution of scores for all questions is very similar for each confidence level. Only the "not very confident" group shows notable, though not overly large, differences. This suggests the confidence a student has in his or her ability to effectively search the Internet does not significantly affect the student's performance.

Back to Top

Conclusion

Clearly, students consider the Internet a primary source of information. The results presented here suggest many students have difficulty recognizing trustworthy sources, though perhaps the underlying problem is a lack of understanding of the Internet as an unmonitored source of information. All future educational ventures must focus on teaching users the Internet is an unmonitored method of sharing information. Specifically, this instruction should equip users to use search engines effectively, and this requires an awareness of their technological and financial constraints. This is not to recommend teaching students that all search engines are devoid of useful information, but rather to promote a better understanding of the actual service provided by search engines.

Students are also not consistently able to differentiate between advertising and fact. Many responses to vespro.com mentioned that as the Web site was just trying to sell a product, its claims could not be readily believed. However, many of these same students immediately believed claims made by Microsoft on its commercial Web site. Students must understand that all information on the Internet is there for a reason, and it is vital to determine the purpose of the information when evaluating its accuracy.

The very small number of students who double-checked information is also concerning. It is commonly believed the triangle method—locating three independent sources that point to the same answer—produces the most accurate information. This approach does not differentiate a great deal between "good" and "bad" sites, but rather encourages users to double-check information regardless of the source. Students in this study seemed to have a great deal of confidence in their abilities to distinguish the good sites from the bad. Colleges themselves often encourage this attitude as they determine "good" or "trustworthy" Web sites to help students begin Internet research. While it is certainly useful to provide guidance, it is equally important to promote the development of critical thinking skills that will allow students to make use of the entire Internet, rather than a few approved sites.

Our findings also suggest that students across the board have similar difficulties in carefully evaluating information found on the Internet. Older students with stronger traditional research skills performed no better than other students, which suggests these skills are simply not sufficient when evaluating online information. In the past, the greatest problem facing researchers was finding information; now, with the advent of the Internet, the greatest problem is evaluating the vast wealth of information available. Students in this survey placed greater emphasis on the process of finding an answer than on analyzing the actual information. The difficulties students encountered suggest this practice is of little use in determining the accuracy of online information. It is therefore important to develop specific research practices for Internet searches that take the structure and purpose of the Internet into account.

---

All future educational ventures must focus on teaching users the Internet is an unmonitored method of sharing information.

---

As students continue to view the Internet as a primary source of information, without a significant shift in training methods, this problem will only worsen. It is vital that students better understand the nature of the Internet and develop an instinctive inclination for verifying all information. This will allow students to take advantage of the tremendous benefits provided online without falling prey to the pitfalls of online research.

Back to Top

Back to Top

Back to Top

Back to Top

Back to top

The Digital Library is published by the Association for Computing Machinery. Copyright © 2003 ACM, Inc.

---

No entries found