I have recently been criticized for picking Wikipedia as one of “my five favorite brands” in Facebook. A friend pointed fingers to me as the gull who swallows everything he finds on the Web.
That’s funny, as I have been one of the first to criticize the fervor with which the whole world, starting in 2006, embraced the notion of Wikipedia as “almost as accurate as Britannica”.
That conclusion, as it easily turns out if you scratch out the media hype, is plain wrong.
The qui pro quo originated in late 2005, when nature.com published a “study” comparing a sample of equal entries in Wikipedia and Encyclopaedia Britannica, and determined that en.wikipedia.org was only 32 percent less accurate than the blazoned competitor.
(Incidentally, it occurs to me that the French Encyclopédie Universalis is actually superior to Britannica, although perhaps this is not something that you want to state in English).
It may be useful to clarify that journalists, not scientists, were the ones who set the study’s methodology and picked the domain experts to whom the single entries were sent for analysis and comparison.
Nature Publishing Group is the company that publishes, among other things, the scientific journal Nature. However, its web site is not managed or written by scientists, scholars and recognized experts of quantitative methods: nature.com’s content is the work of (excellent) journalists specialized in writing about science.
So, when the media popularized the outcome of “a study carried out by Nature”, many readers took it for a scientific research published in Nature the journal, while in actuality it was an article on http://www.nature.com. Some journalists had selected peer entries from Wikipedia and Britannica and sent them for analysis to a panel of experts who had been selected by them and were asked to reply to questions raised by the same journalists.
Does that disqualify in any way the study? No. It does, however, place things in their true context and it helps understand why the study, as I will show in a moment, was substantially flawed. Had it been a scientific paper submitted to the journal Nature, anonymous referees would have intercepted the mistakes well in advance of publication.
So, what flaws, Magrassi?
Well, plenty. In addition to those brought forward by the publisher of Encyclopaedia Britannica in 2006 (see here), consider the following:
The Nature team did not pick entries at random. They only picked entries from the scientific sections of the two encyclopaediae.
Now, extremely specialist entries, such as the scientific ones, are obviously written by domain experts and academics in both Britannica and Wikipedia. Just check it out yourselves. As wise as crowds may be, who the hell do you think could have written the entries “Homo heidelbergensis”, “Epitaxy” or “Kinetic Isotope Effect” in Wikipedia?
The same type of people write scientific entries in both encyclopaediae, so the sample selected by the Nature team is fatally flawed: it leaves out the majority of entries in Wikipedia (those that are not scientific), and the entries left out are indeed those that we would be most interested in knowing whether they are reliable or not (the entries written by the people at large).
Had the sample been reflective of the content of an encyclopaedia (who doesn’t just talk about “hard science” concepts), it is plausible that Wikipedia would have come out with more mistakes, because common people make more mistakes, when they write, than scientists reporting about the “Kinetic Isotope Effect”.
For example, a statistically sound sample would contain the many mistakes that originate from primary research that –however discouraged by the policy– makes it into Wikipedia, with no reference whatsoever to other primary, reliable sources.
The Nature team picked entries that were similar in length, perhaps in order to simplify the judgment of the experts panel.
It has, therefore, left out the less “encyclopaedic” entries of Wikipedia, i.e. those that are prolix and disproportionately long and that are, arguably, those containing more mistakes.
The Nature team did not consider as errors sentences that were hardly comprehensible because they were written in poor English.
The experts in the panel had ranked those entries as wrong, but the Nature team removed them from the test ex-post.
Now, Britannica does not contain poor English, because there are editors who run through the text before publication, looking not for content but for meaning and clarity. It certainly does not contain a lot of incomprehensible language.
Although language editing is, as a paid professional activity, declining in many cultures and companies, I strongly suspect that most of the odd sentences in the Nature test were coming from Wikipedia.
By removing them from the test, the methodology favored Wikipedia further.
BOTTOM LINE: To say that Wikipedia is only slightly less accurate than a professional encyclopaedia is wrong. First, such belief is based on a single test, which concluded that Wikipedia contains 32 percent more mistakes than Britannica. Second, that test was severely flawed and biased towards Wikipedia. There may be as many as 50 or 200 percent more errors in Wikipedia than in a professional encyclopaedia. We do not know. More serious studies will have to tell.
NB: Despite all this, Wikipedia is a wonderful project that is going beyond my own expectations.