The book has three parts of uneven length. Part I is the shortest, In it, Ritchie sets out his view of what science is and how it is supposed to work. A crucial feature of the enterprise is supposed to be replication: scientists are expected to repeat both their own research and that of others, to see whether the original findings were valid, but this hardly ever happens today, partly because it seems less glamorous and exciting (and therefore less likely to attract citations in the literature), but also because most journals don't publish replications. This is particularly true if the replication fails to confirm results previously published by the journal concerned. (Ritchie gives an example of this from his own experience.)
Part II considers four main sources of errors in scientific papers and books. The most egregious is outright fraud, which is not as rare as one might think and is particularly alarming when it occurs in a medical context. What is most disturbing about two of the cases Ritchie discusses is how far some respected academic institutions (the Lancet and the Karolinkska Institute and Hospital in Sweden) resisted acknowledging that they had been duped for as long as they could, even in the face of incontrovertible evidence. As Ritchie remarks, there are undoubtedly many instances of medical fraud that so far remain undetected.
In addition to fraud, scientific papers may be affected by bias, negligence, and hype, each of which gets a chapter to itself.
Bias may appear in authors' attitudes to their subjects, but there is also publication bias, which affects the scientific literature as a whole. There is a widespread failure to publish negative (null) results. Most journals don't publish null results, which presumably is why many scientists don't even submit them. The result is a serious skewing of the scientific literature.
Negligence is something of a catch-all category. It includes numerical errors, which are surprisingly common, and also the planning and publication of under-powered studies (studies that are too small to detect small effects).
Hype, Ritchie finds, is a pervasive feature of much of scientific publishing and of how science is presented to the public. It can lead to a plethora of papers and media features about subjects that are fashionable at a given moment. The microbiome is a current example—'microbiome mania. But the subject that generates more hype than any other is nutrition.
Given the sheer volume of coverage and the number of conflicting assertions about how we should change our diets, little wonder the public are confused about what they should be eating. After years of exaggerated findings, the public now lacks confidence and is sceptical of the field's research.All these problems seem to be becoming more and more pervasive. The reasons are complex but are mostly due to the relentless pressure on scientists to publish as much as possible; this is what careers largely depend on. But not all publication is equal. Where you publish matters too; there is a journal hierarchy, with some being much more prestigious than others. At the opposite end of the scale are the fake online journals that frequently send emails soliciting contributions; these have negative prestige.
An important manifestation of the quest for publication is the phenomenon referred to as p-hacking. Practically any scientific paper you read will be full of p-values, which are supposed to be indications of a given finding's reliability—'statistical significance' (which is not the same as its actual practical importance). The lower the p-value, the better. A value below 0.05 is conventionally taken to be 'statistically significant', so there is enormous pressure on scientists to produce results that are at or below this level.
p-Hacking describes the sometimes frantic search for ways to achieve the desired level. Scientists use two main ways to 'ever so slightly nudge, or hack', their p-values below that crucial threshold'. One way is to adjust or cherry-pick the data in various ways so as to get a better result. The other is to apply a lot of statistical tests to the data in the hope that something significant till emerge. 'The scientist can then decide, often perhaps convincing even themselves, that they'd been searching for these results from the start.'
Both kinds of p-hacking are instances of the same mistake and ironically, it's precisely the one that pp-values were invented to avoid: capitalising on random chance.If nothing else works, authors sometimes resort to phrases such as 'the result is close to statistical significance' in an attempt to mitigate the failure to reach the desired level.
In Part III Ritchie makes a number of suggestions to try to improve the situation that science finds itself in. As he notes, this won't be easy.
This is about root-and-branch' (or lab-and-journal) reform of the way we do research and of scientific culture—an attempt to master the faults and biases that have crept in, largely unnoticed. The world is rightly proud of where science has brought us. To retain that trust, we owe it something far better than the product of our flawed human temperaments.Although Ritchie may be swimming against the tide, it's clear from some of the authors he quotes that he isn't alone in his assessment of science today; other scientists see the same problems. It's important to say this, to make it clear that Ritchie isn't an isolated critic. As an occasional author and reviewer of scientific papers myself, I certainly found his book stimulating and important. But it isn't only relevant to people who are professionally involved in science. As he makes clear, readers of popular books and articles on science need to be on their guard to spot the frequent hyperbole and reliance on poorly substantiated research that characterise a lot of this material.We owe it the truth.
Ritchie's style is relaxed and informal and the book is easy to read. Full references are, of course, provided, but what I found annoying is that the reference section also contains often quite interesting and important notes on the main text, but these are jumbled in with the references so that they aren't easy to pick out. It would have been better to print them as footnotes to the text.
08-08-2020