You don't have to go very far scrolling the news to find a scattered piece of statistics here and there. And when you think advertisers, both online and in billboards, will spare you from facing maths and statistics, you are yet again betrayed by claims from their marketing departments. After all, we all assume that claims backed up by beautifully coloured numbers and percentages are more reliable than those that do not display any data, isn't it? And that is exactly the problem!
Stats are everywhere, hiding in plain sight at your nearest supermarket, newspapers' headlines and billboards.
Image Credit: Alexas_Fotos, shamirandres, and kalhh @ Pixabay.
Seriously! Back in 2016, when the American Press Institute (API) ran a detailed study on "What makes people trust on news" they found that 85% of Americans say accuracy is a very important reason to trust the news; and 70% consider presenting expert sources and data mattered a lot in their decision to believe on what the news are telling them.1
So, again, what is the problem? Well, you see, we are not particularly good at dealing with numbers... And no, I am definitely not talking about the fact that 4 in 10 Americans hate maths - but yes, this definitely was a reference to the widely circulated snippet from a news report with an apparently made up headline2 which reads "'We hate math', say 4 in 10 — a majority of Americans".
What I was actually hinting at were oddities about our inability to properly deal with numbers. Although part of the people actually enjoy maths - yes, if you guessed I am amongst them you are right -, humans generally tend to perceive numbers such as 2.99 as much smaller than 3.00;3 we are also used to getting a sense that numbers written like 1650 are smaller than their alternative format 1,650;4 and, on top of that, people tend to perceive values like 1954.00 as being bigger than when they are written as 1954.4 And, by the way, advertisers know all about that! Oh, did I mention that a poll from Ipsos MORI, Perils of Perception, shows we get nearly everything wrong about how proportions in the world around us really are?5
Now, are you inclined to see headlines like these under the same light?
News headlines6 from the past week containing stats. But are they trustworthy?
After all of that, you might be thinking: "Wow! This is messed up. At least we can trust on the scientists, these people must know their stats very well." I don't want to be that guy who always ruins the parties - and I am definitely not on the side of the science deniers and conspiracy theorists - but I have bad news for you...
For a start, back in 2005 professor John Ioannidis, now at the University of Stanford (USA) , demonstrated "most published research findings are false"7 - and scientists are cool with that. The paper obviously unleashed the typical sensationalist reports in the media, but by no means Ioannidis's work falsifies the scientific method.
Then, enters Dr Daniele Fanelle, who showed papers that report positive results - these are the ones in which researchers find data that supports their original hypothesis - are more common as we go from the physical and chemical sciences to the biological ones, and are even more frequent in the social and psychological sciences.8 Why is that even relevant? Because of confirmation bias, that is, the tendency humans have to trust facts and data that confirm what they think is true.9
But the cake batter began cooking when we learnt that some psychology journals were allowing scientists to be sloppy with their statistics;10 and the cherry was finally placed on top of this foul-smelling cake when 61 out of 100 psychology findings could not be replicated by other researchers.11 Well, despite the journal Nature's assurances that things were actually fine, and that for the most part the scientists who had seen their papers fail to replicate were grateful to be called out,12 media outlets received the report from the Open Science Collaboration with a large, grim smile... The hubbub even motivated Dr Derek Muller to discuss the issue in detail at his YouTube channel Veritasium13a - by the way, if you don't know Derek's work I thoroughly recommend you to check his very interesting videos.
Veritasium's video about the controversy.13b
The point I want to make is that it is only human to get things - and especially statistics - wrong sometimes. You and I are no different! Yes, I had received scientific training to know how to deal with my inherent biases, and how to seek accuracy and responsibly use statistics; but at the end of the day my brain is still as fooled by large numbers and probabilities as anybody else's is.
But there is light at the end of the tunnel - and no, it is not the train coming to get you. Mathematicians, statisticians and scientists began to tackle the issue of statistical rigour in research with all their might. This nearly-miraculous joint effort - just kidding, there is nothing unusual about scientists being collaborative - resulted in a series of changes. From the American Psychological Association adopting stricter statistical checks in their Publication Manual;14 to the Engineering and Physical Sciences Research Council (EPRSC) from the UK updating its expectations about raw data collection, retention and availability of data for future researchers.15 All in the name of greater access to science and better reproducibility.
I hope we all learnt a lesson today: that numbers don't lie... Jokes aside, we should all be more aware of our natural tendency to misjudge numbers, in special when they try to deceive us with their percent signs and their vanishing digits, forcing us to change our perceptions about them. Numbers and stats shape the way we think about the world around us, for better or worse. It is our responsibility to interpret them the best we can, educate others about the limitations of our brains, and hope for a society where people are more aware about the dangers lurking behind figures, digits and percentages.
If anything can be given to you as a consolation prize, here is a list of resources to help you learn more about statistical bias, and stats in general.
Additional Resources
A thorough guide on misleading statistics can be found in this awesome Datapine blog post.
One handy infographic from the journal Nature on what are the steps in the statistical design, and why P values are just the tip of the iceberg, is right here waiting you to check it...
Talking about the way scientists should interact with statistics, here are two great videos on the scientific method. One from SciShow - you should expect I would know about the excellent work done by the Green brothers and their team - and another one from the equally nice PBS channel It's Okay to be Smart.
While we are rambling about stats, check out the good people of Crash Course, who devoted an entire series to statistics.
EDIT: Turned out reviewing and releasing this post at nearly 2 am was not the best idea after all...
Some typos and grammatical errors have been reviewed and updated. Letters in boldface below show where corrections were implemented; or... where I had been utterly wrong.
TL;DR: Mea culpa!
"(...) numbers such as 2.99 are much smaller than 3.00;3"
"(...) - and scientists are cool with that."
"(...) sensationalist reports in the media, (...)"
"(...) scientists to be sloppy with their statistics;10 (...)"
"(...) the scientists who had seen their papers fail (...)"
I hope we all learnt a lesson today: that numbers don't lie... Jokes aside, we should all be more aware of our natural tendency to misjudge numbers, in special when they try to deceive us with their percent signs and their vanishing digits, forcing us to change our perceptions about them. Numbers and stats shape the way we think about the world around us, for better or worse. It is our responsibility to interpret them the best we can, educate others about the limitations of our brains, and hope for a society where people are more aware about the dangers lurking behind figures, digits and percentages.
If anything can be given to you as a consolation prize, here is a list of resources to help you learn more about statistical bias, and stats in general.
Additional Resources
A thorough guide on misleading statistics can be found in this awesome Datapine blog post.
One handy infographic from the journal Nature on what are the steps in the statistical design, and why P values are just the tip of the iceberg, is right here waiting you to check it...
Talking about the way scientists should interact with statistics, here are two great videos on the scientific method. One from SciShow - you should expect I would know about the excellent work done by the Green brothers and their team - and another one from the equally nice PBS channel It's Okay to be Smart.
While we are rambling about stats, check out the good people of Crash Course, who devoted an entire series to statistics.
EDIT: Turned out reviewing and releasing this post at nearly 2 am was not the best idea after all...
Some typos and grammatical errors have been reviewed and updated. Letters in boldface below show where corrections were implemented; or... where I had been utterly wrong.
TL;DR: Mea culpa!
"(...) numbers such as 2.99 are much smaller than 3.00;3"
"(...) - and scientists are cool with that."
"(...) sensationalist reports in the media, (...)"
"(...) scientists to be sloppy with their statistics;10 (...)"
"(...) the scientists who had seen their papers fail (...)"
1 a) For the original report, see: Associated Press-NORC Center for Public Affairs Research, A New Understanding: What Makes People Trust and Rely on News, Associated Press Institute, April 2016, 48p.; b) For the associated news at API, see: American Press Institute, What makes people trust and rely on news, 17/04/2016, https://www.americanpressinstitute.org/publications/reports/survey-research/trust-news/single-page/#the-meaning-of-trust-in-news, retrieved 08/04/2019.
2 a) Although a poll has actually been conducted by the Associate Press (AP) back in 2005, the title of this made up news blooper traces back to a CNN report on the the AP poll. You can check the original article at: CNN.com, The most unpopular school subject, http://web.archive.org/web/20050913070431/http://www.cnn.com/2005/EDUCATION/08/17/school.subjects.ap/index.html, retrieved 08/04/2019; b) Credit for the data digging goes to Philip Olson in his now inactive blog: Olson, P., We hate math, say 4 in 10 — a majority of Americans, http://blog.roshambo.org/306/, retrieved 08/04/2019.
3 a) For a piece of news covering the topic, see: University of Chicago Press Journals, Penny Pricing: How Consumers React to Prices Ending in 99, Science Daily, https://www.sciencedaily.com/releases/2005/06/050607030351.htm, retrieved 09/04/2019; b) Original study: Thomas, M.; Morwitz, V., J. Consum. Res. 2005, 32, 54; c) For a valuable discussion on the topic, see: Basu, K., J. Econ. Manage. Strat. 2006, 15, 125.
4 a) For the original study, see: Couter, K. S.; Choi, P.; Monroe, K. B., J. Consum. Psychol. 2012, 22, 395; b) For news covering the report, see: Howell, R. T., Commas and Cents: Why $1,999.00 is More Than $1999, Psychology Today, 18/08/2012, https://www.psychologytoday.com/intl/blog/cant-buy-happiness/201208/commas-and-cents-why-199900-is-more-1999, retrieved 08/04/2019.
5 a) For the original poll, see: Ipsos, Perceptions are not reality: what the world gets wrong, Ipsos MORI, https://www.ipsos.com/ipsos-mori/en-uk/perceptions-are-not-reality-what-world-gets-wrong, retrieved 08/04/2019; b) For charts associated with the Ipsos MORI poll, see: Ipsos, Perils of Perception 2016: A 40-Country Study, Ipsos MORI Social Research Institute, 2016, https://www.ipsos.com/sites/default/files/migrations/en-uk/files/Assets/Docs/Polls/ipsos-mori-perils-of-perception-charts-2016.pdf, retrieved 08/04/2019; c) For The Guardian report on this poll, see: Duffy, B., Why 2016 is the year the public got things wrong, The Guardian, https://www.theguardian.com/society/datablog/2016/dec/13/why-2016-is-the-year-the-public-got-things-wrong, retrieved 08/04/2019.
6 a) Rosenbaum, E., IBM artificial intelligence can predict with 95% accuracy which workers are about to quit their jobs, CNBC, https://www.cnbc.com/2019/04/03/ibm-ai-can-predict-with-95-percent-accuracy-which-employees-will-quit.html, retrieved 08/04/2019; b) Scott, E., Nearly 70 percent of Americans are okay with a gay president, poll says, The Washington Post, https://www.washingtonpost.com/politics/2019/04/04/nearly-percent-americans-are-okay-with-gay-president/?noredirect=on&utm_term=.c0782db1a0ca, retrieved 08/04/2019; c) Reuters, Dubai says foreign direct investment increased 41 percent in 2018, https://www.reuters.com/article/emirates-dubai-fdi/dubai-says-foreign-direct-investment-increased-41-percent-in-2018-idUSD5N21601G, retrieved 08/04/2019.
7 Ioannidis, J. P. A., PLoS Med. 2005, 2, 696.
8 Fanelli, D., PLoS ONE 2010, 5, e10068.
9 Wikipedia, Confirmation bias, https://en.wikipedia.org/wiki/Confirmation_bias, retrieved 08/04/2019.
10 Bakker, M.; Wicherts, J. M., Behav. Res. 2011, 43, 666.
11 Nosek, B. A., et al., Science 2015, 349, 943.
12 Baker, M., First results from psychology's largest reproducibility test, Nature, 2015, doi:10.1038/nature.2015.17433, https://www.nature.com/news/first-results-from-psychology-s-largest-reproducibility-test-1.17433, retrieved 09/04/2018.
13 a) YouTube, Veritasium: an element of truth, https://www.youtube.com/channel/UCHnyfMqiRRG1u-2MsSQLbXA, retrieved 08/04/2016; b) YouTube, Veritasium - Is Most Published Research Wrong?, published 11/08/2016, retrieved 08/04/2019.
14 American Psychological Association, Publication Manual of the American Psychological Association, 6th ed. American Psychological Association, 2009, 272p.
15 Engineering and Physical Sciences Research Council - EPRSC, EPSRC policy framework on research data, https://epsrc.ukri.org/about/standards/researchdata/, retrieved 08/04/2019.
No comments:
Post a Comment