Statistics and Wrongness

1. Statistics Done Wrong is a very readable excursion through common statistical mistakes, with an intuition  and reasoning substituting for equations, whenever possible. It is now also available in a book format.

Statistics Done Wrong is a guide to the most popular statistical errors and slip-ups committed by scientists every day, in the lab and in peer-reviewed journals. Many of the errors are prevalent in vast swaths of the published literature, casting doubt on the findings of thousands of papers. Statistics Done Wrong assumes no prior knowledge of statistics, so you can read it before your first statistics course or after thirty years of scientific practice.

As an example of the tone of the book consider the exposition on "little extremes":

Suppose you’re in charge of public school reform. As part of your research into the best teaching methods, you look at the effect of school size on standardized test scores. Do smaller schools perform better than larger schools? Should you try to build many small schools or a few large schools? 

To answer this question, you compile a list of the highest-performing schools you have. The average school has about 1,000 students, but the top-scoring five or ten schools are almost all smaller than that. It seems that small schools do the best, perhaps because of their personal atmosphere where teachers can get to know students and help them individually. 

Then you take a look at the worst-performing schools, expecting them to be large urban schools with thousands of students and overworked teachers. Surprise! They’re all small schools too.

Smaller schools have more widely varying average test scores, entirely because they have fewer students. With fewer students, there are fewer data points to establish the “true” performance of the teachers, and so the average scores vary widely. As schools get larger, test scores vary less, and in fact increase on average.

2. This second article I find fascinating, because it debunks a persuasive claim from Kahneman's "Thinking Fast, Thinking Slow" that I held to be true.

The original idea was that performance on a cognitively demanding task improved if the task was presented in a poor font, because it mobilized System 2.

Turns out that the original dataset with N=40 was an outlier.