Second-Level Thinking

James Holzhauer's epic run on Jeopardy came to an end.

While the human drama around that is entertaining, one of the best pieces of analysis was this write up "Did James make the right Final Jeopardy bet?" on one of my favorite websites.

At one level, his explanation made perfect sense, and did play out in the way the leader (Emma) bet.

But here's the "she knows what he knows what she know's" second-ordered thought (what the author calls the "mindfuck" strategy).

But Emma could have figured out his thinking, and bet a smaller amount, ensuring she won regardless of whether she got the question right or wrong (essentially sealing her victory).

Of course, one could take this type of reasoning one or more levels higher.

Antifragile and Hormesis

So yeah, Nassim Taleb and Nate Silver are having a spat on Twitter on what probabilities mean. Unfortunately, Twitter a great medium for expressing outrage, not so much for persuasion.

Here is a more thoughtful take on Taleb's concept of Antifragility. Eric Falkenstein argues that it is an old idea repackaged in a shiny box.

While I concur with Taleb on most issues philosophically, I think the blog articulates what I have thought about most of NNT's big ideas (excellent marketing that oversell old ideas). As someone said, most of his work can be summarized as "shit happens; fuck you; because probability!"

Central Limit Theorem and Wisdom of Crowds

There is an illuminating correspondence between the central limit theorem and the notion of wisdom of crowds.

Informally, the central limit theorem says that the average of many independent trials is normally distributed around the true mean. A similar idea that animates the wisdom of crowds: consensus estimates obtained by aggregating or averaging the estimates of a number of individuals are superior.

The central limit theorem, while powerful, rests on three premises: the trials are independent, identically distributed, and have finite variance. If any of these three conditions are violated, then all guarantees are void.

In the context of the wisdom of crowds, the three premises loosely translate as follows: independent means that each trial (or individual estimate) does not know or care about other trials or estimates; identically distributed implies they are looking at the same underlying problem; and finite variance implies estimates are not so stupidly wild, that a single bad trial or estimate has an outsized effect on the average.

Joel Greenblatt presents a fascinating experiment in which he brought a jar of jelly beans into a class, and asked students to guess the number of beans, and to write it down on a piece of paper. The average of the student estimates was surprisingly close to the true number of the jelly beans in the jar.

But before he disclosed this fact to the class, he repeated the experiment. This time, instead of independently writing down their estimates, students were instructed to announce their guesses to the class. The estimates were no longer independent, and the consensus was way off.

The Other Side of Behavioral Bias

Jason Collins has an evolutionary take on how most behavioral biases "make sense" from an evolutionary standpoint.

Here is a writeup, and an (accompanying) presentation.

His blog which dwells on related issues is super-interesting.

I first heard him on the Rationally Speaking podcast.

Simplex and George Dantzig

We talked about the Simplex algorithm in class last week. It was invented by George Dantzig as a tool to solve large linear programming problems.

The story of how Dantzig, as a graduate student, mistook two unsolved problems in statistical theory for homework assignments is truly inspiring. It sounds like something you would see in a movie (like Goodwill Hunting), except it is true (Snopes).

Don't Judge a Book by its Cover

Situations or observations often have an "obvious" first-order explanation. These explanations are attractive and complete.

Sometimes, however, a deeper and far more interesting second order effect lurks under the surface.

Consider this military example of survivorship bias:

During World War II, the statistician Abraham Wald took survivorship bias into his calculations when considering how to minimize bomber losses to enemy fire. Researchers from the Center for Naval Analyses had conducted a study of the damage done to aircraft that had returned from missions, and had recommended that armor be added to the areas that showed the most damage. Wald noted that the study only considered the aircraft that had survived their missions—the bombers that had been shot down were not present for the damage assessment. The holes in the returning aircraft, then, represented areas where a bomber could take damage and still return home safely. Wald proposed that the Navy instead reinforce the areas where the returning aircraft were unscathed, since those were the areas that, if hit, would cause the plane to be lost. His work is considered seminal in the then-fledgling discipline of operational research.

Or perhaps this example of the law of large (or small) numbers from Statistics Done Wrong, that I brought up previously on the blog. The mean is not a reliable metric, when the variance is large.

Suppose you’re in charge of public school reform. As part of your research into the best teaching methods, you look at the effect of school size on standardized test scores. Do smaller schools perform better than larger schools? Should you try to build many small schools or a few large schools?

To answer this question, you compile a list of the highest-performing schools you have. The average school has about 1,000 students, but the top-scoring five or ten schools are almost all smaller than that. It seems that small schools do the best, perhaps because of their personal atmosphere where teachers can get to know students and help them individually.

Then you take a look at the worst-performing schools, expecting them to be large urban schools with thousands of students and overworked teachers. Surprise! They’re all small schools too. 

Smaller schools have more widely varying average test scores, entirely because they have fewer students. With fewer students, there are fewer data points to establish the “true” performance of the teachers, and so the average scores vary widely. As schools get larger, test scores vary less, and in fact increase on average.

Or this example from Kahneman on whether praise or criticism improves outcomes, when an Israeli air force instructor claimed:

“On many occasions I have praised flight cadets for clean execution of some aerobatic maneuver, and in general when they try it again, they do worse. On the other hand, I have often screamed at cadets for bad execution, and in general they do better the next time. So please don’t tell us that reinforcement works and punishment does not, because the opposite is the case.”

The underlying second-order principle that was operative was regression to the mean. Praise or criticism had nothing to do with the observations, and were merely nuisance variables.

Or perhaps the grisly observation that sales of ice-cream and the number of homicides in cities are strongly correlated. Here, the first order explanation might be to dismiss the correlation as spurious.

However, a more careful look might point out to an important hidden variable, warm weather, which helps us come up with a causal explanation. Warm weather makes people buy more ice-cream. Warm weather also brings people outdoors, which increases the odds of murders.

Multitasking Doesn't Work

Yesterday, I saw a YouTube video in which we are asked to complete two tasks in serial, and in parallel (multitasking). While I am not sure if the test is representative of multitasking in everyday life, it is obvious even from this simple exercise that multitasking is counterproductive.

Switching costs decrease efficiency, quality of experience, and accuracy, while raising stress levels. Multitasking on people degrades relationships.

[...] evidence suggests that the human "executive control" processes have two distinct, complementary stages. They call one stage "goal shifting" ("I want to do this now instead of that") and the other stage "rule activation" ("I'm turning off the rules for that and turning on the rules for this"). Both of these stages help people to, without awareness, switch between tasks.  

Although switch costs may be relatively small, sometimes just a few tenths of a second per switch, they can add up to large amounts when people switch repeatedly back and forth between tasks. Thus, multitasking may seem efficient on the surface but may actually take more time in the end and involve more error. Meyer has said that even brief mental blocks created by shifting between tasks can cost as much as 40 percent of someone's productive time.

It causes collateral damage beyond that inflicted on the multitasker. Maria Konnikova writes in the New Yorker,

When Strayer and his colleagues observed fifty-six thousand drivers approaching an intersection, they found that those on their cell phones were more than twice as likely to fail to heed the stop signs. In 2010, the National Safety Council estimated that twenty-eight per cent of all deaths and accidents on highways were the result of drivers on their phones.

The vast majority (~98%) of us cannot multitask well, and shouldn't delude ourselves.

I like the quote at the opening of Christine Rosen's essay,

In one of the many letters he wrote to his son in the 1740s, Lord Chesterfield offered the following advice: “There is time enough for everything in the course of the day, if you do but one thing at once, but there is not time enough in the year, if you will do two things at a time.” To Chesterfield, singular focus was not merely a practical way to structure one’s time; it was a mark of intelligence. “This steady and undissipated attention to one object, is a sure mark of a superior genius; as hurry, bustle, and agitation, are the never-failing symptoms of a weak and frivolous mind.”

Taubes, Sugar, and Fat

Last week, I listened to Shane Parrish's interview with Gary Taubes on the Knowledge Project podcast. Taubes provides an informative historical perspective on some aspects of research in nutrition science.

His view is not charitable. Perhaps, deservedly so.

I have to confess that I have't read the book "The Case Against Sugar", but I have followed Taubes' arguments for quite a while. His thesis, essentially the same as his previous two books, is that we ditch a "low-fat high-carb" diet, for a "low-carb high-fat (and protein)" diet.

The points he make are provocative, and interesting.

That said, I wished Shane would have challenged Taubes more, and held him accountable.

This counter-point by Stephan Guyenet points to numerous reasonable flaws with Taubes' thesis. It is worth reading in its entirety, if only for the balance it provides.

A couple of other rebuttals are available here and here.

More is Different

Last week, I read a nearly 50 year old essay by P. W. Anderson (h/t fermatslibrary) entitled "More is Different" (pdf). It is a fascinating opinion piece.

• "Quantitative differences become qualitative ones" - Marx
• Psychology is not applied biology, nor is biology applied chemistry.

This other essay on the "arrogance of physicists" speaks to a similar point:

But training and experience in physics gives you a very powerful toolbox of techniques, intuitions and approaches to solving problems that molds your outlook and attitude toward the rest of the world. Other fields of science or engineering are limited in their scope. Mathematics is powerful and immense in logical scope, but in the end it is all tautology, as I tease my mathematician friends, with no implied or even desired connection to the real world. Physics is the application of mathematics to reality and the 20th century proved its remarkable effectiveness in understanding that world, from the behavior of the tiniest particles to the limits of the entire cosmos. Chemistry generally confines itself to the world of atoms and molecules, biology to life, wonderful in itself, but confined so far as we know to just this planet. The social sciences limit themselves still further, mainly to the behavior of us human beings - certainly a complex and highly interesting subject, but difficult to generalize from. Engineering also has a powerful collection of intuitions and formulas to apply to the real world, but those tend to be more specific individual rules, rather than the general and universal laws that physicists have found. 

Computer scientists and their practical real-world programming cousins are perhaps closest to physicists in justified confidence in the generality of their toolbox. Everything real can be viewed as computational, and there are some very general rules about information and logic that seep into the intuition of any good programmer. As physics is the application of mathematics to the real world of physical things, so programming is the application of mathematics to the world of information about things, and sometimes those two worlds even seem to be merging.

Gauss and Ceres

Car Friedrich Gauss was an intellectual colossus, whose work informed or revolutionized broad and seemingly unrelated swathes of science and math. In computational science, his name is attached to numerous methods for solving equations, integrating functions, and describing probabilities.

Interestingly, perhaps two of his most enduring contributions - Gaussian elimination to solve systems of linear equations, and normal or Gaussian distribution are linked through the fascinating story of how Gauss determined the orbit of Ceres (great read!).

While there is plenty of geometry involved, this example illustrates how multiple observations of the asteroid by astronomers, lead to an over-determined system of equations. Assuming that these measurements were tainted by normal or Gaussian error, Gauss built the resulting "normal equations" and solved for the orbit.

When Ceres was lost to the glare of the sun, he was able to use these calculations to direct astronomers to the part of the sky where they should point their telescopes.

Pascal's Wager

I enjoyed this recent conversation between Julia Galef and Amanda Askell on the nuances of Pascal's wager. According to wikipedia:

Pascal argues that a rational person should live as though God exists and seek to believe in God. If God does actually exist, such a person will have only a finite loss (some pleasures, luxury, etc.), whereas they stand to receive infinite gains (as represented by eternity in Heaven) and avoid infinite losses (eternity in Hell).

I always thought this was something of a tongue-in-cheek argument because "of course" the argument fails the smell test. However, if we take it seriously, we find that it resists simple attempts at tearing it down. This blog post ("Common objections to Pascal's wager") outlines some of the rebuttals. It makes for interesting reading.

One of the things from the podcast that stuck with me was a comment about whether belief in climate change maps neatly onto Pascal's wager. Simplistically, let C be the claim that climate change is true, and ~C be the opposite claim. Let A denote action (taken to avert C), and ~A denote inaction (business as usual).

Then, we have the following four possibilities, A|C (action given climate change), A|~C, ~A|C, and ~A|~C.

A|C = mildly painful

An analogy might be something like an appendectomy. There is a problem (inflamed appendix or climate change), and appropriate corrective action is applied (surgical removal, CO2 reduction).

A|~C = mildly painful

An analogy would be unused insurance. You buy home insurance for a year, and nothing happens. You had to fork over premiums (which is mildly painful), but you accept that as reasonable risk against catastrophe.

~A|C = catastrophe

Piggybacking on the previous analogy, here your house is in flames and you realize you skimped on fire insurance. The external "shock" is bad (climate change or house catching fire), but your "penny-wise but pound-foolish" behavior made a bad situation much much worse.

~A|~C = mildly pleasurable

An analogy (which strikes close to home) might be skipping the annual dental checkup, and finding out nothing is wrong with your teeth. As someone once remarked to me, sometimes "pleasure is simply the absence of pain."

Note that the catastrophic outcome 3 (~A|C), with its "infinities", crowds out the others.

Hence, Pascal might argue that we should believe in both, God and climate change.

Implicit Bias Test

I thoroughly enjoyed this Jesse Singal interview on Rationally Speaking on the problems with  the "implicit association test" for diagnosing implicit bias.

The following Dateline video shows how the test was sold to the public as scientifically robust.

For fun, you can take the test yourself.

For the problems with the test, check out Jesse Singal's piece from earlier this year, "Psychology’s Favorite Tool for Measuring Racism Isn’t Up to the Job". It is a thoughtful essay, that should be read in its entirety.

A pile of scholarly work, some of it published in top psychology journals and most of it ignored by the media, suggests that the IAT falls far short of the quality-control standards normally expected of psychological instruments. The IAT, this research suggests, is a noisy, unreliable measure that correlates far too weakly with any real-world outcomes to be used to predict individuals’ behavior — even the test’s creators have now admitted as such. The history of the test suggests it was released to the public and excitedly publicized long before it had been fully validated in the rigorous, careful way normally demanded by the field of psychology.

Singal is careful to point out that just because IAT is flawed it doesn't imply that implicit bias doesn't exist. I liked an analogy he used in the podcast. If a thermometer is flawed, you can't use it to determine if a person has a fever. The person may or may not have a fever, but the thermometer should probably be tossed away. 

Statistics and Gelman

Russ Roberts had a fantastic conversation with Andrew Gelman on a recent podcast. It covered a lot of issues and examples, some of which were familiar.

A particularly salient metaphor "the Garden of Forking Paths" crystallized (for me) some unintentional p-hacking by people with integrity.

In this garden of forking paths, whatever route you take seems predetermined, but that’s because the choices are done implicitly. The researchers are not trying multiple tests to see which has the best p-value; rather, they are using their scientific common sense to formulate their hypotheses in reasonable way, given the data they have. The mistake is in thinking that, if the particular path that was chosen yields statistical significance, that this is strong evidence in favor of the hypothesis.

This is why replication studies in which "researcher degrees of freedom" are taken away have more reliable scientific content. Unfortunately, they are unglamorous. Often, in the minds of the general population, they do not replace the flawed original study.

Gelman discusses numerous such examples on his blog. These include studies on "priming" and "power poses" that have failed to replicate. Sure there is the element of schadenfreude, but what I find far more interesting is the response of scientists who championed a theory react to new disconfirming data. For instance, Daniel Kanheman recently admitted that he misjudged the strength of the scientific evidence on priming, and urged readers to disregard one of the chapters devoted to it in his best-seller "Thinking Fast and Slow". Similarly, one of the coauthors of the original power poses work, Dana Carney, had the courage to publicly change her mind.

That is what good scientists do. They update their priors, when new data instructs them to do so.

This brings me to another health and nutrition story doing rounds on the internet. It suggests a 180-degree turn on how to deal with rising incidence of peanut allergies. Instead of keeping infants away from nuts, it urges parents to incorporate them into early, and often. I haven't looked at the original study carefully, but my instincts on retractions and reversals of consensus tells me to take the findings seriously.

Not so Golden?

We discussed Golden section search method for optimizing functions in 1D last week. Naturally, we had to talk about the golden ratio (GR) and its appearance in the cultural zeitgeist.

However, there are many misconceptions/misunderstandings about the golden ratio, as researched in this eminently readable 1992 article by George Markowsky (pdf). For example:

• neither the great pyramid of Cheops, nor the Parthenon, were designed to conform to the GR
• da Vinci did not use the GR in his paintings
• the golden rectangle is not obviously the most aesthetically pleasing rectangle
• connection with the dimensions of the human body are exaggerated,
• etc.

Keith Devlin explores this misconception in this video:

The Need for Narratives

Neal Koblitz writes in the Chronicle:

The common element in all of this is knowing how to tell a story. Contrary to popular misconceptions about science and technology, a good piece of technical work is not a disembodied sequence of formulas and calculations, but rather is part of a narrative that has a long plot line and a large cast of characters. [...] Story-telling is a fundamental part of being human, from the time we are little children.

I couldn't agree more. The ability to weave a compelling story through a presentation or journal article makes a truly memorable one stand out from the run-of-the-mill kind.

You should check out the rest of the opinion for why STEM majors need grounding in the humanities.

On Existence and Uniqueness

In engineering, it isn't uncommon to approach problems with an implicit assumption that they can be solved. In fact, we take a lot of pride in this problem-solving mindset. We ask "how", not "if"?

In a non-traditional department like mine, I have colleagues and collaborators from applied math. Their rigor and discipline prods them to approach new problems differently. 

Instead of asking "how can I start solving this?", the first question they ask is usually, "does a solution exist?"

If there is no solution, there is no point in looking for one. You can't find a black cat in a dark room, if it isn't there. 

If there is a solution, the next question to ask is: "is there one unique solution?", or are there many, perhaps, even infinite possible answers.

If there is a unique solution, any path that takes us to Rome will do. In practice, there is a preference for a path that might get us there fastest. We can begin thinking about an optimal algorithm.

If there are many possible solutions, and we seek only one, perhaps we can add additional constraints on the problem to discard most of the candidates. We can try to seek a solution that is optimal in some way.

There might be lessons from this mindset that are applicable to life in general.

When faced with a new problem, we might want to triage it into one of the three buckets.

Does this problem have a solution? If there is no solution, or the solution is completely out of one's control, then there is no point in being miserable about it.

As John Tukey once remarked, "Most problems are complex; they have a real part and an imaginary part." It is best to see isolate the real part, and see if it exists.

If there is a unique solution, then one should spend time finding the best method to solve the problem.

If, like the majority of life problems (who should I marry? what career should I pick?), there are multiple solutions, then one has to spend time formulating the best constraints or optimal criteria - before looking for a method of solution.

Extraneous Roots

Mr. Honner takes issue with the grading rubric for a math problem:

Mistakes, even if they are made by the examiners, are useful learning grounds. Here are a couple more problems that he unearthed.

James Tanton also addresses some of these issues (issues with domain, extraneous solutions etc.) in a readable essay (pdf).

Counting by Fingers

Interesting post on allowing kids to use fingers to count. I never quite understood the opposition in the first place.

Hidden near the end is an interesting technique for multiplying by 6, 7, 8, 9, or 10.

To multiply 8 by 7, say:

Algorithm:

1. Touch the "8" and "7" fingers
2. The number of fingers including the touching fingers (5, here) forms the tens place (5 * 10 = 50)
3. Multiply the number of fingers remaining on either hand (3 * 2 = 6)
4. Sum step 2 and 3 (50 + 6 = 56).

Can you figure out why this technique works?

Fama and Thaler on Market Efficiency

This is a fantastic moderated conversation between Eugene Fama and Richard Thaler on the question of "Are Markets Efficient?"

While it is fashionable to bash the efficient market hypothesis (EMH) these days, the wonderful discussion highlights many of the nuances.

Fama posits that the EMH is a useful model, even if it is not perfectly true all the time. Pointing out occasional anomalies doesn't invalidate the model. Furthermore, one has to be careful about hindsight bias (bubbles for example) before rejecting the EMH.

It should be understood that the EMH is not a deterministic model in the same sense as physical laws or models (example: Newton's laws of motion). Instead, it bears resemblance to probabilistic or statistical models (example: weather models).

A single anomaly can completely reject a deterministic model.

If a model says "A implies B", and you find a counter-example, where "A does not imply B", then you have to reject or amend the model "A implies B".

A real example might be the belief that heavy objects fall faster than lighter objects (in the absence of air resistance). A single example (or thought experiment) is enough to destroy the model.

On the other hand, anomalies don't necessarily eliminate probabilistic models.

Consider a model that says "A often implies B", such as "cigarette smoking often implies lung cancer". You find someone who smoked a pack everyday and lived to 90. That example is treated as an anomaly, or "the exception that proves the rule".

EMH, perhaps, belongs to the second group.

If you think like a Bayesian, your belief in the model should decrease as the evidence against the model begins piling up.

The Amazing Tadashi Tokieda

I enjoyed this experiment and explanation by Tadashi Tokieda (YouTube link if the video below doesn't play)

If you have some time to spend, you cannot go wrong with this playlist!