Linsanity of Complexity
I would not give a fig for the simplicity this side of complexity, but I would give my life for the simplicity the other side of complexity - Supreme Court Justice and Pragmatist Oliver Wendell Holmes
The majority of the problems we try to solve are complex. A simple solution to a complex problem will not work. The phrase "correlation is not causation" is a fundamental principle in statistics. Statistics students are repeatedly taught this. Simply because two things correlate does not mean they cause each other. In 1897, Karl Pearson introduced the idea of spurious correlation. In this case, two or more events are associated but not causally connected. It is either due to coincidences or to the presence of a hidden factor. Among their examples of "spurious correlations," Pearson and his assistant cited a relationship between chocolate consumption per capita and the number of Nobel Prize winners in a country. The internet is full of hilarious correlation causations stories. My personal favorite is an analysis of data by a Harvard criminology student showing a clear correlation between Nicolas Cage's movies and the number of people who drown in their swimming pools every year. Complex systems typically have hidden factors behind their cause.
Adam Smit on complexity in Part Six of his "Theory of Moral Sentiments,” regarding what he calls the "Man of System."
He seems to imagine that he can arrange the different members of a great society with as much ease as the hand arranges the different pieces upon a chess‐board. He does not consider that the pieces upon the chess‐board have no other principle of motion besides that which the hand impresses upon them; but that, in the great chess‐board of human society, every single piece has a principle of motion of its own, altogether different from that which the legislature might chuse to impress upon it.
Daniel Kahneman, professor of psychology, winner of the Nobel prize for economics, and author of Thinking, Fast and Slow, tells a story of Israeli flight instructors and the psychology of effective training. Kahneman suggested that praising the students rather than criticizing might improve their performance. However, the flight instructors said they had tried both. When they commended a student, they wound up doing worse, and when he criticized them, they did better. Kahneman heard the same from other instructors. After looking at the data statistically, it indeed showed that after a compliment, the student performed worse, and after criticism, the student did better. This correlation could not be the causation in this case. The flight instructors tried to make sense of this simple correlation data and were not looking for other hidden factors. The praise and criticism data going up and down randomly is what Kahneman calls regression to the mean. According to the regression to the mean concept, if a variable is extreme, subsequent samples of the same variable will be closer to its mean (average).
Out of high school, Jeremy Lin didn't receive any athletic scholarships for basketball and wound up playing basketball in the IVY league for Harvard. After college, again was not drafted by any professional NBA teams. In 2010, Lin signed with his hometown Golden State Warriors. He barely played and bounced between the NBA D-League (minors) and the professional team in his rookie year. Eventually, the Warriors let him go. The NY Knicks picked him up in 2011. On Feb 4th, with two Knicks starters, Carmelo Anthony and Amare Stoudemire, being injured, Lin was bought in the game. Lin, to that point, had only 55 minutes of playing time all season. He caught fire and led the Knicks to a seven-game winning streak. For the next 26 games, he was catapulted to international fame in what was called "Linsanity." Being a Knicks fan from an early age remembering the world champions in 1969, I can tell you we haven't had a lot to cheer about over the years. After coming off the bench, Lin scored 25 points as a starter. In the next game, he scored 28 points. At the height of "Linsanity," he scored 38 points off the hall of fame Kobe Bryant of the Lakers. After the 26 games, he had surgery and missed the playoffs. In the off-season, Lin was let go by the Knicks. Lin bounced around over the next ten years between seven different teams. He signed with the Chinese Basketball Association (CBA) team Beijing Ducks. Jeremy Lin was a good basketball player; however, he was not the great one who had played with the Knicks during his "Linsanity" period. Lin's story is a perfect example of regression to the mean.
According to Deming's System of Profound Knowledge, one way to discover unseen factors in complex systems like the Israeli flight instruction program would be to try and understand the Theory of Variation. With a control chart, the data showing common cause variation of the student's performance will show that praise and criticism data as random, meaning it is normal and not causal. It goes up, then comes down. It comes down and then comes back up. It is considered a stable process when this randomness occurs within the control limits. Upon understanding the common cause variation stable process, they could then focus on possible causal variables such as the weather, communication, instrumentation, etc. Statistical Process Control allows them to analyze these non-random variations as possible reasons for performance trends.
