Deming Point #6 and the Flaw of Averages
Our culture encodes a strong bias either to neglect or ignore variation. We tend to focus instead on measures of central tendency, and as a result we make some terrible mistakes, often with considerable practical import. — Stephen Jay Gould
In a famous cartoon by Jeff Danziger, titled the Flaw of Averages, an incident is portrayed regarding a statistician who drowned while crossing a river that was, on average, only three feet deep. Although the average depth of the water was three feet, the man who drowned was seven feet underwater.
Assumptions based on averages are incorrect on average.
At a seminar once, Dr. Deming was heard laughing as a manager said that all their employees were above average. Dr. Deming believed that an organization's training needed to be reconstructed from the ground up based on basic knowledge of science. Leaders need to move from snap judgment cognitive thinking to more analytical thinking involving uncertainty and risk to transform an organization. Leaders need to institute new ways of thinking that involve analytical uses of statistics.
Dr. Deming focused more on Inferential Statistics (aka Analytical Statistics) than Descriptive Statistics (aka Enumerative Statistics.) Descriptive Statistics asks, "what?" Inferential Statistics asks, "why?"
A typical measurement in IT operations, DevOps, is Mean Time to Recover (MTTR), basically the average time of an IT outage. This metric has the essential characteristics of the flaw of averages. Not all outages are equal for one, and averaging all the outages over some time only tells you the what, not the why. An analytical approach might include more advanced statistical analysis to try and understand why the outage occurred or make the "out of process" patterns visible. Statistical Process Control is a tool that enables an organization to understand the nature of these patterns through the lens of variation. Lloyd Nelson summarized that the purpose of management in all its aspects, including planning, procurement, manufacturing, research, sales, personnel, accounting, and law, is to understand the meaning of variation and extract information from variation.
