DevOps and Operationalism
John Travolta played George Malley in the 1996 movie Phenomenon, where he was mysteriously turned into a genius. Brent Spiner (Data in Star Trek) plays Dr. Bob, who needs to evaluate George Malley. Dr. Bob asks George to respond quickly to the following questions:
Dr. Bob:
Answer as quickly as you can... how old is a person born in 1928?
George:
Man or a woman?
Why?
George:
Specifics, Bob.
Dr. Bob:
Okay, one more time. How old is a MAN born in 1928?
George:
Still alive?
Dr. Bob:
If a man is born in 1928, and he's still alive, how old is he?
Georgey:
What month?
Dr. Bob:
If a man was born October 3rd, 1928, and he's still alive, how old is he?
George:
What Time?
Dr. Bob:
10 o'clock... PM!
George:
Where?
Dr. Bob:
Anywhere!
George:
Well, let's get specific, Bob! I mean, if the guy's still alive, born in California, October 3rd, 1928, 10 PM, he's 67 years, 9 months, 22 days, 14 hours, and 12 minutes. If he was born in New York, he's 3 hours older, now isn't he?
New Measures
Nobel Prize physicist Percy Williams Bridgman was also concerned with specifics when creating synthetic diamonds using extreme pressures. His gauges kept breaking down when he worked under extreme pressure, so he had no idea what pressure levels he had reached. This work led him to describe a general philosophical doctrine called Operationalism. It is based on the idea that we can only know the meaning of something if we have a way of measuring it. In 1927 Bridgman's published, The Logic of Modern Physics. Bridgman was inspired by Albert Einstein's special theory of relativity, where measurements used in time and space were different than conventual means of measurement. As a result of Einstein's work, Bridgman examined how scientists define measurements in more detail.
Length
In "The Logic of Modern Physics," Bridgeman discusses the concept of length as it applies to measurement. Bridgemen discusses how scientists use different measurements in different domains. When measuring physical space like a house, we might use a simple example of square footage. When measuring the distance to a planet, we use light years.
"To say that a certain star is 10^5 light years distant is actually and conceptually different from saying that a certain goal post is 100 meters distant." - The Logic of Modern Physics.
Operational Definition
This work inspired Dr. Walter Shewhart and Dr. Edwards Deming's ideas around what Bridgeman coined as an Operational Definition. Deming defined an Operational Definition in his New Economics as a procedure agreed upon for translating a concept into a measurement of some kind. Deming further uses an example of asking students to count the number of animal crackers based on category (cows, horses, and pigs). One student asks how she should count the broken pieces. Should she try to put the broken pieces back together like a puzzle or just count the main body parts?
From my upcoming book - A Journey of Profound Knowledge
Deming gives us a great simple example of variability. If you ask three people to count the number of people in the room, you might get three different answers. The answers depend on each counter's definition of "the room." Should the count include the people serving food or just the guests? Should the count include the open patio attached to the room? There are many questions to which there are no exact answers. Deming believed that many things cannot be measured yet must still be managed and that managers must make decisions.
Deming describes an operational definition as one that can be used for business; for example, safety, reliability, or other qualities must be communicated in the same way. When constructing an Operational Definition, Deming described three elements:
Criteria - Standards against which to evaluate test results. Provide judgment criteria. The "Plan" portion of PDSA aligns with this. A Theory of Knowledge (how do we know what we think we know) is also an element of profound knowledge.
Test - Measuring a characteristic of a material or assembly procedure. How is clarity determined, and who performs the test? This aligns with the "Do" portion of PDSA. Furthermore, it aligns with the Theory of Variation element of profound knowledge. To study a situation, Deming believed analytical statistics were necessary. An Operational Definition should include testing for Assignable or Special Cause variations.
Decision - Whether the object or material met the criteria. Test results are used to determine if a characteristic meets a criterion. Clarify a documented statement. In line with PDSA's "Study" portion. Analytical statistics via SPC emphasizes that Special Cause variation is the starting point of a required "Act."
Job Description
Deming said job descriptions should do more than prescribe motions, do this, do that, do it this way, do it that way. It must state what the work will be used for and how it contributes to the system's goals. He often used the notion of a clean table. Without an Operational Definition, a worker has no idea what a clean table means. What are the criteria? Does the table serve food, or is it used for medical procedures? A moist rag will suffice for a restaurant table. The operating room table should be covered in lint-free cloths and disinfectants.
Criteria
In an upcoming series of discussions, I would like to explore the operational definitions of DevOps metrics. We often use the term "time" when discussing DevOps Research and Assessment (DORA) metrics. Some of the metrics are commonly prefixed with lead and restore. When we examine the standard DevOps definitions of Lead Time and Restore Time, we find that they are mental models constructed from discourse comprehension. In 2018, Accelerate, the Shingo award-winning book on DORA metrics, was considered ground-breaking. Accelerate is typically considered the conical source for the DORA definitions. However, after five years, taking a closer look at DORA metrics through a lens of Operational Definitions might be an interesting exercise. Lead time is referred to differently throughout Accelerate: Lead Time, Change Lead Time, Delivery Lead Time, and Design Lead Time. Lead time is also defined in two ways that are at odds with each other.
The time it takes for work to be implemented, tested, and delivered. (pg14)
The time it takes to go from code committed to code successfully running in production. (pg 15)
The former describes the practice of Continuous Delivery, where code changes are prepared for release. The prior describes the Continuous Deployment process, which involves deploying the code changes to the production environment.
In Accelerate, Time to Restore does not have a conical definition. It introduces a new term, "Mean," with Mean Time to Restore, Mean Time to Recover, and Mean Time to Repair. Overloading the meanings of Mean, Restore, Recover, and Repair. The latest 2022 Accelerate State of DevOps Report adds to the discourse. The 2022 report defined Lead Time for Changes and Time to Restore Service but also referred to the Restore Service as MTTR (with the M being Mean.)
Rather than criticizing Accelerate, this is intended to illustrate the need for better Operational Definitions in DevOps, both at an industry level and within local delivery teams. If we accept that Lead time is a commit to production, what are the operational definitions of commit and production? Are we referring to any branch or only trunk when we measure time from commit, or are we referring to pull requests? Are we using the Continuous Delivery or Continuous Deployment definition to calculate time? Do we account for deployment strategies like feature flags or dark launches? On a restore, how should we know when to restore the service and when it has been restored? The point isn't whether we have the correct definitions; instead, we use Operational Definitions when working with the metrics.
Test
How do we test the data if we have clear operational definitions for Lean Time and Time to Restore? Just referring to mean time without analytical statics will not add value (see post Enumerated and Analytical Statistics (Part 1)). Control charts are a form of analytical statistics suggested by Deming. A control chart for Operationally Defined Lead Time and Time to Restore value would show which values are statistically in control (Common Cause) and which are out of control (Special Cause). Here, we can use DevOps metrics that can be communicated within an organization as a standard.
I want this post to start discussing how we should look at Deming’s Operational Definitions for DevOps, DevSecOps, and IT Risk metrics.
