S2 E4- Bill Bensing- Supply Chain and Security
In this episode, Bill and I discuss operations research and supply chain concepts and how they apply to security. Bill gives an overview of his work with DOD on project DEDSORD. He also gives a great overview of DevOps Automated Governance and usage of Sigstore. We also touch on SBOM's. Bill can be found mainly on LinkedIn here: https://www.linkedin.com/in/billbensing/
Resources:
The DevOps Handbook - Referenced as an influential book in the DevOps movement.
The Goal by Eliyahu Goldratt - Mentioned as an important book on operations management and theory of constraints.
Beyond the Phoenix Project - Referred to as a follow-up to The Phoenix Project, discussing value stream mapping.
The Fifth Discipline by Peter Senge - Mentioned as a good book on systems thinking.
DoD Enterprise DevSecOps Reference Design - A 2019 document from the U.S. Department of Defense on DevSecOps practices.
Sigstore - An open source project for digital signing and verification.
Rekor - A component of Sigstore for providing transparency logs.
Investments Unlimited - An upcoming book on automated governance that Bill and John are working on.
DevOps Automated Governance Reference Paper - A paper written by John and others in 2019-2020.
Transcript:
John Willis: 0:06
Hey, John, once again, this is another profound podcast I have a friend and I could save over the last year and a half my partner in crime and and all this sort of automated governance and DevOps and Dev SEC ops. And so, Bill, you want to introduce yourself?
Bill Bensing: 0:23
Yeah, Bill, fencing, fencing here from our work, John and I are working together at Red Hat.
John Willis: 0:30
Yeah, you know, so what's your what's your, you know, what do you do? What's your background?
Bill Bensing: 0:36
Our background is when you saw at Red Hat right now, what I do is, I'm the Managing architect for the Software Factory out of the North American public sector. So that's a lot of word itself. But I'm on the consulting side of the house focused in delivering consulting services, specifically around software delivery. For this organization. My background is interesting, because I'm not traditional computer science, I actually spent three years going through a mechanical engineering degree and just gave up at Diffie Kyo, and decided to go do business. My mentality back then is you know, I can own I can own a bunch of engineers one day and make a lot of money instead of be one myself. And I think that sort of came full circle, as I got into industry and got got into software I've been, we've been writing software as a first language, PHP and MySQL, since since I was in high school built the first online, I call it an online signup form for it was a computer class. So it was online registration, and that was it, my senior year of high school got into that and sort of been a bit of a just a bit of a just playing with software and building software through college. And then, as I got into industry took it a bit more seriously. And, you know, got a big name from buildings from shadow IT organizations at large, large aerospace company, and that sort of started my career and where I'm at today, was really like yourself, I use your words, and we talk Meat, meat and potatoes, you're talking today about meat and potatoes type folks, sort of meat, potatoes type individual that just, you know, built software to solve my own problems, my team's problems and eventually figured out that solves more than ours. And then I asked myself one day around 2013 15 timeframe, like, I should just probably go try to do this stuff myself and started down the entrepreneurial path and haven't looked back since then. So I've been in and out of industries since 2015, two to three times. So most of my professional life, half of its been out of industry has been in industry. So it's been pretty interesting,
John Willis: 2:27
or cool, you know, it's, you know, you know, meet a lot. And it's interesting to meet a lot of a lot of electrical engineers, but people who literally sort of went right into software, you know, don't really sort of talk much about, you know, supply chain operations research, right. And then, or you'll meet people who are sort of older times that allow, they're in project management, or like, almost, you know, my brother in law is like that he's, you know, he worked at like Tyson and, and he years of sort of, you know, running, you know, sort of supply chain with SAP and all that now, he's sort of a DevOps guy, but it's, but it's, but he's not like, he doesn't, per se write code, or, and it was interesting about your background. I mean, he is, you know, like, I'm always fascinated people that have those type educations and operations research and, and in supply chain and all. But, you know, the conversation sort of dries up quick. And then when I met you, you know, you have this very technical I mean, the right sort of heavyweight code heavyweight implementations of software. And so, so let me sort of publicly pick your brain about like, you know, what, what have you learned from that world that has been very helpful in our software world?
Bill Bensing: 3:40
Yeah, I mean, I like to think is Jenny, as you pointed out, my formal background is supply chains and operations management. I actually, my MBA was an emphasis on supply chain and operations management, and I end up going over and doing a degree in Europe and focusing on international business. And actually, my thesis was on the bullwhip effect in the beer industry. And so it's, it's everybody talks about software supply chain s bombs. And it's interesting about how like, the technology industry is borrowing terms from other every other industry, it's, you know, it's what 6070 7080 ish years old, that competing science industry itself, so it's fairly young in the scheme of things, but it's, it's one to one when I think about supply chain, and I think about sort of what we're doing like especially DevOps, like I got into this because reading the goal I remember my first operations research classmen ba was reading the goal and going through it and analyzing it understood
John Willis: 4:36
ever heard about string soon, right? Before any
Bill Bensing: 4:39
of that this was 2009. So I was a graduate of the of the of the recession. And so of course, I get out, I'm going to business management degree, and you can make more money bartending than you can do going into getting a formal job at that time. So that's why I want to do my Masters but you know, as a student, I was reading the goal and of going through this and not until a couple years later did I really really mash up what I learned in supply chain operations management with what what's going on in technology. And I think what was it 2009 As actually around the timeframe, I believe, like you and that you in the community, we're building out like DevOps days, and like this whole concept of DevOps are starting to take off. But yeah, it's it's one on one, like, the biggest thing I came from, because I came from that background was I realized, as you're getting into software, and I remember, like in 2011, talking about how do you apply, like, these concepts of lean and theories of constraints like back office, and in in those seems like in in software's I look at building software, sometimes as back office, people may not think about it that way. But it's not as scientifically for it's not approached in the scientific way, that maybe building like an aircraft is in certain certain areas. And maybe that's an overgeneralization as parts of the industry, not certain other people. But it's it is that you think about it's just an exchange of value that the other day and like you think about get back to Deming, and all it is is mapping the exchange of value, and understanding where the value additions happen, and then optimizing for that flow. And I think if you take some of those, some of those just fundamental aspects, and then start applying that to any type of human process. And then of course, technology is just an implementation of human process, then you can support it through technology, whether you build that technology, or augmented, do things like automation, it really hit me as I was doing these, these things. And, as I say, as Boeing as a Aircraft Company, and its procurement agent, buying stuff, buying plate, things are planes. And like, all our job was really manual. And there's things that could have automated expression, like things around pricing analysis that were highly complex. They had a lot of essential complexity to them. So I go back to like the mythical Batman a lot of decision complexity. And so as you as you're looking dude is like, Okay, how do you apply this now we call or whatever call or a lot of work, remote was robotic process automation, or RPA is when the whole term is now. Yeah, RPA be getting into the supply chain is like, how do you do the exchange of value and like, I started building the software like shadow IT organizations actually built this piece of software that was a prescriptive and predictive analytics piece of software for for that that was my first foray into sort of the IT realm. And like, it was really interesting, breaking into the it wrong, like almost to say you weren't wanted, if you weren't in it, you weren't wanted. And so that was a very interesting, I get it. So it's an ask you a question, I get how you bring this like full circle, it's like, that's when I sort of got this bug. I'm like, Okay, I want to help people build software anytime. But what I need to do is, I need to understand how to really do this better, not better than the folks that you know, aren't really open to me becoming part of their club. It's the and then that was that may have been a small cultural thing where I was at that point in time, but like they're actually had, they had some folks had a good point. It's like, how do you take and I think this is like the IT business alignment? How do you take, you know, somebody who's like hands on as close to the customer the problem as possible, and get them to contribute their knowledge towards something like a piece of software or whatnot, but do it in a way that's effective for the organization. That's, that reduces risk. And I think and then going back to your supply chain, like that's the whole fundamental aspect of supply chain is stitching value together in a way that manages and mitigates non systematic risk? So I'll stop right there, because that was what was going on. And I think it's
John Willis: 8:21
good, because I think it brings up two points that that I think about one is, I think you said it really good, like the it club meal. So your perspective is they wouldn't let you in, in an a pre DevOps mindset. That was basically the way the world worked, right? Like, oh, no, no, no, we do this, you know, and so, it was, so it was so non software, so non supply chain, right, the idea like, why would you not work directly with the people, you know, like, let us go buy the product for you? And we'll we'll just give it to you. And, you know, don't we don't want your input? So it is it sort of you're right, there was this it club mentality? You know, and that's sort of why shadow it created, and we can, I don't really want to talk about that. But then I think the other I think the other interesting thing was, you know, we had a great opportunity, you know, you know, so in hindsight, like which, which is sort of impossible, but like, there would have been a great opportunity to sit back and say, like, why don't we think this as just another value supply chain. And we literally had to go through this sort of war of DevOps, to get to a place where even when I was doing DevOps in the earliest days, I wasn't sitting there going, oh, you know, it's all based on Lean principles. I mean, I didn't even know that. It wasn't until I started researching sort of Deming or got to meet Jean and, and I realized so I think and, and I won't speak for the whole community. But at least the sort of the, the mindshare that I was sort of evolving with AI in the early days, you didn't see a whole lot of discussions about oh, DevOps is, is an implementation of lean. I mean, even agile and DevOps are sort of having this Currently battle of like, no, no, it's DevOps. No, no, no, we're gonna do Agile, you know, I mean today, like, I think we're much, but I think those are two interesting points. And I guess there's a third point to which sort of gets my goat which is this idea that people make this binary decision that you can't map. industriousness, economy ideas like supply chain to knowledge economy, because, you know, I guess in their mind is there's sort of this variation thing that it doesn't matter. And, and I won't say clearly maps 100% in our areas, but But I do think you coming from a supply chain background and immediately not looking at sort of, like, we can't do this, as opposed like, why aren't we doing that? So that was a mouthful for me. But yeah, no,
Bill Bensing: 10:44
I mean, I, you hit one thing there, you talk about like the early battles of like DevOps and agile, and he talked about the, the industry like, AIDS, it's hit me a couple times, and I'm gonna try to say it. I know we've talked about this a couple of times, historically, but I don't know why people in technology, don't look to the other industries. It's so so young, why aren't we looking out and saying, Okay, here's how the, here's how the physical world operates? Because I'll get back to all technology is, is an implementation of a human system. So why are we looking at how we manage human systems broadly through different domains?
John Willis: 11:17
That DevOps sort of can open that but I think we still sometimes lose? The broader but yeah, go ahead. Definitely. No,
Bill Bensing: 11:24
that's your point. DevOps, I think DevOps like that, that kind of that that was the first look, that was like, as you're getting into it and trying to meet you think about AWS, everybody trying to solve this scale technical problem, like that was really the first look at this thing. Okay, how do we, how do we look at how other people have done it, and like you talked about, so now people are looking at supply chain. And I think that's actually a risk when people are trying to, I want to say reinvent the supply chain, that's probably not the right word. But they're using terms like supply chain and s bomb without understanding how Bill materials possibly are used, and are conflating it with quality management of hard goods, procurement, supplier management, and then what a true bill of material is, and the different concepts and specifications like there's a whole little thing I want to write about one day and start juxtaposing like, you know, where's your specifications, like people talk about open source, like, you go to an airplane, an aircraft, for example, it goes all the way down to the smallest specs. And like even the bolts like saying in three bolt has an SAE specification that get multiple suppliers for. And as we talk about supply chain and things like that, it's like, where's the spec for some of our open source, of course, it doesn't exist because it was an implementation first before design. But there's a lot of things as we talk through software supply chain that I want to say, is cart before the horse, maybe the way to say it. But there's a lot of I think concepts missing, and almost my challenge to the industry. And one thing I want to try to do is to say, Alright, hold on a second, let's not slow down. But let's look at and say when you see Bill of Material, do you understand how Bill materials operate in a hard good So and, you know, now you want to talk about build processes, and you want to you want to conflate build processes with build materials? Like, is that proper? Because you go to a hard? Yeah, so that's, that's where I'm, you know, you
John Willis: 13:06
know, you've talked about this, like, I mean, I mean, we're both fans of the industry sort of getting interested, exciting. You know, I think you've Linux foundation now is saying that the war is over as bombs are, you know, but but but I think, you know, that's great like that, like, let's something is better than nothing. But to your point, you know, like you, you've talked about this before, and educated like the whole idea of spesification, right, that plays right deep into operations management, and Deming, and, you know, sort of statistical process control, like, in other words, like, I mean, like, if we're just saying, like, let's look at libraries and dependencies and look for, you know, and be able to quickly identify, you know, if we were really going to be not a bunch of baby, you know, a very immature industry. We'd have like, go no code tolerance. Like, I mean, again, like, I don't know how to do all that. But like, don't just walk around and throw the word s bomb and just say, Oh, we've got this now, right? Because exactly how you build an airplane?
Bill Bensing: 14:08
Exactly. Like we're just blows your face. If you think about like designs, and like a people talk about design, they're like, where's that spec? And where's that reusable specification and who cares? Who provides that you can now get down into a supplier quality and procurement realm around that and the the analogy of procurement and software, but it's like, who can provide this spec, like you should have five providers? Like, there's a lot of people that provided them three bolt and so to your point, it's like, how do you build it? And maybe that's the next step because we start thinking about supply chain, start thinking about adversaries that are going through and exploiting just specific implementations. You know, what if we were to take some of these popular concepts that p this popular open source and just create a specification around it, and then allow people to then the market to then provide competitive offerings, that specification, because now you can then start to test a quality based upon that spec and as long as you know that specification meets this level of quality But you're willing to use, you can now then go in and start to test the quality of the suppliers. I mean, that's, yeah, I go down like, I'm done, man. I'll start I'll start I'll start going down rabbit holes as we started No,
John Willis: 15:10
no, I mean, again, you know, I think it's sort of the the net net of it all is we don't emphasize operations management and in software, right like that, like, it's almost like, like everybody should be I mean, I'm not even going to come close to professing that, like, you've got a masters in it. Like, I, you know, I spelunk the internet about it. Right. But the point being I know enough about it now that like, says that this is a discipline that we're missing.
Bill Bensing: 15:42
You say this, like, what if I said, What a hard to make assessment that just software engineering is a discipline that's missing? Like, there is no discipline of software in general. Right? Because I, to your point, like a discipline, but yeah, the right the operations management. Yeah, that's, that's what I've been thinking about lately. Because like, was it the first time software engineering, a software engineer made appearances like was, it was a 1962. Speech and I forget her name, but off the top my head, but she's on the first dimension, I wrote about software engineering as a concept. Like, maybe that's what like Nutella ASP. Net? Maybe that's the point to push force for, but
John Willis: 16:19
not to dump on an industry that's been very, very good to us. But, um, is that you know, that, you know, I don't really I think, you know, even I sort of joke every once in a while, like, when we hear the word computer scientists, how non scientific, you know, most of our industry behaves, right, like we don't, you know, we're sort of terrible at the sort of scientific method. I mean, well, yeah, let's put in perspective, I think some of the large scale organizations now are good, have been very good at, you know, the Googles. And in the Amazon's, of course, right, like, but in general, you know, most likely just, uh, you know, the, my common joke is, you know, if you think of PDSA as sort of the, the cookie cutter version of scientific method, you know, we're really good in this industry plan, do plan, do plan, do plan do, right? We kind of stink at Study Act, right? And but, yeah, so I think, you know, when we sort of say we're like, software engineers or so like, we challenge people and say, Okay, what do you mean by that? Do you mean that somebody writes Python code? Or do you mean somebody that literally applies, you know, operations research manager,
Bill Bensing: 17:26
I want to point out one thing there, this is the only industry you can call yourself an engineer and like, legally get away with it. Because think about you have the Professional Engineering certifications. When you're going to like a municipality or whatnot. A lot of their engineers, they have a P certification, they have an independent industry body that validates they meet some level of criteria. Same thing with an architect, you can't just call yourself an architect when you're doing buildings. I mean, there's liability behind it. You know, is the is the is the IT industry getting captured?
John Willis: 17:57
In fact, we, you know, you we can talk a little later about the project we're working on for it got loose. Yeah. But um, yeah, I've been doing now seven or eight years, in one year, it was sort of a false start. But we tried to work on a paper. So we write like, 4050 people meet virtually of last couple years. But But hopefully, maybe next year, we'll get back to going to Portland, I think jeans, wife calls it jeans, pajama party. It's some of the smartest people in history to come for two or three days to work on hard problems. But we tried to work on a project about the ethos of software, like because you're right, like there's an ethos like a pilot has, you know, sort of there's a certification, there's, there's sort of an ethos, there's a commitment, you know, like a doctor is the Hippocratic Oath. Like, there's, there's sort of, if you look at almost every industry, you know, a pilot, or even a police officer, there's sort of a, there's a certification, there's, you know, there's sort of an ethos of like, like, the pilots is that like, he controls hundreds of lives, in his or her hands or the doctor, you know, you know, the Hippocratic oath, and even even the police officer has an ethos of like, you're handed a badge and a gun, like, you know, and we have nothing like that in our industry to your point. So
Bill Bensing: 19:17
it openly say this, and I say this, because that's how I started out but I you think and this is not a criticism, but like, it's the difference between mechanics like, I'll use the analogy car mechanics, and like, a true engineers. It is a PE, it's almost like it's interesting how like, our industry is full of frankly, mechanics, we call them we'll say it's an angular engineer or a Java engineer. Really, those are just tools frameworks are just tools no different than like a Ford mechanic or Chevy mechanic and that is, is we ask maybe there's a question like when I look at we look at supply chain, we talk about building. So a bit more rigor in the industry. Is there a way we should be going like now we're starting to bring in these concepts and like think about the we are truly now getting to where software can be a a exploits in software are now affecting your life. What do you think about those oaths, a lot of those oaths are predicated on protecting the individual they're serving. Now, we're looking at this to where,
John Willis: 20:15
like, and again, I think you'd agree with this is as we're talking this through, right? Like, again, I think what's going on with, you know, Linux Foundation, the hash bomb, the executive order, all those things are great, because something is better than nothing. But the thing that the thing that I think, you know, like, possibly, you know, we've talked about, we'll talk about automated governance a little bit, and where we think there's sort of party as far as missing a good portion of the point. But then maybe the meta meta point is, you know, instead of sort of trying to dive into, like, how do we fix continuous delivery? How about we fix the industry, of how we think about, you know, like, you can, you know, an ethos for software. But, yeah, I'm gonna dive into, you know, when I first came on Red Hat I had been working with, you know, the, I mean, very excited and still excited about this concept of DevOps, automated governance. And, you know, we'll talk a little about it. And, you know, and I saw a lot of things going on in, in Red Hat, you know, but nothing seems to sort of click, and then I was told to sort of, you know, go talk to this guy, Bill, he's done some really cool stuff with the government. And I saw what you were doing, I was like, this is exactly what I think automated governance or DevOps, automated governance. Do you want to talk a little about what you did with this? I guess it's called Dead sword or No, what's the problem? Yeah.
Bill Bensing: 21:39
So we built it. So there is a bit of a market problem there market opportunity with Red Hat Do you have the DoD as a 2019, November ish, the United States, the department defense released, the single the DoD enterprise DevStack, ops reference design, and I'm pretty sure if you read the references, John, your names in there three or four times, because what they did is they took a lot of the top thought leadership of the industry, whether it's books, publications, and they compiled it into this sort of 100 page document that was meant to be given to any DoD organization or agency or department saying, okay. They've, I think they've referenced the DevOps handbook there a couple of years ago, back in the pages, but they're going through and they're referencing this and what they did and what I what I liked about it was it's an amalgamation of everything that you didn't think that that everybody should understand. And no, one thing that may be I don't think is unique to the to the department offense, I just think that non digital native companies, I'll just say, the traditional IT companies, I mean, their culture is such that nobody's gonna say nobody's continuously learning. But continuous learning isn't a normal course of what how those those organizations operate, have been part of these organizations. It's just not there. And that's what by Google, Facebook, Netflix, like, that's their differentiating cultural value there. But what it is they brought this in together, and it was an easy way for people to get up to speed to say, Okay, here's what sort of modern software delivery looks like. And then now looking at that the biggest thing is it described, describe sort of what an in state could look like. But it didn't give anything really fungible to the organization such as, okay, there's always tools out there. And there's approaches and processes. So that's cool. I sort of have a 40 lane highway. Now, I know where my left and right bounds are. But like, which lane do I pick. And so there's a lot of I can compare it to a coloring book, like it was a coloring book, but it still needs the mediums and colors. And so what we decided to do when we st created an upstream was we focused on where a lot of people don't focus. And I would argue, don't focus on software delivery is on the business architecture, the business process, the value stream, like nobody said, Nobody talks about the value stream, besides before anybody before Nick, published project a product. I don't remember too many people. But I take that back, I do remember listening to be on a Phoenix Project, which by the way, we could talk about that whole, like my infatuation, all that and but beyond the Phoenix Project, where you talk about the value stream mapping, and actually, some of the books out there, but like nobody's talking, nobody thinks of it from a term of a value stream. So what we did is okay, what's the business idea? What's the value stream? Because that's what people miss? What do you need to do to properly deliver software? And that's not just build unit test, go to an artifact repository and go to production? There's a lot of things in there. What are the validations and it's where we overlapped with the concept of automated governance, because those are the things those are the what people don't pay attention to because they focus on the developer experience. Hence just developer not the development, which is the whole aspect of the value chain of going from idea to production. That's more than just the developer. And so as we started doing this, we build up PI ghosts, which is a was a community and upstream and a tool that sort of enforces this allows you to think workflow first and business architecture and value stream first, and then codify it in a way that it's structured for the organizations you provide the concept of paved and golden paths, and like like Net focus has on a paved path. And so it gets down to some of the concepts. The operation management is variability. When you have different tools, you have different approaches, you introduce variability, we have variability in your system, you increase risk, and people want to understand why they're getting, you know, why they're getting, why they're getting owned by adversaries on a daily if not weekly, on a weekly if not daily basis, you start looking at the variability in how you do your software. And I think this is a lot of where we aligned on the automated governance, because it was there's like, okay, so how do you start to identify, and then in reinforces value streams? But yeah, that's where that's where you and I got that got hooked up. And
John Willis: 25:37
yeah, it was play gross. What that you know, so that was, you know, I think it's gone through a couple name changes, but play goes, the thing I saw the immediate value in, in sort of my description of it was, it had an opinionated interface, but a non opinionated implementation, which was perfect for the automated governance thing, like in other words, like, there's some structure to how you would have to do this, to create different workflows, you know, but that's a small price to pay. But how you wanted to implement either, you know, you know, Jenkins, or bamboo or, you know, whatever, sort of, you know, Sona type, or, you know, whatever sonar cube or sonar is whatever, the, you know, it didn't have an opinion about that, it allowed you to, and I thought, wow, you know, and, and it had the infrastructure of things, you know, so, you know, they sort of get into a little bit of the automated governance, the idea was, you know, back in 2017, topple power, right, this brilliant blog article about how Capital One had done their pipelines, and they talked about gates, you know, and then this idea of, like, there were these, you know, if you could prove as a developer that you could evidence, these type of things, and we gates, and in conversations, he was going to jeans, sort of forum paper thing, and we got to know each other, and I was like, wouldn't that like, be a great place to put evidence, you know, at the stations? So we're in, you know, I think it was 2019, we wrote a paper, you know, they were sort of at governance about the sort of active stationary data. But it was, it was very loose, like, you know, you know, sort of some of the guys at PNC genre's Toski, they went out and started building this their own. But there was no sort of handoff to somebody, like, how do I start? Well, you're going to have to write a web hook architecture, you're going to have to sort of write something that captures the sort of web hooks and then go to at the station data store. And to me ply ghosts, not only solve the sort of interface structure, but you'd even tell me about things like, like, one of the things that you might do for evidence is, like, if you did a scan, or even sort of the build logs, you could tar them, and then SHA them, and then turn that into a digitally signed evidence. And plug us would like, have a directory structure for those things. Those were requirements as part of government. So all those things like that somebody would have to build themselves before they even got into the automated governance. Like, it seemed to me like that's, you know, sort of fell in love with playgrounds because of that.
Bill Bensing: 28:15
That's awesome. Yeah, exactly. And you talked about, it's treating everything, you know, it's, it's giving those things first class citizen capabilities. And that's the thing about the development lifecycle itself. It's like that as part of the cycle. So give it a first class capability.
John Willis: 28:28
So, so that, you know, I was thinking about this just now, as you were talking, like, it's a good opportunity. So, you know, when when we first first wrote the paper, you know, so Google had introduced an open source project called Graph ehess, which was actually designed for gestational data, but something they used internally, although it was very purposeful for the way Google wanted to use it. And it was really only a reference architecture was very small lines of code. And, and I know, on a couple of projects, we tried to sort of get that working and nothing against the people support that project just for what we wanted to do in mass scale for delivery and creating at the stations for like, for every type of thing, like Did somebody you're appearing on a pull request? What was the evidence of the build? What was the evidence? It just didn't scale? And then in parallel, another project, right, and again, to Red Hat's credit, like, we build a lot of open source tools, right? A project called six star started. And, you know, mean, you sort of looked at it, you went deeper into the technical side, but it took me about six star and why it works and what what opportunities are for like, this idea of digitally signed evidence or at the stations for audit.
Bill Bensing: 29:39
Yeah, the what I liked about six, because six store has itself is a sort of a it's an open PKI if you recall that, but it's based upon same concepts that that Let's Encrypt and know and certificate transparency is based upon, but one other component had was something called a record, which is I think, bleep creek for record, which is of course it's Merkle tree structure. So what you can start to do is store your store signed cryptographically signed with keys from the sixth or PKI inside the structure, and you can cryptographically validate that what's been signed is tamper evident. And so first off, you have two things first with the sixth or product, because it's certificate transparency. And so I'll explain that for a second. Like Let's Encrypt, I can go request a private public key pair, I'll get my key pair back, but then the public key is posted for everybody to see. So sort of a line of defense to where if if something was signed with my private key, or one was issued to me, and it was signed, but I didn't actually request it. Now I know, I've been for some reason somebody got a hold of my identity. And it's tying that everything back to an identity, which is, which is key and then store itself, I believe it used to be 20 minutes, I believe it still is 20 minutes, like the key pairs are only valid for 20 minutes. So the idea is not to give these long live Key Pairs just to give long enough, but you also now have that history, we can go back and see, okay, here's what the public key was signed with, who was it issued to, and you could track things back to an identity, which is, you know, very important if you enter zero trust concepts. And then you have the recourse of the things I can say the records of what I signed can be stored in there. And what I like about record what we found good, which was different from a little bit different than graph ehess Was that we took the evidence, so say, the outputs, we would if we didn't get an output from something like GitHub or GitLab, or sonar cube, we could create an output automatically. And then we would hash it and sign it. And then that we create a record of that with the the private key, the private key, we sign there, we create a record of it inside of record. And then we take just that that tar, that tar bundle and just said little zip file, and stick it somewhere where it doesn't matter. And what happened is we didn't have to figure out how do we protect and keep things from being tampered with? Because now we have something that's tamper evident. I think that's the big key when people try like, Okay, we got to figure out how to protect this from being tampered with. It's like, okay, if you tamper with it, what's the impact of our business, frankly, none. But at that point in time, we want to know that we can't trust whatever that body of evidence is anymore. And so then we have to do some other procedure. And so that's where the record store came in. They have two other things, one called Full CEO and CO sign of its to their components. Full CEO, I think is the open root cert and then co signers, for signing containers based on the OCI image. But it's sort of that ecosystem of being able to create transparency. And as I like to talk about, like sort of doing blockchain without
John Willis: 32:30
Yeah, that was the thing. Because I mean, originally the original conversation I had had with, you know, the group of people that were originally sort of starting this, like, what can we do with this evidence, you know, like through blockchain, and then we found that like, a number of problems with Blockchain in the enterprise, which was, you know, usually got some group within certain financial, like, who's doing blockchain, you know, that's ours, you know, next thing, you can't you weren't, but second, it was more of a hammer, you know, to sledgehammer to sort of, you know, swat a fly. Right. And, and then it, you know, the thing I liked about about Riccar, and, you know, sort of six star Riccar, which is the Merkle tree, which is really sort of a very sort of an easier implementation of what blockchain does, because the Merkel tree itself, is an immutable structure. So is that having a blockchain and correct me if I'm, like, totally off, but the blockchain is like this sort of list of digital signatures that's in sort of that can't be broken? Well, if you had all those digital signatures in a Merkel tree, it's the same thing. If anything in that sort of Merkel tree is broken, the whole thing's broken. And it's not, you know, not trustworthy.
Bill Bensing: 33:39
Exactly. It's open trust. I used to think about sort of like what blockchain is, at the end of the day is a distributed ledger. This is you know, something like $6 not distributed, like blockchain depends upon having community participants to take on the computing and supporting of the actual the system. Holy speaking, six store doesn't, but it provides that same transparency capability provides transparency, and cryptographic validation of that transparency. And so like, and I think that that's key for any organizations who are looking at doing this kind of stuff, it's like, yeah, blockchain has a lot of fun things to play with. But if you if you do, but if you set up your own blockchain, but you don't have any other community participants, besides your own servers, it really blockchain based so it is, yeah.
John Willis: 34:25
So we, you know, we started, you know, one of the things I love about you know, so to your you know, your enthusiasm, you know, is that so I feel like I lit a fire under you on this automated governance, you know, like, you had been doing sort of the dead sword stuff, which was really good. And it was sort of almost everything that I, you know, and I came in and started telling you about, let me tell you about what we've been trying to do. And, you know, it seems like now you've been come a really good, you know, strong evangelist for this DevOps, automated governance. So, we're, I guess we're, yeah, I think people have told us something times is that when you use that phrase automated governance, traditional security people think you know of, you know, sort of older products where sometimes we refer modern governance. But give us a quick overview what your thoughts are about this sort of modern governance or automated, sort of made it go with us.
Bill Bensing: 35:19
Yeah, I mean, that's a Kobe, my thoughts is, like the it club I talked about earlier, that was always the rhetorical thing. Well, it's not secure, and it's not compliant, we'll tell you what a secure and compliant mean, and of course, nobody can ever tell you. And so it's always implicit, I go to two different people get two different answers. So at the end of the day, if people didn't want to deal with what I was building, it was always insecure or non compliant. But you couldn't tell me how to become secure and compliant, right. So that's why I like the concept of modern governance, like when you and I started talking, it was and I don't want to about shadow it. But at the end of the day, like there's people out there who are great are calling the great mechanics that don't really have the engineering backgrounds, the meat and potatoes, people that that have just enough technical knowledge, to build some valuable piece of software for an organization. But the biggest hurdle to move it through security compliance. That's why I like about automated modern governance and moving from implicit to explicit governance, they themselves may not know what's going on, I look at a sort of like middleware for a process. Like if you think about like middleware on a value chain, that's how I look at automated governance, they may not know or, nor should they need to have that skill at that point in time, they may learn over time, but they need the feedback. If it's non compliant, where is it at? So then they can learn how to make it compliant, right? Or they can understand that that's what I love about the automated governance as we talk about, you know, flow feedback and continuous improvement from the DevOps handbook. Like, it's feedback, that's the feedback that's stopping a lot of stuff. And like in the public sector, there's this concept of authority to operate, and to attain authority to operate could be months, sometimes years on different platforms. And like, how does that help you when you're trying to whether it's defense or whatever? It's not like, why is it taking so many years, it's just simply the feedback cycle, and the implicit nature of it. So that's why it does, like, if you could be explicit, you can shorten the feedback cycles, then you can do, of course, with the core concept of agility is to get to market quicker, but that's not the thing. And I think that's what a lot of people miss, when they talk about agility is just getting summer. It's also the continuous maintenance and support of it over time, like software lives amongst most a lot of people over a long period of time nowadays, it's like when you think about modern governance, and automated governance is now okay, constantly checking, am I still compliant? Am I still doing the things I need to do to ensure that like, even if you aren't making changes to the pieces of software is not become a security factor or a compliance vector anymore. And it is not in the need to be fixed. And that's what I liked about it. Because like, for my background, so that meat potatoes just like wanting just to write software, even though it's probably the most horribly written stuff, if I went back and looked at what I wrote, I would probably cringe. But the end of the day, like, I delivered value to 1000s of people across large swaths of organizations that did billions of dollars of business. And so like that, right there, I think, is the epitome and I look at modern governance and automated governance. That's the reason why when you when I was first talking, I gravitated towards it because I realized if I ever have the stuff, and people like me, have this, that we can be empowered to really deliver, and figure out how to build more value. And that's why like, is
John Willis: 38:21
the abstractions right, in other words, you know, like, I think, you know, um, you know, the sort of the, the waste if we go back to lean sort of the waste and how we deliver things with, you know, sort of risks and regulatory control. And all that is, it's just the sort of the communication and non agreement about what things mean, right? Because we have different people you're trying to collaborate on, like, what is this particular reg number of psi DSS or something in this? What does it really mean? And so it's, it's, it's really almost like the pre DevOps days, it's a throw over the wall, you know, make sure you adhere to this, you know, and then they're like, Okay, I'll try to do that. And like, you didn't do it, you know, and, and, you know, in a perfect world, you know, which I think, you know, we both believe nothing's perfect, but, like trying to get to a true north have a better way of like, could we, or I will say that you'll be like, imagine a world where risk developers and operations don't even know what sonar cube is. All they know, is that there's sort of a language of DSL, yes, that has a contract. Basically, that is it was certified to get the accurate data and like, and just like, we don't have to worry about like, how we, you know, get our money in an ATM machine. We trust that there's, there's some mechanisms that move it to the Spencer, we get to that place and, and really, then is the sort of transaction between risk operation Development says Do we agree that there's a trusted source of this information? And do we agree that this particular set of, you know, stances or, you know, yellow code or whatever, will basically get us as close as we can to, you know that. And and then we sort of remove all that nonsense. I mean, you look at, you know, when we talk to people, and they tell you about, like, long debate, so we're a risk person, and you know, and developer or are just sort of debating and argue about like, like API transactions that come from sonicu. Like, we are Toyota Ono's worst nightmare. Right? So,
Bill Bensing: 40:38
it's like, you're sitting there, you're talking, it's I am getting back to like, the Moodle, Mara and Marie at the end of the day, right. So you get back to some of those concepts of total credit. Everybody focuses on Moodle and the seven types of ways you know, is what we're talking about here is really Murrah the lack of uniformity. Exactly what you're talking about. Nomura causes Muna right at the end of the day, it causes the different types of wastes but like also like cognitive load, which you're talking about there's like, Why does an audit person have to understand the sonar cube x y&z and you may have five different artifacts,
John Willis: 41:08
or have to know to I don't know, I don't want my effect, except for the operations people who install it and make sure it operates in it. It actually creates the certifiable data. I don't even really want them to know the operational aspects of it.
Bill Bensing: 41:21
And I would argue, would you put it that's Marie. So like, our software processes are full of Murray, which is just overworking people, we have this, we have this cognitive load, which they're having to work in areas. So yeah, exactly. I get, I mean, I get down to like, I always go back to ERP systems. And actually, when I, when I listened to beyond the goal, the second chapter, the second part of the on the goal, when Goldratt talks about the ERP systems and MRP and ERP systems like and that's when it really hit me was like, if you think about like technology, and some of the stuff like the ERP system was designed, like before, then everybody had a different modules. It's almost like, you know, whether it's finance, accounting, inventory, management, whatever, it was similar to what we have here, artifact repository, but what do you abstract that away? And you just focus on, you talk about, like, what's critical to your job to executing, you know, the value you do? It's not knowing sonar cube, it's just knowing or it's not knowing it's just knowing that Okay, did I? What is it? What is our agreement on the C code coverage or agreement on maintainability? Or something? And then who cares? Who evaluated that? That's, that's irrelevant.
John Willis: 42:25
Like to it to bring it all the way back to apply ghosts? That's an you know, that is an implementation, you know, exact subject or, you know, piece, right, like, because doesn't matter, like, like, like, you know, and again, I think that takes us all the way to like, the whole idea of Linux, you know, sort of chain command chain, right. Like, like, I like, like, I get this output. And like, I can interchange the things I do, you know, maybe that's not the best example. But the point being, but I can I, you know, if it's bamboo tomorrow, or it's Jenkins, tomorrow, or whatever, it's trusted, and it delivers the information that we need, at the abstraction level. Exactly. And so what's, what's some sort of interesting, I guess, the one other thing, like, I think, we talk a lot about this, like, again, big fans of s bomb, but, but this idea, and we talked earlier about, like, sort of SVM maybe, you know, maybe there's a bigger picture, like basically, setting up practices for software engineers to actually have like operations, like, we covered that. But I think the other thing about s bomb is like this idea, like, if we just do as bombs, like, everything's gonna be great. And then, and this is why I think a sponsor without automated governance, you know, as far as we'll be good at maybe defining, you know, what dependencies but not to the spec level, but like, again, you know, everybody talks about the lock lock for J. Sort of fiasco, right? Like maybe, you know, zero days can be identified quicker, and I agree. But there's nothing in an F bomb and you're going to people when think they just sort of talk about it. I'm just solves it all, just this this bomb the hell out of it. And like it'll be all fixed. And I don't know. So everybody's saying that, but it sure sounds like they say that. There's an S bomb tell you that somebody did a peer review, on a pull request, there's an S bomb tell you that there was you know, a clean bill is as bomb tell you that you had your cyclomatic complexity less than a certain level, does it tell you that you had 875 or 80% test coverage? Right, like and all those things, right? No, it doesn't tell you any of that. Because that's not what it's designed to. So yes, allowing me to understand a better software Bill material at a high level is great. But thinking that that by itself solves all the sort of the cyber threat problems is I just think I worry about the over rotation of that.
Bill Bensing: 44:51
I'm there with you, I think is the over rotation mean ever since some of the solar winds and things that are happening now that people do that mean the success and the prevalence of of being able to like not having to actually actually attack you as an organization, all I got to do is just go downstream a little bit inject a couple pieces of
John Willis: 45:10
skirmishes here Yeah. To be. And so
Bill Bensing: 45:13
it's, it's over rotation but like that's why I like you know my been limping on me doing like sit down look at how like a proper supply chain works when I say a proper supply chain look at a physical good and look at how they secure and validating security, I will argue is just a form of quality, I look at quality as the overriding term and security and compliance are just forms of quality. And look at how quality is validated in a supply chain and and not specifically a value chain just not within an organization but within a true supply chain between disparate parties. And then look at also sensitive supply chains and how people manage sensitive supply chains. I think once you start looking at that and start saying, Okay, how do we apply those concepts and those Hillstone on the physical exchange of goods with what's happening in the email exchange of bits and bytes, then S bomb will take on a different a different view. Because like to your point, like I'm seeing people trying to fit everything and ask people just looking at Spam was a bucket to dump everything about their piece of software. And that's not what it is it was if it's trying to solve everything, then it is actually not a solution for anything. And so it's like, okay, let's use it for a bill materialist talk about that, let's get down to okay, what's a composed of where did it come from? Man? And the question is, should even where it came from be in the s bomb, because we talked about specifications. If I have something that's an mp3 bowl, like there is actually different systems called supplier management systems that determine you know, where you can get your components from. So you can create a graph, ultimately a graph of what it looks like, of the reality of your state of an application, which is really where I think people are going, they're using the term s bomb, to describe some graph that decided to cut that has like, a declaration of what your piece of software is
John Willis: 46:56
No, and I think we got the we're pointing in the right direction. And, you know, again, to give our industry credit, you know, the, the cyanotypes in the J frogs, and, you know, I've been, you know, hovering around, you know, you know, giving us better Intel in, you know, I won't say at the level of a graph, although they would probably say it was, but we're, you know, we've got all the ingredients to get in right there. And I just, I think sometimes in our industry, we sort of over rotate on, we get this new hot idea, and like, that's gonna solve everything. And then we just saw that the oxygen for everything else is lost. The one last piece that I wanted to sort of talk about is, you know, I talked about what he wrote the DevOps, automated governance, reference paper, but it revolution 2019 2020, COVID, we just didn't do anything on it, we decided 21, we, we sort of start back up what we originally thought was going to be version two, you know, I really enjoy working with Bill, I, you know, you can already tell at this point, his thoughts are pretty awesome. So I invited him onto the project. And, and we did such a good job, they decided probably about the mid summer last year that like, they decided not to just put it out as a reference paper, to turn it into a book. And so we decided to do it in sort of a novel format. Instead of just a rough, we thought about not as many people read the reference architecture that like as I thought they would, because it was boring as hell. Right. And so instead of like, let's turn into sort of a Phoenix Project, textile, so give us a little overview of investments unlimited, and, you know, what, what can we expect? And we'll see that next summer sometime, I guess.
Bill Bensing: 48:33
Yeah. So investments I'm learning, it's a it's the story of the hurts and the identification of a way to people think that an organization thought they're doing DevOps, I thought they're doing these great things. And and, you know, the, in what they were doing was good, but it's also it brings into the reality is like DevOps can be narrowly scoped between just development operations and not other concerns. And then all of a sudden, they're hit with a large with a large MRA material matter requiring attention. And so a large compliance issue or like
John Willis: 49:03
a failed audit, but in a very specific financial. Yes,
Bill Bensing: 49:07
exactly. And so now with this, they've realized holy cow, like there's a big part of our process that's missing. We've we've missed the these organizations. And, you know, the story is going through, and it's representative of also a lot of our co authors have have experienced in industry sort of that, I'll say the socio technical approach the socio technical issues, more on the cultural issues of what does it look like to include other people in the development process? So it's not just the technology, like there's things around attestations technology, like how do you make that work? But a lot of it too, is like how do you start to change the minds and how people operate? When you go to an automated governance or modern governance approach? You are fundamentally asking people to change traditional ways of operating, you don't have you know, you're taking security compliance and taking them from just looking at pieces of paper and doing checkboxes to people that are now thinking about okay, like, what's a CTRL V to be considering? Like, yes, some people do that, but you've changed them from now. Just execute process to how do we continually design and improve on our risk management process? What are the things we need to be thinking through? And so as everything goes through through the investments Unlimited, it is that story of how it plays out as the trials as to tribulations. It's the, you know, we think we're doing good and all of a sudden, we get our legs caught back out under duress, because we did one thing good. But guess what it's, you realize it's more complex than just the one or two things we can do. So, yeah, it's, it's, it's, it's, I feel like, you know, the goal that the goal of that book is, as people read it, definitely wanted to ring true home, like the Phoenix Project did where, you know, sometimes we're like, wow, how are you? You know, anybody in a financial institution. And it's not just for, you know, engineers, and it folks, a lot of what we're riding towards, is audit, because the goal of this audit and governance, the goal of this book is to raise awareness, but also present a model for collaboration to like the DevOps, automated governance did that. So how do you look at clarity, because I think that's probably where a lot of the industry stuck is like, yeah, the automation sounds good. Like, how do we even get to automation? It's like the dead sword we talked about earlier. There's a lot of things out there to say, here's how you want to do it. But nobody's saying like, specifically, how do we take that first step? And I you eyes that are in investments and limited? A sort of that that the you know, the whole goal is like this, this, this is what your first steps could look like? This is literally a model of your company, you probably the most people read it, though. They'll feel it. And then okay, here's how this company now gets to automated governance.
John Willis: 51:29
Yeah, no, I think one of the things we talked about a lot nicer. So one of our problems our industry right now is we don't do both you have big fans and sort of system thinking and complex systems is that it seems like the approach that we're taking in sort of DEV SEC Ops is a very sort of reference architecture structure, Dev SEC ops reference architectures where, you know, do you have this in the pipeline? Do you have this in the pipeline? Do you have this in the file, and all these three, this is their own format, their own DSL, their own logs? And, and what we're really doing is making the translation and conversation between risk and developers exponentially more complex. Right, like, you know, where, you know, either way, a common one that I hear a lot, which is, you know, the auditors want to understand, what's the difference between the sonicu blog in this, you know, and Nexus lock? And like, well, you know, you don't understand they sort of look like they tell you the same things, but they don't write in like, like, again, we shouldn't be having those kind of conversations, you know, like, you know, a developer or developer calling operations to say, Hey, can you explain the risk? Why the next Syslog doesn't match this on a keylock will never match doesn't, that's about what they're idiots. You know, like, yeah, like, we shouldn't be having those kind of conversations.
Bill Bensing: 52:41
I say to the scenes, it's unnecessary cognitive dissonance that creates cognitive overhead and cognitive and a psychotic load. And like, that's you, at the end of day. I'm rereading The Fifth Discipline, I know, it's gonna be pretty heavy. But like, it's, I think, that's, you know, if people can trudge through, that's a good book, because to point point out systems thinking is, you know, we use the word engineering an architect, but there's not, that lacks a lot. And as you point out, like, we're not here to I'm not necessarily bashing the industry, I think sometimes I want to say is the industry is over rotating, sometimes I have a tendency to over rotate and some of my the way I say, simply because I just see vendors, like all of a sudden, like s bombs popular, so let's put what we do now. So it's like, so like, hold on a second. Like, there's they're there. There's, there's not that they do or don't have this capability. Some of them do, but it's like, alright, let's,
John Willis: 53:32
let's start. The thing that's, you know, sort of, like I said, take it with a grain of salt, because it is what it is. But it's amazing how many vendors after the lock for J. shell script or sort of module was exposed, all explaining how they could have caught it. Yeah, he did. Yeah, nobody did and like, but every vendor now talks about like, that's their first page. Oh, you know, the letter per day like we have a solution that, like, Hey, Bill, how do people find you if they want to sort of have more conversations with you and
Bill Bensing: 54:06
always LinkedIn, hit me up on LinkedIn, you'll find Bill Benson on LinkedIn. I do have a Twitter handle at Belbin Singh. By I admit I'm trying to get better with my tweeting. But LinkedIn is where I tend to LinkedIn conversations
John Willis: 54:20
activity is these days. So me too. I used to tweet like a madman. And these days, I find, I just have better conversations on LinkedIn, but I'll put all those up in the shownotes. And also, it's always a blast talking to you. I you know, I get to talk to Bill quite often as part of my job, which is sort of a treat, so but anyway, thanks, Bill for for a great podcast.
Bill Bensing: 54:46
Yeah, thank you very much for the invite. Cheers. Good.
