EP 08: The Kubernetes Effect: Transforming Cloud Infrastructure with Jim Bugwadia

David McKenney
Hello, everyone, and welcome to this episode of Cloud Currents. I'm Dave McKenney, and today I'm joined with Jim Bugwadia. And Jim and I are going to be talking about containers today, more specifically Kubernetes and governance and that whole world. So how are you doing today, Jim?

Jim Bugwadia
Very good. Thanks for having me, David.

00:31 - Jim Bugwadia's Background

David McKenney
Awesome. So we always start off a little softer. Let's talk about kind of a little bit of your history. You've been in the industry for quite a while, but most of the recent 1015 years has been with your new company and your model, which we're going to spend a lot of time talking about. But part of that, you were at Cisco, and I looked at the timeline there and that one, it registered with me because I know some of the stuff that was going on at Cisco 2010, 2013, and you were in cloud. Can you talk a bit about what you were working on? Were you working on some of the VCE stuff or the app centric infrastructure?

Jim Bugwadia
What were you doing there? Yeah, absolutely right. So just maybe a little bit about my background and know I'll lead up to the time at Cisco. So I'm a software know. I started my career more in the telecommunication space out in the Midwest at Motorola, Lucent, Bell Labs, then moved out to the San Francisco Bay area in the late 90s, early 2000s, worked in a variety of different startups, big companies building around virtualization, data center, cloud, and then eventually ended up at Cisco, like you mentioned, in 2000, this was like around 2000 and maybe eight or nine until 2013. And at that time, this was still early days of cloud computing.

David McKenney
Right.

Jim Bugwadia
So folks were still at that time, people were thinking about IaaS and infrastructure as a service and platform as a service and software as a service. And Cisco, of course, was evolving from virtualization and data center to mostly like private cloud and then some partnerships on public cloud. So my role at Cisco was more on the consulting side, worked with a lot of customers, both enterprises as well as service provider customers, who were building this kind of modern cloud computing style infrastructure for themselves.

David McKenney
Yeah, that's awesome. So as a customer of Vblock back in the day, VCE was definitely doing a lot of really good things. And it's interesting because it led the way for a lot of what we've come to use in converged infrastructure. It wasn't without his fault, but it was definitely interesting time. So this was 2010 to 2013 time frame, and this is also when a little thing called Kubernetes starts to rear its head. So was there stuff that was going on at Cisco that had you say, this idea of containers is really the next thing, I want to be part of that. Or were there certain other aha. Moments that had you change over to containers or cloud native?

03:13 - Shift in Containerization

Jim Bugwadia
Yeah. So containers existed, as you probably know, in our industry, containers have existed for quite a few years, like even a decade or so before some of this evolution to cloud, right? And with things like PSD and Solaris, they had zones, and BSD had a concept called jails. So it was ways of. And at one point there was a term used called OS virtualization, if you remember that it was a different flavor of virtualization from what VMware popularized. More to slice up. Exactly, right. So it was more slicing up an operating system into secure containers, zones, jails, whatever the term, the different operating systems used. But it was interesting as virtualization and type two hypervisors were getting popular, companies like Google and others who just completely bypassed that level of virtualization.

They were packaging their application as containers, running these on bare metal type servers at scale, using other orchestration software. And Google at that time was using something in Trottle called Borg. And Kubernetes emerged out of those endeavors and all of the learnings and all of the things. And when in Linux, of course, there were things like namespaces introduced for segmentation. Other features went into the Linux kernel, which ultimately allowed containers to evolve, right? And the interesting thing at that time was a lot of the companies offering platform as a service. So there was cloud foundry, there were other companies like Mesos, with Mesosphere. They were all using containers underneath, but the interface was developers would submit their code, whether it's Python, jar files, other things, and then the Paz layer would containerize the code and run the code.

And as that was happening, a company called Cloud, from which Docker emerged, they started experimenting with saying, hey, what if we shift this containerization step left in our pipeline? What if we containerize outside of the platform at build time? And now you can take that same container image and you can run it on any path, right? And that idea resonated enough in the industry where know Docker launched as a separate company, separate from cloud. And containers became much more popular. And it was that act of freeing containers from the platform layer, giving it back to developers who could build it early in the lifecycle, make it available to run anywhere. That then eventually led to orchestration systems like Kubernetes. And again at that time, there were others like Cloud Foundry and besos. And Docker had their own orchestration system called swarm you're right. And it took a few years for the industry to settle on Kubernetes at de facto standard. So very interesting times. And going back to your question, yes, we started learning about these technologies while during my time at Cisco, but then it was also kind of working with some of the hyperscalers, working with companies like Netflix and others who were building this different style of architecture, a different style of application architecture, which could run really well on cloud computing systems where servers at that time, cloud computing in the early days, maybe your server disappears or maybe something happens, but your app needs to stay running. Right? So those were the type of problems being solved which ultimately led to the evolution of Kubernetes and other technologies around it.

7:38 - CNCF and Kubernetes

David McKenney
Yeah, your description there, the shift left, that whole line of thought was probably one of the best descriptions, examples of how that all played out that I've heard. That was really well done. I'm going to have to steal some of that. But I was going to ask you. So the time, you're right, there was sort of that, I can remember a time, even in public clouds, where your choice of engine, I'll just use that as a blanket statement, was either mesosphere, a swarm, or Kubernetes, that it was still very unknown, sort of in the late 2000 and teens who was going to win. And I say that lightly because they're all still in use. But Kubernetes really did rise. So was there ever a thought that you would be supporting just Kubernetes with your platform or that you would be supporting all of these?

Or did you really see that Kubernetes was the way things were going to go?

Jim Bugwadia
So we understood the importance of convergence in that layer, right? And just like, of course, with any other kind of evolution we have seen in our space, it's important to have some standardization and some convergence of one sort of de facto winner being emerging, right? So we did not feel that adopting all of these or having a choice of orchestrators really added much value to end users.

David McKenney
Right?

Jim Bugwadia
Because as a developer, why do I care? Vendors may care. It's like VHS versus Betamax. Ultimately, I want to watch a movie, right? Why do I care about what the format of the tape is? Of course, price points, features, things like that make a difference. And in our industry, we've seen in these kind of battles, I guess the best technology usually prevails, or the most widely used and widely adopted marketing budget doesn't hurt either. That is very true. That is very true. However, in this case, I do feel Kubernetes was able. Then there are a few different things that happened, right? So everyone working in the space kind of understood that one company, one organization owning this was not going to work. And nobody also wanted fragmentation, where every vendor tried to do their own and really differentiate at that layer.

So like I was mentioning, Kubernetes emerged out of Google. There was a team at Google that was looking at how containers were evolving and they kind of would step back and said, hey, we have been using containers for ten plus years. We know how to orchestrate and manage these at scale. We have internal systems, right?

David McKenney
Google app, we should do something. Early days of, right?

Jim Bugwadia
Yeah, as a platform, as a service. So they started working on what eventually became Kubernetes.

And the really smart thing they did was they donated Kubernetes to the Linux foundation and what eventually became the CNCF, the Cloud native Compute foundation, which is where Kubernetes, Prometheus and a lot of other projects, including Kiverno, which is Nirmata's project reside, and our own. So what that did is it allowed different vendors to collaborate, contribute, work on Kubernetes, and really evolve that as that next set of standards for orchestrating containerized applications. And one interesting trend I'll highlight quickly, and we can discuss in more detail later, is Kubernetes. Over time, it added features for extensibility, which allows it to go a lot beyond just containerized application.

Kubernetes has become the platform for building platforms where you can orchestrate almost any service request for its operation, even if you're trying to provision cloud services, vms, et cetera. In a lot of cases, the substrate tends to be Kubernetes, which is really exciting for our industry.

David McKenney
Right.

Jim Bugwadia
It now gives not only APIs for developers to standardize on how to describe and operate their applications, but it also gives operators, security teams a standard set of construct with which they can now operate not just applications, but any workload, any service they want to orchestrate and offer to their end users.

David McKenney
Yeah, we've said it probably a couple of times now without saying it directly. But for me, I think one of the most fascinating things regarding containers is the timeline. So in my career, we all got that steady dose of adopting software defined compute, first and foremost with hypervisors, and then software defined storage. Well, actually, I think software defined networking probably came a little bit before software defined storage, but either way, they all kind of came in their own relative chunks, let's say. So adoption was quicker with each one that came out, but they were all kind of came after one or the other containers. You pretty much got all this all over again out of the gate, right? Because all these are pretty much set.

And like to say, as somebody who's worked in hypervisors, you're going to really be looking at doing just about everything you did at that level, one level deeper with containers. But it's just fascinating how fast it rose to what it is today and just how it continues to expand. And I know you just mentioned CNCF, I think it's a fantastic site to go visit. So anybody who wants to just go see what the world of containers is like and all the different projects that are working on it, some that are very heavily adopted, some that are very new. The CNCF landscape is a fantastic view into the world of what everybody's working on. And you can spend the better part of a week just clicking through everything.

So maybe talk a little bit more about you, because I think you work with CNCF directly, correct? Yeah, talk a little bit more about what CNCF does and the importance you see in the container world.

Jim Bugwadia
Absolutely, yeah. So CNCF was formed to be the organization, the open source organization that manages Kubernetes. But then very quickly, Kubernetes is not meant to be complete. Right. And that's part of its appeal. Like I mentioned, it's mostly used as a platform for building platforms, by enterprises directly. And there's this trend called platform engineering, which is very fascinating too, as an evolution of DevOps and devsecops. But Kubernetes, to complete it and to put it in production, you need other tools, other projects you need. Just like with a data center, you need storage, you need networking, you need some of these other components. But beyond that, Kubernetes, again by design, is not secure by default, right?

You need to put in your guardrails, you need to put in other things into place to be able to secure it, harden it, and to be able to offer it and apply it at scale. So very quickly, what happened in the CNCF is as Kubernetes evolved, there was this need for other projects. And some, like Prometheus, et cetera, just got very wide adoption out of the gate.

David McKenney
Monitoring solutions, backup solutions, all those extensions.

Jim Bugwadia
Exactly. So those came about first, and there were solutions for networking, solutions for storage and in the ecosystem. The way CNCF works is there's a lifecycle to every project, right. And it's fairly straightforward to submit a project as a sandbox project as long as it meets the goals of the CNCF, their charter and what they're trying to accomplish. But the barrier to entry is not too high. Right. You need it to show traction. You need to show why it meets the cloud native, I guess, philosophy, how it complements the ecosystem and what unique value proposition your project might bring. But as long as there's some adoption, some traction, and you can check these boxes, you can get a CNCF sandbox project, but that's where the real kind of journey begins. Right, because sandbox really means that, yeah, this is interesting.

Maybe you have 2030 people using it organizations using it, but it's not at a scale or not at the level of adoption of Kubernetes or Prometheus.

David McKenney
Or in enterprise grade that some might call.

Jim Bugwadia
Right, right. So there's a fairly steep curve to go from sandbox to incubation. Right. And incubation means now CNCF. So really, if you kind of map it to the classic product journey of crossing the chasm, incubation is where you have your biggest gap. So you're still working with that early majority, but you're on the brink of crossing that chasm.

David McKenney
Right.

Jim Bugwadia
And you have now proven that your project, your technology is being used in production. There are enough adopters, there's maintainers, perhaps from different organizations, and there's widespread usage and interests of the project. And the final step is graduation, which is more kind of, again, making sure your project has the right governance, it can survive independently of any one organization. And of course, you're kind of showing more progress from incubation or anything that had come up during due diligence for incubation. You're resolving it before graduation. So that's typically the journey of projects. And there you mentioned my company, Nirmata. We are part of the CNCF as one of the member companies, but we also contribute to a variety of projects as well as working groups within the CNCF.

David McKenney

So you saw a need, though, with, I mean, I do agree, like that it's fantastic to see how many solutions that are out there, some that are coming and going. But when you decided to start Nirmata, outside of the ones that you mentioned, Prometheus, some of the ones that have been around for a while, there was clearly a need around security and governance. Is that what sparked your interest to start Nirmata and products like.

Jim Bugwadia
Initially and again, just from a lifecycle of our company, we work with early adopters of Kubernetes Microservices, tried a few different things, where at that time, some of the pain points customers were seeing was around cluster provisioning, workload management, things like that. So we built a few tools around there. But one of the things we built was the soil policy and governance module, and that really resonated deeply. And as we kind of looked at our early adopters of Nirmata and we would ask them, what is it that you feel is compelling here? What would you like to see more features around, et cetera? It was overwhelmingly governance, right? And you're like, okay, just tell us, what does that mean? What exactly are you trying to do? So few interesting things we learned along the way.

First of all, Kubernetes, along with all of the other goodness it brings. If you think about it, Kubernetes is the first platform in our industry that really is built for Devsecon ops, right? The whole DevOps movement and bringing together developers, operators and security teams and giving them one set of constructs as code, as a YaMl manifest, or however you are as a JSON file, however you want to express your applications. That was never done.

David McKenney
That's not just one person at the company. I thought that was just one individual.

Jim Bugwadia
No, I mean, you obviously need expertise from several different folks. So the problem quickly became how do you get all these different roles to collaborate on this? How do you get them to agree on this, right? And what we saw is, look what Kubernetes is doing for applications. Why can't we do the same for policies and governance, right? And in the industry, of course, we've talked about policy and code before, but Kubernetes, with its extensibility, with its powerful features of plugging in custom controllers, really now gave us that opportunity to take something. And by the way, Kubernetes has this other great feature called admission controls, where every API request that flows through Kubernetes, and being a modern application, everything in Kubernetes is driven through API requests.

So any API request you can intersect, and you can apply some policies or checks on and you can even trigger other workflows from those API requests. So it really becomes this powerful extensible system which a policy and rules based engine can take advantage of for security, for automation and operations. But really, we saw this opportunity where there were several companies focusing on different aspects of Kubernetes that the missing piece to the puzzle was governance and this convergence of security and operations, right? And that's what we doubled down on as a company, as a product team. And out of which came Kiverno and some of our product offerings at Nemata.

21:43 - The Multifaceted Role of Policies in Kubernetes Governance

David McKenney
So where would you say let's unpack the policies a little bit more here. So when I think governance, I think of very popular public cloud things these days. So compliance, security and cost are very relevant that you'd think some of this stuff would normalize over time, but it just keeps growing and expanding. There are new services, new ways to save changing regulations. So the governance space just year over year continues to be right at the top of challenges. And I can only imagine that's very true here in containers. So when we talk about this policy as code policy engine that you've brought up, are these policies rooted in compliance security costs? You mentioned Devsecops, so I'm assuming there's policies related to operations and what is the consensus mechanism for how these policies are created?

I'm sure some come from like industry standards, but then there's probably also something across these various cross functional groups you've mentioned. How are they coming together with a tool like this to say, I agree, this is the policy we should put in place for what we're doing.

Jim Bugwadia
Right? Yeah, great question.

David McKenney
Right.

Jim Bugwadia
And you're very right, policies address a broad set of concerns across operations, security, even cost management. Right? So resource management, cost management. And really one of the patterns we started seeing is in the security space we've kind of thought about as, okay, you start with vulnerability management, runtime security, but when it comes to systems like Kubernetes, which are very real time, very fast, based on your declarative manifest, your instructions that you give the system, they react very quickly to things and this push towards self-service, right? So of course every company, every business today is a software business at some level or another. And every business wants to deliver value quicker to their customers, which leads to developer experience. Developer agility are key concerns. And even same thing for data scientists, right?

With all of the gen AI and big data and other things, data scientists fall into that same category, right? How do you empower them quickly? So what happened here in Kubernetes is there are certain policies required to take real time actions in cluster which you cannot do through other traditional control loops or governance mechanisms or ticketing systems, things like that, which we did in the past. So you want, for example, if a developer wants a namespace, and a namespace is, think of it as a virtual cluster, right? So it's a slice of a cluster with some compute, some capacity, things of that. They want to do it in real time without any interaction with it, and they want to get a secure provision namespace in which they can deploy their applications, try out things, et cetera. Right?

So to do that, though, you need policies in place to quickly generate resources as well as validate configurations that developer is putting in and saying, hey, maybe you require labels, or maybe your pod requires a security context before we can allow it to run in production. Maybe because your workload doesn't have a default network policy, the policy that the security team has put in will automatically generate one which denies traffic to external system, right? These are things which can be organizational rules put in. Like you mentioned, there are compliance standards, industry standards like PCI, HIPAA, even like guidelines from NIST 853, NSA hardening guidelines for containers which have been translated into these policies.

So there's the whole regulatory compliance aspect of this, and then there's policies which might be just designed for best practices to say, look, you can run a pod without quotas and limits and things like that. And a pod, by the way, is just a basic unit of workload. In Kubernetes, it can have multiple containers, but you're running this pod in Kubernetes, you need to define requests and limits for how much CPU, how much memory, things like that, right? So policies can address these wide range of constructs, but to really get them to share different teams, to coordinate, collaborate on these, becomes tricky, like you said. So some of these are just industry standards defined as controls in specifications.

Kubernetes has a set of pod security standards in the similar spirit where they publish the Kubernetes authors, that the community publishes a set of security standards for pods. But then there are other policies. You want your teams to agree to customize and also kind of manage over time. So the innovation that Nirmata brought in is to say, why not manage this the same way we're managing Kubernetes, right? So make policies native to Kubernetes, make policy reporting native to Kubernetes, make policy exception management native to Kubernetes. And the thing that were betting on is that more and more developers are being trained on Kubernetes. They understand how to use Kubernetes, what it means. It's almost become like the operating system across clouds and lots that I kind of saw. There's about like six to 7 million developers already who are certified on Kubernetes.

And that continues to grow year after year. So making it native to Kubernetes and allowing now different roles to collaborate, really just like what Docker did for the container. We're kind of trying to do the same for policy and saying, let's standardize, let's use common tools, common languages, an easy way each one of these roles within an enterprise can understand, view what a policy is, apply the best principles there, and collaborate on this. Yes.

28:01 - Integrating Policy and Security Early in Development

David McKenney
So how do you to steal from what you mentioned or talked about earlier about shifting left and moving stuff more to the build time? Can clearly see a goal here is to enforce much of this better way of doing things earlier on, whether it's security or policy or any sort of performance guarantees that you're trying to strive for. But ideally, you're adhering to this as early into the build cycle as possible. But in our world as an MSP, we often see that tools like this, I really don't mean to call it a tool, but these solutions rather they aren't always a first thought. When companies are going out there shopping, it usually comes in as an afterthought and then they see the value like oh yeah, we should have done this from the beginning. So where do you think this sits with developers?

Do you see many adopting a solution like Nirmata right out of the gate and ingesting it in their sort of their development process? Or is it more so that people are hitting bottlenecks, or they can't push past that next scaling barrier? Or maybe there's security issues that they just can't get past. Where do you see people adopting the tool today? I'm sure this is all going to evolve over time, but where are you seeing the footprint today?

Jim Bugwadia
So there is growing awareness, especially in the cloud native space. So if you ask folks maybe three years ago, what's your checklist for putting Kubernetes into production policy governance would not be on that checklist, right? If you ask them today it very much is. And this is some of the work that we've been doing in the community. I co-chair in the policy working group in CNCF. We collaborate with other groups like the security, as well as other automation and operations teams to build this awareness, to kind of make sure folks understand the security model of Kubernetes, what it can do. I mean it's extremely powerful, but it's only powerful if you use it, right? So how do you kind of enforce this usage? And that's where this trend towards platform engineering comes in.

David McKenney
Right?

Jim Bugwadia
And we've had operations, of course, as long as there's been any infrastructure need, there's been an operations team. In the beginning, developers and operations were completely siloed. In my early days of my career, I had no idea who was putting my code into production. Right. It was some team that did their job, and my job was to kind of complete the feature or fix a bug or that was it, right? As once it got submitted to a build system that doesn't work anymore. That's not the best practice in our industry developers. I think it was Adrian Cockroft at Netflix at that time. Or maybe it was somebody from AWS that said you build it, you run it, right?

So kind of the mantra is if you're not responsible only till your code is compiled and tested in a build system, you're responsible for your code running in production. Of course it doesn't mean you need to root credentials in every production system, but what it means is at least at the application layer, you have to take ownership of some of the SLAs and runtime and of course production issues as well.

David McKenney
Yeah, it seems so simple, but it's definitely a pivotal, I mean, it's very tangible. In our timeline, that actually was a major shift into the way we deployed and especially the agile modes of sprint releases and all that stuff.

Jim Bugwadia
Absolutely. And then that brought about a lot of the cultural shift for DevOps. Right. And at that time then folks said, well, what about security? We really need security to be part of this conversation. And you need to, like you were mentioning, start thinking about that as early as possible too. So that sort of led to, well, maybe it should be devsecops. But that never really caught on, right? Because the challenge was developers. So look, DevOps is not about getting developers to do operations. You still need an ops team; you still need experts at the different layers of the infrastructure stack. And that doesn't go away, right? The complexity only increases with every additional technology and layer and other things we're adding. And same thing with cloud. There are so many cloud services now, somebody needs to know what to do.

But platform engineering kind of now says, well, let's take the best of what we learned in DevOps, but as an organization for your developers, and this is for mid to large organizations, right? If you're a 20 person team, you don't need to think about platform engineering. But if you're 1000 person, 2000 person team, where within the enterprise you have several applications, you have several different clouds maybe, or infrastructure stacks you're using, then you need to start thinking of the platform you're building internally as a product. So really the shift here is to say, treat your platform as a product. And Kubernetes, because of its standardization, its extensibility, and its broad adoption across every major infrastructure stack, becomes that ideal platform on which you can build your own, what Gartner calls itps, or internal developer portals, internal developer platforms for your organization.

David McKenney
Right.

Jim Bugwadia
So what we're seeing is this move away from a one vendor platform, right. To say in the past it's like,

okay, I have to pick one vendor or the other towards composing a best in class platform using CNCF components, using Kubernetes as that underlying substrate, and picking and choosing how you want, networking, storage, policy, security, all of these components now you can pick and maybe one of them evolves over time, right? But you're not changing the fundamentals of your platform and that's the beauty of a composable architecture and this style of architecting platforms for enterprises.

David McKenney
Yeah, and I love the choice. I know sometimes it can be paralyzing the number of choices that are out there, but I love what you said about Kubernetes not trying to be a complete platform because it lends to the decoupledness of bringing other solutions in. But I did want to highlight. So a little earlier you mentioned that Kubernetes is not secure by default. And when you've got governance tools being layered on it, maybe pointing out where some things could use improvement, would you say that Kubernetes has the mechanisms to become secure? Or would you say that there's often a need to go back to the ecosystem and find a tool that maybe does a thing that Kubernetes can't to make it secure? I'm sure there's a blend, I'm sure it does some of this.

But yes, if you want this compliance, maybe we need to go get another solution. But how much of this can be done with tweaks to Kubernetes, as it says, deployed, whereas you might need to go find some more solutions to bring it up to a truly secure and compliant level.

Jim Bugwadia
So you can do a lot with Kubernetes and containers by itself. Right. So I would say like at least about 40% to 60% of it can be achieved directly with Kubernetes. And this is a very rapidly evolving space. There are efforts underway, which we are part of to even push this policy governance further down into the stack, into itself. Right. And the idea is the more broadly folks can use it, the more programmable, the kernel, if you will, which the API server kind of acts as the kernel of Kubernetes is, the better for everybody because now we can innovate, we can bring different technologies and tools, but make Kubernetes as programmable, as extensible and as easily usable as possible.

So there are efforts to make this as native as possible, but there's a balance, because at some point you want Kubernetes to be stable and mature. There are efforts underway which are now proposing long term kind of release cycles for Kubernetes. Because so far Kubernetes, first initially it was every three months, you would see a minor release and then it became every four months. And now enterprises are saying even that's too much to keep up with, right? Because enterprises don't want to upgrade. You need stable abstractions; you need things like Kubernetes. Maybe you upgrade once a year, but applications, of course, you want to upgrade more frequently and patch more frequently and do things like that, right? So both of these are sort of counter forces in the community which need to be balanced out.

How much do you put into the core of Kubernetes itself without destabilizing it? Can you expand certain performance cycles, things like that, in critical components? And where do you need that next level of plugin, if you will, that needs to operate with Kubernetes. So the good news is Kubernetes is designed, again, the kernel of Kubernetes is designed for extensibility. So it's very easy to try things out as an extension. And once that, let's say concept or idea sticks, then try to bake it further and further into Kubernetes. And we've seen that evolution. In fact, even on the networking side, we've seen that evolution happen in Kubernetes because there were things like ingress and ingress controllers, and then there were approaches of managing traffic, like inbound flows and outbound flows. But now we have an evolution called the gateway API, which still allows extensibility.

You still need some tool to provide that networking layer, but the concept of that is baked into Kubernetes itself. Similarly with policy and governance, one of the things we're doing with Nirmata is we're looking at policy management and governance as a whole and saying what components can we bake in further into Kubernetes? So one thing that immediately came out is even if you have different ways of writing policies, even if you want to use different tools, languages, whatever, why not have the same reporting so much like with open telemetry did for observability? We're kind of saying let's have an open reporting standard which any scanner, any security tool, any policy engine can take advantage of and report things the same manner. So any user, whether it's a developer, an operator, security team, now you can standardize at that layer. Right.

So those are the sort of things that are happening as we speak, which is very exciting in the community itself.

David McKenney
Yeah, the counterforce's comment there, because I would imagine that. So containers bring about one of those promises that we largely wanted but was missed with virtual machines and that was that promise of portability. Like I can click and drag and drop a vms from here and drag and drop it to some other building over there and it works. And I'm sure some have achieved something near that, but containers definitely execute on that. I'd say better, but the counterforces hit me with kind of that upstream compliance that you're always wanting to maintain your ability to be upstream compliant and that way that you can achieve that portability. Now, Nirmata, you guys work with multi cluster, multiplatform governance here. What are you seeing? It could be challenges or best practices, but just let's kind of double click on the topic because it's an important one.

I mean, most people who run containers out of a dev test or lab environment are running them in more than one production environment, if I'm not mistaken.

Jim Bugwadia
Absolutely.

40:28 - Navigating Multi-Environment Challenges in Container Deployment

David McKenney
Where are some of these challenges? Where did things sort of get exacerbated, if you will? Because you're running now containers in more than one environment where maybe it was hard enough to do it at scale in one environment. Yeah.

Jim Bugwadia
So certainly there's the lifecycle of the application, like going from dev test to staging to production, and those could be different environments, right. But then there's different cloud providers, each with its own differentiated service set for containers, for applications, for even things like databases or AIML, whatever you wish to kind of consume. So the level of portability that we see is Kubernetes now offers a standard API across these, right? So your developers don't have to be experts in every cloud provider service, whether you're using AWS or Google or Azure, you can focus on that Kubernetes layer. Your OpS team can also standardize on that layer. But of course, Ops has to know the next layer down, which becomes the cloud services too.

So at that point, the difference to a developer, if they're dealing with Kubernetes constructs and APIs, whether that cluster was provisioned on AWS or whether the cluster was provisioned on Azure or Google, matters very. There's minimal difference, right? Or no difference if you're standardizing on Kubernetes. So that's the advantage of standardizing at that layer and then the Ops team or the platform team can shield developers, data scientists from the complexities of each cloud and the pitfalls for governance, security and all of those. But what we see is it's not so much as taking an application that was built on, let's say, AWS, and then dynamically running it on some other cloud like Google or Azure oracle or others, but it's more that ability to do that if needed, right?

So we're not seeing the industry is not at a state where the same application may be running on completely different cloud providers at the same time. But it's more about that standardization of concepts of APIs, things like that. And to say if we do need to migrate, it's going to be fairly straightforward, right? And that we have seen, we have worked with several customers who have migrated across major cloud providers, and it's been minimal effort compared to what would have happened if they had directly used cloud provider APIs, right? It's been very seamless.

David McKenney
So does some of the non-portability or where you have to look a little more under the lens, is it when you get to those extension? So I might see that I need persistent storage. I've got my persistent storage extension. I might be using S3 or ebs in Amazon, but I make a move to Azure and now I'm working with blobs. So to the developer I'm probably removed from that portion of it. But is this where some of the portability becomes more problematic, is where the extensions are leaving off and picked up by the native platform side?

Jim Bugwadia
That's exactly right. Right. So if your application is directly using S three or RDS or something like that, of course porting it to a different cloud will become a challenge. Now there are tools and just kind of giving a plug for another awesome tool. In the CNCF ecosystem, there are tools like cross plane which let you model cloud services using Kubernetes itself, so much like Nirmata is doing for policies. With cross plane, you can model infrastructure using Kubernetes manifests and you can create these composite definitions. So now if you're using that cross plane composite, your application really doesn't care what type of object store is used underneath. It's using things at a higher level of abstraction so you can do things.

And the other awesomeness there is also the provisioning and state management of that IAC component is done completely in Kubernetes, just like any Kubernetes resource, right? So you can apply policies to it, you can do a lot of other things to it, make it part of your GitHub and CI CD pipelines. But that's kind of the next level of evolution for enterprises that need that. In a lot of cases. If your application again is just using standard Kubernetes constructs, then you can map the storage like in Kubernetes itself. There are ways of expressing storage and every cloud provider's managed Kubernetes service will expose their native storage to it. Right, right.

So that's the kind of trade off to think about is trying not to use cloud provider services directly in the apps, using it through Kubernetes constructs or using projects like cross plane which bring that abstraction into Kubernetes in a generic fashion.

45:45 - Outro: Securing the Software Supply Chain with Nirmata

David McKenney
It just makes it so much easier to consume the storage from that provider though. Yeah, so we talked a little bit about pod security and compliance. There was one concept that we hadn't talked about yet that I know you guys’ reference and that's software supply chain security. And I think it's a fantastic set of words all put together. Maybe talk about that and how Nirmata enables that supply chain security and why maybe those newer into the development world what it is and why it's so important. I think a lot of us have learned what supply chain and what blockchain have done, but kind of bring us around here and what you're doing with it.

Jim Bugwadia
Yeah. So you're aware when we're talking about software supply chains, we are really talking about the whole pipeline from development to testing, then pushing things in production. And more and more these pipelines are getting automated because of technologies like containers, Kubernetes, et cetera. Right. So a few interesting things have happened. First of all, we have gotten a lot better as an industry in securing production environments, right? We know what to do to secure production. There are enough tools, great tools available for that. But we also kind of have now with the shift left, with the codification of everything infrastructure as code policy as code, more and more responsibility and power has shifted to these build systems, right? So you might have a Jenkins server sitting in some data center which is actually deploying to a bunch of production systems.

And to do know, you might have put keys to the production system on that Jenkins server. So what I guess attackers got a little bit savvy about is to say instead of trying to break into production systems, if we can penetrate that build server and put a malicious package or something over there, I can immediately get access to hundreds perhaps of clusters and production systems much easier. Right, right. If I could just get access to.

David McKenney
The compiler, I can run everybody's code.

Jim Bugwadia
Yeah, attackers are shifting left as well, and they're seeing the power of the ability to, again, use these systems to break into a lot of other systems. And by the way, if you manage to compromise a popular vendor or a popular open source package, which is now deployed to thousands of millions of other things, guess what? You suddenly have a much bigger exposure and breach, which is exactly what we saw with attacks like SolarWinds and log forging and others which are highly sophisticated and designed to take advantage of these things. So supply chain security is about saying, how do I make sure not only my production systems are secure, but my build systems are secure. How do I make sure that just like we talk about immutable infrastructure, I have immutable builds. Nobody can compromise my build environment.

It has to be hermetic; it has to be sealed, and I can prove that. So when I run something in production, like a container image, how do I get proof for that? Right? So doing that, the technologies that have been developed in the open source community, there are projects like six store and notary, which are solving these problems. And Nirmata and Kiverno integrate with those, use those. So the whole idea is when you're building container images, you can sign container images, you can also build these proof manifests, right? To say, I can prove where my build originated. I can prove that I have done a security scan, a vulnerability scan. I can prove that I don't have any critical flaws in my software.

I've done a code review, and I can sign all of these as digital documents and attach these attestations into my container image. And then policy engines like Kiverno and Nirmata can check for these at runtime and say, okay, I see that this was built on a system I trust. I see that you have followed your best practices for your git repo, you have organized things correctly. And I see that this has been signed by somebody who has the ability to deploy into these clusters, right? So only then I'm going to allow this image to run. And that all happens as part of admission controls, right? So that's what we do to help secure software supply chains and make sure that what you're running in production is trusted, is properly built, has the integrity, and hasn't been tampered.

David McKenney
So are there any regulatory mechanisms right now that really showcase that? So that if somebody says, we really love your software, but I need you to prove to me that it's signed, certified, immutable, everything you just described from start to finish, everything below deck is good. All right. What would somebody be looking for to identify that?

Jim Bugwadia
Yeah. So there are efforts underway to define those standards. There is one emerging standard called sALSA or SLSA, which stands for supply chain levels for software artifact. Bit of a. SalsA is easy to write. SalsA is good. So, yeah, what they try to do is they define different levels of compliance for your software supply chain. Right. And it's mostly around, again, are your repositories, are your build systems, are your things coming out of these? Are they properly built based on best practices and open source projects like Kiverno and several others, Kubernetes itself, we have started doing things like signing every image, creating s bombs, software buildup materials, which shows exactly what's in every image, what's in every container. Exactly. And things like adding vulnerability scanned reports as attestations. So open source is certainly innovating quite a bit in this area.

It's showing how to kind of do this. And we're seeing a lot of regulated industries, like in financials, healthcare, utilities are extremely, of course, not just interested, but also actively putting this into production by starting to sign images, check for these signatures, and looking at salsa compliance as that level of industry sort of regulation or standard.

David McKenney
Yeah, I'm going to definitely do some more research on this aspect because I'm very green here. But one final item, and this is probably going to turn into a long one, and that's okay. We'll just see where we go. Maybe a follow up. We can't get through a full conversation without talking about artificial intelligence.

And it struck me when we're talking a lot about policy and there's a lot going on, especially in the storage world, because storage is so important when it comes to AI and people who are training models and the desire to put more and more, whether it's things like classifications and labels and triggers, closer to the storage and shift left, to use your words, are you seeing anything particular to what you are doing in the governance and policy space in Nirmala that you find are going to be very applicable to people who maybe are, whether they're doing training or inference or what have you, but just anything in the AI ecosystem, where do you see some of the power that Nirmata can bring?

Jim Bugwadia
Great question, right? And you're absolutely right again, any business in this space is looking at AI ML at some level or another. And Gen AI is of course exploded in terms of popularity. So the interesting thing is almost all of these AIML workloads at some level or another run on Kubernetes, right? If you're consuming it as a managed service, the cloud provider is kind of providing some abstraction, but in many cases, they're running on Kubernetes underneath. What's happening is these managed services tend to get used in the early experimentation phase, but then eventually doing anything at scale requires. The same platform engineering team were talking about is also starting to service data scientists by providing data services on top of Kubernetes for NAI ML services. Right. So there are emerging frameworks like Ray or Kubera.

There's another framework called Kubeflow which lets you do everything from training to inference serving all on top of Kubernetes in a very native manner. And certainly for that, policy governance becomes key. Everything from like, how do you share GPU resources? Am I allowed to kind of do it? Am I keeping into the budget?

David McKenney
It's expensive hardware. Yeah, I want to make sure I'm using every last inch that I paid for, but not a drop more.

Jim Bugwadia
Extremely, yeah. So things like that become super critical. And then there is the data governance. Am I secure? Am I not leaking data on some public buckets? Things like that becomes very critical. So certainly putting in those guardrails at the operational level is what we do and where we provide value into these AI ML workloads.

David McKenney
That's fantastic. I'm going to fight the urge to ask a follow up because I really want to. But Jim, this has been awesome. Thank you so much. I've certainly learned a lot. Got some notes to go do some searching on my own and I'm looking forward to a follow up sometime. So appreciate your time today.

Jim Bugwadia
Thank you. My pleasure. Thanks for having me, David.