3. Daniel Feldman, Cloud Security Architect at HPE, Service Authentication with SPIFFE and SPIRE

Welcome to the Security Podcast of Silicon Valley. I'm your host, John McLaughlin, and I'm here with a very special guest, Daniel Feldman, who is the cloud security architect at HPE. And formerly, he was at, how do you pronounce it? It's called Citel.

Citel. Citel. Yes. If you Google the pronunciation for that word, you will get a different pronunciation, but that's how we pronounce the company.

Citel. io, recently acquired by HPE, so I'm sure that was a very exciting adventure to be part of an acquisition. It's been a wild ride over the last couple of years to build this product and then get some adoption and then get acquired and then get a lot more adoption all of a sudden. Well, I'm super happy for you, and thank you again for joining.

Just to put this into context, you were recently speaking at KubeCon Europe, were you not? Yes. So I did a lightning talk at KubeCon Europe, and then I've been doing a number of talks and workshops in the HPE booth at KubeCon Europe. KubeCon, if you're not familiar, is the big conference for cloud-native technology.

So Kubernetes and then everything that surrounds Kubernetes, like Istio, Prometheus, Jaeger, Thanos, all these enormous projects that have a lot of momentum building cloud-native technology. Every cloud provider does presentations there. There are a thousand different startups showing off their new ideas. Of course, most of them won't exist next year.

That's the dark joke of KubeCon. And it's been a huge privilege to participate in, I think, every KubeCon for the last three years. It's twice a year, so that's six KubeCons, either doing presentations or staffing our booth or just attending and having a lot of fun meeting people in the community, learning about this technology. Well, thank you for being on our show, the Security Podcast in Silicon Valley.

We are asking the question, what is service identity management and authentication, and where is it going? And I know you've been knee-deep in this problem for quite some time, Daniel. So very interested to hear your perspective. Let's start with the basics.

So when we're talking about identity, it's the answer to the question, who are you? User identity, and of course, this is authentication, logging in, all of the things that we as humans do to prove who we are. And then there's service identity, which is a completely different beast because you can't ask a service really to just save a password. But service identity is something every engineer has struggled with that's ever built a microservice architecture or worked with a microservice architecture.

The basic question is, what is service identity when you just cut right down to it? So first of all, I'd like to start with a bit of a history. This is some research that I've been doing into the history of service identity as part of the talks I've done at KubeCon and other conferences. Back around 2010, there was an enormous attack against Google infrastructure.

That was really the genesis of all this technology. There were advanced persistent threats. We think they were maybe Chinese spies, but it's very hard to prove, that were able to capture enormous amounts of data from Google data centers. And Google, of course, is a very smart company, employs a lot of very smart people.

What they decided to do was create a service identity technology for every service at Google. And you can Google, you can do an internet search for the attack, which is called Operation Aurora. And then you can also do a Bing search or a Google search for the technology they developed, which is called LOAS, or L-O-A-S, which is apparently a huge part of the infrastructure at Google. I personally have never worked at Google, and I certainly wasn't there in 2010, but a lot of the key people who developed this technology were involved in that.

So the idea that they came up with was to uniquely identify every service in their infrastructure, and not just have every service claim its own identity, but be able to trace that identity back to the cluster they're running on, the hardware they're running on, and the data center they're running on, and then have these centrally managed identities that are granted to each service, such as an X. 509 certificate or a JWT token. And if you have that X. 509 certificate, then any connection or request that that service makes can be encrypted just using mutual TLS encryption.

Or if it's a JWT token, the JWT token can contain a bunch of additional metadata that allows the service's requests to be authenticated. And then you can combine that JWT token for the service with a JWT token for the user, and then any request can have its authentication and authorization checked by all the other services in the microservices architecture. So when you're looking at a company like Google or Netflix or Facebook that may have tens of thousands of services, they need this technology in order to be able to make sure that every request is authenticated and traceable. to an originating service and an originating user.

And of course, those companies have very large budgets and a lot of very smart people working for them, so they were able to implement this technology themselves, and they did. Each one of those companies has teams working on service identity. Then what happened in around 2017 when Citadel was founded was several people who were alumni of those companies got together and decided to make an open source service identity layer. And that was the genesis of Spiffy and Spire.

So the initial document for Spiffy was actually written by Joe Beda from Google, and he wrote a white paper proposing the idea for Spiffy. It's still on our website. It's a little bit outdated now because it dates back to 2017. Spiffy is an open standard for implementing service identity.

So it provides an API that allows any service to retrieve its own identity and then also trust bundles that allow it to validate identity from other services. And what do I mean specifically when I say identity? I mean an X. 509 certificate that lets you establish mutual TLS authentication or a JWT token.

You get both of them. So the reason it's standardized, of course, is you don't want to be writing your services to just acquire identity in like 12 different ways. You want to be able to have one API that each service can implement as a client and fetch its own service identity, and then you could have multiple implementations of that standard. So the initial idea would be there would be a whole bunch of different implementations of Spiffy.

What ended up happening is there's the Spire implementation, which is the only real complete implementation, and then a number of other projects have just taken the Spire open source code and incorporated it into their existing software. And more of that is going on as we speak. There are more projects that are just wholesale adopting the Spire project as a component. Does that make sense so far, John?

Yeah, that makes perfect sense. And so was it really a shock that no one else really picked up the standard Spiffy and ran with it in their own implementation, or was there a push to sort of open that up and get developer interest in writing those? Well, honestly, the challenge there has really been around the workload API. The other parts of Spiffy have actually been widely adopted.

So, for example, almost every service mesh that's available uses Spiffy compatible identities. The issue is they aren't real Spiffy implementations because they don't provide the API. Instead, when you initiate a connection, for example, in Istio, it's automatically creating a mutual TLS tunnel for communication with your service, and it uses a Spiffy identity on both ends, but it never provides the workload API so you can access your own identity. Oh, I see.

It didn't get adoption. Very specific pieces of it got adopted, very wide adoption. Yes, I would agree with that. Specific parts of Spiffy have been very widely adopted.

What hasn't been widely adopted yet, what we're working on, is getting real adoption of the concept of this workload API that allows you to, as a service, fetch your own identity and then use it for establishing authentication with other services directly. And we are working on that. Right now, the only way to do that is to install Spire yourself, though. And I love that Spire was the full implementation and is also open source.

Yes, it's not only open source, but the code is owned by the Cloud Native Computing Foundation. So it's super open source. It's not open source owned by one company. It's open source that anyone can contribute to.

Anyone can join the meetings. Anyone can join the steering committee. You have to be voted onto the steering committee. Anyone can have an official role in the project.

That sounds like a great opportunity for anyone that wants to get their hands dirty. So how exactly does Spire identify a service? So I can give you a quick walkthrough of how the Spire server works. So first of all, you have a Spire server, which is running centrally, and that contains all your signing keys, both for JWT tokens and for X.

509 certificates. And then there's a Spire agent that's running in each node, each Kubernetes node or each virtual machine or each bare metal box. The Spire server is continuously generating these X. 509 certificates and JWT tokens and then pushing the right one down to the right Spire agent.

Now, when you start up a new Spire agent, it has to authenticate itself to the Spire server. So it has to effectively log into the Spire server and prove its identity in order to get these certificates and JWT tokens. How does it prove its identity? Well, if you're running in the cloud, you have an instance identity document.

So Amazon or Google or Azure all provide an instance identity document that's a signed document. from the cloud provider, you can download that within the node and then provide that to the Spire server. And then it can verify that signature, verify that that Spire agent is allowed to get certain identities, and then pass them down to the Spire agent. Then once you start up individual services, they connect to the Spire agent using a Unix domain socket, which is a local network socket.

The Spire agent validates the identity of the service that's connecting to it. So, for example, if you're supposed to give a certain identity to a service that's running in a particular Kubernetes service account, the Spire agent, first of all, when a connection is opened, it can verify the process ID that's connecting to it, then contact the Kubernetes services and verify that that process ID has a particular service account, and then push the right service identity to the right service. All of this is continuously changing. It's dynamic.

So you can constantly change your policies for granting identities to particular services, and then they're constantly refreshing. So every 30 minutes or hour or two hours, we're regenerating these certificates, regenerating the JWT tokens, and pushing them down to all the services. And that's why you need an API, so it can be continuously pushed instead of having just a file that's deployed alongside the service. That makes perfect sense, Daniel.

Thank you so much. Very impressive identity layer built on top of all of this really great infrastructure that we all have access to when we're using cloud service providers, but it is tricky to use effectively to solve identity problems. But here's a layer that will just piggyback on it and do it for you. Absolutely.

And if you're running everything in AWS on AWS instances, not in Kubernetes, you probably don't need this because you already have those instance identities and you can just use them. But what Spire gives you is an identity layer that works in Kubernetes. It works on AWS instances. It works on GCP instances.

It works on Azure instances. It works on-prem. And those identities are all generated by the same server and they're all compatible with each other. And when you talk about large organizations, of course, they have heterogeneous infrastructure.

They have a lot of stuff going on. And they want it all to be compatible because you want your service that's running in Kubernetes on AWS to be able to talk to your database that's running on an on-prem server. And Spire enables that. I was just going to mention that by abstracting out your service authentication layer away from your cloud service provider, now you can host and run services across different cloud service providers, the same service around across different cloud service providers and not have to worry about reimplementing authentication mechanisms that were tightly coupled to one cloud service provider or another, right?

Exactly. And it's not just multi-cloud because some large companies are multi-cloud, but a lot of smaller companies are able to standardize entirely on one cloud. But still, they have multiple AWS regions and each AWS region is effectively a different cloud. They don't really share anything.

And still, they're going to maybe occasionally have an on-prem resource that they have to talk to. They're going to have some stuff that's in Kubernetes and then some stuff that's not in Kubernetes, like an RDS database or some other managed SaaS service that doesn't run inside Kubernetes and is impossible to run inside Kubernetes. Even if you're all in one cloud, you still need some kind of identity plane for everything to talk to each other. So the cool new thing in the security world is this zero trust stuff.

Is Spire zero trust? Would you consider it zero trust? That is a great question, John. Zero trust is a buzzword.

It's a marketing term. So the initial idea of zero trust was to create networks where there was no perimeter, where every service and every user has its own unique service identity or user identity, and every connection, every request is validated and verified based on that identity. Unfortunately, what's happened is that term has been a little bit subsumed into marketing, and now you'll see it used in all kinds of different ways. So there are VPN companies that claim their technology is zero trust.

There are monitoring companies that monitor all your infrastructure and make sure that only the right services are connecting to other services, and they claim to be implementing zero trust. There's even a company that claims it has zero data security, which was a new term for me, but what that means is they're actually taking vital data like credit card numbers or social security numbers and storing it somewhere else, and they're calling themselves zero trust. So the term, unfortunately, kind of like cloud, kind of like SaaS, kind of like every hype term in technology, has a little bit blown up, and because it's not trademarked, everyone uses it to mean slightly different things.

So as a consequence, I just try not to use the term zero trust. Sure. Perhaps a better. .

. The question would be, do you have to trust your underlining cloud service provider when you're using Spire? That's a great question. And you do have to trust whoever is running the Spire server.

There's no way around that because they have all your private keys. They can generate the identities. But if you want to trust AWS, say, and then you want to delegate a little bit of trust to Google Cloud by running some services on Google Cloud, Spire will help you do that because you can create that Spire agent running on Google Cloud talking to the Spire server on AWS. Or you could even have a federated Spire server with one running on Google Cloud and one running on AWS.

So they don't have to trust each other completely. So it is one step towards not trusting your cloud provider. I'm really looking forward to enclave technologies like NSJL and SGX. And there are a few others out there that let you run code on your cloud service provider and isolate your code from your cloud service provider.

That's sort of a new frontier in security. And I'm watching that closely. Yeah, that's exciting stuff. So how did you get involved in this originally, Daniel?

So I was previously working at a company called Veritas, which makes backup software. It's actually the number one enterprise backup product used around the world by tens of thousands of different companies. And what happened is actually a little bit similar to that Operation Aurora that I was talking about at Google. What happened is a number of zero-day attacks were found against our backup product.

And there were actually things that we already knew about internally, but essentially we had always trusted that companies would deploy our product inside a secure perimeter. So there were ways to convince the backup service to provide files that it shouldn't be able to provide. There were ways to convince the backup server to allow you to restore files you shouldn't be able to restore. But essentially because we trusted that it was always running inside a secure perimeter and these companies knew their users, that that wasn't an issue.

Defense in depth. Exactly. Yes. So what happened is a number of zero-day attacks were publicized by a security research team against our product.

And suddenly all our customers demanded zero-trust security for our backup product. There's that buzzword again, zero-trust. Exactly. So fortunately, the company prioritized this.

And for several years, we pretty much focused on security exclusively at the expense of other features that we could be providing. There were hundreds of people working on different aspects of security for that backup product. As an engineer there, I was a principal engineer in the CTO office. I took on parts of our security strategy for certificate management, certificate distribution, CRLs, establishing mutual TLS connections between services, that sort of thing.

So they deployed maybe about 50 different services at each customer site, depending on the exact configuration. Because it was a legacy product, there were a lot of different communication protocols, a lot of different programming languages, a lot of different layers of technology that dated back probably two decades. So there was a lot of complexity in deploying that zero-trust security for backups specifically. And then once I was wrapping that project up, we actually did complete it.

And we were able to say that that backup product is secure, even if you run it in an untrusted environment now. That's incredible. That's an incredible achievement. It was an enormous amount of work.

That diverse ecosystem of legacy software. It took years of work for hundreds of people. I can't even say how much money, but it must have been hundreds of millions of dollars in engineering costs to redo a lot of the communication in this product. And then we deployed it.

We released an update. Tens of thousands of companies got that update. And it mostly went okay. So it automatically deployed certificates across all their backup nodes.

Some of these companies had hundreds of thousands of backup nodes. It automatically deployed certificates to all of them. And then the communication continued seamlessly. Mandatory mutual TLS with identities that are tied to the specific installation.

It was an enormous project and definitely proud to have worked on that. And then once we were wrapping that up, again, it took several years. So this was actually a pretty big part of my career. We moved on to the other things.

And I was thinking, this technology is really cool. There has to be applications outside just one company, outside just one project. And I started looking around and that's when I learned about Scytale. That's when I learned that there was this huge problem in the cloud-native infrastructure world where you have a bunch of services and they don't have secure identities.

They don't have any way to establish secure trust between each other. And that's when I joined Scytale. I was an early employee. I wasn't one of the founders.

So employee number two or three or something like that. That was probably three or four in there, something like that. But even by the point I had joined, they already had a very solid strategy and a pretty good POC product that we've built on. And then over the next couple of years, just the adoption has been really interesting to watch.

A number of the hyperscalers have really latched onto this technology and have whole teams working on it, which is really cool to see. Nice. Which hyperscalers are using? I don't think I can say that.

Oh, okay. There is a list on our website of conference presentations about Spiffy Inspire, and you can look at the names and see what companies they work for. That would be a good hint about which companies are using Spiffy Inspire. I've been told by our trademark marketing people that I'm not allowed to say the names of specific customers.

And then just about a year ago, just actually the week before the pandemic, we got acquired by HPE, which has been a fascinating journey because HPE has this worldwide marketing and sales presence, and they were able to present this technology to all kinds of companies that are not hyperscalers, that are maybe not in the cloud native ecosystem yet, but are really interested and get them interested.

And then I've been consulting with them, doing all kinds of incremental additions to the product, a lot of Helm charts, a lot of installers, a lot of YAML files, a lot of white papers, a lot of documentation, that kind of thing, to try to really get this adoption in more old-fashioned companies that haven't really adopted cloud native yet. So when you were acquired by HPE, you really were able to take advantage of their sales and their marketing operational readiness. It sounds like you were instrumental in leading a lot of those efforts, your Helm charts and just making it easy to consume. Exactly, exactly.

And the combination of a tiny startup with some really new technology that does a lot of new things, and then a pre-existing company that has that marketing distribution, sales channel, vendor relationships around the world, it's a really powerful combination because the big company has a lot of trouble developing and releasing new products internally. Not because there's any problem with the company, but just inherently at a big company, if you release something, it needs to be nearly perfect. You can't just experiment. But then a small company, a tiny startup, has a lot of challenges around marketing and distribution.

Every time you attend a conference, it costs a lot of money. Every time you fly up to visit a customer, it costs a lot of money. Building that momentum is really hard at a tiny company. Yep.

Actually pretty incredible that your entire experience at Veritas led you to Scytale, which really, it sounds like it laid the foundation for exploring and understanding the nuances of service identity and machine identity at a very low level. It's been a fascinating journey, but what I like most about it is not even necessarily the service identity or Spiffy Inspire or even security related. What I like is that I got to see the entire process from the idea for a product, the prototype for a product, building momentum around the product through a lot of marketing, a lot of conference presentations, a lot of white papers, a lot of discussions.

And then finally, we've gotten to the successful point where people have adopted the product. So I've been able to see it end to end. And of course, this isn't the end to security, but hopefully we'll have more security products in the future that maybe do other things in the zero trust cloud native space. And I'm hoping to repeat that experience where we come up with the idea for something, build something, and then drive it to successful adoption.

I think at the end of the day, a lot of people can relate, would really enjoy being part of something bigger than themselves and bigger than any one company because this problem spans so many different components and people. When I think about successful movements, a lot of them are based on open source technologies and open standards, Spiffy being one of them. Another one that comes to mind that's in a related space is actually OpenID Connect, which really seems to be helping folks solve the authentication problem with end users and logins and authentication between different sites.

And so, you know, when we want to log in with Google to a non-Google site, we're almost certainly using OpenID Connect, or at least I hope we are. Do you think that there's a future where Spiffy and OpenID Connect merge or what's the relationship between these two open standards that you see? Sure, I love talking about this topic, actually, because OpenID Connect is an amazing technology and it meshes really well with Spiffy. So what OpenID Connect is, is it's a standard for creating JWTokens so that one service can authenticate to another service.

So, for example, if you want to log into AWS or if you want to command line to manipulate AWS, you can create an OpenID Connect provider for your company. That will serve up a root certificate bundle, essentially. It's not technically root certificates, but It's similar to root certificates. It will serve that up over a secure HTTP endpoint, then AWS will continuously check that endpoint, download it, and then anyone who tries to log into your AWS account with a JWT token that's signed by that root certificate bundle, that connection will be authenticated against that root certificate bundle.

So you still need the provider. You still need something to create the bundles. You still need something to create the JWT tokens. And Spire can already do that.

Spire already supports JWT tokens, and it already supports serving OpenID Connect endpoints. So that wasn't the original purpose of Spire, but it's a perfect application for Spire, and it's something that a number of users are really interested in because it's a huge problem. How do you securely authenticate to APIs? And by that, I mean public APIs, not internal APIs.

Who hasn't accidentally taken an AWS secret key and accidentally checked that into a public Git repo or even a private Git repo where it's not supposed to be, or it's been on a server that the wrong people have had access to? Those secret access tokens are a huge problem. They're probably the biggest barrier to cloud adoption.

So by using something like Spiffy and Spire to generate identities continuously and then serve them up via OpenID Connect, and then because we have that node attestation and workload attestation, those identities can't easily be stolen because any service that's using that identity has to have passed the workload attestation step and have passed the node attestation step. So you know it isn't just using a secret access token that maybe was accidentally checked into a public Git repo or stolen off some other server that wasn't supposed to have it. Awesome. So maybe we even will be entering a world here pretty soon where secret management is a thing of the past.

I'm certainly hoping that we just have no more secrets. Actually, one of our competitors in this space is called Secretless, which I think is such a great name. And unfortunately, we can't use it because they create market. But it's a good name.

And I wish I could use that term because what we are trying to do here is get rid of static secrets. If you have one less secret and no secrets at all, there's nothing to steal for an adversary that breaks into a cloud system, given a zero day or whatever. Security and depth is always a good thing, but it's only as good as the weakest link in each of the chains. And what we're doing here is we're helping cloud adoption because one of the biggest barriers to cloud adoption is that secret step where you have these access tokens that have to be managed securely for each cloud, for each region.

And a lot of companies just aren't capable of doing that in a secure way. And they can use secret managers, but then someone has to set up the secret manager and manage the secret manager and then grant access to the secret manager. And that's been holding back a lot of companies, especially in regulated industries and security conscious industries and federal government from adopting cloud infrastructure because they just can't manage the secrets. It's difficult to manage secrets because they're so sensitive and HSMs are painful to set up if you need to go there.

And if there's no secret at all, that's even better, right? There's nothing then to steal. Get rid of your secrets. That's my motto.

Just get rid of the secrets. It's like nuclear weapons. No nukes are good nukes and no secrets are good secrets. Yeah.

Or you could even say that secrets are like toxic waste. They just contaminate any infrastructure they touch. That's so true. And you have to bury them deep underground for no one to find or accidentally bump into.

Yep. So everything that we've talked about so far has been a centralized technology, but a lot of the trends, especially with all of these cryptocurrencies running around trying to incentivize certain decentralized architectures, it's really starting to focus on decentralized technology. I'm not sure how you see the future in terms of how this, something like Spiffy or Spire could work in a decentralized mode, but do you see any overlap between these two different worlds of Spiffy and Spire and the decentralized approach to service authentication? Absolutely.

And this is a question I get all the time. Can we use blockchain? Can we somehow use a distributed ledger as we're generating these secrets in order to validate the secrets across the distributed infrastructure? Right now, that's extremely challenging.

And it's challenging for one reason, which is we're generating a lot of secrets here. Again, we have customers who have thousands or tens of thousands of nodes continuously generating secrets. There is no distributed ledger technology quite yet that can keep up with that in a super reliable way, because remember, if it fails to update the ledger or it fails to generate a secret and be able to save it, secret, then your communication starts to fail. So it does need to be really, really secure.

There are some companies working in this space, but they have a long way to go in order to achieve a distributed ledger that's operating at the performance and reliability that you need for service identity. There is one baby step towards decentralized service identity, and this is the same thing that's used on the public internet, which is certificate transparency. So as these are generated, they can be added very efficiently to something called a Merkle tree, which is a hashed ledger of all the secrets that have been generated. And then each secret contains a pointer into that tree.

So then if you have a couple of different servers hosting the certificate transparency, it's called a certificate transparency log, and you have a couple of clients that are validating these, then you achieve a little bit of a decentralized service identity. And this is used on the public internet already. Any certificate you get for any website is going into a certificate transparency log. So it does have the performance and reliability that we need.

We're working on implementing that for Spiffy. It isn't really decentralized because you still have to trust these CT logs, but at least you're trusting that at least one CT log hasn't been compromised instead of only trusting one source. So we do think a lot about decentralized service identity. There's a huge challenge to really making it a consensus distributed identity in the same way that like Bitcoin is a consensus-based distributed identity.

Right. Well, I mean, you were talking about the technical challenge of building a blockchain that has the capacity to issue the X. 509 certificates or the job certificates at a rate that would satisfy your demand on the network. But when I think about decentralized technologies in terms of a B2C space, like, okay, sure.

But in a B2B space, then you have to sell that to large customers or to enterprises, and you have to meet SLAs. Maybe it's the same problem, but it turns into a sales and marketing problem. How do you ensure to these companies that they have control over their networks? Exactly.

Exactly. And we're operating in a very conservative space. We're operating in enterprise data center infrastructure. So back when I used to work on backup software, we would release a backup product.

It might take a year and a half or two years for a company just to test the product before releasing it in their enterprise. That's normal in that space. That's completely normal. Because if you install a product on your bank's infrastructure and the bank goes down even for 10 minutes, that's a huge problem that could cost hundreds of millions or even billions of dollars.

So they are extremely conservative, and we have to meet those demands. Our code has to be essentially perfect. We're operating in the red zone. If we fail to issue an identity that should be issued and a service crashes, that could bring down your bank.

That could bring down your hospital. So the opportunities for doing something like decentralized service identity, they're there, but you have to figure out how to do that and not only make it extremely reliable, but make it reliable enough for enterprise infrastructure companies to adopt. Exactly. Especially if it's going to involve other people's computers and you're going to go to a bank or to a hospital and say, your systems will now be dependent on thousands and thousands of other people's machines.

Now, there are some interesting companies operating in this space. I was just thinking that Helium IoT is doing decentralized identity for IoT devices. There may be some others. I'm sure I've heard of others.

But again, they're mostly in a consumer space where if it fails, it may be bad, but it's not going to crash the airplane. You're not losing billions and billions of dollars for an outage. Exactly. Right.

Exactly. So we do need to be very conservative about how futuristic we can be with these technologies. But regardless, in terms of just the technology itself, it's a very interesting question, right? How do you approach that?

Yeah. And I do think there are a lot of opportunities in this space. We're still in the early days for service identity. We still have a lot to figure out, and there's a lot of opportunities to try something new.

And speaking of the opportunities, what do you see as the future for the service identity space or security space in general? I would say we're still in the early days of computer security in general. If this were a time in aerospace engineering, we'd be in the 1950s when there were jetliners, but there were also random planes crashing all the time. There were some planes where the windows would just fall out and people died all the time, unfortunately.

We're in those days with computer security where there are some organizations that are extremely advanced, that are flying that 737 that is exactly like the one you'd see today. And then there are other organizations that unfortunately are crashing the plane constantly. So that's where we are. And we clearly have a very long way to go in this realm of establishing security with corporations.

That's true. I couldn't agree more. You know, the global economy lost $1. 2 trillion, $1.

2 trillion just in the First quarter of 2020 to data breaches. It's an astronomical number. Those airplane crashes, yeah, we see them. I think that the most promising directions to take the entire world of service identity and security between services is really in the area of availability and usability of the technology.

If you look at Let's Encrypt, that wasn't inventing anything new. That's just creating certificates. But it made creating certificates so much easier to do, and of course free instead of costing a few hundred dollars, and instant instead of taking a few weeks. Everyone adopted certificates once Let's Encrypt became available.

Again, it wasn't inventing too much new technology. They did have to create a new protocol called ACME for obtaining these certificates automatically. But the primary innovation of Let's Encrypt was usability and availability of the service and cost. The actual security technology, the encryption technologies, the identity technology exists today that can solve these world problems.

We just need to put them together in a way that's easy to consume, easy to understand, and easy to use. And that's certainly the direction that I'm interested in working on. Spiffy and Spire already exist. They are free, so we're meeting that cost criteria.

They are hard to set up. There's a big cognitive load to figuring out how you're going to attest your nodes, how you're going to attest your workloads, how those workloads are going to use the workload API. There's just a lot of effort involved in setting up this technology. And you can pay my company for consulting services, and we'd be very happy to provide consulting services, but the real frontier is going to be in making this automatic and easy to use and instant and also free.

Just like Let's Encrypt was phenomenally successful distributing certificates to every website in the world, essentially. There's a huge frontier in providing service identity to every service in the world. So are you working on anything to help with the setup overhead, the cognitive overhead? Is there a simplified GUI?

Is there even a way to remove it? Some of it may be. Yeah, I mean, that's the great question. And we are working on a number of things.

We actually do have a GUI for, we actually may have two GUIs for Spire right now in different states of development. I don't think a GUI is the answer. I mean, if you look at what happened with Let's Encrypt, again, phenomenally successful security product. It wasn't successful because it had a GUI.

It was successful because it involved very little effort. And it was doing something that was previously difficult and doing it really well. A company in the space that I would definitely watch is called Smallstep. And what they are doing is literally taking the Let's Encrypt approach to service identity and doing it inside companies.

And they've really focused on usability. I really admire that company in particular. Maybe I shouldn't say that because they are a competitor. They're really focusing on small and medium business, while Spiffy and Spire are focusing on large business.

And they take some different approaches on some of the security questions. But I would definitely look at their general approach to usability and documentation and ease of configuration, because I think they're really innovative on those fronts. Just a shout out to a friendly competitor in this space. A friendly competitor.

I mean, if you're doing good things, you're doing good things, right? And just as a side note, Smallstep has the best content marketing of any security company I've ever seen. Their blog posts are incredible, and they're often near the top of Hacker News, near the top of Lobsters, all over Twitter. They're actually amazing at getting the word out about their technology.

Another one to admire. Yes, yes, which is very difficult to get right. It makes me wonder if they have a podcast. I actually don't know.

Maybe they can be your next guest. Perhaps. Not to pit the competitors against each other, but. .

. As I said, they're our friendly competitors. They're mostly focusing on small companies, while Spiffy and Spire are focusing on big companies. Different market segment.

There you go. They're not competitors anymore. All right, Daniel. Well, do you have any final thoughts to share with the audience?

So thank you so much for having me on, John. This has been an amazing experience being on this podcast and being able to talk about Spiffy and Spire and having so much recognition for the little bit of work I did over the last couple of years building out these products along with a big team of brilliant people at many different companies. I'm really grateful for the experience, and I'm looking forward to the future of developing cloud-native and security products that really change the way that companies relate to their security infrastructure.

So Spiffy and Spire are fundamentally changing the way that large companies distribute service identity to their services, and they're changing what's possible because now you can communicate between multiple clouds. You can communicate between cloud and on-prem services, and you can do it in a completely secure manner with uniformly generated certificates. kits and JWTokens that every service within your enterprise can use, and that vastly expands the horizons of what you can do safely within cloud and within your company. And I'm looking forward to doing more of that, to creating more things in the future that fundamentally change what's possible in the security realm.

If you have any ideas, don't hesitate to get in touch with me. I'm sure John will put my email and contact information in the podcast notes. And I really look forward to working in this space in the future and keeping in touch. If you'd like to chat with me, I'm always on the Spiffy Slack channel, and it's an open Slack channel.

You can go to spiffy. io and sign up for the Slack channel. And I'm always answering questions on there, along with many other people who are from a number of different companies and are mostly smarter than I am about how to architect zero trust security for cloud native architectures. Again, thank you so much, John.

Daniel, it was an absolute pleasure to have you on the show. Thank you deeply and sharing your insights, your experiences, your thoughts, because this is a podcast, just so everyone knows how to spell Spiffy, it's S-P-I-F-F-E. And we'll stay in touch. I'm excited to learn what you'll be building next, and we'll be in touch, huh?

Great. Keep in touch, John. And if you're attending KubeCon in the future, either right now at KubeCon Europe or in the fall at KubeCon US, which will hopefully be the first on-prem conference that many of us have attended in a few years. On-prem and in-person.

On-prem, on-prem conferences. Yes, instead of cloud conferences. It'll be in Los Angeles in the fall, and I'd be delighted to grab a beer and chat about cloud native security. All right, well, perhaps we'll have some folks take you up on that.

All right. All right, Daniel. Thank you so much again.