38. Nick Sullivan on Cryptography, Cloudflare, and Building a Better Internet

Hello, everyone, and welcome to another episode of the Security Podcast in Silicon Valley. I'm here today with an awesome guest and a good friend, Nick Sullivan. Hi, John. How's it going, Nick?

It's going well. I'm really happy to be here. Thanks for having me on and excited to chat with you. Just for our listeners, Nick has built an entire career around security and cryptography.

Over the many years, you studied at Waterloo. Is that a focus on cryptography and security? I studied pure mathematics, actually. So it has a connection to security and cryptography, but it is much more abstract and much more theory-focused.

Well, you got to get those foundations solid before you can get into the bit smashing, right? True. Yep. And so when you entered the industry, it looks like you were a cryptography researcher for a while, and then you jumped into Symantec as a security analyst, and then you were at Apple as a mathematician.

I love the title. We worked together. Just for all of our listeners' context, we were both on the DRM team. Nick worked a little bit more on the cryptography, the actual business logic, and I was over in the harder-to-talk-about coffee-skating compiler team, but we worked very closely together.

Those were the glory days, weren't they, Nick? Yeah. Yeah, that was fun. Apple media empire is, you know, built on the foundations of having secure ways of enforcing business logic.

And that's what DRM is, I know. That's what DRM is, yeah. There's a very strong, like, political opinions out there, I don't know if it's good or if it's evil. Maybe it's both, but we both worked on it.

Without even really having to dive into the politics behind it, the technical challenges were very open and interesting. But Nick had the good sense to get out of there and join a young, budding, promising cloud flair, which you have spent more than 10 years there now. And you started off as head of security engineering, and very quickly that turned into head of cryptography. You facilitated a lot of really great public talks, which I actually participated in, which was amazing.

And you transformed into the head of research there at Cloudflare. And you are still associated with Cloudflare adds one of their advisors, which is spectacular. And now you have a cryptography consulting LLC, along with many advisor roles to lots of promising companies. Yeah, yeah.

It's, you know, Cloudflare was quite an amazing adventure. I joined when it was a small company, under 50 people or so. And what Cloudflare does is it basically sits as a proxy between people and websites or web resources. And just that general generic framework makes it incredibly powerful.

So you can do things like security monitoring. You can use web application firewalls. You can stop denial of service. You can do all sorts of things if you're the middle party between the user and the website.

And as we know, the web is, you know, constantly, constantly under attack. Constantly attackers are finding new ways to exploit technical loopholes to try to take advantage of companies, take advantage of people, stealing identities, stealing money. You know, the world is a wild and crazy place. And having some sort of security blanket over your website and web services is really helpful.

And so, you know, when I started there, I was doing some engineering, some pretty hardcore engineering, solving some of the, some of the bigger, you know, security architecture things that were, that were missing. So I built Cloudflare's public key infrastructure and, you know, built a small team to open source tool called CFSSL for creating certificates and managing all sorts of internal trust things. And, and yeah, it really helped what was a fledgling company at the time take off. And so it's, it's been an amazing journey there for, for 10 years.

And, you know, I can go a little bit more into some of the different things that I touched there, but it was, it was an amazing opportunity to be able to take the ideas of security and cryptography and privacy and, and all these wonderful things that the tools that the technical tools that we build and work on create and bring it to a wide audience. And not only bringing it to a wide audience as part of a product, but I spent a lot of time working on the industry challenges themselves directly. So I worked with standards bodies. I've worked with academia on validating new ideas, built prototypes.

I worked with all the folks at Cloudflare to kind of bring these exciting new ideas that were nascent in the industry and explored in academia, but not to from like it's academic rigor, but not necessarily from a technical scaling perspective, able to be deployed and kind of help bridge that gap. And that started with, you know, my, my own personal history and focuses is on cryptography things. And that's, you know, where the head of cryptography role is. We participated in TLS 1.

3, invented stuff like privacy pass. And, you know, that model of taking less of a product focused approach and more of a research long-term technology approach that is based on collaboration across the industry and not competition through venues like the IETF, through conferences, through working with, with academics. And, you know, we built up this amazing internship program with probably 40 interns at least going through the program over the last couple of years. And, and, and this made a lot of sense to go beyond security and cryptography.

And so we expanded the team from research into like cryptography and protocols to broader research. So we brought in things like DNS, network measurement, distributed systems, privacy enhancing technologies. And, and yeah, I still consult with Cloudflare and, and the team is still going strong, putting up some, some amazing stuff. We recently published a paper from my work there studying the, basically the effect of, it could be called censorship, but it's more, it's more broadly, if you were to categorize as tampering as different participants on the network, stopping different traffic from getting through in various parts of the world.

And, and we had some pretty interesting insights there where, you know, almost 10% of the traffic that's, or the requests that's, that were intended for Cloudflare, you know, millions and millions of customers. It makes up almost 20% of the web at this point, 10% of those, at least we're getting locked. One reason, one reason or the other in global. And it's a lot higher in certain countries.

And so there's a lot of different reasons you could imagine that someone would want to stop a connection. There's, you know, corporate security, there's lots of monitoring protection there so that, you know, employees don't go to phishing sites and, and, and this sort of thing. But there's also geopolitical reasons. And there's, there's things like schools, like schools that have content filtering mechanisms that come into play.

So being able to be at Cloudflare, which is central to, you know, a big percentage of the web gave us amazing data. You really have some interesting data, right? Yeah. You have just tons of data, I imagine.

So were there any patterns that really stuck out that struck you as unexpected? Patterns with respect to like product development or, or with like what has worked and what didn't work? Oh, I mean the, the censorship. Oh, and the censorship.

How could you detect that there was something being censored, like within a school, you know, that's tricky to be able to detect, right? Because if the packets are not making it to Cloudflare's CDN or, or connection points, like. Yeah, no, it's a good question. It's very counterintuitive that one would be able to learn this from only having the server side of the connection.

And historically up until, to this point, a lot of the measurements with respect to the censorship and with respect to connectivity and availability of, of content online has been active. Meaning that companies and groups like UNI and these other measurement measurement groups who try to, to measure these things are putting devices in people's homes around the world or like letting people opt in. And then the measurements would come to like a preset list of names and just to see whether or not these ones were, were available or not. And, and, and with our study, um, so the folks at the university of Maryland came up with this brilliant machine learning algorithm to analyze TCP flows.

And what happens is that there's a number of different signatures that you can see on the server side where one packet will get through and it'll be indicative of the entire flow being blocked or delayed or, you know, whatever. Like from a, from a very high level for those like who understand TCP, if you get an inbound SIN, then you can just send a, you can just send a FIN if you're in between, right? There's no authentication on any of these TCP things. So, so someone will send a request.

It'll have a website name in what's called the subject name indication. It's part, part of the, the raw records on, on the wire is like, I want to go to example. com and it's in the handshake. You can see it having a, like a network monitoring device.

And if you match it against, you know, no, no, no words, you can send a, send a reset packet, or you can do all sorts of different things just to confuse the state machines on either side. And so because like the first request, maybe not that, I mean, maybe the handshake will get through or, and that's how we can determine which sites are being tampered with is that the website handshake goes through. It says, I'm going to this site. And then you get a random, you know, reset or a different package packet that just tells the TCP connection to shut down.

And so that's, that's a strong indicator that something's going on and there's, there's a lot of signatures that are pretty high confidence and correlated with particular products and devices that people sell that do this sort of thing. Yeah. And so, yeah, it's the, the paper was just presented at SIGCOM, a huge ACM conference. And yeah, that's just one of, you know, a million different things you can do with this amazing global dataset.

And I always say that's one of the superpowers of Cloudflare is, is having a free customer base, having the ability for, you know, millions and millions of customers to just join on and use the service. And, and, you know, they get a lot of value out of that. And, you know, there is cost at Cloudflare, but Cloudflare gets a lot of value in terms of being able to see what threats are going on on the internet. So it's kind of like an immune system, if you will.

You see some sort of threat against a free customer and then you can study it, apply mitigation and then protect, you know, the very large enterprise customers. It's a really good symbiotic relationship. So there's value being added on both sides, unlike Google where they're using the data to sell something. Yeah.

And, and some of the, some of the work that I did most recently at Cloudflare was only facilitated by the fact that the company is not based on monetizing individuals or trying to track people and sell information, all of these sort of things. That the business is based on, let's just make the network work the way that it should. And so in, in doing that, it enabled us to do a bunch of projects. Actually, one was with our former employer at Apple Cloudflare participated in iCloud private relay, which is a privacy proxy that leverages CDNs and network providers like Cloudflare and Akamai and Fastly to provide a layer of anonymity for the client IP address.

And it's native on iOS now, and that, that kind of happened through five years from when we, when we came out with privacy pass and published it. A lot of behind the scenes conversations, convincing people of, you know, the benefit of secure DNS and, and, and privacy and all these sort of things where Apple and Cloudflare were very aligned. Neither company is trying to monetize individuals. Right.

So they can build these more robust privacy primitives and privacy protocols to help improve user privacy. And Cloudflare was happy to do that. And, and that's, that's what I would consider this, the kind of wins that you have to expect when you have a research team and also the timelines that you have to be comfortable with as an executive team funding something like a research team. Right.

Right. This is a five year process. And actually most of the successful projects at Cloudflare took, you know, three to five years to become fully robust. Did you find that that was a difficult sell when you had to interact with the executive teams and, and catch these projects and the potential payout from a business sense for Cloudflare, but also in terms of like, let's just make the internet better and more secure and more private.

Yeah. Yeah. The, the, the mission statement was very much in support of doing these sort of things. Cloudflare's mission statement is to help build a better internet.

And so better means a lot of different things. It can mean you're more available. It can also mean more private. Um, and so.

You know, Cloudflare was fantastic in supporting myself and the research team in doing these long-term things in pursuit of the mission. Rather than in pursuit directly of the product. And, you know, research teams are a cost center and you have to be aware of that when, when building a company other, you know, whenever you're growing as fast as Cloudflare has. For the last 10 years, consistently growing significantly year over year.

There's always a million types of tensions that have to come into play. Healthy tensions. Yeah. That's healthy.

How things are funded, how things are prioritized and how people communicate, especially as you know, you go from a hundred people to 500 people to 2000 people. It's a very different systems that have to be in place, but, but given the strong support from leadership that, you know, Cloudflare was, is serious about the mission of helping build a better internet. And, uh, uh, uh, these long-term projects, although like lean as they were, we're very successful. So that was really fun.

And I think it's, it's, it's. It's something that not a lot of companies are comfortable taking a big swing at is long-term research because. Just how precarious the situation is with, with raising revenue and, you know, becoming cashflow positive and all these sorts of things that businesses have to deal with. Making sure that growth continues at.

A, a rate that's. If it's not exponential, that it's, it's, it's, it's repeatable and in line with the investor's goals. But, but I think we showed that. Research can, can pay off even in a smaller company, even in a growing company and took the model.

Took some, some lessons from different models at research groups at places like Google or Microsoft, or even going way back to the Bell Labs days. And yeah. And yeah, it's, it's a different way of operating than you will at a product focus company where it's very, you know, you're, you have very short deadlines and sprints and everything has to come together in terms of schedules and, and having an aspect of your company that doesn't have. Clear short term deadlines.

Meaning like within even, you know, six months, there may not be guaranteed deliverables or outcomes. It, it relies it, well, it forces you to rely more on, on, on relationships, relationship building within the company and communication. Making sure that people know the value that research brings and will bring and want to work with you. And, and so like one of the, the slogans that I came up with for the research team was we make friends.

And there's, there's absolutely no way to, to do some of the more difficult long-term things that take away from other priorities that come with research without, you know, without human support. And without people, of course, of course, I feel the warmth and like the, the Canadian, like welcoming spirit. When you, when you said that to me, it's just like, and being a Minnesota, it's well received. Yeah.

I like that. You know, we make friends. I've always mentioned that there's two types of security people in the world. There's one type of security person, you walk into a meeting room and, and everyone looks at you and they go, oh no, like so-and-so is here.

They're going to tell us we have to implement all these strange standards and they're going to push the deadline back by, you know, a couple of months. And like, oh, it's just going to be roadblock after roadblock. And, and then there's another type of security person. They walk into exactly that same meeting room.

They have exactly the same expectations of what they want to see in the product, but the other people in that meeting, they see this person walk in the room and they're like, oh, I'm so glad like Nick is here. He's going to help us understand how to build this new thing in a way that aligns with our security goals and our customer expectations. And just make the world a better place for everyone and maybe more private, more secure. And Nick is the guy who's going to help us like understand and navigate that complex space as an ally.

I'd like to think that, that I try to embody that second person, like perfectly and perfectly. And I, and like, even your slogan of like, we make friends. Yes. I, I, I feel that.

Yeah. Well, thank you. Yeah. I, I really don't think that it's, it's possible to be very successful in a company.

If you're, if you're, if people's expectations coming into the meeting is like, I don't want to see that person. They're going to cause trouble for me. So you've lost already, like at that point. Yeah, of course.

And so, you know, my approach is, is very collaborative, very inclusive, very helpful. Like I encourage everyone on my team to, you know, as much as possible, if people are looking for help or, or you have some expertise that you can provide. Help people out. I mean, it's going to help the company and it's going to help the position of the goals of the group and, you know, and there's specialized knowledge within research groups or security groups where some people have years of experience and understand things in the ways that, you know, a typical engineer will just not know at all.

And we'll make very easy to find mistakes or, or go in directions where just, you know, a five minute conversation with somebody who, who's an expert may help turn them in different directions. And if you're not a resource for your company, then, you know, you're potentially a burden, right? Especially if, if the goals of, of your group and you, and the people that you're working with are, you know. Are perceived as being like against the goals of the company, right?

Right. Yeah, that's right. Just dangerous place to be. Yeah.

Especially in a contracting market, I would say. But maybe that has nothing to do with like how a company might contract its like resources as well. Like, I'm sure like both, both types of security folks are, you know, considered when you just have to contract it, the size of a company with layoff server or whatnot. Seems to be going on a little bit more often these days.

Yeah. Yeah. Yeah. I mean, efficiency is super important.

And if you can do more with less than you're ever going to want that. Yeah, for sure. For sure. So is there any story about how you got into security?

I know you, you went down the mathematics road and cryptography maybe piqued your interest and it was a little bit more applied mathematics when you got into the cryptography space. But was there a moment or like something hit your passion button that tipped you into that security space? You know, I wish I wish I could identify a specific moment, but I don't think there was one. It was kind of a series of events and doors that opened up.

Um, you know, in my last semester at Waterloo in my pure math degree, I'd done pure math, combinatorics, all this very, very pen and paper, deep abstract stuff. And, and then the Alfred Menezes, who was the, one of the cryptography profs at Waterloo actually invited me back to give a guest lecture a couple of years ago, which was super nice. And I taught this intro cryptography class and everything just clicked for me and that like, I was able to, you know, take my deep hard learned, you know, intuition around prime numbers and discrete mathematics and all those sort of things and put it into a concrete application.

And cryptography to me is, it's like magic, like it's allows you to provide things, uh, digitally that would, could only be reproduced physically. So, and then the intuition is really weird. So, so it's like, you can take data and make it so that no one in the world can see it. Right.

Right. And right. Yes. Like, okay, well, we can physically do that by putting it in like an unbreakable safe, but you can do that digitally.

And some of the, the, even the very basic things like digital signatures, they, they enable these capabilities that really help with designing the architecture of communication systems. And so after that, I decided to do a master's degree in cryptography and, you know, expanded from math to doing computer science. Mm-hmm. .

. . And security, like, like just the, the entry level into like, okay, you know, there's a, there's a way that you think software works and then there's a way that software really works. Yes.

And these two things really tied together really nicely. And so once I finished my master's degree in elliptic curve cryptography, I, you know, started getting into some implementation and then also studying things like threats on the internet. And Symantec had this great role for security analysts. They have a annual report called the internet security threat report.

And, yeah, they hired me as, as one of the coauthors of this. And this was, this kind of opened the door to like massive data problems and data as a tool for explaining the world of, of security. And so, you know, you have this cryptography angle, which solves security problems by unlocking these like magical primitives. And then there's the, just the, the fact that every single person's not every single person, but like the majority of the world is online and living their lives online and building things online.

And there are security threats to this and nothing was built perfectly. Everybody's taking shortcuts when one way or the other, we're all just trying to make it all happen. And so that was just such a fascinating field for me. And, you know, combining those two was.

The perfect fit. Yeah. Perfect fit for Nick. That's amazing.

Yeah. That's really cool that you came in from the mathematical site. Did you ever, did you ever, here's a crazy question for you. Did you ever just like sit down at a coffee shop and be like, well, let's see if I can figure out a different way to think about how to factor the product of two large prime numbers.

Or the discrete algorithm, like efficiently over a modulo prime or. Yeah. Like that. I mean, I definitely, definitely in the early days before I knew the theory played around with, with things like this.

Right. Like what sort of algorithms could you do? Uh, but you know, once I learned about the quadratic SIV and the general number field SIV, which are the most efficient ways of solving. Solving factorization over a certain size.

Yeah. It was like, Oh, this is really deep. Like this is like took all of my undergrad pure mathematics and algebra classes to like really understand what's happening. And so I think my creative juices in terms of being a cryptanalyst, it, it, it, it was a little, a little too, it involves like some real creativity and deep, deep knowledge of this, that, you know, I wasn't in a position to spend my life.

Yeah. Just let go of that and like go over here and do this, uh, industry stuff for a little bit. Yeah. Well, once it was clear that I could do something like at Apple, like, okay, let's all the content in the world with encryption and do interesting things for authentication and for device attestation and for obfuscation or reverse engineering.

Like, it's just such a tangible, physical landscape to me, at least like when I, when I think of software development, I, I mean, I, I think of it as like a physical, a physical thing. So it becomes something that you can explore like a landscape. And so when, when exploring these sorts of things at Apple and then, you know, at Cloudflare is like, okay, I can upgrade the cryptography for 20% of the internet. And it's, it's this amazing impact and, and, you know, people, the academics and cryptography are amazing.

They are so brilliant and they spend their whole careers on things. Like there hasn't been a new factoring algorithm for 20 years. Right. Um, and so.

I don't know. I'm not the kind of sit in a room for seven years and solve the hardest problem in the world type of person. I'm very much a let's work with people to get, get, get a better understanding. All of these amazingly smart people together and like, let's see what happens when the rubber hits the road.

And then once you do figure things out or there's new concepts explaining it, you know, I'm a really big fan of communication. One of the things that I emphasized in my job at Cloudflare is, is blogging. So I, I wrote, you know, dozens of technical blog posts, just deep dives into this, these ideas so that. You know, something that engineers tend to do, which is.

I guess this is, this is the true end. This is true for everyone. And for any discovery and realization of new, new knowledge is that once you learn it, it becomes more second nature. So it becomes harder to teach people how to learn what you learn.

And so taking the time to break down something that you recently learned that's new and innovative and novel in a way that's either a general audience or technical audience can follow. It not only reinforces your understanding of where this concept can be applied in other places, but it brings people in and brings people into the industry. And that's, that's something I'm very passionate about. I've mentored high school students who then became interns and developers and I, and I have spoken at universities.

I've given guest lectures at classes and all these sorts of things. I really think that this stuff is really fascinating to me and a lot of people will find it fascinating, but it's also. Critically important for our society going forward to have folks working in security who understand these things and. You know, can help.

Build up this society and this technical infrastructure to, to be better. Build up this society and this, you know, it's not enough. I think that that's right. It's not enough.

I think there's, there are a lot of interesting problems. And like, as you mentioned, like one of the great. I don't know, responsibilities or it's the right word. Privilegiance of being a leader is that you get to share that.

You get to pass it on. You get to communicate that. And it's, it's part of being a leader. You get to a point in your career, you're giving back to the community much more than you get.

You're consuming from it. You're becoming a leader. You like you're right there at the tip of the sphere. And to be able to mentor and then watch other people grow and realize like all of their potential.

And then I continue learning and growing. It's a great privilege of being, you know, in a leadership position. Absolutely. Absolutely.

It, it brings a different type of contribution back to the community, you know, not just technical. Right. It's. And at the end of the day, like all of this is about people.

If we're not focused on how we can have an impact on people's lives and improve people's lives. It's like, I always try to take that as a red flag and like step back and, and ask like, okay, how is this actually going to impact people? And does this matter? Right.

And if it does, and you can connect it back to people in building a better world, because we all see like there's issues with the world. Everyone sees issues with the world. And yeah, it's important to call those out. But I think it's even more important, especially for people like that can do something about it, to do something about it.

Like to be the change that you want to see in the world, to help nudge the world in that direction. And depending on where you are in life and your career and the community, the communities that you're part of, like you can take on different roles to do that. It sounds like you've just sort of implicitly like pick that up and really value that as well. So that's special.

And, and thank you. Thank you. Not everyone picks that up quite as quickly or integrates that as, as a core like importance, but I've always tried to as well. Yeah.

Well, I know that John from your history and you're definitely someone who, uh, who does what they can to help folks and, uh, and, you know, see the bigger picture. So I really appreciate that about you. And, and, you know, there's a lot of ways of giving back. I'm, I'm still, you know, although I've moved on from full-time at Cloudflare to being an advisor, I'm still involved in a lot of volunteer efforts to help improve the internet.

So I'm the co-chair of a couple committees at the IETF, including the Crypto Forum and Messaging Layer Security, which is a new protocol for end-to-end secure group messaging. Is that what Signal is based on? Yeah. Yeah.

So Signal is, was sort of the OG, right? It was the original, like we, we are able to do end-to-end secure encryption between two people and they were able to scale it up to larger groups through, I guess, essentially a pairwise handshake. So it doesn't scale very well. Every participant of the, of the, of the communication has to, you know, chat with everyone else.

Do this handshake thing. Yeah. You do this handshake thing, but it had some really nice, Signal introduced some really nice characteristics, like forward security, post-compromised security. So like if a key's compromised in the past or present or it's currently compromised, there's like ways to recover, which is, which has been fantastic.

And so MLS Messaging Layer Security's goal was to build on these really strong primitives and security characteristics that something like Signal will bring in the two-party case, but to bring it to larger parties, a hundred, thousand, ten thousand, in a way that the key agreement can scale with the size of the group. And so we recently published an RFC on this and it's used by WebEx already for end-to-end secure communication. And that's something that I'm passionate about as well is standards in a way. I'm not finding a, finding a way to have like a very polished and fully thought out version of a tool to solve a particular problem.

Because we all run into similar problems in our building companies, building products. We do. Like we, we all bump into the same problems over and over again. And, and people have built the same solutions for the same problems in multiple different places.

And it's often. Guilty as charged. Yeah. I mean, I've, I've done that as well in, in, in certain times in my history, but if you want things to be actually global and interoperable, it's, it's great to concentrate the, the efforts of like some experts to come up with a final document.

And so I'm, I'm really big and on, on that type of community collaborative process or coming up with like standards and final documents there. I also spend some time helping with security conferences. So through my time at Cloudflare, I've managed to work with a number of amazing people, co-authored for a dozen papers about, you know, interesting topics. And, you know, even though I don't have a PhD, I still was invited to like help review some of these high-end conferences and security.

And that keeps me fresh. Like I really have to do the background reading to understand some, some of the latest papers that have been coming out. But I do enjoy taking what I've learned and my expertise and, and sharing it with in ways that, that, that help help improve the product of, you know, papers and standards. And, you know, I, I, I don't like to be pushy and take my design philosophy or particular like way of doing things.

I think every, every person has their own quirks and we've all got our things. Yeah. So I, I've been more, I've been more leaning into, you know, enabling the group or people to participate in, in, in a group and, and me being less of, less of a final reviewer, but more of like sanity check and making sure that the process goes on and the people are involved in all these sorts of things. So I think all that's super important and, and, you know, as much as technical work is fun and it's great to just do it.

Like you can scale your impact by enabling others and sharing knowledge. And, and, you know, I think that's just something that's come very natural to me over, well, I guess over my last 10 years of being in the public, whereas, you know, at Apple, we weren't allowed to be that public. So, and you know, like when I, when I was on my way out of, of Apple, I looked internally to Apple to see like what sort of security opportunities there were. And I, I talked to that global security team and I talked with like ex CIA, FBI, NSA people that were like, oh yeah, here's what we do.

And I was like, oh. There's, I mean, there's different, different types of security, right? There are different types of security. Yes.

They take corporate security very seriously over there at Apple. It's a very, well, we've worked in a secretive group inside of a secretive company that just was so secret. Yeah. It's really hard to do technology the way that I've learned has been effective at a place where you're in a secret group within a secret board, because to me, communication is so important.

At the end of the day, I think another way to say it is that security based on like keeping what you do a secret is not secure. You increase the cost of the attack, but you're cutting out other people from being able to contribute and move the needle like from the bigger picture. And you're not participating in like a global community. And sure, it might be a short term temporary solution to a very specific business problem or challenge that you face.

But in the grand scheme of things, it could be totally misaligned with your, your values. And that's what I feel like we, we both shared that, like, and are talking about Apple. I recall a moment in which, you know, the power of the crowd became very, very obvious to me versus the power of like a red team or like an internal group reviewing things. And this is when the heart bleed vulnerability came out, which was, I think, nine years ago now.

And maybe 10 years ago. In any case, this was a bug in very popular cryptography library, OpenSSL, which was in almost every web server of like 80% of web servers in the world. And it had this really silly bug that was written by a grad student that allowed an attacker to send a single packet and then grab memory contents. Wild.

This is like absolutely wild type of type of attack. And especially since it was enabled on all web servers. And so, you know, when I was at Cloudflare, we were internally debating like, you know, we, we passed it ahead of time. But we're internally debating like if someone knew about this earlier, could they have gotten the private keys for all of these websites and all these customers?

And it wasn't really clear. We tried to attack it ourselves. We spent, you know, like one day doing some experiments. And in the end, it was like, this is a lot of work.

There's a, you know, this is a top news story in the world. Someone's got to have a better way to do this. So what we did is we came up with this thing called the Cloudflare Heartbleed Challenge. And I remember the challenge.

Yes. Yeah. We set up a vulnerable website and said, okay, world, you know, get the key from here. Let's see if you can do it.

And a lot of really funky stuff happened. But we were kind of overconfident that it would take a while for people to do it. And in less than a day, there was a dozen people had solved it and some in very different ways. And so, I mean, at that point, it was like, okay, well, this is, there's no way we can get this much brainpower from, you know, an in-house group, especially as a small company with limited resources.

So the power of the crowd is, is strong. There are brilliant people out there. And if you can get them to pay attention to your problems, they might be able to help you. Absolutely.

When you step back, there are a huge set of problems that we all face. And if we collaborate instead of compete over the solutions of those particular problems, like we can all help each other make the world a better place much more quickly, much more quickly. It doesn't always work that way though, right? I mean, sometimes you have to make a strategic and bold announcement to take advantage of timing.

That's true. Yeah. Sometimes. There's a certain type of problem where it's not the one size fits all sort of approach, of course.

Right. And I know that you, you also get back, well, you also participate in the security community as an advisor for many startups too. So how's that going? What sort of advisor are you?

Yeah. So I'm a technical advisor for a number of different small companies. And, you know, this has been, this has been great for me. So since moving away from Cloudflare first full time, they're doing great.

I've been trying to explore more of what's happening in the up and coming startups. Insecurity is one area, but basically everything that my expertise or everything that I touched at Cloudflare, if there's a company doing something innovative there, that solves the problem. That I know that I've seen through 10 years of working at Cloudflare and working with Cloudflare's customers. Then, you know, I, I'm, I'm happy to, to talk with the founders and work through different problems.

And so what I've been doing is some technical advising as well as, as sort of matchmaking, if you will, for technical roles or investors or, or these sorts of things. But yeah, I mean, there's security protocols and, and everything sort of in, in all sorts of these little companies that are, are really trying to solve important problems. And, and, and yeah, if I, if I can be helpful and I love the idea, then, you know, this is, this is sort of what I'm doing with my time right now is, is helping these entrepreneurs succeed in whatever way I can. And so this has been really fun.

I think there's a lot of opportunity in the industry. It's obviously kind of a tricky time in terms of fundraising and in terms of. An economy. Yeah.

Yeah. Like to be more efficient. Like what's going to happen. Right.

Macro. We don't really know there. I mean, we could be in for a tough couple of years where there might be. Who knows, but in any case, having the right foundations for a company, finding product market fit, actually, you know, providing something novel from a security or other functionality.

Yeah. It's, it, it, it's been really great. And I, and, you know, I love entrepreneurs. I really think that taking, taking on the world and trying to build something by yourself is, is really laudable.

And so, and I think they're often pretty fun people as well. So yeah, I, I've had a great time at advising companies and, you know, I'm actively talking to companies and talking to VCs about, you know, whether my advisorship technical or otherwise could be useful. And, and yeah, met some really great folks. Amazing.

There we have it for all the founders that are listening. Might have an interesting conversation with Nick. Could reach out. Yeah.

It's also nice, it's also nice for them to have me on like the pitch deck for raising money as an advisor and whatnot. I bet that helps significantly. Like those VCs will just be like, oh, it's Nick, Nick from Cloudflare. Okay.

Right. Good. Yeah. Imagine that your, your network is quite extensive as well too.

Like it just helped connect the dots. And a lot of times that's what it takes to close a sales deal or people talk a lot about currency and the security space, but our main currency, I think it's trust. Definitely. And that's built up over time.

And it's, it's shared between people, not companies. We evolved with those. But I think it's a good idea. If you're a smart person, you're a smart person to try to ignore them or to try to escape them is to almost deny our humanness.

Yeah. And it's a diverse, diverse group of professionals and even non-professionals working in security. And the, the communities are not always overlapping too. So having exposure to folks who, well, corporate security versus an app sec versus cryptography in application or.

what do you think will be the next huge change that come down the pipe in cryptography have any predictions i really think quantum is going to break everything maybe homomorphic encryption will come pull through we can encrypt our operate on encrypted data or something different maybe you know i think there's there's three different prongs and you touched on them all that are exciting right now one is post-quantum cryptography which is being rolled out currently yeah a lot of places my understanding is like it's going to be part of phips in about a year and a half yeah so this this is amazing i've been part of this journey uh for standardizing post-quantum cryptography for a long time because uh it's important to get out ahead of it um the threat with quantum computers is that someone in the future will go back and take your encrypted data and decrypt it so like we need to start encrypting things with future-proofed algorithms now in case you know 10 years 20 years who knows when a quantum computer if and when it will ever happen but you know and the second one is privacy enhancing technologies i think there is a sea change happening in terms of how the internet is monetized advertising and personal privacy and tracking are at odds and there are currently a lot of interesting proposals for using cryptography to enable some of the functionality that would have been provided by tracking but in a privacy preserving way and this is happening in a lot of different venues it's yeah there's a project called dap which is uses a multi-party computation system to help collect analytics in a privacy preserving way this is just at the very beginning i think it's going to be a huge wave going forward um and then the last is is confidential computing which you mentioned homomorphic encryption is the you know the the superpower holy grail of that field still performance isn't that great but there's other categories of types of things that do involve advanced cryptography that give you different useful bits of this like private information retrieval private set intersection searchable symmetric encryption all these sort of things give you some level of um assurance that your date your your database provider can't you know accidentally read your data service provider yeah service provider at all i mean like all of this is tied together with trust with contracts with the ability to uh to just promise that you're not going to do anything with the data but these new privacy preserving algorithms and these confidential computing mechanisms they tie these things together you know in a way that you know it's it's a it's a lot more elegant you don't have to trust as much you can you can trust all you like but this it gives you something cryptographically verifiable which right right that's the magic yeah one of the things that i've heard someone's approach they can rename nameless here for a moment but they propose just to keep all the data on the devices and make since the devices are getting more powerful faster smaller memory is cheaper the silicon is cheaper of like you should do more and more stuff on the device and i i thought about that and i was like yeah but privacy right i mean oh yeah restoration yeah with this onslaught of new machine learning llm applications yeah data the size is humongous it's not something that fits on phones anymore so you're going to have to have a hybrid architecture no matter what if you're going to be doing anything with ai sure they're going to be ginormous and maybe a little bit slower and that can't you just going to be optimized for those machines up there in the cloud and there's all sorts of reasons why you would want to use a centralized sas service and to be able to do that like and preserve your privacy at the same time is ideal so yeah and i i'm hoping and expecting the industry to move in that direction i don't know how far it'll go or it'll succeed but it's it's going to be a talking point is there anything that does not exist today that you just wished someone would sit down and build it already a little bit of a leading this is the let's inspire your audience to uh come up with an idea segment yeah it's a little bit of a leading question for all the the board entrepreneurs out there yeah i i don't have a i don't have a an idea to to put on the table right now um no there's so many ideas and and so like finding that yeah so a lot of different things that i've ran into a cloud floor my colleagues have run into these are a lot of the companies i'm advising is are like hey this whole thing is a huge pain i'm going to just sit down and build something that that that solves it and actually a lot of the the best ideas come from people who are just banging their head against the walls inside of a big company trying to get something done that just requires you know a sas system or like some piece of software to to illuminate all the manual work and those are the types of companies that some of the types of companies that inspire me is like operators operating companies saw like having generic problems that they know aren't being solved somewhere else and just going out and solving it amazing if you could go back and visit your younger self and share a piece of advice with your younger self would you and what would that advice be this is such a hard question uh i was sorry for putting it on the spot it's fine it's fine no but it's it's good it's a good one right i mean like it it goes to the core of like mentoring well yeah mentoring yourself or like what is your personal narrative or like where did you get to where you are it is like butterfly effect some sort of change in the past and and and you know i'm i'm relatively happy where i am right so it was it was a it was a good path but i i feel like i'm much more of someone who learns through experience and through making mistakes than from listening to someone who listening to some old guy like tell you what to do yeah some somebody in their 40s telling you what to do as an early 20 something but and also you barely have the context to be able to take advice to heart unless you really think about it and so i'd have to really think about it but i i know that you know as as a young person i was always in pursuit of novel ideas and in pursuit of exciting things that are able to change the way that people interact or people interact with technology and i would probably tell myself to continue to lean into that be curious my young self because um unlocks unlocks the world to you right and unlocks your ability to know yourself so get out there follow your follow your curiosity i would say great advice nick sullivan everyone thank you so much for joining on a episode of the security podcast in silica valley nick this is amazing thanks john and i i really enjoyed this conversation and good luck and i hope to see it online soon thanks for all of our listeners as well for tuning in thanks everyone thanks thanks john