24. Aman LaChapelle, Early Engineer at Modular AI, on Redefining AI Infrastructure

Hello, everyone, and welcome to another episode of the Security Podcast in Silicon Valley. I'm here today with a very special guest, Amin LaChapelle. Would you like to share with our audience today just a quick intro of yourself? Sure, yeah.

In college, I was doing physics stuff, so I spent about a year and a half as a research assistant in a physics lab. Was gonna go get my PhD, ended up deciding not to after realizing that software engineering was pretty awesome. So I spent about a year at Unify ID doing machine learning stuff adjacent to security. That's how I met John.

Then spent a little bit at SambaNova doing compiler backends for CGRAs. Some time at Apple building compilers and runtimes for heterogeneous systems. And now I'm at Modular rebuilding high infrastructure for the entire world. Wow, that is a lot of experience.

It's great to have you on the show. Thank you so much for joining. Thanks for having me. So I know that you just shared a little bit about yourself, but would you like to share maybe any particular parts of it that really helped shape who you are today?

Sure. I think my college years were fairly formative in that sense, just in that I went from height elementary, middle, high school, where I think a lot of people have a similar experience of you don't have to work super hard, and then going into college and all of a sudden I had to work really hard. And so that was a very illuminating experience. And so I feel like that's where I learned to really sit down and put in the work to do something.

And that's, I think, a super important experience to have. And so I'm very glad that I had it. And I think the other thing that I got in college was just this sort of deriving things from first principles and understanding things at a deep level, enabling you to build more complex ideas off of very simple ones and gaining an appreciation for that. And I have, personally, I feel like I have my physics background to thank for that, just because it always made sense to me in a physics context when you take something simple and build up to more complex concepts.

And so I feel like I carry that forward into my software engineering today. I love the first principles and getting into the grit and just getting your hands dirty a little bit. Yeah, it's great. And as it turns out, in order to build something really awesome at a high level, you do have to understand how things underneath work.

You can't pretend it's a black box because it's not. That's right, it's not. And oftentimes those small decisions that are made in the guts of the machine have huge impacts on very high level experience or system behavior make a big difference. Do you brand yourself as a security person?

No, not really. I actually don't really brand myself as anything in particular. I just run into a lot of security problems and I care about security and I care about privacy and protecting, basically, I care about preserving people's choice with what they do with their data. And because data is all around us, it means that no matter what I do, I run into these types of issues.

And so I spend a lot of time thinking about them, basically. I would say if I was anything, I'm a bit more of a compiler person, but honestly, I like everything. That's right. I remember, can I share the story of that morning in Unify ID?

So Amin and I used to go into the office like crazy early in the morning, 7 o'clock in the morning, which is very strange for San Francisco and Silicon Valley in general. So here we were in the office one morning, we have this ritual of making coffee. And it was a very important ritual. And so Amin came in, and I have to admit, your feathers were a little bit ruffled this morning, that morning in particular.

And so we started making our coffee and getting to talking, and you were really upset about an optimization pass inside one of the compiler that we were using to compile something for Unify ID. And you were really upset because it did something like, I forget what it did. Do you remember what it did? Like unrolled a loop.

I think it was, well, no, I think it was, I think the problem was I wanted it to vectorize. I wanted it to unroll and vectorize, and it was just like, it was not doing it, and it was not giving me helpful diagnostics at the time. Yeah, it was not doing it. And we have this conversation, and as we were making coffee, I kind of trolled you just a little bit, like with all of the love in the world, I said something like, that poor compiler, it's just a piece of software like anything else that we write.

And it's only doing what someone. . . All that to do.

And by the way, it's an open source project, so if you really wanted to, you could go in and massage it and give it a little bit of love and improve it, make it better, not just for yourself, but for everyone. Do you remember? Yeah, I do remember. That harkens back to that thing that you used to say, computers only do exactly what we tell them to, which like, it's so true.

This is why computers are so simple. And so a couple of days go by, and then you came up to my desk at some point a few days after that, and you were like, Hey, John, you want to see something? I was like, okay, sure. Oh yeah, this was the thing.

Was this the thing where I like manually generated that LLVM IR to unroll the loop and then generate the vectors? I don't think I ever ended up committing to the upstream at that point. No, but I was so impressed because no one had ever taken me up on like the challenge to go dig into a compiler. And so it connected perfectly with your first principles thinking and your understanding, like the world from the ground up that you just shared that sort of made you in your formative years from college.

So I always have had the absolute most respect for you. Like machines just bend to your will. You realize that, right? Or something.

So you don't brand yourself as a security person, but you do care deeply about people's privacy and people's data. And as a security or as an engineer in general, you bumped into this all of the time because as we write software, like software, it always operates on data. So we have data flowing into our systems. We compute things on our data.

And you mentioned that you care deeply about this. And so when you bump into these problems, maybe you'd be willing to share with our listeners an instance, a specific instance where you were really faced with a challenging security question or a privacy question. So at a lot of my sort of jobs after Unify ID, we weren't dealing directly with user data, especially at SambaNova or like at Apple, it wasn't really anything that I didn't have direct access to any user data. And honestly, I'm like, I'm cool with that.

But at Unify specifically, there were a few times where I was, there were some, we were storing some like pretty personal data from people. And it was the kind of thing where it was like, I understand that we are, like, I understand that we need this data in order to do our business magic, right? Like we need all this like raw data from people's phones to do the stuff that we're trying to do. And that's awesome.

What's the data also? It was like accelerometer, gyroscope data. This was the, this was like tracking people's like walking, right? Yeah, the gait analysis.

Yeah. And it was the kind of thing where it just was like uncomfortable for me because it was like, this is something, so it's interesting because we need this, but also it's not something that someone can change, right? Like you can change a password, you can rotate a key, but you can't like change the way you walk. And so to me, this biometric data is something that's super sensitive that we really need to be careful about.

But, and it was in an encrypted bucket and we were being careful with it, but also everybody in the company had access to it and things were flying around a little bit in a way that just was a little uncomfortable. And so that was partly why I was pushing to get things done, like on device was to preserve some of that like privacy. Because if you don't, and I think this has sort of been my guiding principle is you can't leak data that you don't have. So like the goal is to have as little as possible and not that you would leak it intentionally or anything.

I don't think anybody's intent. Well, that's a bad statement. Some people intentionally leak data. I think good, like, I think most people are good.

Most people don't want to leak data. Most people are fundamentally good. Most people fundamentally don't want to leak data. And so when it happens, it's an accident and you can't guard against accidents.

They're accidents. The only way to protect against that is to not have it. You can't lose something you don't have. Right, right.

No, that makes perfect sense. I remember those days very well. I remember you trying to advocate to do all of the processing on the devices and the internal, I wouldn't say the internal pushback that we had, but more on the business needs were pushing us to think holistically about the system and delivering product to market as quickly as we could. And I remember finding encryption code like caked into the Android client and having to rip it out because it was breaking and preventing us from getting good data.

And It had no key rotation in it or anything, and it was like a hard-coded key, and I was like, what is this? It's unfortunate because I think Silicon Valley, as it exists right now, really incentivizes this move fast without regard for the follow-on effects for what you're doing, if that makes sense. I think, and I think Unify was a bit of just like a, like a victim of that almost, because like, I'm sure, I like, I wasn't privy to these conversations, but I'm sure there was a ton of pressure from investors and folks who were like, hey, you need to get product market fit. You need to get the product out into the market.

Now you need to get customers. And so like from that standpoint, I understand fully do things the easy way, just get things done. But at the same time, I also think that's the wrong incentive. And as Unify was not that long ago, but I feel like I've learned a lot since then.

And I've, I think I've seen that sort of consistently and that like technology and people that build technology in Silicon Valley are consistently incentivized to not build the right thing, if that makes sense. Like there's this overarching incentive to build the fast thing that works, not the right thing that you can extend and modify and fix and maintain fundamentally secure. Yeah, exactly. Yeah.

It's great that you're thinking about the full life cycle of a piece of software or even a product as a whole, which I agree with a lot of people miss that when you move fast and you think of shipping as, okay, we're done. Nope. That's actually when the fun really begins. And then you just enter another phase of your life cycle.

You have to be able to maintain those things. And that's when security starts really mattering because if you're successful and if you're lucky enough, you're going to have a giant bullseye painted on the back of you because you only get hacked or breached or you get that type of attention when you become successful. So yeah, absolutely. 100%.

And that's where good design and like good engineering principles really come in. And actually when you nail that security problem, like the way that you phrased it was you can't leak what you don't have. And this idea that less is more, that pays off in engineering architecture as well, not just from a security architecture point of view. Because if you, if your distribution, or that's one way to distribute a system that's highly computationally heavy, if you don't want to spend all of that money and maintain all of those EC2 instances up in AWS or you can't, or you don't have the bandwidth or the capital to be able to facilitate that.

And assuming no one else out there is building a SaaS that can help you scale your AI, then shoving it into a device is perhaps a pretty good compromise there. Oh yeah. And this sort of touches on something that I think is just, you know, lacking in software as a whole, which is we all assume, and I've fallen into this trap too. We all assume that everybody on earth has iPhone generation N minus one at the oldest and the latest laptop with effectively unlimited RAM.

Like, I don't know if people realize how much a gigabyte is. A gigabyte is so much. Like 32 gigabytes of RAM, that's more bytes of RAM than, what is it, than a signed integer can hold, than an unsigned 32-bit integer can hold. That's so much RAM.

That is so much memory. We went to the moon on what, like 100K or less? Or was it 32 or 64K or something? Like, the amount of memory that Google Chrome uses to show me cat photos is like mind-boggling to me.

And so this is another thing that's like very dear to my heart, is we really need to get software to shrink, if that makes sense. Like, software as it exists right now is super bloated and very, like, computationally and memory heavy, and much more so than I think it needs to be. And so if we shrink it, then that enables us to put stuff on devices. Once we put stuff on people's devices rather than centralizing it, then now you can have all these nice security guarantees.

Hey, I want to be able to do, in the case of AI, I want to be able to train a model on lots of people's data, but I don't want to learn what their data is. You can't do that right now because training a model is incredibly computationally expensive. You need a supercomputer to do it effectively. But if you can make that cheaper, if you can write software to make it so that you could deploy that to people's devices, even if it's not the whole computation, even if it's just part of it, now you're in a position where, again, you can't leak what you don't have.

And obviously I'm way oversimplifying the problem here, right? There's. . .

A lot of nuance, and it's not super obvious how to do this, but like, at a very high level, you push the computation to where the data is, not bring the data to where the compute is. I love that idea. Maybe we're just waiting for a platform to be able to write services that can run on our mobile devices. And then you could deploy all sorts of different things onto devices, perhaps.

And your idea of cutting the computational burden, let's call it, splitting it between a cloud and a device where you keep maybe the sensitive stuff on the device and you put the less sensitive things up in the cloud, that's also very interesting too. Hybrid model. Sure, yeah, you always need a cloud. Or, sorry, you don't always need a cloud, but having stuff in a centralized place lets you move fast.

And like, I'm very, I'm sensitive to the fact that once you deploy code to a user's device, updating it becomes difficult. That's just a fact of life. Even like iPhones, I think, are better than Android for this, but like, in general, once you deploy some code, it's hard to update it. That's why design is important, right?

And designing something where it's configurable and modular and changeable that you can put on the device so that your centralized server can give instructions to the device on what to do without actually receiving the data. Instead of moving the data around, let's move around the computation. Let's move around the programs, the services. Yeah.

You're making the case for flipping the paradigm on its head. And in the, the advocating the privacy position and a different type of architecture, a different type of system architecture for scale. Sure, yeah. Sure.

No, I love that. I love that. A lot of times people centralize things just because you have to go up to the server to hit another service that then facilitates like the actual business transaction. Sure.

So like a bank has to interact. If you move money around a bank, you do have to go to where the money is being tracked and saved. But I get it for all of this other stuff, there's maybe something that can be had there. Sure.

Like, I think another good example here is like Uber or Lyft as an example here. So you flag your Lyft, right? But no, I hesitate to use that as an example because I don't understand fully how their architecture works. I don't want to like speak to it.

Assume, yeah. Yeah, exactly. I don't want to assume, although I imagine that they do some AI. And, oh, I'm sure.

And what they're doing is they're tracking like your location. They're tracking like what sorts of rides you've done from that location at that time of day in the past. And they probably come up with a probability matrix of where you want to go. And then that's how when you open Lyft, like maybe it tells you, like, oh, do you want to go to SFO?

Do you want to go to like to the Castro? Where do you want to go today? Here's where you've gone before, dude. And that data doesn't necessarily need to live in the cloud.

Yeah. Yeah, exactly. I think that, frankly, I don't think it should live in the cloud. I think it should live on my device or your device or each individual person's device.

No, I love it. I love that perspective. In fact, I'm, I live in San Francisco and I see all of these billboards of Apple advertising that privacy is a core value of the company and it's baked into all of their devices. And all of these advertisements just have an iPhone like being held by someone and you can't see who it is because the iPhone is covering their face, signifying like privacy is iPhone.

And with your stint at Apple, did you really feel that privacy mentality guide your engineering? Honestly, actually, yeah. Like, I think it's, I don't, I don't want to sound like I'm like parroting the company line or anything here, but I think from a fundamentals perspective, Apple is a hardware company. Like Apple sells iPhones and MacBooks and they sell hardware based on chips.

And so because of that, I think they're very focused on the hardware and getting things to run on the hardware. And so everything I did at Apple was not about centralizing it to some, was not about centralizing things to some server. It was a hundred percent about deploying to the device, which fits right in with my worldview. So I, I actually very much enjoyed my time at Apple for largely those reasons.

It was just partly those reasons, partly I had a great team and the work was super interesting and stuff. So it just fit very well. I am so tempted right now to ask you what you worked in. And for, yes, he's smiling and laughing.

I know you can't answer that question. I'm so sorry. That's okay. I told you, I have told you what I'm allowed to say.

And we appreciate your sharing that much. Even with the secretiveness, you still got a lot out of it. Apple, I think Apple was a great experience for me. I really enjoyed it.

Okay, speaking of being an engineer, and you know, you've been in a couple companies now, and you've worked on a bunch of different teams. Let's talk about interviews for a second. Oh, okay. Giving interviews.

Do you have a favorite interview question that you like to ask potential candidates? I have a couple, but so I used to just do A to I, just string to int in some strongly typed language. I now have started using like circular buffer or lock-free queue to test concurrency. And it's interesting because I find that simple questions like that often work the best.

So the trick is you got to find a question that's like simple enough that people are guaranteed to at least get some part of it, but has enough range that you can make it more complex. So I really like A to I because for like more junior folks, you can sometimes even ask it to them in like a language that they don't know super well. So I frequently will do like, give me A to I, but in C, right? These are folks that like have maybe only ever interacted with C like once or twice.

And so I fully understand that there's a lot of difficulty there, but what you get to see is like problem solving. So in the like more junior folks that haven't worked with C a whole lot, obviously I'm not going to hold it against them if they're like, hey, how do I get the length of this string? Or like, or some like C is something like, that's fine. I haven't interacted with it before.

What I want to see is the problem solving of like, how do you like, how do you approach this problem? How do you, how do you work collaboratively? Do you ask questions or do you just try and bulldoze through? Or then do you take feedback, that kind of stuff.

For the people that have a little bit more experience, you can sort of take it up to the next level. You can be like, hey, okay, so what about negative numbers? What about hex numbers? What about base 23 numbers?

If I wanted to come up with my own number system, like what, how would I handle that? So there's a lot of like up, there's a lot of range to go up the experience spectrum. Which even just a simple beginning can change into all sorts of interesting questions, it sounds like. Yeah, absolutely.

Unlike one, one interesting one that comes up occasionally, it always makes me happy when it comes up anyway, is a string can hold a number much larger than like even a U. S. 64. What do you do then?

What do you do then? Those are the questions, right? And so that's when you start talking to the, that's when, that's when the candidate and like talking to the candidate becomes super important because you can be like, so let's talk about what's the system around this thing, right? So does the system around this thing expect to be able to have these humongous numbers?

Or is it very likely to be an attack? And then depending on that, you can decide, okay, let's modify the API to instead of returning it in, it can return an error or if it's too big for an int, then you should reject. Or maybe we say, okay, it has to be a double or something like this. And even then maybe you get outside the representable range of a double and then it's okay, do we want big int?

How do we do big int? All that kind of stuff. No, I love it. So it goes back to just an interesting discussion and you get to really see how the candidate is thinking through things and you have an opportunity.

Yeah, yeah, you get to think through the, a complex situation with them, almost like working with them. Yeah, exactly. Which like, that's ultimately what we're trying to test. I don't know.

I may be totally wrong. I may not have the best interview practice out there, but it has worked for me so far. So now I'm happy that you got something that works for you and it, I don't, honestly, I think that when it comes to interviews, like there's some unease on both sides of every interview. And it's always there and it's always like lingering, but no one ever talks about it.

And I'm not sure that either side realizes all of the time that's actually what's going on, but what helps significantly is just, you know, melting away a lot of the sort of the expectation of looking for a right answer or do you know how to like code in this language and you forgot a curly brace over here and just having a fun conversation with a new person. Yeah, yeah, no, absolutely. Yeah, I really like it when people get like excited in interviews. So that's always my goal is to like push a button that gets someone like talking really fast.

Yeah. Yeah. Did they light up like a Christmas tree? Oh, that's it.

That's it. Yep, exactly. When then you haven't talked for the last 10 minutes, that's when you know, like, that was a good question. Because then you really get a sense of who they are, what they like, that kind of stuff.

Where's that passion? The passion. Yeah, the passion-focused interview. Very nice.

No, I have a very similar style based on the passion. I would say strong, it's based on the passion. Yeah. Where do you think I got it from?

I got this from you. Oh, that makes me really happy. That's right, because we did a couple of interviews. We pair interviewed a few times at Unified ID.

Oh, that's right. We used the string to int question. Yep. Yep.

I think we did. I don't remember if we ever did Dijkstra, but it was mostly just string to int. Oh, yeah. Yeah.

Maybe we did. Maybe we did not. I usually save that one for more senior candidates. Yeah, but honestly, I've given string to int to some very senior people, and like, sometimes they still struggle with it.

Like, and it solidified this idea that, yeah, I think a simple, like a simple question is best, because struggling is not a bad thing. It's just really hard to distinguish what is struggling with the question versus what is struggling with the best answer. Because I think what a lot of people do is they'll be like, they'll like, not overthink it, but they tend, and I do this too, you like have a problem and you come up with a whole bunch of solutions, and then you spend a bunch of time analyzing which is the best solution. And so it's nice to have something simple, like string to int, just like this thing, not a lot of complexity.

You can do it in like linear runtime. Like, it's not a secret. It's not tricky. There's not really, there are some gotchas.

There's some gotchas. Sure. But like, whatever. Yeah, yeah, yeah.

But for the simplest case, it's simple. It constrains the sort of like, the thing that your brain does. I think my brain does it too, where it just explores all possibilities, and it's a little bit overwhelming sometimes. And it can lock me down if I don't pick a direction and just move forward.

So what about one tool or service that you wish someone would just build already so that you could use it in your future? Oh, God. So I think, honestly, if I was going to boil it down, the like very basic thing is, I want a cross-platform way to deploy code that allows like fine-grained permissions editing and is nice and lightweight. And I think a lot of people will be like, that's what Docker is.

But you can't use Docker and things similar to it on things like mobile phones or embedded devices. When I say cross-platform, I want to be able to write some code and deploy it on an Arduino and the same code on like some beefy x86 thing in like GCP. Like, I don't want to have to, and I feel like I shouldn't have to, because the like, so some cases there are platform-dependent code, absolutely. But in a lot of cases, for a lot of people, there's just like the process of deciding which code is platform-dependent versus platform-independent is basically like, which instructions does this target have?

Which like, it turns out compilers are really good at that and have been doing it for many years. So it doesn't, it feels like it's not something I should have to do. Yes, compilers have been around longer than I've been alive. Yes, me too.

Yes. Let's just let the compilers figure this hard problem out for us. Can we please reap the benefits already? Yeah, exactly.

And you mentioned something. You mentioned also fine-grained access control. Oh, yeah. So tell me about that.

What does that mean? So like, basically, and when I say fine-grained access control, I don't just mean like security type access control. I also mean things even more fundamental. Hey, I want to take this code and deploy it to somewhere that doesn't have libc.

Or it has parts of libc, but it doesn't have malloc or something like this. And I want a system where I can write some code and I can just describe my target. And I can say, this target is ARM64. It is bare metal.

These are the specific functions that you can expect to have from whatever your hosted services or potentially even, this is unhosted. You are running on bare metal. And so then the idea is, all that stuff just gets injected into your module and you just go. And so then I think the nice thing about something like that is, and I'm aware that what I'm describing is like a unikernel.

I think unikernels are. . . Interesting for different reasons, because I think part of the problem with the unikernel is they tend to focus really heavy on the embedded system side, and I can't take a unikernel application and run it on iOS, for example.

Like, there's a lot of stuff that's just not necessary because I'm running in a hosted system, so I don't need the like extra kernel bits. But like, on a bare metal system, I do need the extra kernel bits. Anyway, and so like fine-grained access controls are like, so I don't want to have to worry, like, I would like to be able to say, this thing is not allowed to access the network. This thing is not allowed to, like, or even more fine-grained, this function, this symbol is the only thing that's allowed to call out to the network.

Because then that allows me to super lock down my code. I can say, hey, access to this function is extremely tightly controlled. And so now that just makes it real hard for the black hats, basically. Amazing.

So you're looking for object level or even function level access to system capabilities. Exactly, like symbol level access to system capabilities. I'm just thinking about how a system like that might actually be designed. Once you gain access to a system, though, like any adversary with enough time will probably be able to reverse engineer it unless you go back to trusted compute modules that we're starting to see, the secure enclaves and stuff like that.

So I suppose you could design it. We're in a place now in terms of the technology where you can glue all these pieces together and probably come up with a secure mechanism to build. And I don't want to say that it's going to be the most secure possible or like it's going to be perfectly secure because nothing is perfectly secure. You're right.

Any adversary with enough time and effort can figure it out. But you can make it really hard. If you have a system, and just off the top of my head, if I have a system where I have symbol level control over network access capabilities, then, yeah. And so then what I can do is anything that comes in from the network, I can spend a lot of cycles validating it because now I know, okay, only this function can call out to the network.

So I can call this function in cases where I don't mind having a little bit of extra overhead. So I can do a lot of extra validation, be super careful with what I accept and with what I send. And so then if someone manages to get past that, I can also have capabilities that are like, and nothing on this, there are no symbols inside this box that are allowed to access things like the shell. Or potentially, there is no shell.

And so that makes it even harder for someone to perform an attack. Basically, this is about defense in depth. It's not about making it perfect. It's about making it really hard.

That's right. That's the game that we play in security. And I love to hear that mindset reiterated from an architecture elegance point of view, too. Because I think that stripping things out of a system is really the hard thing to do.

It's easy to bloat a piece of software, to bloat a Docker image, to bloat something that's running in the cloud or on a device. And you look at it and you're like, why is this thing like 600 megabytes? You know, it's Java or whatever. It's like, and that's like, that's kind of sad.

Yeah, and you could take the same thing and you could rewrite it in C and now it's, what, a megabyte, two megabytes tops? I mean, if that. Depends what it is. Yeah, exactly.

I love that you're striving for the simplicity, which is actually much harder than what everyone else tends to do when we move fast and bloat and break things and everything. Yeah, and that's also where you get your security from. You can't patch it. It's not there.

Right? If there's no shell, guess what? You're not going to have like a root compromise on that device. Because there is no root to compromise.

There is no shell. There you go. Go away. You maybe can have like arbitrary code execution or something, but the other thing that you can do with a system like that is you can say, hey, this system is not allowed to run any code other than this exact code.

And you can use cryptography to enforce that. And then guess what? That's really hard. That's really, that gets even just a little harder to bypass.

And it's just, I'd love to end up in a world where capabilities are default off rather than default on. And you only add what you need. Exactly. And when you add a dependency, like the compiler does not download half the internet to support it.

I love it. Wouldn't that be nice? What would we call this package manager? Oh, I don't know.

Names are so hard. Names are very important. That's true. That's true.

All right. Would you like to leave any of our listeners with parting words of wisdom? Yeah, how about something along the lines of, do you really need that thing that you're adding? Are you sure?

Are you really sure? So three questions that we can go ask ourselves now the next time we do an import or something. Yeah, exactly. The next time you do a pip install, ask those three questions three times.

And the ghost of Amin will be lingering over your left shoulder. Yeah, exactly. I'll whisper in your ear. Amin, it's been a great pleasure to have you on the show.

Thank you so much for joining me for an interesting discussion and a fresh perspective on security and privacy. Yeah, thanks for having me, John. It's been a great time. And thanks to everyone for tuning in for this episode.

Stay tuned for another episode. Thanks, everyone.