15. Salesforce Security Engineer, Benjamin Schmoker Scott, Navigating the Cyber Security Landscape

Welcome to the Security Podcast of Silicon Valley. I'm here with a very special guest today, Benjamin Scott, who focuses on threat intelligence at Salesforce. Welcome to the show, Ben. Thanks for having me, John.

I was just looking at your LinkedIn, and you've made a hell of a career out of all things security. I can really feel the passion just going all the way back, even to your internship with Union Pacific Railroad. You were involved with security operations and then cyber defense at Sandia National Laboratories. You were a security engineer out in Washington, D.

C. , a couple of different places. You were a technical investigator in Virginia for a while, and then you had a gig at ThreatConnect, Inc. You helped with security at Tile back in the Bay Area, then security at Yahoo, where you also were advising companies, it looks like, through Raven Flights.

But now your day job is as a security engineer in the threat intelligence team at Salesforce. So I love the focus. I love the drive of the passion on security. Welcome to the show.

Yeah, thanks absolutely for having me, John. And just to provide a little bit of color on that, I'm from Omaha, Nebraska originally, so I was able to go to University of Nebraska at Omaha, which has a really great security program. And, you know, I was able to take some classes at the essentially graduate level when I was an undergrad, and they were testing out the new classes. So I got to do reverse engineering, physical security, rewritable card, like the NFC card, got to do a lot of operating system internals.

And I really liked that program. And some of my buddies have come through there, come through Sandia, that same internship program, and really done some really amazing things. So even in school, you were focused on security. That's great.

I did the same sort of thing, University of Minnesota. It sounds like you really love it. Yeah, I think security for me, I started pretty early. My dad helped me build some computers out of age 11.

So I had like full access to the hardware, software, everything. And at the time, there was a lot of like video game type modifications that were popular. Got into that for things like hex editing, reverse engineering, or you just figure out how to make things work, like weird driver things, weird hardware things. So I got that experience pretty early.

And then during and after the CS degree, focused more on like, how do you defend against issues like vulnerabilities? How do you deal with bad guys that are doing things on platforms? The field really evolved since I got into it. It used to be like Flash, Internet Explorer, Java bugs.

Now that stuff is like, you know, those entire platforms are deprecated. Now we're dealing with WhatsApp, no click, like exploits. It's been pretty interesting. And I think that one of the things for me, I think growing up in the Midwest and having a good access to education, Omaha has a lot of military intelligence connections.

So I was able to make that transition to DC a bit easier, I think. Was any of the military connections in your family? No, no, no. More the university also.

So there's a huge base there called Strategic Command. So I worked for Strategic Command as an intern, did some really cool projects like write-ups and got to present to three and four star generals. So that was a really unique and exciting experience and really showed me, hey, it doesn't matter. At the time I was like 20.

It was like, it doesn't matter how old you are, like something unique, even if you are still in university. And that really helped, I think, going forward because I felt that I could become like the quote unquote expert in something because it was such an emerging field. Yeah, yeah, definitely. And I remember we bumped into each other at B-Sides and you brought that same drive and spirit to the table when we ate lunch together.

I forget what we had, what our conversation was over that table, but I just remember feeling inspired. Yeah, absolutely. I'm sure we talked about cryptography and your company and things going on at the day. Yup, yup.

And so fast forward a little bit. Thank you for sharing a little bit about your childhood and where you came from and adding that color that I had missed just skimming your LinkedIn profile. But at Salesforce, you focus on threat intelligence and it looks like your title is security engineer for threat intelligence. So what is that for our listeners who are interested in security, maybe less familiar with what threat intelligence is?

How would you describe it? Sure. So threat intel is a field. I see it as applied knowledge.

So it's something that's very connected to the way that military intelligence looks at things, right? Like you have information, which is just, hey, this site information. This site is suspicious, and then information is like, this domain points to this IP, and intelligence is like, this domain pointed to this IP on the same day that a URL to that site was attached to a phishing email sent by this address. And then intelligence might also be like, we know that this particular guy sends emails through that, or we know that this phone number is attached to this particular intrusion group or something.

So it's about bringing for the organization that sense of making information actionable and giving context around it and helping them with these really high-impact incidents. So like we act as the escalation point sometimes for incidents. And then there's our team is mostly in DC actually at this point. So there's a lot of that X-mil experience that I think you have to buy yourself some of that.

Like every tech company I think has done this, like Facebook, Google, like they all have these big presence on the East coast and the DC area because they have to have people that understand that content. I have experience with that area as an analyst. I also as a CS guy do programming. So I do security engineering in my day job where we want to build scripts.

We want to build tools. We want to build platforms. I use a lot of AWS. I use a lot of serverless at this point.

We have other internal platforms built by other teams. Like we have a huge security engineering team that just builds like larger things. So integrating with those. And I've done at this point, database design, a lot of these other kind of more sweet things like software engineer type design tasks.

So that's been really cool to pivot from pure security operations, Intel analysts, malware reversing more into kind of this programming building role. Cause I think frankly, it has more legs as a career. Yeah. So you're more, so you're getting more into the product security side of the house where you're building security features that either end up protecting parts of the infrastructure or stitching together the actionable items that can come out of like threat intelligence, like all of those logs.

They tell a story about what's happening out there on the internet and how it relates to your internal systems. And that, yeah, I would imagine that requires a lot of engineering work to stitch together some actionable items out of all of the logs. Or are you more on like the product security where you build security features that end up inside CRM systems? More, more the first one, we have a bunch of logs, like you said, that come from, so our team focuses on both employees and security, corporate security.

And then we also have some overlap with like product and user security, although that one is a bit more each dedicated product team. And we do jump in and help them when we see something. Like if we see, hey, a bunch of bad guys are abusing this product, or we got a bunch of phishing reports on this platform. Like sometimes they don't have the context and like they're busy doing another actual job and they're not reading blogs or whatever.

What does that look like when there's platform abuse? So where do you get that in? You read blogs and stuff you mentioned, or you just read like hacker magazines like 2600 maybe, or? No, it's maybe that one might be a little bit more technically in depth, right?

But no, I think that a big part of our team, so we have part of our team for collections. So that's just getting the data that we turn into intelligence. And that might come from vendors like FireEye, CrowdStrike, doing private publications, might come from public publications, government outputs, other people in the community saying, hey, like we're seeing a lot of abuse coming from X place, or hey, by the way, FYI, we want to help you guys. So here's something we're seeing.

So having those connections and knowing how to parse that data, because it is like a really large amount that comes in. Are those data sources, are some of them like open source and then it sounds like some of them are just proprietary. You have to pay for them and they're just streams of data. Yes, both.

So we have these both these long form kind of prose reports, as well as indicators, like here's a bad IP or whatever. And we treat those differently. It's just a feed of, we have a bunch of things that are just feeds of like, you know, URLs or IPs. You have to sometimes say, okay, how do I know this isn't a false positive?

Or I need some more context around this. So that's part of what I work on is adding context for have other places reported on this? What, you know, if I enrich this using like virus total. Passive total, what does that tell me?

Is this a hosting provider where we probably don't want to blacklist their IP because it's just a hosting provider? Or is this some VPS in the Netherlands that we're fine with blacklisting because like. . .

Because who knows what's coming out of the VPS, right? Yeah, users are not going to, we're not worried about our employees going there, but if it's like some GoDaddy IP or AWS IP, yeah, maybe it's doing something bad, but it's probably going to break something if we just block it. And so it sounds like at least gathering up all of those different data sources and stitching things together, you could automate a lot of the context building, if that's what you would call it, like the context, you called it enrichment. Yeah, if, let's say you just have a domain name, so that domain name is just some information, some data.

So if you want to get some context on it, you might ask passive total about who registered it, when was it registered, the TLS certificates are associated with the IPs that it points to. When were those minted? Has this been seen? Is there anything running on that IP?

Is there a server or is there just nothing? Is there like a RDP server running? And there's all these kind of different things you can add to it. And you might ask virus total, hey, what files and what digests of those files were downloaded from that IP?

Because they know that info. So in the context of like phishing or some kind of like malware dropper, that's helpful to know. And then when it comes time to take action, is that generally, is the, I guess, I imagine a couple different ways forward. You could block internal employees from hitting that from within your own VPN, or you could just drop all external traffic that's coming from those bad IPs, including things that may involve potential customer integrations or because you don't want to break customer experiences.

That's the whole name of the game, along with like your, the operational flow of just day-to-day operations in Salesforce. Yeah, and we can do a couple of things. We have our corporate firewalls. We can add things to.

It's different teams, like product teams that handle whether or not they like filter different IPs, but we do influence, influence that process on their side because they trust us to tell them, hey, here's some example data. A lot of the time, they're not working with like specific indicators. They're working with more of a machine learning type thing where they're saying, here's known bad data. Here's like a corpus of data.

Can we use some ML to like figure it out on the fly? Like how suspicious this is and whether we need to escalate like other authentication methods. Like Google does something very similar to that. I'm sure they do a lot more complicated stuff too.

Oh, that's interesting. So you could, you can detect traffic, inbound traffic that may be legitimate and just start raising the security bar one step at a time and require like a 2FA token be used if there's a 2FA token on the account coming from suspicious IP trying to log in. Yeah, and that's something that a lot of companies do now. And using Google as an example, right?

They check source IP. They check for teleportation. If you're logged in in Seattle and then you've teleported to Ukraine, like that might not be legit. Even if your credentials are valid.

Like something I remember from B-Sides talk was Google saying that some huge proportion, they had valid credentials and that might be from keylogging, credential stuffing. Yep. Like, so you basically can't really treat username and password as this is the user anymore. It's more like a profile of the user.

And a lot of places, I think when you log in from a new IP, will say, hey, I've texted you a code. Will you put that in to validate this IP? Like places that really care about your user security. Like I say really care, meaning like it's worth like a million dollars to you, like Coinbase or something.

You have to, yes, yes. This is getting back to a business case, which is, this sounds like some really heavy duty security. It makes a lot of sense for the largest SaaS service in the Bay Area, which is Salesforce, to care about these things. And I'm super happy that they do it.

And same for Google. You're mentioning the B-Sides talk from Google. When do you think is the right time to start caring about these things? I think if you're, like if you're a startup or a smaller company.

So there's this idea that Facebook started talking about called abuse engineering, where you build your product with the abuse cases in mind, which I think is a really cool way to think about it, right? Like I've started thinking of it that any useful product will be abused. And there's a corollary there that if your product is abused, that means. .

. It's useful. And I think that both of those are true. It's absolutely true.

You will get hacked only until you're big enough to matter to get hacked, right? Or abused, to use your language. Yes. Yeah, I think that startups have to move fast and break things.

And I think that if you can use a built-in, what I would do if I was doing a startup, I think, would be use like understood login platforms. I'd probably just use Google authentication, like using SAML. And I think that if I had some break glass, like the user needs to do something major on their account, I think that having their phone number is a good workaround right now. So you can text them.

I don't necessarily think that requiring 2FA for every account for every use case isn't necessarily a good idea. If you're a bank, like you should probably have that before you can do a wire transfer, you have to put in like a 2FA code, even if you've already logged in. But if, but I don't want to do that for my random, you know, just some SaaS company that I'm using. And I think that's a little overkill, but.

Right, some consumer thing and Stu Dads and Trinkets and nothing really is at stake and it's a barrier to entry perhaps or additional friction sometimes. Yeah, I think you have to look at exactly what's, is there something that keeps the users from coming back or, and looking at your data and saying, are they bouncing because I asked them for their, can I ask them for their phone number after they log in, after they use the system? Maybe I can gate features. I've seen places start doing that where they gate, hey, you can send messages to other users to validate a phone number, or you can do these other parts of the features of the platform.

So I think that's something you can also get around it because like people using that as part of the freemium service there. They get it. It's a trade. It's, it's maybe a fair trade.

You put your phone number in, you get more features. You don't put your phone number in. Well, okay, that's okay. You just don't get those features.

Yeah. That's very creative, actually. I guess it's Salesforce. What is the business case in the huge company to do this?

Of course, reinforcing the trust with the end customer. Yeah, I was, I was going to mention because you mentioned CRM. So you can have IP allow listing set up for your account to say you can only log in from such and such IP, point that to your, you know, VPN egress range or whatever. I think that is something that we can't really mandate places use because we don't know what their ranges are.

And it would be some friction to like just push it on them. But that's something where if it is a security conscious organization, we can tell them about that feature. We can make it easy to use. Maybe we can pre-fill things.

So maybe we can like support other integrations. So I think that's something that the core CRM does. 2FA is similar. It's opt in.

We, at least for that product, because it's a very like historically used product and it's hard to tell people, hey, you've been using this for 20 years. Like now I'm going to move your cheese around. Like hopefully that's fine with you. Exactly.

What, what has worked in the past should probably continue working, especially if there's a lot of people using it, unless there's a serious security need to stop using it. I was going to mention, we have these other products that kind of operate as other companies to a large extent, right? We have Heroku, we have Marketing Cloud slash Pardot, which was two different companies that merged into one. And so there's kind of, you can imagine these different legacy code bases, workflows, different authentication, different abuse models.

Like Heroku is a huge abuse model because for the price of a free email address, you can get some compute is the way that they talk about it. So trying to deal with that. That's a very challenging one, isn't it? Yeah.

And we have email marketing platforms and that's part of marketing cloud. And those have subtly different kind of login flows, backends, because they are different products at the end of the day. Maybe you can have a free trial in one of them that is a little bit more free than the other. Maybe one of them does more validation.

Maybe you can convince the customer success representative you're talking to that you're really going to pay them a month from now, but you just need to send like a billion emails now. So there's some social engineering and some kind of human aspects, right? Where the sales, salespeople, like they want to close deals. They want to do right by customers.

And so they're incentivized to give out things. They have the exact opposite incentive of security guys, where if you want to keep, make it harder to use. Yes, please use our platform. Reduce the amount of abuse on the platform.

Yep. No, I've done that. I've tried to get good deals and negotiate with sales folks on certain SaaS services. And they're usually willing to accommodate it, at least at some.

So just having been on the flip side, I can sympathize. Nice, with the legitimate group of customers. It's the same conversation, I'm sure, that the malicious users are sharing with those same sales folks, though, too. And they'll set up like, yeah, and they'll look like a legit company.

They'll set up a website and they'll do whatever, but they don't look any different. Have a phone number. Yep, yeah, I'm sure they do, especially if they want access to the emails or the free compute. So for anyone that's listening to the podcast and is specifically interested in learning more about these sorts of things, these sorts of tools and techniques to prevent or mitigate platform abuse, would you recommend any resources?

Yeah, so I would say, like I mentioned, Facebook's abuse engineering presentations. I think that there were some talks also at ShmooCon about similar topics like dealing with abuse. Netflix has a really interesting way of thinking about it as well, where they, I think they approach it from a developer side and they say, let's make it easy to write secure code or have secure deployments. And they do like a paved road that makes it easy to do that.

And then if you want to deploy your own, if you want to go off-road, you can, but as a developer to build something, but they want you to take the nice paved road that has everything set up for you. Guided path through. I think, don't they use Spinnaker? That's where Spinnaker has come from, Netflix, isn't it?

Like they're one of the major contributors. Yeah, for sure. And so suddenly instead of using like some custom authentication for your particular use case, you're just like, okay, I'm going to integrate with the main one that we have or things like anti-DDoS or filtering or security response, things like that. Nice prepackaged solutions.

And they do a great service for the security community too. They open source a lot of those things. Yeah, and they have some talks. That's also what I'd recommend is some of their, some of their talks there on SlideShare and besides YouTube talks and stuff like that.

Okay, so what's the best day that you've had at Salesforce so far in your journey? Yeah, any day that I can push out production code and it's stable is a good day, I think. Well, I know that feeling. And then I'm doing something exciting and helping mitigate bad guys and keep users safe.

I remember we had a case where I was dealing, I was helping investigate some large scale phishing and that was just nice because I like feeling like I have an impact on real people, which actually is unique in the security space because you can have that and you can feel that way. You can say, oh, like I, by whacking this bad behavior, I've probably prevented some people from getting compromised, losing money, potentially losing their freedom. It's like some big impacts. So I try to look at that.

So much of our lives are handled by computers and the information inside of them that protecting that stuff, you are having, you're touching people's lives. I like I could make some compiler slightly faster, but I don't necessarily care about that so much at this point.

And I think that was something I really enjoyed about Yahoo as well was I wasn't necessarily, given the context of the overall company, like there were some issues, but I think that helping with Yahoo Mail and dealing with phishing, dealing with e-crime, like that was really exciting because you would see things happen and be like, oh yeah, okay, this is going to be really impactful on this person if they click on this or they interact with this. And then going through and helping mitigate that was really nice. Okay, so how about, have you had a worse day in your journey at Salesforce so far that you might be willing to share with us? I don't know.

It's not really like too bad of a place to work, honestly. There's a whole building next to us receding and tilting thing. Oh, that's right. The leaning tower of San Francisco or.

Millennium Tower, yeah. The Millennium Tower, yeah, that's right. But I think always like when you had a couple of notes on here around, I think vendor tools and stuff like that. When vendor tools break or their APIs kind of don't work as you expect and then it's some like small thing that you just need to change happy to glad and some hair and you're like, oh, I just spent five hours on some documentation could have fixed or something.

Yeah. Yeah, I know. I know those feelings. And you can smell it when it's going to be something like that before, like right around hour number four, you start scratching your head and you're like, oh, I know this should work.

And what is it? And it turns out to be some little thing, some config. Configs are my favorite. Yeah, absolutely.

Okay, so how about a juicy question here? How about what's your favorite interview question and why? Sure, so there's a great one that's on GitHub. It's called What Happens When?

And it has a really comprehensive write-up of all the different aspects of what happens. What happens when you type Google. com in your web browser and hit enter? And I like people, and you give them a limit.

You say, okay, spend 5-10 minutes telling me what happens. It's not like a free form hour-long thing. So I like to look at where do they start? Where do they start at DNS?

Do they start at HTTP? Or do they talk about TLS? They talk about certificates. They talk about cryptography.

If they're a front-end developer, what's up? Or maybe even the, I was going to say the sin, but TCP is not the first thing that goes out. It's the DNS request for the domain. Yeah, yeah.

I even skipped that because I think in terms of, I was thinking HTTP wrapped on it, but yeah, yeah. There's so many different places you could take that because that's a really deep, you could go so deep. Yeah, or you can go broad too and be like, okay, site loads and some HTML is rendered and some JavaScript is loaded by an interpreter. And I think that it really helps to figure out also how they explain things.

And there's a Feynman quote that's, if you understand something, you can explain it to a 12-year-old. And I think that I try to do that where I'm like, okay, how do I make this basic enough to explain it? Can I explain it without buzzwords and without kind of jumping over parts of it? Yes, this is the mark of a great communicator.

You take complex ideas and simplify them and use analogies and help folks that are smart, but maybe not necessarily experts get a solid handle on what's going on. Yeah, and even experts, you forget about things, right? I probably couldn't recite like the whole OS, what's the, what's that called? The OS stack?

Because in practice, you know. Because what do you, what do we, how often do we have to like know, okay, what's a layer four or five or whatever? So. Yeah.

Like at this point, JSON parsing is one of those layers probably. And yeah. And so I was going to mention something I skipped over. We talked about business cases a bit.

So I just wanted to say that I think that's something at Salesforce that we really got in front of was saying like, trust is our number one value. And so I think drilling that into people's heads, both internally and then user-wise, really helped them look at it and go, oh, you guys take it seriously. You're spending a lot of money on it. I think big companies are looking at security more as a bottom line issue now, where it's like a significant risk, especially like when you see SolarWinds, like the pervasive IP theft intrusions as well and like espionage-related intrusions.

You ask, what if it was me? If you're a small company, it's just existential. If people lose confidence in your site because it's compromised, like that's it. Done.

So I think it's. Yes. Yep. No, I love how Salesforce uses it as a top-end growth proposition.

They put it out there. It's part of their market distinguisher. It's part of their identity as a company to be secure or to be trusted. And I agree so much with you.

So often I bump into folks. Their view of security is the bottom end. Go, how do I protect myself from losses? How do you think about amortized loss over the likelihood of a very expensive event happening?

So they want to amortize it over time. And then there's your budget for how much you should be spending on the mitigation mechanism to come out ahead in terms of the business. Just very complicated formulas and questions. If you just embrace it and you put it out there and you use it to attract top-end growth, then you really flip the whole thing on its head and position yourself in the company, not just to do the right thing for your customers, but really put a significant dent in the war against cyberterror.

Yeah, I think it used to be a little more, a little more based on the bottom line. Like big banks would just kind of take carding, card fraud as a bottom line cost. Like, yeah, it'll cost X million dollars to reverse transactions. And some of them will fly out the door and moved around liability when they were like, okay, bank fraud, we'll reimburse you.

If we can find out that you were actually like a willing money mule or a different company that, say, wired money out, can we move the liability to you? It was kind of more of that. I think in the modern day with ransomware and stuff, like they're not able to do that anymore. I got ransomed.

Can't move that liability around. We're finding out cyber risk insurance. We're testing cyber risk insurance. I have some opinions on it, but.

Oh my goodness. What are your opinions on cyber risk insurance? I think it's not great. I think it's not great.

It's, I think it was sold as something. It's, it has a hard time paying out for, and I think it has a lot of edge cases and clauses in there for reasons why it shouldn't pay out or why it won't. So I think that it's, it was something I think people scrambled to buy to cover their liabilities and make the financial people happy, but I don't think in practice it's really that great. And I think It's hard because when you become uninsurable also, like if you've been ransomwared, are you now just uninsurable?

Like That's a great question. I remember back to when Target got hacked and they lost all of that credit card information, and it was, it basically was an embarrassment for the company. And they just doubled down, and now they have one of the strongest security teams of all of the companies out there. They're work daily with the FBI to grab threat intel.

They're a very strong team at Target. And so they've recovered. I guess companies can change. Yeah, I think, I think there is some kind of pervasive brand damage, but maybe, maybe it just takes time for that to go away.

I think outside of, maybe outside of security wonks, like nobody remembers that. You know, if you ask people on the street, maybe they're like, what? I don't care. But then at the time, it would have been all the news outlets, and so they would have been like, oh my God, yeah.

Yeah, yeah, there are buckets of folks out there that think that maybe it's okay to buy the insurance to cover the gaps for the resources because they don't have the resources to pull the trigger on a lot of the internal projects that they would like to. And it's when I look into the future, I see much more automation and self-service tools and really full stack solutions that can help with a lot of these things. But with your perspective and all of your experience, Ben, in the threat intelligence space, do you see a lot of that being automated or do we got a long way to go before? Oh yeah.

Oh yeah, and that's why I'm focusing on engineering instead of analysis as well, is that I see the writing on the wall of having rooms full of people staring at the news and clicking buttons is not going to scale and something that's going to, something that's not going to exist outside of like big government socks. I think you have to automate a lot of this too. How many people can you hire also? There's three digit number of people that can actually do this work in like the entire West.

And I think that if you can pay some company to automate parts of this away, you know, that's nice. Kind of the same way as you've automated away parts of like your email infrastructure and hosting infrastructure, you know, like. Exactly, exactly. As these things become services, then we can start talking about scale.

Then we can put a real dent in a lot of these security problems that everyone faces, but as you put it so elegantly, like there's just not enough security folks to go around at the end of the day. Yeah, and I think that you'll, you see like AWS do a lot of things as a service now that previously were individual companies and things like that around data security. They have a snowflake use case clone, and they have IP filtering. They have, you can now run your AD infrastructure in Azure.

So that's just an entire threat model, not necessarily mitigated, but pushed off to someone else. Someone else can patch your AD. Someone else can manage it. Maybe that's not perfect.

We've seen like now that moves all the nice juicy data to one place in Azure, like there was, you know, recent Azure break-ins, but maybe that's better. Maybe it's better to accrete that in a pool rather than having a bunch of easily accessible pools that in each company. That probably trusts Microsoft to run that more than I do myself. I'm with you on that one.

That's why services are great from a security perspective. Let the folks who wrote the software run the software. They know what needs to be updated. They know how to run that stuff.

Let them run it. They're the experts. Thank you so much for tuning in to this episode of the Security Podcast of Silicon Valley. Ben, it was a pleasure.

Yeah, absolutely. Thank you, John. And I'm glad to do it whenever you want to. Awesome.

We'll have you on again. All right. Have a good one, guys. Thanks.

See you.