In this week’s episode of The Future of Security Operations podcast, I'm joined by Matt Johansen. Matt is a security veteran who has helped defend startups, the biggest financial companies in the world, and everything in between. Alongside his day job as Head of Software Security at Reddit, he teaches companies how to protect against cyber attacks, and coaches entrepreneurs and CISOs that need help with infrastructure, application, cloud, and security policies. He also writes Vulnerable U, a weekly newsletter that talks about embracing the power of vulnerability for growth.
Matt and I discuss:
Moving from a large security team at Bank of America to a small one at Reddit
Embracing scrappiness and doing more with less
Overcoming sunk-cost fallacy
Why the 2014 Sony hack was a pivotal time for AppSec
Running the threat research centre at White Hat
What he looks for when hiring in AppSec, the SOC and beyond
His decision to start creating content about mental health in security
Moving past imposter syndrome
Renouncing superhero culture
Paved paths and guardrails, and what comes next after "shift left"
Lessons learned from Reddit's 2023 security incident
The power of automating incident response
The Future of Security Operations is brought to you by Tines, the smart, secure workflow builder that powers some of the world’s most important workflows.
Where to find Matt Johansen:
Where to find Thomas Kinsella:
Resources mentioned:
The Tech Professional's Guide to Mindfulness by Matt Johansen
Matt's piece on developer experience in the Vulnerable U newsletter
Reddit's post on a February 2023 incident
Collaborative Incident Response Best Practices: Don't Rely on Superheroes by Matt Johansen
Threat modeling depression by Matt Johansen
In this episode:
[02:14] Going from long-time Reddit user to employee
[04:50] Running AppSec at Reddit
[07:30] Being the internet's punching bag and boxing gloves
[10:30] Building a team from scratch at White Hat and lessons learned from the 2014 Sony hack
[15:10] Matt's approach to hiring
[21:15] His decision to create content about mental health in security
[23:20] Turning his Twitter network into his IRL network
[27:55] Moving past imposter syndrome
[30:00] Tools for safeguarding your mental health in incident response
[36:20] Preserving work-life balance for his teams at Reddit
[39:15] Moving past "shift left", and paved path to production and guardrails
[47:40] Lessons learned from a February 2023 incident at Reddit
[51:20] Renouncing superhero culture
[52:20] Automating incident response
[54:12] Connect with Matt
TL:DL? Read Matt's take on…
Securing Reddit:
“Some of the challenges of just being Reddit are that every decision our leadership makes gets under the microscope of the like Internet. And they all have a megaphone on our own platform, right? Sometimes it's hard to be the Internet's punching bag while you go to work in the morning.”
Getting hired in security:
“If you're looking to start out, just prove passion. If you really want it, prove that you want it. Go out and do the extra learning. If anyone came in with any sort of experience in an open source project or like, 'Hey, I did this conference talk' or bug bounty... nowadays that's a perfect way that you can prove your knowledge on something you don't know. There's no interviews for bug bounties. Just go find the bug bounties and then put that on your résumé."
Nailing a job interview:
“A lot of questions that I ask in interviews don't have right answers. I like to build a scenario and just see how you walk through it. It’s not like, 'Here's the A+ answer.' It’s like, did you mention some important things? Did you at least recognize the problems that I was laying out? Did you hit the right stuff and approach it the right way? That's worth way more to me than knowing the textbook.”
A lot of really good security practices need to basically just be development practices. It's really blurring the line. And so some of your best security engineering teams on the planet right now are engineering teams that also kind of know security.
Imposter syndrome:
“Picture the highest levels of politics, or your career ladder, or whatever it is, you go into those rooms. They're having the same conversations. They're having the same doubts about their own feelings, their own decisions, and all this kind of stuff, and then they work through it, and they come to a decision. And that's it. They just go forward. That's the difference between the top and the bottom of the chain, is just the ability to continuously move forward.”
Promoting work-life balance on his teams:
“It's not good to be a load-bearing team member. You gotta test the structure here that you've built. And we can only test that if you're gone. We try to encourage folks to take the time off, and then, when they take the time off, like, 'Uninstall Slack. Don’t try to like stay in touch, or anything like that. Really, really check out. Really go and recharge the batteries.' That's the most important thing that I can say.”
The pressures of working on Blue Team:
“We work in an industry where you're gonna have the pager, right? Especially on Blue Team, things are gonna hit the fan. It's gonna happen a lot on US holidays cause the adversaries know when US holidays are. I used to go camping every July 4th, and I think, like 3 out of 5 years, I was searching for cell service in the woods to be on an incident bridge, because adversaries know that the team is thin on these holidays. So that's taxing.”
In incident response, speed is what matters... The incidents that you read about that have the smallest impact, and the fastest time to fix, it's because they've automated something and likely Tines was involved.
Cross-team collaboration:
"At Reddit, because we're a small security team and a large engineering org, we do have to rely a lot on our engineers. We also find a really good partner with SRE, which is basically DevOps. They're really good security champions for us as well... We also have very, very good partnership with our legal team, which made incident response, much, much easier too, by the way. So pro tip to all you incident responders or SOCs out there. Go make friends with your lawyers."
Lessons learned from a security incident at Reddit:
“Document the sh*t out of everything - excuse my language! Document everything as part of the process, that makes everything so much easier, especially, as you transition. It was a wake up in the middle of night situation. You have shifts. so you're handing off to next shift, 'cause you have to go take a nap. If you didn't write anything down that whole time, that handoff is gonna be super hard. And handing off to legal or PR, or anyone like that. I was incident command on this incident you're talking about, and the highest praise I got was, 'This running incident doc that you had just made everything so much easier.'"
We can't be playing whack-a-mole with with a small team. You don't have a hundred people to throw at... It's like, okay, what are the guardrails? Make the default path that the developers take the secure path, and they'll take it.
Superhero culture:
“It was actually a blessing for us that one of our superheroes on our security team that winds up taking on a whole lot, was sitting on a beach with his family at the time of that incident, so we couldn't call him. That was a blessing honestly for us, because it was like, 'Hey, this is life without him, let's go.' And every team I've ever worked on has those superheroes that it's like, no matter what, you wind up calling that person - don't! Figure out how not to. They're gonna leave at some point."
His advice for teams working in incident response:
"If you don't have that automation before the incident, you're toast. These threat actors are very fast. They're very, very fast. I've read incident reports that said we had them shut down within 5 hours - too slow. Their goal was done in the first 30 min. Automate beforehand, practice before it happens. Test your automations, build them and test them.”
Listen to more episodes of the Future of Security Operations podcast.