Victoria and Will interview Rishi Malik, the Founder of Backstop.it and VP of Engineering at Varo Bank. They talk about Rishi's recent adventure at DEF CON, the renowned annual security conference that he's attended for six years, and describes how it has transformed from a mere learning experience into a thrilling competition for him and his team. The conference = their playground for tackling an array of security challenges and brain-teasing puzzles, with a primary focus on cloud security competitions.
They talk about the significance of community in such events and how problem-solving through interaction adds value. Rishi shares his background, tracing his path from firmware development through various tech companies to his current roles in security and engineering management.
The vital topic of security in the fintech and banking sector highlights the initial concerns people had when online banking emerged. Rishi navigates through the technical intricacies of security measures, liability protection, and the regulatory framework that safeguards online banking for consumers. He also highlights the evolving landscape, where technological advancements and convenience have bolstered consumer confidence in online banking.
Rishi shares his unique approach to leadership and decision-making, and pearls of wisdom for budding engineers starting their careers. His advice revolves around nurturing curiosity and relentlessly seeking to understand the "why" behind systems and processes.
__
- Backstop.it
- Follow Backstop.it on X.
- Varo Bank
- Follow Varo Bank on Instagram, Facebook, X, YouTube, or LinkedIn.
- Follow Rishi Malik on LinkedIn.
- Follow thoughtbot on X or LinkedIn.
Become a Sponsor of Giant Robots!
Transcript:
VICTORIA: This is the Giant Robots Smashing Into Other Giant Robots podcast, where we explore the design, development, and business of great products. I'm your host, Victoria Guido.
WILL: And I'm your other host, Will Larry. And with us today is Rishi Malik, Founder of Backstop.it and VP of Engineering at Varo Bank. Rishi, thank you for joining us.
RISHI: Thanks for having me. I'm excited to be here.
VICTORIA: Yes, Rishi. I'm so excited to talk with you today about your security background and get into your role at Varo and Backstop IT. But first, I wanted to hear a little bit more about your recent experience attending DEF CON. How was that?
RISHI: It was awesome. I do have quite the background in security at this point. And one of the things I started doing early on, as I was getting up to speed and learning more about the security-specific side of things, was beginning to attend DEF CON itself. So, I've now gone six years straight.
And it started out as just kind of experiencing the conference and security and meeting folks. But it's progressed to where I now bring a team of people where we go and we compete. We have a good time. But we do get to kind of bring the security side of things into the software engineering and engineering leadership stuff that we all do on a day-to-day basis.
VICTORIA: Yeah. And what kind of puzzles do you solve with your team when you attend DEF CON?
RISHI: There's definitely a lot of variety there, which I think is part of the fun. So, DEF CON frequently has electronic badges, you know, with random puzzles on there that you have to solve. Some of it are cryptographic. Some of them are kind of random cultural things. Sometimes there's music challenges based around it. Sometimes, it's social and interactive. And you have to go find the right type of badge or the right person behind it to unlock something. So, all of those, you know, typically exist and are a ton of fun.
Primarily, in the last few years, we've been focusing more on the cloud CTF. So, in this case, it's our team competing against other teams and really focused on cloud security. So, it's, you know, figuring out vulnerabilities in, you know, specially designed puzzles around AWS and GCP, the application side of things as well, and competing to see how well you can do.
Three years ago, the last couple of years, we've not won it, but we've been pretty competitive. And the great thing is the field is expanding as more and more people get into CTF themselves but, more importantly, into cloud infrastructure and cloud knowledge there. So, it's just great to see that expansion and see what people are into, what people are learning, and how challenging some of these things can be.
VICTORIA: I love the idea of having a puzzle at a conference where you have to find a specific person to solve it. And yeah, I'm always interested in ways where we can have these events where you're getting together and building community and growing expertise in a field but in a way that makes it fun [laughs] and isn't just life-draining long, like, talks about random stuff.
RISHI: [laughs] I think what you're touching on there is crucial. And you said the word community, and, to me, that is, you know, a big part of what DEF CON and, you know, hacking and security culture is. But it is, I think, one of the things that kind of outside of this, we tend to miss it more, you know, specifically, like, focused conferences. It is more about kind of the content, you know, the hallway track is always a thing. But it's less intentional than I personally, at this stage, really prefer, you know. So, I do like those things where it is encouraging interaction.
For me, I'd rather go to happy hour with some people who are really well versed in the subject that they're in rather than even necessarily listening to a talk from them on what they're doing. Simply because I think the community aspect, the social aspect, actually gets you more of the information that is more relevant to what you're doing on a day-to-day basis than just consuming it passively.
VICTORIA: I agree because consuming it passively or even intentionally remotely, there are things that you didn't even think to think about [laughs] that aren't going to come up just on your own. You have to have another person there who's...Actually, I have a good friend who's co-working with me this week who's at Ticketmaster. And so, just hearing about some of the problems they have and issues there has been entertaining for me. So yeah, I love that about DEF CON, and I love hearing about community stories and fun ways that companies can get a benefit out of coming together and just putting good content out there.
RISHI: Absolutely. I think problem-solving is where you get the most value out of it as a company and as a business.
VICTORIA: Yeah, maybe that's a good segue to tell me a little bit more about your background and how you came to be where you are today.
RISHI: Yeah. For me growing up, I was always that problem-solver type of person. So, I think that's what kind of naturally gravitated me towards tech and, you know, hardware and software engineering. You know, so, for me, I go back quite a while. I'd been doing a lot of development, you know, in the early days of my career.
I started out doing firmware development back in the days of large tape libraries, right? So, if you think about, like, big businesses back before cloud was a big thing and even back before SSDs were a thing, you know, it was all spinning disks. It was all tape. And that's kind of the area that I started in. So, I was working on robots that actually move tapes around these giant tape libraries that are, you know, taller than I am that you can walk inside of because they're so big, for big corporations to be able to backup their data on an overnight basis. You have to do that kind of stuff.
Then I started going into smaller and smaller companies, into web tech, into startups, then into venture-backed startups. And then, eventually, I started my own company and did that for a while. All of this is really just kind of, you know, software engineering in a nutshell, lots of different languages, lots of different technologies. But really, from the standpoint of, here's a whole bunch of hard problems that need to be solved. Let's figure out how we can do that and how we can make some money by solving some of these problems.
That eventually kind of led me down the security path as well and the engineering management side of things, which is what I do now, both at Backstop...is a security consulting business and being VP of Engineering at Varo Bank.
WILL: How was your journey? Because you started as an intern in 2003.
RISHI: [laughs]
WILL: And then, you know, 20 years later. So, how was your journey through all of that? [laughs]
RISHI: [laughs] You know, I hadn't actually put it together that it has been 20 years this year until you said that. So, that's awesome. It's been a blast, you know. I can honestly say it's been wildly different than what I imagined 20 years ago and interesting in different ways. I think I'm very fortunate to be able to say that.
When I started out as an intern in 2003, technologies were very different. I was doing some intern shifts with the federal government, you know, so the pace was wildly different. And when I think of where technology has come now, and where the industry has gone, and what I get to do on a day-to-day basis, I'm kind of just almost speechless at just how far we've come in 20 years, how easy some things are, how remarkably hard some other things are that should honestly be easy at this point, but just the things that we can do.
I'm old enough that I remember cell phones being a thing and then smartphones coming out and playing with them and being like, yeah, this is kind of mediocre. I don't really know why people would want this. And the iPhone coming out and just changing the game and being like, okay, now I get it. You know, to the experience of the internet and, you know, mobile data and everywhere. It's just phenomenal the advances that we've had in the last 20 years. And it makes me excited for the next 20 years to see what we can do as we go forward.
VICTORIA: I'm going to take personal offense to someone knowing that technology being too old [laughs], but, yeah, because it really wasn't that long ago. And I think one thing I always think about having a background in civic tech and in financial tech as well is that the future is here; it's just not evenly distributed. So, now, if you're building a new company, of course, the default is to go straight to the cloud.
But many companies and organizations that have been around for 60-80 years and using the internet right when it first came out are still in really old technologies that just simply work. And maybe they're not totally sure why, and change is difficult and slow. So, I wonder if you have any experience that you can take from the banking or fintech industry on how to make the most out of modern security and compliance platforms.
RISHI: Yeah, you know, I think most people in tech especially...and the gray hairs on me are saying the younger folks in tech especially don't realize just how much older technologies still exist and will exist for quite some time.
When you think of banking itself, you know, most of the major companies that you can think of, you know, in the U.S. especially but kind of across the world that are the top tier names of banks, and networks, and stuff like that, still run mainframes. When you swipe your credit card, there's a very good chance that is processed on a mainframe. And that's not a bad thing. But it's just, you know when you talk to younger engineers, it's not something that kind of crosses their mind. They feel like it is old-tech.
The bulk of businesses don't actually run on the cloud. Having been through it, I've racked and stacked servers and had to figure out how to physically take hardware across, you know, country borders and things like those lines. And now, when I do want to spin up a server somewhere else, it's just a different AWS region. So, it's remarkably easy, at this point, to solve a lot of those problems.
But once you're up and live and you have customers, you know, where downtime is impactful or, you know, the cost of moving to the cloud or modernizing your technology is substantial, things tend to move a lot slower. And I think you see that, especially when it comes to security, because we have more modern movements like DevOps bringing security into it. And with a lot of the, you know, the modern security and compliance platforms that exist, they work very, very well for what they do, especially when you're a startup or your whole tech stack is modernized.
The biggest challenges, I think, seem to come in when you have that hybrid aspect of it. You do have some cloud infrastructure you have to secure. You do have some physical data centers you have to secure. You have something that is, you know, on-premise in your office. You have something that is co [inaudible 10:01] somewhere else. Or you also have to deal with stuff like, you know, much less modern tech, you know, when it comes to mainframes and security and kind of being responsible for all of that.
And I think that is a big challenge because security is one of those things where it's, you know, if you think of your house, you can have the strongest locks on your door and everything else like that. But if you have one weak point, you have a window that's left open, that's all it takes. And so, it has to be all-inclusive and holistic. And I think that is remarkably hard to do well, even despite where technology has come to these days.
WILL: Speaking of securities, I remember when the Internet banking started a couple of years ago. And some of the biggest, I guess, fears were, like, the security around it, the safety. Because, you know, your money, you're putting your money in it, and you can't go to a physical location to talk to anyone or anything. And the more and more you learn about it...at first, I was terrified of it because you couldn't go talk to someone. But the more and more I learned about it, I was like, oh, there's so much security around it. In your role, what does that look like for you? Because you have such a huge impact with people's money. So, how do you overcome that fear that people have?
RISHI: There's, I think, a number of steps that kind of go into it. And, you know, in 2023, it's certainly a little bit easier than it used to be. But, you know, very similar, I've had the same questions, you know, and concerns that you're describing. And I remember using one of the first banks that was essentially all digital and kind of wondering, you know, where is my money going? What happens if something goes wrong? And all of those types of things.
And so, I think there is kind of a number of different aspects that go into it. One is, you know, obviously, the technical aspects of security, you know, when you put your credit card number in on the internet, you know, is it encrypted? You know, is it over, you know, TLS? What's happening there? You know, how safe and secure is all that kind of thing?
You know, at this point, pretty much everyone, at least in the U.S., has been affected by credit card breaches, huge companies like Home Depot and Target that got cards accessed or, you know, just even the smaller companies when you're buying something random from maybe something...a smaller website on the internet. You know, that's all a little bit better now. So, I think what you have there was just kind of a little bit of becoming comfortable with what exists now.
The other aspect, though, I think, then comes into, well, what happens when something goes wrong? And I think there's a number of aspects that are super helpful for that. I think the liability aspect of credit card, you know, companies saying, you know, and the banks "You're not liable for a fraudulent transaction," I think that was a very big and important step that really helps with that.
And on top of that, then I think when you have stuff like the FDIC, you know, and insurance in the U.S., you know, that is government-backed that says, you know what? Even if this is an online-only digital bank, you're safe. You're protected. The government's got your back in that regard. And we're going to make sure that's covered. At Varo, that's one of the key things that we think about a lot because we are a bank.
Now, most FinTechs, actually, aren't banks, right? They partner with other third-party banks to provide their financial services. Whereas at Varo, we are federally regulated. And so, we have the full FDIC protection. We get the benefits of that. But it also means that we deal with the regulation aspects and being able to prove that we are safe and secure and show the regulators that we're doing the right things for our customers.
And I think that's huge and important because, obviously, it's safety for customers. But then it changes how you begin to think about how you're designing products, and how you're [inaudible 13:34] them, and, you know, how you're marketing them. Are we making a mobile app that shows that we're safe, and secure, and stable? Or are we doing this [inaudible 13:42] thing of moving too fast and breaking things?
When it's people's money, you have to be very, very dialed into that. You still have to be able to move fast, but you have to show the protection and the safety that people have because it is impactful to their lives. And so, I think from the FinTech perspective, that's a shift that's been happening over the last couple of years to continue that.
The last thing I'll say, too, is that part of it has just come from technology itself and the comfort there. It used to be that people who were buying, you know, items on the internet were more the exception rather than the rule. And now with Amazon, with Shopify, with all the other stuff that's out there, like, it's much more than a norm. And so, all of that just adds that level of comfort that says, I know I'm doing the right things as a consumer, that I'm protected. If I, you know, do have problems, my bank's got my back. The government is watching out for what's happening and trying to do what they can do to regulate all of that.
So, I think all of that has combined to get to that point where we can do much more of our banking online and safely. And I think that's a pretty fantastic thing when it comes to what customers get from that. I am old enough that I remember having to figure out times to get to the bank because they're open nine to five, and, you know, I have to deposit my paycheck. And, you know, I work nine to five, and maybe more hours pass, and I had no idea when I can go get that submitted.
And now, when I have to deposit something, I can just take a picture with my phone, and it safely makes it to my account. So, I think the convenience that we have now is really amazing, but it has certainly taken some time. And I think a number of different industry and commercial players kind of come together and make that happen.
MID-ROLL AD:
Now that you have funding, it's time to design, build, and ship the most impactful MVP that wows customers now and can scale in the future.
thoughtbot Liftoff brings you the most reliable cross-functional team of product experts to mitigate risk and set you up for long-term success.
As your trusted, experienced technical partner, we'll help launch your new product and guide you into a future-forward business that takes advantage of today's new technologies and agile best practices.
Make the right decisions for tomorrow today. Get in touch at thoughtbot.com/liftoff.
VICTORIA: I appreciate that perspective on approaching security from the user experience of wanting safety. And I'm curious if we can talk in contrast from that experience to the developer experience with security. And how do you, as a new leader in this financial product company, prioritize security and introduce it from a, like, building a safety culture perspective?
RISHI: I think you just said that very eloquently. It is a safety culture. And cultural changes are hard. And I think for quite some time in the developer industry, security was either an afterthought or somebody else's problem. You know, it's the security team that has to think about it. It's, you know, and even these days, it's the red team that's going to go, you know, find these answers or whatever I'm shipping as a developer. My only thing to focus on is how fast I can ship, or, you know, what I'm shipping, rather than how secure is what I'm shipping.
And so, I think to really be effective at that, it is a cultural shift. You have to think and talk about security from the outset. And you have to bake those processes into how you build product. Those security conversations really do need to start at the design phase. And, you know, thinking about a mobile app for a bank as an example, you know, it starts when you're just thinking about the different screens on a mobile app that people are going to go through. How are people interpreting this? You know, what is the [inaudible 17:23], and the feeling, and the emotions, that we're building towards? You know, is that safe and secure or, you know, is it not?
But then it starts getting to the architecture and the design of the systems themselves to say, well, here's how they're going to enter information, here's how we're passing this back and forth. And especially in a world where a lot of software isn't just 100% in-house, but we're calling other partners for that, you know, be it, you know, infrastructure or risk, you know, or compliance, or whatever else it may be, how are we protecting people's data? How are we making sure our third parties are protecting people's data? You know, how are we encrypting it? How are we thinking about their safety all the way through?
Again, even all the way down to the individual developer that's writing code, how are we verifying they're writing good, high-quality, secure code? Part of it is training, part of it is culture, part of it is using good tooling around that to be able to make sure and say, when humans make mistakes because we are all human and we all will make mistakes, how are we catching that?
What are the layers do we have to make sure that if a mistake does happen, we either catch it before it happens or, you know, we have defense in depth such that that mistake in and of itself isn't enough to cause a, you know, compromise or a problem for our customers? So, I think it starts right from the start. And then, every kind of step along the way for delivering value for customers, also let's add that security and privacy and compliance perspective in there as well.
VICTORIA: Yes, I agree. And I don't want to work for a company where if I make a small human mistake, I'm going to potentially cost someone tens or however many thousands of dollars. [laughs]
WILL: I have a question around that. How, as a leader, how does that affect you day to day? Because I feel like there's some companies, maybe thoughtbot, maybe other companies, that a decision is not as critical as working as a bank. So, you, as a leader, how do you handle that?
RISHI: There's a couple of things I try and consider in any given big or important decision I have to make, the aspects around, like, you know, the context, what the decision is, and that type of stuff. But from a higher level, there's kind of two things I try and keep in mind. And when I say keep in mind, like, when it's a big, impactful decision, I will actually go through the steps of, you know, writing it down or talking this out loud, sometimes by myself, sometimes with others, just, again, to make sure we are actually getting to the meat of it.
But the first thing I'm trying to think of is kind of the Amazon idea of one-way versus two-way doors. If we make this decision and this is the wrong decision, what are the ramifications of that? You know, is it super easy to undo and there's very little risk with it? Or is it once we've made this decision or the negative outcome of this decision has happened, is it unfixable to a certain degree?
You know, and that is a good reminder in my head to make sure that, you know, A, I am considering it deeply. And that, B, if it is something where the ramifications, you know, are super huge, that you do take the time, and you do the legwork necessary to make sure you're making a good, valid decision, you know, based on the data, based on the risks involved and that there's a deep understanding of the problem there.
The second thing I try to think of is our customers. So, at Varo, our customers aren't who most banks target. A lot of banks want you to take all your money, put it in there, and they're going to loan that money out to make their money. And Varo is not that type of bank, and we focus on a pretty different segment of the market. What that means is our customers need their money. They need it safely and reliably, and it needs to be accurate when they have it.
And what I mean by that is, you know, frequently, our customers may not have, you know, hundreds or a thousand dollars worth of float in their bank accounts. So, if they're going and they're buying groceries and they can't because there's an error on our side because we're down, and because the transactions haven't settled, then that is very, very impactful to them, you know, as an individual.
And I think about that with most of these decisions because being in software and being in engineering I am fortunate enough that I'm not necessarily experiencing the same economic struggles that our customers may have. And so, that reminder helps me to think about it from their perspective. In addition, I also like to try and think of it from the perspective...from my mom, actually, who, you know, she is retired age. She's a teacher. She's non-technical.
And so, I think about her because I'd say, okay, when we're making a product or a design decision, how easy is it for her to understand? And my biases when I think about that, really kind of come into focus when I think about how she would interpret things. Because, you know, again, for me, I'm in tech. I think about things, you know, very analytically. And I just have a ton of experience across the industry, which she doesn't have. So, even something as simple as a little bit of copy for a page that makes a ton of sense to me, when I think about how she would interpret it, it's frequently wildly different.
And so, all of those things, I think, kind of come together to help make a very strong and informed decision in these types of situations where the negative outcomes really do matter. But you are, you know, as Varo is, you're a startup. And you do need to be able to build more products quickly because our customers have needs that aren't being met by the existing banking industry. And so, we need to provide value to them so that their lives are a bit better.
VICTORIA: I love that focus on a specific market segment and their needs and solving for that problem. And we know that if you're at a certain income level, it's more expensive [laughs] because of the overdraft fees and other things that can cause you problems. So, I really appreciate that that's the mission at Varo, and that's who you're focusing on to create a better banking product that makes more sense.
I'm curious if there were any surprises and challenges that you could share from that discovery process and finding out, you know, exactly what were those things where your mom was, like, uh, actually, I need something completely different. [laughs]
RISHI: Yeah, so, [chuckles] I'm chuckling because, you know, it's not, like, a single kind of time or event. It's, you know, definitely an ongoing process. But, you know, as actually, we were talking, you know, about earlier in terms of being kind of comfortable with doing things digital and online, that in and of itself is something that even in 2023, my mom isn't as comfortable or as confident as, you know, say, maybe the three of us are.
As an example, when sending money, you know, kind of like a peer-to-peer basis, like, if I'm sending my mom a little bit of money, or she's sending me something, you're kind of within the family. Things that I would think would be kind of very easy and straightforward actually do cause her a little bit more concern. Okay, I'm entering my debit card number into this so that it can get, you know, the cash transferred into my bank account. You know, again, for me, it didn't even cross my mind, actually, that that would be something uncomfortable. But for my mom, that was something where she actually had some concerns about it and was messaging me.
Her kind of personal point of view on that was, I would rather use a credit card for this and get the money on a credit card instead of a debit card because the debit card is linked to a bank account, and the security around that needs to be, you know, much tighter. And so, it made her more uncomfortable entering that on her phone. Whereas even a credit card it would have given her a little bit more peace of mind simply because it wasn't directly tied to her bank account.
So, that's just, you know, the most recent example. I mean, honestly, that was earlier today, but it's something I hadn't thought of. And, again, for most of our customers, maybe that's not the case and how they think. But for folks that are at that retirement age, you know, in a world where there are constant barrages of scam, you know, emails, and phone calls, and text messages going around, the concern was definitely there.
VICTORIA: That happened to me. Last week, I was on vacation with my family, and we needed to pay my mom for the house we'd rented. And I had to teach her how to use Zelle and set up Zelle. [laughter] It was a week-long process. But we got there, and it works [laughs] now. But yeah, it's interesting what concerns they have. And the funny part about it was that my sister-in-law happens to be, like, a lawyer who prevents class action lawsuits at a major bank. And she reassured us that it was, in fact, secure. [laughs]
I think it's interesting thinking about that user experience for security. And I'm curious, again, like, compare again with the developer experience and using security toolings. And I wonder if you had any top recommendations on tools that make the developer experience a little more comfortable and feeling like you're deploying with security in mind.
RISHI: That, in particular, is a bit of a hard question to answer. I try and stay away from specific vendors when it comes to that because I think a lot of it is contextual. But I could definitely talk through, like, some of the tools that I use and the way I like to think about it, especially from the developer perspective.
I think, first off, consider what aspect of the software development, you know, lifecycle you're in. If you are an engineer writing, you know, mostly application code and dealing with building product and features and stuff like that, start from that angle. I could even take a step back and say security as an industry is very, very wide at this point. There is somebody trying to sell you a tool for basically every step in the SDLC process, and honestly, before and after to [inaudible 26:23]. I would even almost say it's, to some extent, kind of information and vendor overload in a lot of ways.
So, I think what's important is to think about what your particular aspect of that is. Again, as an application engineer, or if you're building cloud infrastructure, or if you're an SRE, you know, or a platform team, kind of depending on what you are, your tooling will be different. The concepts are all kind of similar ideas, but how you go about what you build will be different.
In general, I like to say, from the app side of things, A, start with considering the code you're writing. And that's a little bit cultural, but it's also kind of more training. Are you writing code with a security mindset? are you designing systems with a security mindset? These aren't things that are typically taught, you know, in school if you go get a CS degree, or even in a lot of companies in terms of the things that you should be thinking about. So, A, start from there.
And if you don't feel like you think about, you know, is this design secure? Have we done, you know, threat modeling on it? Are we considering all of the error paths or the negative ways people can break the system? Then, start from that and start going through some of the security training that exists out there. And there's a lot of different aspects or avenues by which you can get that to be able to say, like, okay, I know I'm at least thinking about the code I write with a security mindset, even if you haven't actually changed anything about the code you're writing yet.
What I actually think is really helpful for a lot of engineers is to have them try and break things. It's why I like to compete in CTFs, but it's also why I like to have my engineers do the same types of things. Trying to break software is both really insightful from the aspect that you don't get when you're just writing code and shipping it because it's not something you have time to do, but it's also a great way to build up some of the skills that you need to then protect against.
And there's a lot of good, you know, cyber ranges out there. There's lots of good, just intentionally vulnerable applications that you can find on GitHub but that you can just run, you know, locally even on your machine and say, okay, now I have a little web app stood up. I know this is vulnerable. What do I do? How do I go and break it? Because then all of a sudden, the code that you're writing you start to think about a little bit differently.
It's not just about how am I solving this product problem or this development problem? But it's, how am I doing this in a way that is safe and secure? Again, as an application side of things, you know, just make sure you know the OWASP Top 10 inside and out. Those are the most basic things a lot of engineers miss. And it only takes, again, one miss for it to be critical. So, start reviewing it.
And then, you start to think about the tooling aspect of it. People are human. We're going to make mistakes. So, how do we use the power of technology to be able to stop this? You know, and there is static scanning tools. Like, there's a whole bunch of different ones out there. You know, Semgrep is a great one that's open source just to get started with that can help you find the vulnerable code that may exist there.
Consider the SQL queries that you're writing, and most importantly, how you're writing them. You know, are you taking user input and just chucking it in there, or are you sanitizing it? When I ask these questions, for a lot of engineers, it's not usually yes or no. It's much more of an, well, I don't know. Because in software, we do a really good job of writing abstraction layers. But that also means, you know, to some extent, there may be a little bit of magic in there, or a lack thereof of magic that you don't necessarily know about.
And so, you have to be able to dive into the libraries. You have to know what you're doing to even be able to say something like, oh no, this SQL query is safe from this user input because we have sanitized it. We have, you know, done a prepared statement, whatever it may be. Or, no, actually, we are just doing something here that's been vulnerable, and we didn't realize we were, and so now that's something we have to address.
So, I think, like, that aspect in and of itself, which isn't, you know, a crazy ton of things. It's not spending a ton of money on different tools. But it's just internalizing the fact that you start to think a little bit differently. It provides a ton of value.
The last thing on that, too, is to be able to say, especially if you're coming from a development side, or even just from a founder or a startup side of things, what are my big risks? What do I need to take care of first? What are the giant holes or flaws? You know, and what is my threat model around that?
Obviously, as a bank, you have to care very deeply right from the start. You know, if you're not a bank, if you're not dealing with financial transactions, or PII, or anything like that, there are some things that you can deal with a little bit later. So, you have to know your industry, and you have to know what people are trying to do and the threat models and the threat vectors that can exist based on where you are.
WILL: That's amazing. You know, earlier, we talked about you being an engineer for 20 years, different areas, and stuff like that. Do you have any advice for engineers that are starting out right now? And, you know, from probably year one to year, you know, anything under ten years of experience, do you have any advice that you usually give engineers when you're chatting with them?
RISHI: The advice I tend to give people who are just starting out is be the type of person that asks, "How does this work?" Or "Why does this work?" And then do the work to figure out the answer. Maybe it is talking to someone; maybe it's diving into the details; maybe it's reading a book in some aspect that you haven't had much exposure to.
When I look at my career and when I look at the careers of folks around me and the people that I've seen be most successful, both in engineering but also on the business side, that desire to know why something is the case is I think, one of the biggest things that determines success. And then the ability to answer that question by putting in the right types of work, the right types of scientific method and processes and such, are the other factor. So, to me, that's what I try and get across to people.
I say that mostly to junior folks because I think when you're getting started, it's really difficult. There's a ton out there. And we've, again, as software engineers, and hardware engineers, and cloud, and all this kind of stuff, done a pretty good job of building a ton of abstraction layers. All of our abstraction layers [inaudible 32:28] to some degree.
You know, so as you start, you know, writing a bunch of code, you start finding a bunch of bugs that you don't necessarily know how to solve and that don't make any sense in the avenue that you've been exposed to. But as soon as you get into the next layer, you understand how that works begin to make a lot more sense. So, I think being comfortable with saying, "I have no idea why this is the case, but I'm going to go find out," makes the biggest difference for people just starting out their career.
WILL: I love that advice. Not too long ago, my manager encouraged me to write a blog post on something that I thought that I really knew. And when I started writing that blog post, I was like, oh boy, I have no idea. I know how to do it, but I don't know the why behind it. And so, I was very thankful that he encouraged me to write a blog post on it. Because once you start explaining it to other people, I feel you really have to know the whys. And so, I love that advice. That's really good advice.
VICTORIA: Me too. And it makes sense with what we see statistically as well in the DORA research. The DevOps Research Association publishes a survey every year, the State of DevOps Report. And one of the biggest findings I remember from last year's was that the most secure and reliable systems have the most open communication and high trust among the teams. And so, being able to have that curiosity as a junior developer, you need to be in an environment where you can feel comfortable asking questions [laughs], and you can approach different people, and you're encouraged to make those connections and write blog posts like Will was saying.
RISHI: Absolutely, absolutely. I think you touched on something very important there as well. The psychological safety really makes a big difference. And I think that's critical for, again, like, folks especially earlier in their career or have recently transitioned to tech, or whatever the case may be. Because asking "Why?" should be something that excites people, and there are companies where that's not necessarily the case, right? Where you asking why, it seems to be viewed as a sign that you don't know something, and therefore, you're not as good as what you should be, you know, the level you should be at or for whatever they expect.
But I do think that's the wrong attitude. I think the more people ask why, the more people are able and comfortable to be able to say, "I don't know, but I'm going to go find out," and then being able to be successful with that makes way better systems. It makes way safer and more secure systems. And, honestly, I think it makes humans, in general, better humans because we can do that.
VICTORIA: I think that's a great note to start to wrap up on. Is there any questions that you have for me or Will?
RISHI: Yeah. I would love to hear from both of you as to what you see; with the experiences that you have and what you do, the biggest impediments or speed bumps are when it comes to developers being able to write and ship secure code.
VICTORIA: When we're talking with new clients, it depends on where they are in really the adoption of their product and the maturity of their organization. Some early founders really have no technology experience. They have never managed an IT organization. You know, setting up basic employee account access and IDs is some of the initial steps you have to take to really get to where you can do identity management, and permissions management, and all the things that are really table stakes for security.
And then others have some progress, and they have a fair amount of data. And maybe it's in that situation, like you said before, where it's really a trade-off between the cost and benefit of making those changes to a more secure, more best practice in the cloud or in their CI/CD pipeline or wherever it may be.
And then, when you're a larger organization, and you have to make the trade-offs between all of that, and how it's impacting your developer experience, and how long are those deployed times now. And you might get fewer rates of errors and fewer rates of security vulnerabilities. But if it's taking three hours for your deployments to go out [laughs] because there's so many people, and there's so many checks to go through, then you have to consider where you can make some cuts and where there might be more efficiencies to be gained.
So, it's really interesting. Everyone's on a different point in their journey. And starting with the basics, like you said, I love that you brought up the OWASP Top 10. We've been adopting the CIS Controls and just doing a basic internal security audit ourselves to get more ready and to be in a position where...
What I'm familiar with as well from working in federal agencies, consulting, maintaining some of the older security frameworks can be a really high cost, not only in terms of auditing fees but what it impacts to your organization to, like, maintain those things [laughs] and the documentation required. And how do you do that in an agile way, in a way that really focuses on addressing the actual purpose of the requirements over needing to check a box? And how do we replicate that for our clients as well?
RISHI: That is super helpful. And I think the checkbox aspect that you just discussed I think is key. It's a difficult position to be in when there are boxes that you have to check and don't necessarily actually add value when it comes to security or compliance or, you know, a decrease in risk for the company. And I think that one of the challenges industry-wide has always been that security and compliance in and of itself tends to move a little bit slower from a blue team or a protection perspective than the rest of the industry.
And so, I mean, I can think of, you know, audits that I've been in where, you know, just even the fact that things were cloud-hosted just didn't make sense to the auditors. And it was a struggle to get them to understand that, you know, there is shared responsibility, and this kind of stuff exists, and AWS is taking care of some things, and we're taking care of some other things when they've just been developed with this on-premise kind of mentality.
That is one of the big challenges that still exists kind of across the board is making sure that the security work that you're doing adds security value, adds business value. It isn't just checking the box for the sake of checking the box, even when that's sometimes necessary.
VICTORIA: I am a pro box checker.
RISHI: [laughs]
VICTORIA: Like, I'll get the box checked. I'll use Trello and Confluence and any other tool besides Excel to do it, too. We'll make it happen with less pain, but I'd rather not do it [laughs] if we don't have to.
RISHI: [laughs]
VICTORIA: Let's make it easy. No, I love it. Is there anything else that you want to promote?
RISHI: No, I don't think there's anything else I want to promote other than I'm going to go back to what I said just earlier, like, that culture. And if, you know, folks are out there and you have junior engineers, you have engineers that are asking "Why?", you have people that just want to do the right thing and get better, lean into that. Double down on those types of folks. Those are the ones that are going to make big differences in what you do as a business, and do what you can to help them out. I think that is something we don't see enough of in the industry still. And I would love for that to change.
VICTORIA: I love that. Thank you so much, Rishi, for joining us.
RISHI: Thanks for having me. This was a great conversation. I appreciate the time.
VICTORIA: You can subscribe to the show and find notes along with a complete transcript for this episode at giantrobots.fm. If you have questions or comments, email us at hosts@giantrobots.fm. And you can find me on Twitter @victori_ousg.
WILL: And you could find me on Twitter @will23larry.
This podcast is brought to you by thoughtbot and produced and edited by Mandy Moore. Thanks for listening. See you next time.
ANNOUNCER: This podcast is brought to you by thoughtbot, your expert strategy, design, development, and product management partner. We bring digital products from idea to success and teach you how because we care. Learn more at thoughtbot.com.
Support Giant Robots Smashing Into Other Giant Robots