The Current State Of Protecting Industrial Systems and Safeguarding Civilization Today-An Interview with Joe Slowik, Adversary Hunter at Dragos
The Current State Of Protecting Industrial Systems and Safeguarding Civilization Today, With Joe Slowik.
In this episode, we welcome Joe Slowik. Joe is the threat and adversary hunter at Dragos and has extensive network security and computer network operations experience spanning the military, intelligence, and nuclear communities. In this episode, Joe takes us beyond the often sensational headlines and misconceptions to walk us through the real challenges and current state of protecting industrial systems and critical infrastructure in our world today. We learn more about what we should aspire to when it comes to industrial control systems and why we need to develop a more analytical approach to threat behavior. Joe shares with us why the Dragos company motto is “safeguarding civilization,” as well as their methodology to detect threats and provide the context, tools, and knowledge to respond to attacks with speed and confidence. Could we be safer than we think? Take a listen to find out more.
Key Points From This Episode:
• Learn more about Joe Slowik and his non-traditional CS Background.
• Joe gives his overview of the current thought around industrial controls.
• Find out how we defend industrial control systems today.
• How can attacks be actualized to impact an ICS environment?
• Script locking and reevaluating credential storage and credential use.
• Adopting a strategic perspective and designing network defense.
• Discover more about the Perdue model and what this means for defense.
• Tackling the misconception that the attacker only needs to get it right once.
• Who are getting industrial control systems right and what to aspire to.
• Why we need to develop a more analytical approach to threat behavior.
• How to empower individuals to respond and react to threats as they arise.
• Learn more about the Dragos company motto of safeguarding civilization.
• And much more!
Links Mentioned in Today’s Episode:
OpCde – http://www.opcde.com/
Joe Slowik on Twitter – https://twitter.com/jfslowik
Joe Slowik on LinkedIn – https://www.linkedin.com/in/joe-slowik/
Dragos – https://dragos.com/
In this interview CSD sits down with Joe Slowik, adversary hunter at Dragos, Joe takes us beyond the headlines to walk us through the real challenges and state of protecting industrial systems and critical infrastructure.
Interview Transcript
Joe, thanks for coming on. Just introduce yourself.
Sure, my name is Joe Slowik, currently, I’m an adversary hunter at Dragos. Dragos is an industrial control system security startup right now, been there for about eight months, adversary hunter’s kind of a loaded term, what does that mean. Really, it’s threat intelligence work but really taking more of the view that the adversary is something that can be hunted and pursued as supposed to just simply ingesting information passively and reporting on things that had already happened.
But trying to adopting more for landing approach and that’s something that ties in fairly nicely to how I’ve worked in information security for my career so prior to joining Dragos I was running the incident response team at Los Alamos national lab. That was a fun and rewarding time, frustrating position, had a great team there but really a lot of the emphasis in what we’re trying to do is to adopt that same sort of forward leaning, more active defense role in terms of not just responding to things but actively seeking out and trying to stay ahead of the adversary.
Which is something I think is near and dear to your heart as well. Prior to joining the lab, I was a US naval officer, and information war officer where I did some network defense, some network other things and a little bit of time on ship, et cetera and before that even, sort of the long and winding road to get into information security.
I worked for an industrial supply company in suburban Chicago doing data analytics work and before that, I dropped out of graduate school as a philosophy student. It’s been a different sort of course to get here in terms of not having a traditional CS background.
Although, I would say the majority of the people we chat with in this world don’t have that traditional background, right?
Right.
It’s partly what makes this space really fun and interesting. Yeah, I mean, industrial controls are kind of certainly top of mind for a lot of people these days, right? For someone who is not as immersed in that world, what are the things that you are seeing people kind of scared about, talked about, excited about, right?
Right, the concern always, especially when you start getting into individuals who are less informed or don’t have as complete of an understanding of the underpinnings of things like the electric grid or waste water treatment in water treatment is oh my goodness, something network related is going to result in cascading blackouts across the country or poisoning water or cause machines in the factory to spin out of control and kill someone.
You know, certainly valid concerns to a point but the – this is walking the company line so to speak but it’s one that I share as well and all the reasons I work there is always adopting a nuance for the threats are real but putting the risk in the context of we have at least in the United States and you know, most of the western world very resilient critical infrastructure.
It’s not a trivial sort of action, there are some adversary to say, turn the lights out for the entire eastern seaboard that would be quite an amazing thing to actually execute and practice the amount of pre-planning, coordination and just getting the decision right multiple times across multiple infrastructures would be something that I would be very curious to meet that adversary and probably be quite in awe of them.
“while we understand that there are risks out there, certainly we’ve seen an increase in identified industrial control system attacks, I say identify because I’m almost certain that there are others out there that we just haven’t found or even attack the organization might not even be aware ”
Really making sure that while we understand that there are risks out there, certainly we’ve seen an increase in identified industrial control system attacks, I say identify because I’m almost certain that there are others out there that we just haven’t found or even attack the organization might not even be aware that some event was a cyber sourced incident that we’re seeing more and more attention paid to this space whether it’s in the general media or within the security community because it is a concern.
These are things that we care about and near and dear to our everyday life.
Yeah, I mean, I think that certainly, it’s good to hear someone who is deeply involved, you’re not as concerned about that sort of disaster scenario, at least for Joe Q Public, which I am going to include myself in for the most part, that’s comforting. I do thing the scale of – the potential issues and sort of essentially taking cyber attacks to a kinetic to an actual sort of world, that is something that we’re starting to see around the world, right?
Whether that’s the Ukraine, some of the stuff that happened in Saudi Arabia as well. I was reading an article you were talking about, some of the approaches to potentially defend against those, right? Particularly the two new exploits that came out with chips, essentially,
For those people who didn’t have a chance to read that, what was your sort of takeaways or how do you defend those sort of industrial control systems?
Industrial control systems are always a difficult spot to be in because our typical advice is, there’s a vulnerability, go patch it. Well, in an ICS environment, you can’t simply take a system offline that’s in charge of either monitoring the safety, reliability or operating a steam turbine in a power plant. Instead, you have to wait for a scheduled maintenance period which maybe that’s six months from now.
Maybe it’s two years from now, maybe you can never patch because it’s a system that was built to a very specific specification and wasn’t designed to ever really be updated until it’s brought out of that environment and replaced. When thinking about something like Specter and Meltdown which those have been interesting as well because the patches in play have had a rather interesting affect in environments where the –
The blue screen of death.
Yeah, among other things. Really, my advice for that with the understanding and expectation that asset owners wouldn’t be able to patch their systems that really, it’s approaching it from a defense in depth perspective that you know, hopefully, this isn’t always the case, you don’t have these systems as being internet facing.
The sorts of threats that you would need to worry about from a web application server that’s sitting, open up to the entire internet or from a standard user doing normal browsing activity. Those scenarios shouldn’t apply, if they do, that’s a concern in and of itself that needs to be addressed. Then, looking at the next step, well how could these attacks be actualized to cause an impact within an ICS environment.
Well, one way like potential attack scenario I brought out for meltdown inspector, specter especially which is one of the ones that would be unpatched in that the patch itself, efficacy is a little debatable is you know, you have an engineer who is on their work PC, laptop, et cetera and they are using a web based VNC client in order to configure and manage something within the industrial control network which is traditionally separated from the regular IT network.
Well, in another browser tab, they’re browsing Reddit or some other form. It could even be something perfectly legitimate like a forum for vendor questions or something and there’s a malicious I-Frame embedded into it that loads up code to start dumping memory from the browser process and the tab, the separate browser is writing in the same process and so you can dump credentials for that remote communication attempt.
Well, be aware of that risk so both as a hygiene perspective, you know, pushing things like added script locking, I know a lot of people hate saying that because that defeats how many places can monetize having content hosted on the web but considering some of the risks, you probably want that on your critical personnel though that might be doing this sort of processes or reevaluate credential storage, credential use, evaluating things like two factor authentication so that even if you do find yourself in a situation where credentials have been linked that there is a fallback that provides an additional layer of security.
Although it’s interesting from the session cookie capture that even that doesn’t necessarily hold because you can replay the session as long as that session is still active. A number of things but at least identifying whether it’s a Specter in Meltdown exploit which I still haven’t seen any of that appear to be more than a proof of concept but that could just mean we haven’t seen it yet, it’s out there.
You know, to something like a crash override or a Tricis event that you know, these things need to migrate from that sort, some attacker owned infrastructure which is setting somewhere on the internet, get in to the IT network for the target organization, identify where to drill down from the IT network into your industrial control network and maintain some sort of command and control through that entire process and have enough understanding once they get to the ICS environment for how to operate and how to deliver a malicious impact.
That entire chain is a non-trivial amount of work which is why when I say earlier that I’m not really worried about someone taking out every major power station or distribution station in the eastern United states because one, these environments are somewhat different from each other but two, just the level of work and effort required to maintain what would be a relatively trivial initial footprint than an IT network involves almost like another set of pivoting exploit and gaining access into another network.
Now, certainly when you have misconfigurations and this is something that is always a concern to us and recommending that asset owners identify these, you have a vendor product, that beacons home directly via a GPRS link that’s not terribly well secured to receive periodic updates or to send telemetry, that’s a problem and these are the things that need to be addressed but when it comes to some of the things that need to be addressed but when it comes to some of the really critical infrastructure items.
Those don’t seem to be as big of an issue instead of saying maybe a medium sized factory environment or some other use cases, it’s always exceptions to these but at the same time, for every scary article you see about, my goodness, we found all of these devices accessible on showdown. Yeah, you might have but there’s an awful lot more that are in a much better spot.
Maybe the story’s not quite as bad as the headline.
Right, sometimes you look at those stories too and you look at the devices that are identified and like half of these are actually honey pots but I’m not going to cast aspersions on anyone’s research there but it’s good to know when you’re returning these results just going to take the meta data but get what the actual results that includes.
No, I mean, wow, you hit on so many of the things that we talk about, think about in terms of how do you segment your potential network, right? The choices you make, right? If you’re going to network things, you don’t necessarily have to take it all the way to the internet, right? You don’t have to make it open and particularly any of that sort of critical infrastructure you're concerned about.
But still provide access, right? Then something that we have a lot of conversations about is can you provide access in a way, whether that’s a separate browser or you know, we even talk about using sort of virtual machines, do you provide a clean environment, a sort of completely segmented sort of clean build so that you don’t have that sort of cross scripting issues, right?
Even if you had a root access and the potential.
Or implementing things like jump posts, Bastian hosts one of the concepts that I like to really hit hard on, I’ll be referencing it, I talk here, teaching a two day class on this at the op code event in Dubai coming up in April is adopting a strategic perspective and designing network defense and when I say strategic, it’s really moving out of the day to day operations of how do I respond to events in question but it’s taking it to a higher level of how do I organize my environment and my response procedures so that I’m tuning what I can do and how I will respond to my likely threat environment.
“Increasing either physical or logical segmentation, probably logical since that’s easier to create more choke points within the network where you can do things like identify sensors, and establish white and black lists or access controls.”
One example of that would be you know, increasing either physical or logical segmentation, probably logical since that’s easier to create more choke points within the network where you can do things like identify sensors, establish white and black list or access control this and then the same sort of natural difficulty already in doing in IT to ICS migration, increases further by creating sub nets and that’s simply the logical model for how an ICS network is setup, it’s something called a Perdue model.
Which is a zero to five, zero being your physical controllers up to layer five is your corporate IT network. You know, not having those layers be flat in themselves but you know, further dividing up these so that it’s not trivially easy to move between one or the other.
And you would get the added bonus of potentially adding monitoring solutions to make up for a lack of host visibility between any one item because you’ll have the possibility of capturing lateral movement to attempts to communication between them.
Yeah, no. I think really interesting stuff in terms of how do you make it potentially more challenging and expensive for an attacker to penetrate and move throughout your system? It’s not, are we secure or not secure, it’s not a binary sort of decision, its’ really, how do you kind of make the challenges for your most critical, your most sort of sensitive items, the most kind of challenging expensive for an attacker to hit.
Right, you know, one thing, my boss in Dragos and founder of CEO Rob Lee, myself and other in the company always kind of repeat is you know, there’s this misconception that the attacker only needs to get it right once and the defender needs to get it right every time.
That’s not necessarily the case because you know, you flip that script and really you, especially if you had a background in doing offense that the adversary really needs to get it right every step of the way and if you're just talking about detecting them, the defender really only needs to get it right once.
If you increased the number of times that that adversary must get it right, you know, whether you’re talking about just a natural IT to ICS migration. Making it even harder by increasing the amount of effort, work and steps required to achieve an effect, you’ve just enabled the defender now, it’s like, I don’t just have to possibly catch you at maybe three or four steps but rather 15 or 16 sorts of instances where the adversary must get those right and sustain that success and the defender hopefully catches things earlier. That’s always preferable but at the very least, they are getting multiple bites at the apple for an analogy.
Yeah, I mean that is sort of defense in depth in practice, right? Yeah I think that’s really interesting and we see that in the people that we talked to right? Like thinking about multiple steps that you need to jump through. We are often trying to implement sort of dynamic piece as well, right? So even if you’ve unlocked 10 of the keys suddenly key number 11 is different today than it was yesterday and it will be tomorrow.
So it’s just incredibly frustrating for an attacker to get there and then suddenly all the work that they did is suddenly useless, right? That’s really interesting. I think it’s when you think about the kind of people who are doing this well and you know in the space you can’t ultimately name names but sort of what are the things that you are seeing particularly in industrial control systems whether that’s particular vendors or particular users.
What are they getting right and who do you put up and examples you put up as sort of things to aspire to?
Okay, I am not going to get into specifics unfortunately sorry.
Oh I didn’t expect you to.
Yeah but as a community especially as you are talking about a community that consists of a large if not a majority of members who are not IT, IT security backgrounds but engineers, process engineers, mechanical engineers, etcetera that operate with these environments or even more dirt under the fingernails like we know how to operate the machinery and make the environment work but cyber security is something new is the amount of awareness concern and desire to solve these problems is it’s there.
So you see a lot of these sort of headline grabbing stories about these environments or insecure, they’re insecure by design. We are setting ourselves up for disasters that really if you look at the individuals who are involved in the day to day operation of these environments, they are aware of these issues and working diligently to resolve them as best as they can given the limitations required and how these environments operate and what amount of flexibility exists.
So certainly and like I said, it’s industry wide or at least vertically wide that people are paying attention to stories like crisis, like crash override, like the 2015 Ukraine event then realize that, “Huh this is something that it might be at the other side of the world but they’re not taking the view point of all of California goes dark at one point in time” but a single plant that’s very doable and there is the recognition of what do we need to do.
How can we get better to make sure that that isn’t us and I think that that is something that’s being sustained and hopefully is getting pushed forward to identify ways to solve these problems.
Yeah, I mean it’s a challenge right? Though I heard somebody else that I was talking with the sort of typical life, a lot of these equipment is 35 years right? I mean you imagine you roll back the clock that gets us to about to 1980 and so a pretty scary way. If you think about what are their computer systems that we are running in 1980 and not really expecting to be networked for sure.
Right but even then just having an idea for – you know there is an advantage to an ICS environment and that the systems are designed to do one or a few things. It’s not like say Windows work station that someone uses it for their day to day work where they’re playing solitaire, using Excel, browsing the internet and using one or two or three chat platforms. Instead these are systems designed to operate within early narrow tolerance is an expectation of what they do.
And having an understanding of not just what those limitations are because then you can go down the road of not only a detection. I think there is some value there but I also think that’s limiting but it also lets you identify from a threat base perspective like, “Well what are the things that I observe or what are the things that an adversary must do in order to enter that environment and then start doing something?” and that enables an informed detection.
An informed defense based around that threat environment where you really can provide an operator who might not even have much in the way of a security background but it just happen to be responsible for identifying these defense to give them a detailed understanding of like, “We observed this in your network. This is weird or different for hot it normally operates but also here’s the context around what we’ve observed in terms of network activity”.
Host activity we’ve seen it and this ties into the threat environment and maybe some specific threat actors so you know not just, “Oh something weird is going on, what do you do with that” but also, “Something weird, what happened its correlated with these other events which are indicative of this sort of intrusion activity and now you’ve given someone, if you have done this correctly the ability not just to understand that something is wrong but also what does it mean and what do I do about it.
And that really as an industry where we need to go instead of just saying, you know I am not trying to catch this person or any of your products here but just throwing an alert out leads to alert fatigue and especially if it lacks that context because you don’t know where to go next but developing a more analytical approach focused on threat behavior really empowers individuals to respond and react to events as they come along.
Yeah and is that, I mean I’d love to hear about what you are doing directly, at Dragos and what your tackling and where do you guys fit into that environment and into that sort of that life cycle?
Right, so that’s precisely what we aim to do. So my specific goal is adding that threat perspective to the industrial environment. So what threat act, when I say threat actors that always leads to the discussion of, “Oh you mean Russia” or you mean the NSA, well no. The perspective of threat actors that I take and this is something that I talk about frequently at events and personally it’s always like the categorization of behaviors that define some operating entity.
So you might see several sort of distinct activity groups that may all correlate to some in person organization but from a defender’s perspective, while it’s nice to know if you can actually figure this out like so and so is responsible really what I need to know is, “How does this group operate and what are their tactics, techniques, procedures and how do I counter them?” and that information gives you all you need in order to conduct defense.
So filling in that part of the picture, that allows us at Dragos to apply this with the platform aspect of our company to, “Okay, we’ve got an idea for what our threat environment, our bad actors look like, how they behave, what information do we need now to hunt them down, find them where they exist and notify people that such and such is going on?” and that’s where the platform fits in, not just to tell you that, “Hey something weird has happened but we observed several data points that all correlate to crash override like attack”.
Trisis like attack or pick one or the other ICS focus groups that we track. So that individuals responding aren’t washed in the dark so to speak when it comes to any single data point but rather presented in a more complete picture of what’s going on in their environment.
Right, we’ve seen this movie before. This is usually kind of how it plays out, yeah.
Yeah because even looking from a response perspective, it’s not just like, “Hey here is something that happened” but also “Here’s something happened and here’s the playbook for how you might respond or how you might want to respond. These are the additional data you need to arrive at a clear decision. These are the things to look for next” because you know, having come from an incident response background one of the critical things isn’t just finding out that something happened.
It’s also okay, now what? How did they move, did they or if they did, how did they move literary in my environment? How do I track them to truly understand the attack footprint and then how do I roll that up in order to not just wipe and rebuild one machine if you are even able to do that but then roll that infection of it all the way up so that I kicked them off individual posts and then deny every other point of access that they’ve managed to come across.
Yeah, so from that you guys don’t do the remediation piece but you sort of or do you?
No, so really it’s providing the information. So visibility into the environment and awareness from a threat oriented perspective on what it means.
Is that remediation piece, are there other kind of players in the space who are tackling outside of the issue or is it so bespoke based on the system and the team that they are really having to handle it inside?
So yes and no. So I sort of misspoke, you know at Dragos we have a threat operation center that includes and sort of incident response retainers and you know some very skilled and very intimidating in my opinion. They’re really nice people but they’re really smart people, individuals with incident response backgrounds that can provide that as a service but even in looking at it, we have seen a variety of things like certainly there is Mandiant out there.
They were involved in the Trisis/Triton event. We’ve even seen vendors get involved in these sorts of remediation questions. So the same event for crisis Shneider Electric got involved and actually they were – you know to give them a lot of credit and praise for how they handles the situation, they were very open and engaging with the community and willing to work with others in order to solve the problem so to speak. So we are seeing more of that but otherwise, one of the questionable things is from a traditional incident response security operation center perspective.
Those who are engaged in those operations at an organization typically have an IT only view and don’t understand or may not even have access to that ICS environment. So, having that level of expertise, it’s a knowledge gap. It is being addressed and I think there is awareness around it but we are not there yet for having a lot of that to be internalized.
Yeah, I mean I have heard that in several with my conversations sort of the guys who are running the industrial control systems and the security teams are often not even reporting into the same people or rarely speaking the same language and they really – you know an interesting sort of way I heard it, it talked about was that for industrial control guys the CIA kind of list is flipped, right? That availability isn’t the top whereas confidentiality often sort of the top for people in the security space so.
That’s possible. There are some artificial elements that get introduced in looking at things that way. So I have heard that often as well that inability is key. Well integrity is pretty important too. You know if my controller logic for a safety instrumented system is compromised or altered in some way like say with the purpose of the crisis attack was, that’s pretty –
Problematic.
Yes, exactly and confidentiality that probably is a lesser one although even there, just taking a screenshot from an HMI within an industrial control environment can provide you with a wealth of information on proprietary processes and technology. So it’s never as easy at as it seems I guess is the whole thing.
Clearly, in theory, very simple.
Yes but otherwise though, you know it is an unfortunate disconnect within the security community that like I was saying earlier the ICS guys, the engineers, the plant personnel, they know their systems. They know what they are supposed to do and how they’re supposed to operate and not engaging them when you have sort of silos of excellence within the organization is very dangerous because yeah, your IT SOC might have understanding.
Oh this looks like commander control traffic but what does it mean? How does it apply within the OT space is always a much cagier topic to address unless you engage though is really have a fundamental understanding of these then you’re never really going to arrive at a very good answer. You may arrive at a very bad answers definitely.
I mean this has been great. The floor is yours, anything you want to talk about to let people know kind of this is your mini soap box.
You know my mini soap box is very similar to how we started this conversation is Dragos has adopted a very aspirational company motto of safeguarding civilization which I snickered at it when I first learned of the company like, “They’ve got to be joking” but that mission statement rings true. I mean maybe we’ll change it down the line like Google did with don’t be evil but we’ll see. We are nowhere near there yet but having that aspirational value is important.
Because even though I will say to whoever is asking that’s causing some of the large scale cascading zombie apocalypse events is a lot easier to do that with a nuclear weapon than it is through a cyber-attack. Having said that though, taking on an individual facility or targeting and having an effect on an individual facility or say Washington DC as opposed to the entire east coast, that is more feasible and those are things we need to be worried about and I think the community is moving in the right direction.
At Dragos, we’re committed to helping in that movement even just through talks like this, presentations at a conference around defense to engage with not just the ICS community but even the IT community of building awareness for how these issues resonate and what sort of challenges lie therein because there are a lot of false assumptions in both sides either that all IT wants to do is look at all the traffic in black and white machines.
And on the IT side that these are fragile environments and they just need to be isolated and walk away from it.
Air gap it, just air gap. That’s all we need to do.
Right, if you think that there is air gap then you have a – there are exceptions to this but most of the time if you feel that something is air gapped, you missed something.
I heard them described like unicorns, right?
Yes.
Lots of people talk about them but they’re not. So far you have never seen one.
Right, truly air gapped network is quite the rare thing.
Yeah, Joe this is great. Thank you so much for coming on.
Hey, thank you for having me.