Friday, March 19, 2021

Some additional words on those SOC robots

 The topic on SOC automation is really a fun one to think about, and even after putting my thoughts into words with my last post, I've still kept thinking about it. Some additional considerations came to my mind.

The simplistic question of "Will machines replace humans in a SOC" can be clearly answered with a NO, as I explained in my previous post. As the human attackers are required to evolve the attacking robots, blue team people are required to update the automated defenses.

But things change if the question is asked with some additional nuance. If you ask "will defense actions be automated end to end, from detection to response actions?", it becomes a more interesting question to answer.

The scenario of automated threats that Anton described in his post will, IMO, require SOCs to put together some end to end automation. Having a human involved for every response will not scale to face those attacks. Humans will be responsible for creating those playbooks and monitor their performance, but they cannot be involved in their execution. We need SOC automation that allows us to detect, investigate and initiate response without human intervention. This is challenging, but we must get there at some point.

Andre Gironda commented on the LinkedIn post pointing to my blog post that even with the appropriate tools he still can't fully automate simple phishing response. I could say he's probably being too perfectionist or doing something wrong, but I actually believe him. I believe automation can provide value by reducing human effort in the SOC right now, but full automation, even for some specific threats, is still challenging. But we'll have to get there if we want to stand a chance.





Tuesday, March 16, 2021

The Robots Are Coming!

 The debate around SOC automation has been a fun one to follow. Allie Mellen wrote a short but on the spot piece about it, reaffirming what seems to be the commonsense opinion on this topic today: Automation is good, but to augment human capacity, not replace it.

 

After that Anton brought up a very interesting follow up, confirming that view but also pointing to a scary future scenario, where automation would be adopted so extensively by the attackers that it would force defense to do the same. Does this scenario make sense? 

 

I believe it does, and indeed it forces defense to adopt more automation. But even if Anton says the middle ground position is "cheating", I still think it is the most reasonable one. There will never be (until we reach the Singularity) a fully automated SOC, just as there will never be a fully automated attacker (until...you know). Why? Let's look at the scenario Anton painted for this evolved attacker:

 

 

• You face the attacker in possession of a machine that can auto-generate reliable zero day exploits and then use them (an upgraded version of what was the subject of 2016 DARPA Grand Challenge)
• You face the attackers who use worms for everything, and these are not the dumb 2003 worms, but these are coded by the best of the best of the offensive “community”
• Your threat assessment indicates that “your” attackers are adopting automation faster than you are and the delta is increasing (and the speed of increase is growing).

 

 

Even if it looks scary, this scenario is still limited in certain points. You may have malware capable of creating exploits by itself, but what will they exploit? What is this exploitation trying to accomplishThere is an abstract level of actions that is defined by the creator of the malware. Using MITRE ATT&CK language, the malware is capable of generating multiple instances of a selection of techniques, but a human must define the tactics and select the techniques to be used. Quoting Rumsfeld, there will be more known unknowns, but the unknown unknown is still the realm of humans.

 

A few years ago, I had a similar discussion with a vendor claiming that their deep learning-based technology would be able to detect"any malware". This is nonsense. Even the most advanced ML still needs to be pointed to some data to look at. If the signal required to detect something is not in that data, there's no miracle. Let's look at a simple example:

 

• A super network-based detection technology inspects ALL network traffic and can miraculously identify any attack.
• The attacker is on host A in this network, planning to attack host B, connected to the same network
• The attacker scans for Bluetooth devices from host A, finds host B, exploits host B via a Bluetooth exploit
• The super NDR/NIDS tool sits there patiently waiting to see an attack that never traverses the monitored network!

 

You may claim this is an edge scenario, but I'm using anexaggerated situation on purposeThere’s still many cases that we can relate to, such as breaches due to the use of shadow IT, cloud resources, etc. What I want to highlight is the type of lateral thinking very often employed by attackers in cybersecurity. And the lateral thinking is still exclusive of humans.

 

What I'm trying to say is that fully automated threats are scary, buy they lack the main force that makes detecting threats challenging. Defense automation can evolve to match the same level, but both sides will still rely on humans to tip the scale when those machines reach a balance point in capabilities.

 

What we have today is similar to those battling robots TV shows. Machines operated by humans. If things evolve as Anton suggests we will move to what happens in "robot soccer": human created machines operating autonomously, but within a finite framework of capabilities.





Robot wars vs Robot Soccer

 

 

Threats and SOCs will become more automated for sure. As they automate, they become faster, so each side has to increase its own level of automation to keep up. But when automation limits are reached, the humans on the threat side must apply that lateral thinking to find other avenues to exploit. They need to take the Kirk approach to Kobayashi Maru. When this happens, the humans on the defense side become critical. They need to figure out what is happening and create new ways to fight against the new methods.

 



 

 

So, humans will still be necessary on both sides. Of course, the operational involvement will be greatly reduced, again, on both sides. But they will be there, waiting to react against the innovation introduced by their counterparts on the other side.

 

This may be an anticlimactic conclusion, and it is. But there are some interesting follow up conversations to have. The number of humans required, their skills and how they are engaged will be different. What does it mean for outsourcing? Do end users still need people on their side? If solution providers engage this problem in a smart way, we may be able to remove, or greatly reduce, the need for humans on the end user organization side, for example. The remaining humans would be on the vendor side, adapting the tools to react against the latest attacks. For the end user organization, the result may look very similar to full automation, as they would not need to add their humans to the mix. Will we end up with the mythical "SOC in a box"? Future will tell.

 

Thursday, March 4, 2021

An Analysis of Past Mistakes

 As I was looking for an old email in my archives, I stumbled on discussions about a security incident that happened almost 13 years ago. That was that time when, well, there's no other way of saying it....I was hacked.

The good thing about looking at incidents like that one after a long time is that it helps us understand what really happened and also run a less passionate and unbiased assessment of our own actions. I have to say this case is really enlightening, in many ways. There are good lessons to learn and mistakes to acknowledge from multiple perspectives: Technical, Managerial and even Political. 
  
The year was 2008. I was part of the Board for the Brazil ISSA chapter. We were trying to push for a more inclusive posture of the association, promoting free monthly encounters and other initiatives. Our group took over the board when we felt there were too many security vendors dominating the association, many of them pulling things to where their business would benefit most. A group of friends and acquaintances discussed this and after some deliberation, I was chosen as the head of the ballot. It was an honor for me at that time, as each one in that group was capable of taking the central role. We won the election using our network and a popular email discussion board at that time to spread our word and our plans for the association.
 
So, back to the "breach". We had set up a portal for the association using an open source CMS, Joomla. Joomla was plagued by vulnerabilities at that time, and someone managed to access the user database and crack the passwords. The password for my test account there...well, I was using it in some other places. It was my old password from before I started working with security. I had replaced it almost everywhere, but it was still used on a few places I had forgot about, like LinkedIn and a hotmail account I used to have so I could use MS Messenger. Well, those, and a couple of other services were quickly found by the attackers, and an embarrassing message with all that was posted in that popular email forum, and other places. In summary, an application breach on a website ran by...security professionals, and some pretty lame secops practices by one those guys exposed. 
 
What have I been able to extract from that incident? A lot. Here it is. 

Technical lessons 

The easiest to mention. We were using a horrible tool from a security perspective (Joomla). We had been warned by some people, but some of our group believed we could run it securely by not using crappy plugins and keeping it always up to date. But we didn't have a dedicated security operations team to keep watching it. In addition to it, we knew there were technically competent people out there trying to hack us. So, the threat component was high. It was an explosive combination. In short, we should have made choices that would simplify the challenge of keeping the vulnerability profile low, as we didn't have time to protect it like it should be. 
 
Then, there was my own personal mistake, reusing a password. It is certainly something no one, especially a security professional, should do.  Of course, I was already aware of that, and I was already using unique, different passwords on almost everything that mattered at that time. But this old password ("trustno1", if you really wanna know!) was something I started using long before getting involved with security. As I became more aware of the risks of password reuse I started changing it everywhere, but there were still a few places I had forgotten to do it. To make things worse, I started using it as my "throwaway" password for testing needs. An account I had for testing on the ISSA chapter website was using that password. Bad secops…bang, they got me. 

Management Lessons

This is where I think we can start getting good lessons from the incident. This is about our organizations, the ISSA chapter. How come a security professionals organization be hacked? 
 
We fell for the same mistakes we see in many other organizations. First, the fact that we were all security people caused the "too many cooks in the kitchen" issue. Who was the "CISO" for our organization? That was never defined, so there weren't clear roles and responsibilities defined regarding our own security. I brought the site up and did some of the initial hardening, but at that time I was already moving those responsibilities to other people and completely focused on other issues (I was preparing to move to Canada at that time). People generally know about vulnerability management, but on that case, I believe no one was actually the owner of that process and consciously doing it for us.  

Political, social and relationship lessons

Here's another point from where I extract a lot of personal lessons. When we took over the chapter, our group had as one of its objectives to close the gap between the "security professionals" community (the CISSPs :-)), in fact those dealing with risk management, security policies and other less technology oriented topics, and those with the technology background or IT security jobs. That should also include the "hacking" community (or "scene"). 
 
That divide between the "management people" and the "technical people" was also related to professionals in different stages in their careers. It was very hard to find technical individual contributors in a highly paid position in Brazil at that time. It wasn't interesting to make them part of ISSA for some of the previous directors because there was low value in junior people as potential customers to their products and services. Trying to be more inclusive of professionals with technical backgrounds was really the attempt to make the association useful for people in the early stages of their careers as well.
 
But although I have a technical background, I was never close the underground scene in Brazil. I knew people who were, some volunteers helping us during those days were very connected to that community. Still, I've never been a fan of some of the more juvenile aspects of hacking communities. The use of leetspeak, piercings, crazy haircuts...nothing against that, it's just not my thing.  This, on top of my effort to make the technical professionals voices heard in the community, made me adopt a gatekeeping position, as in my view they were not being helpful in solving the problem I wanted to solve. In more traditional environments, appearances matter a lot. At that time, it was hard to be taken seriously wearing shorts, a mohawk and writing “3 n0iZ M4n0!!”.
 
In the end, I believe we didn't do enough to reach out and include them, and they felt excluded. Our posture about a "professional organization", plus a growing number of charlatans in the market put fire in a "take down a whitehat" movement, which I ultimately fell victim of.
 
I had helped create the animosity against security professionals, then underestimated their abilities and their motivations against me. What a stupid combination, right? Yes, I know. Talk about not having control over the "Threat" component of the risk equation...
 
In summary, that was my collection of mistakes. Technical blunders, classical management mistakes and a dose of simple immaturity. For those also hurt in the process, I'm sorry. I hope I can keep learning from mistakes like those and make better decisions in the future. This is an extremely important part of working in security, knowing we'll never be able to reach perfection.   

Friday, October 9, 2020

Monitoring and Vulnerability Management

 (Cross posted from the Securonix Blog)

Vulnerability management is one of the most basic security hygiene practices organizations must have in place to avoid being hacked. However, even being a primary security control doesn't make it simple to successfully implement. I used to cover VM in my Gartner days, and it was sad to see how many organizations were not doing it properly.

Many security professionals see VM as a boring topic, usually seeing it simply as a "scan and patch" cycle. Although the bulk of a typical VM program may indeed be based on the processes of scanning for vulnerabilities and applying patches, there are many other things that need to be done so it can deliver the expected results.

One of the most important pieces of it is the prioritization of findings. It is clear to most organizations that patching every open vulnerability is just not feasible. If you can't patch everything, what should you patch first? There are many interesting advancements in this area. What used to be based only on the severity of the vulnerabilities (the old CVSS value) is now a more sophisticated process that leverages multiple data points, including threat intelligence. The EPSS research by Kenna Security is a great example of how evolved the practice of prioritizing vulnerabilities is now when compared to the old CVSS times.

But even when you are able to decide what to patch first, there are also cases where the remediation is not simply applying a patch. Some vulnerabilities involve not only a bug, but also other issues such as the existence of legacy software and protocols in the environment. These situations usually require a more complex approach, and that's where an additional component of the VM process, the compensating controls, become important.

Compensating controls are used to address the risk of a vulnerability while the full remediation cannot be applied. Using an IPS, for example, is a typical compensating control. You can use them when you cannot apply the remediation, such as when a patch is not available, or to mitigate the risk until you are comfortable enough (usually after testing is done, during a maintenance window) to apply it. We usually see some security controls that can avoid or reduce the impact of vulnerability exploitation as the ideal candidates for compensating risk, but there is something I always like to bring up during this discussion: Monitoring.

Think about it for a second. You have an open vulnerability that you still cannot patch. The exploit is available, as well as a lot of information about how it is used. Even if you cannot avoid it, you can use all this information to build a security monitoring use case focused on the exploitation of this specific vulnerability. You it is there, and that there is a chance for it being exploited, so why not put something together to look for that exploitation? You can prioritize the alerts generated by this use case, as you know you are currently vulnerable to that type of attack.

A great example of using security monitoring as part of the VM process is what is happening with the new Windows Zerologon EP (ZEP) vulnerability (CVE-2020-1472). The issue is complex and requires more than just applying a patch. Our VP of Threat Research, Oleg Kolesnikov, produced a great write-up about the details and also variants of exploitation and detection. In summary, Microsoft has provided a patch for the immediate problem, but some third-party systems may still use an older, vulnerable version of Netlogon secure channel connections. To avoid breaking functionality of existing systems, Microsoft has introduced new events in their logs to identify the use of these older versions, and signaled they will move to an enforcement mode that will not accept them anymore after February, 2021.

This is where aligning monitoring with the remediation process becomes so important. The new events added by Microsoft can help identify attack attempts and track other vulnerable systems on the network. A pre-established process to coordinate the use of monitoring tools and infrastructure as an additional compensating control for VM can help in situations like this, where the plan to handle a vulnerability also requires monitoring activities.

Monday, September 21, 2020

DDLC - Detection Development Life Cycle

Dr. Chuvakin has recently delivered another great blog post about "detection as code". I was glad to read it because it was the typical discussion we used have in our brainstorming conversations at Gartner. It had a nice nostalgic feeling :-). But it also reminded me of my favorite paper from those times, "How To Develop and Maintain Security Monitoring Use Cases".

That paper describes a process framework for organizations to identify and develop use cases for security monitoring. It was intentionally designed to be tool neutral, so it could be used to develop SIEM rules, IDS signatures or any other type of content used by security monitoring tools. It was also built to mimic Agile development processes, to avoid the capital mistake of killing the required agility to adapt to threats by too much process. I had fun discussions with great minds like Alex Sieira and Alex Teixeira (what's this with Alexes and security?) when developing some of the ideas for that paper.

Reading the philosophical musings from Anton on "detection as code" (DaaC?), I realized that most of threat detection is code already. All the "content" covered by our process framework is developed and maintained as code, so I believe we are quite close, from a technology perspective, to DaaC. What I think we really need is a DDLC - Detection Development Life Cycle. In retrospect I believe our paper would be more popular if we used that as a catchy title. Here's a free tip for the great analysts responsible for future updates ;-)

Anyway, I believe there are a few things missing to get to real DaaC and DDLC. Among them:
  • Testing and QA. We suck at effectively testing detection content. Most detection tools have no capabilities to help with it. Meanwhile, the software development world has robust processes and tools to test what is developed. There are, however, some interesting steps in that direction for detection content. BAS tools are becoming more popular and integrated to detection tools, so the development of new content can be connected to testing scenarios performed by those tools. Just like automated test cases for apps, but for detection content. Proper staging of content from development to production must also be possible. Full UAT or QA environment are not very useful for threat detection, as it's very hard and expensive to replicate the telemetry flowing through production systems just for testing. But the production tools can have embedded testing environments for content. The Securonix platform, for example, has introduced the Analytics Sandbox, a great way to test content without messing with existing production alerts and queues. 
  • Effective requirements gathering processes. Software development is plagued by developers envisioning capabilities and driving the addition of new features. It's a well-known problem in that realm and they have developed roles and practices to properly move the gathering of requirements to the real users of the software. Does it work for detection content? I'm not sure. We see "SIEM specialists" writing rules, but are they writing rules that generate the alerts the SOC analysts are looking for? Or looking for the activities the red team has performed in their exercises? Security operations groups still operate with loosely defined roles and for many organizations the content developers are the same people looking at the alerts, so the problem may not be that evident for everyone. But as teams grow and roles become more distributed, it will become a big deal. This is also important when so much content is provided by the tools vendors or even content vendors. Some content does not need direct input from each individual organization; we do not have many opportunities to provide our requirements for OS developers, for example, but OS users requirements are generic enough to work that way. Detection content for commodity threats is similar. But when dealing with threats more specific to the business, the right people to provide the requirements must be identified and connected to the process. Doing this continuously and efficiently is challenging and very few organizations have consistent practices to do it.
  • Finally, embedding the toolset and infrastructure into DDLC to make it really DaaC. Here's where my post is very aligned to what Anton initially raised. Content for each tool is already code, but the setup and placement of the tools themselves is not. There's still a substantial amount of manual work to define and deploy log collection, network probes and endpoint agents. And that setup is usually brittle, static and detached from content development. Imagine you need to deploy some network-based detection content and find out there's no traffic capture setup for that network; someone will have to go there and add a tap, or configure something to start capturing the data you need for your content to work. With more traditional IT environments the challenge is still considerable, but as we move to cloud, devops managed environments, these pre-requisite setting can also be incorporated as code in the DDLC.
There's still a lot to make full DaaC and comprehensive DDLC a reality. But there's a lot of interesting stuff in this sense going on, pushed by the need for security operations to align with the DevOps environments in need to be monitored and protected. Check the Analytics Sandbox as a good example. We'll certainly see more like this coming up as we move closer to the vision of threat detection becoming more like software development.

Friday, September 11, 2020

NG SIEM?

An interesting result from changing jobs is seeing how people interpret your decision and how they view the company you’re moving to. I was happy to hear good feedback from many people regarding Securonix, reinforcing my pick for the winning car in the SIEM race.

But there was a question that popped up a few times that indicates an interesting trend in the market: “A SIEM? Isn’t it old technology?”. No, it is not. It may be an old concept, but definitely not “old technology”.

Look at these two pictures below? What do they show?

 


Both show cars. But can we say the Tesla is “old technology”? Notice that the basic idea behind both is essentially the same: Transportation. But this, and the fact they have four wheels, is probably the only thing in common. This is the same for the many SIEMs we’ve seen in the market in twenty or so many years.

Here is the barebones concept of a SIEM:

 


 












How this is accomplished, as well as the scale of things, have changed dramatically since ArcSight, Intellitactics and netforensics days. Some of the main changes:
  • Architecture. Old SIEMs were traditional software stacks running on relational databases and with big and complex fat clients for UI. Compare this with the modern, big data powered SaaS systems with sleek web interfaces. Wow!
  • Use cases. What were we doing with the SIEMs in the past? Some reports, such as “top 10 failed connection attempts” or some other compliance driven report. Many SIEMs had been deployed as an answer to SOX, HIPAA and PCI DSS requirements. Now, most SIEMs are used for threat detection. Reporting, although still a thing, is far less important than the ability to find the needle in the haystack and provide an alert about it.
  • Volume. SIEM sizing used to be a few EPS, Gigabytes exercise. With the need to monitor chatty sources such as EDR, NDR and cloud applications the measures are orders of magnitude higher. This changes the game in terms of architecture (cloud is the new normal) and also drive the need for better analytics; we can’t handle the old false positive rates with the current base rates of events.
  • Threats. It was so easy to detect threats in the past. It was common to find single events that could be used to detect malicious actions. But attacks have evolved to a point where multiple events may be assessed, in isolation and together as a pattern, to determine the existence of malicious intent.
  • Analytics. Driven by the changes to threats, volume and use cases, the analytics capabilities of SIEM have also changed in a huge manner. While old SIEMs would give us some regex capabilities and simple AND/OR correlation, modern solutions will do that and far, far more. Enriched data is analyzed with modern statistics and ML algorithms, providing a way to identify the stealthiest threat actions.

With all that in mind, does it still make sense to call these new Teslas of threat detection a “SIEM”? Well, if we still call a Tesla a car, why not keep the SIEM name?

 

However, differentiating between the old rusty SQL-based tool and the advanced analytics SaaS tools of modern days is also important. In my previous life as an analyst I would frequently laugh at the “Next Gen” fads created by vendors trying to differentiate. But I also have to say it was useful to provide a distinction between the old Firewall and what we now call NGFW. People know the implied difference in capabilities when we say NGFW. With that in mind, I believe saying NG-SIEM is not really a bad thing, if you consider all those differences I mentioned before. Sorry Gartner, I did it! :-)

So, old SIEM dead, long live the NG-SIEM? No, I don’t think we need to do that. But in conversations where you need to highlight the newer capabilities and more modern architecture, it’s certainly worth throwing the NG there.

Tesla owners can’t stop talking about how exciting their cars are. For us, cybersecurity nerds, deploying and using a Next-gen SIEM gives a similar thrill.

Monday, August 31, 2020

I'm Joining Securonix

 I’m very happy to announce today I’m starting my journey with Securonix!


I’ve spent the last five years working as an industry analyst, talking to thousands of clients and vendors about their challenges and solutions on security operations. During this time I was able to identify many of common pain points and what vendors have been doing to address them. Some with success, some not much.


Helping clients as an analyst is a great job. It gives you tremendous visibility into their challenges. But it is also somewhat limited into how much you can help them. So I ended up with many ideas and things I’d like to do, but with no right channel to provide them.


That’s why I chose to join Securonix. Securonix has a great platform to deliver many capabilities that organizations need to tackle their threat detection and response problems. I first came into contact with Securonix before my Gartner life, and have been watching it grow and evolve since then. When we produced an UEBA solutions comparison, back in 2016, it was the best one of the batch. But it didn’t stop there.


A few years ago Gartner said SIEM and UEBA would eventually converge. Securonix didn’t miss the trend. Actually, it was one of the main drivers. UEBA vendors first appeared in the SIEM Magic Quadrant back in 2017. Securonix was already there as a Visionary. Actually it was the vendor with the most complete vision at that time. Since then it managed to improve its ability to execute, becoming one of the leaders in the space. It hasn’t missed the major trends since then, adding important capabilities and quickly adapting to offer a great cloud SIEM solution.


Good tools are extremely important to anyone who wants to make a dent on the incredible threat detection and response challenges we face. I’m excited to help with the evolution of the best security operations and analytics platform available today. You can watch this great journey here, , on Linkedin and on Twitter (@apbarros).