Kubernetes Security Blog | RAD Security

The true cost of signatures; and the ROI of behavioral cloud detection and response

Written by Story Tweedie-Yates | Aug 28, 2024 5:46:25 PM

The average cost of a breach was $4.45M in 2023, with a mean time to identity of 204 days and a mean time to contain of 73, which means that every day that a breach goes undetected costs another $16,000. These are enormous costs for companies to shoulder, not to mention the personal impact on those involved, and the recent increased risks of litigation against CISOs. Teams are taking the risks seriously, spending 50% more of Enterprise IT budget on detection and response than in 2017, but despite these efforts, only ⅓ of current incidents are detected by internal tools. 

 

 

When it comes to cloud security, even though documentation and information about the limitations of signature-based approaches through the history of cybersecurity is well-known and available, signatures are the de-facto approach to detection for cloud security. Signatures are also ingrained deeply in the heart of a SOC and security team’s detection and response processes, dictating organizational structure, job qualifications and what any team can reasonably be expected to achieve.

What would teams gain if they could take back a portion of the time they spend on signatures in cloud security? 

To understand this, we need to first get an adequate understanding of the true cost of signatures when it comes to cloud detection and response, and examine whether a signature-based approach to cloud detection and response is really the ‘right tool for the right job.’ 

 

The true cost of signatures

Due to the sheer volume involved in the cloud, detection engineers are forced to rely somewhat on out of the box detections from vendors, who are primarily detecting threats in cloud environments using signature-based methods. 

The human impact of signatures

People and processes are generally more important to the success of a security team than technology, and this is especially the case for the teams responsible for detection and response, including those that run the Security Operations Center (SOC), because of the broad scope, various inputs, and need for speedy responses. When it comes to detection and response in the cloud, to substantially support people and processes given the sheer volume involved, the best support for the SOC/SIEM workflow is:

  1. A higher quality of detection 
  2. Scope that’s in a goldilocks zone - not too broad, not too narrow

The majority of the pain in working with signatures is what people must go through in order to achieve adequate scope to cover their gaps, with the best quality of detection.

 

The security engineer’s time is precious

Both home-grown and out-of-the-box signatures from vendors create an immense burden on the detection engineer or cloud security engineer, who will spend up to half their time on signature creation and maintenance, for example in the following scenarios:

  • Legacy CNAPP Vendor X provides its own rule to detect a reverse shell, but it can’t pick up the reverse shell that the customer has created, or that recently made its way into the environment. The customers must ask for the signature to be changed or simply change it themselves.  
  • Vendors’ signatures create numerous false positives, triggering on normal events in the customer’s environment because they don’t take into account the unique environment of the customer. The customer is left with the work of tuning the rules to reduce the flood of false positives.
  • Signatures for a new threat aren’t created fast enough, so the customer has to step in and make their own.
  • Compliance reports for auditors must be stitched together from multiple sources, some of which contain conflicting data due to static lists versus real-time workloads.

A detection engineer or a cloud security engineer has a massive opportunity cost, in the value of their skillset to the organization, and in many cases, work on signatures ties up ½ of their time.

 

Delaying breach detection costs $

The side effects of signatures include delays in mean time to identify and mean time to remediate (MTTI, MTTR):

  • Trying to piece together signature-based alerts from Kubernetes, identity and workload runtime alerts that are delivered separately, and are inactionable, make it impossible to respond 
  • Even if using eBPF, signatures can pack too much into each node (e.g. libraries, metadata enrichment), increasing compute requirements and causing performance problems, or even downtime, reducing teams’ willingness to deploy and creating coverage gaps
  • Signatures leave entire swaths of valid processes and programs unexamined - they only look at what matches their logic 
  • In the time it takes for teams to turn these fragmented detection pieces into real, working detections that point out an entire sequence of adversarial activity, and variations of it, in their unique environment, multiple attacks might have already been carried out
  • In the case of a new attack, a signature won’t do anything until after the fact, by design

 

Every day that a breach goes undetected costs an organization ~$16,000, and signatures create cumbersome inputs that teams must combine into working detections that drive down MTTI and MTTR. 

 

High quality detections with RAD behavioral baselines

Much of the pain with signatures around achieving the right scope, and a high quality detection, would be reduced or altogether solved with:

 

RAD Security delivers the above with automated, eBPF-powered behavioral fingerprints that can be updated, verified, and changed based on your environment parameters, combined with AI-based investigation and enrichment.

Step 1. Deploy RAD and create initial baseline workload behavior

Baseline workload fingerprint for nginx image

 

Step 2. Detect drift and immediately enrich with real-time identity and infra context 

Drift event with AI-powered investigation

 

Step 3. Update the fingerprint as needed over time with knowledge of your custom environment 

Option to update behavioral baseline

 

With these steps, you can:

  • Easily input your own custom knowledge into your detections
  • Detect an entire sequence of attack events . . . in less than 30 seconds 
  • Know - transparently - what good looks like across your entire cloud environment
  • Get a real-time attack view from the K8s infra & identity to the workload
  • Access detections that already include TI feeds, & novel info that you would otherwise only get from internal red-teaming exercises
  • View attack processes, programs and files that are spawned versus random alerts grouped by image/alert names and labels
  • Send reports to compliance auditors that don't require adjustment or configuration

 

The ROI of RAD behavioral detection 

With RAD’s behavioral detection, teams are getting the right level of scope, and a higher quality detection than with signatures, which saves significant time and improves the time to identify and remediate in the case of a breach.

Increased productivity

The activities involved in writing and tuning signatures, with RAD, would be replaced by automated behavioral fingerprint baselines that can quickly be updated, freeing up ½ the time of a detection engineer or security engineer. Imagine what your team could do if each person had 50% more of their time to dedicate to continuous testing, better alert definitions, expanding to new attack surfaces, documentation, or process management? How much more effective could your team work together across threat hunting, detection engineering, the SOC, and engineering?

 

Saved costs

For the below analysis of the cost of a breach, we only included costs that would have directly related to reducing the time to detect a breach, versus costs that will occur in a breach regardless of how long it remained undetected, like communications and PR. RAD’s solution can detect novel threats, not just known attacks, which means your time to detect and then respond could be reduced to zero. And showing an entire sequence of events, with the ‘Goldilocks’ scope across cloud native workloads, identity and infrastructure, reduces investigation time dramatically. By improving your MTTI and MTTR 70-99%, you can achieve cost savings of $10,000 per day, or $2-2.85M per breach.

 

Easier compliance

Compliance is another area where signatures make it hard to gain a complete picture. You are left piecing together fragmented, stateless runtime data with static Kubernetes configurations and identity permissions, creating a lot of overhead with every audit. If you are being audited constantly, as is the case in many financial institutions, this burden is enough to take up multiple peoples’ weekly output. Based on customer interviews, each audit takes on average around 38 hours across multiple teams; RAD cuts this time in half by automating compliance reports that can be handed directly through to auditors. If each hour of time for a cloud security engineer costs $140, and you have 4 audits over a combined span of 20 weeks, you would be saving the equivalent of $200k in hours that could have been put to use detecting and responding to threats. In essence, you are getting ½ of a work week for one person for each week you are being audited, times the number of audits in a year.

 

 

The human factor in behavioral detection

Just as using signatures for detection and response in the cloud creates endless headaches for people and processes, behavioral detection and response creates desirable consequences for processes and people.

  • Any work with a third party (e.g. an MDR tool) will be much easier because you no longer have to constantly describe and review their representations and customizations for your unique environment; they can use the same tool. Collaborating on behavioral baselines is much easier, and more automated than collaborating on signatures.
  • There would be less need for back and forth between the research teams and the detection teams; the detection teams are already getting the polished production detections for the newest emerging threats, in real-time so the threat teams spend less time figuring out the right format and scope for threat intel and more time keeping up with the emerging threat landscape.
  • Since RAD detects novel threats, this brings the detection engineering team into the picture with threat hunting teams; more hands at the table means a better likelihood of seeing the needle in the haystack. 
  • With behavioral detections, the thresholds for urgency, originally put in place to calm the flow of false positives, becomes less time intensive, and you have better prioritization across the board, getting everyone quickly aligned on other important processes like triage, and continuous testing. 
  • When your environment changes, and the changes aren’t necessarily bad, it’s much easier to see what happened and simply change the behavioral baseline and understanding of what is ‘good’ versus finding out if the new alerts indicate something bad or simply a change in your environment.
  • Testing is also easier because all the visibility is right there, from the event to the cluster to the asset inventory to the baseline fingerprint. 

In the end, we can imagine a team that is finally able to take the time to adequately document, onboard new teammates, and turn the tribal knowledge of their environment into scalable detections, taking time away from signatures to do their part to ensure the SOC runs as smoothly as possible and the SIEM is giving the holistic picture of your risk posture.

 

The Upside 

In conclusion, the shift from signature-based detection to behavioral detection in cloud security isn't just a technical upgrade—it's a transformative change for security operations. By eliminating the time-consuming tasks associated with signatures, like constant tuning and updates, teams can focus on higher-value activities that truly enhance security posture. RAD's automated, behavioral detection approach not only reduces the mean time to identify and respond to threats but also empowers teams to work more effectively across detection engineering, threat hunting, and incident response. The result is not only a more secure environment but also a more efficient and cohesive team, leading to significant cost savings and a better overall ROI in cloud security.