Measuring SOC Performance in the Cyber Range
Measuring SOC Performance in the Cyber Range
One challenge that security operations (SOC) managers constantly have to contend with – outside the threat of a cyber attack – is understanding gaps in the People, Process, and Technology (PPT) Framework and how those affect cyber defense for the organization.
Security operations managers struggle with obtaining meaningful metrics on their team’s performance, leaving them with no clear picture. Incident tracking tools can measure the average time from incident detection to resolution, or hours spent on addressing an incident. What these metrics cannot tell leadership is how well the team is trained and ready to execute on detection and response tasks.
This blog continues the conversation started in Training and Evaluating the Modern SOC. A comprehensive grading and evaluation system can provide cyber security leadership with a clearer understanding of how well your teams are performing across all seven focus areas. Such measurements show what is performing well and where there are opportunities to improve. The goal is to continually hone and optimize the processes that are followed, the technology that is used, and especially the capabilities of your people – as they are the last line of defense.
Cloud Range Overview
As a quick review, Cloud Range designs and executes a wide range of cyber security exercises that focus on the following seven incident handling performance areas:
Technical Proficiency
Detection
Incident Response
Prevention/Mitigation
Communication
Teamwork
Compliance
FlexRange live-fire simulation exercises for teams are facilitated by Cloud Range experts who provide key insights into the team’s incident-handling capability and maturity. Organizations support the team missions with Cloud Range’s OpenRange practice and FlexLabs skill training and development exercises.
Running a regular cadence of training exercises over the course of a year provides the best opportunity for managers and senior leaders to show both the current capability of the team and continuous improvement. Cloud Range achieves this through a combination of realistic, live-fire attack scenarios and an evaluation structure that provides measurable, actionable insight into the team’s performance.
"The sports adage applies to cybersecurity, 'If you're not getting better, you're getting worse.' Cyber defenders need continuous improvement to be ready for today's complex and destructive threats, and the best way to do that is with ongoing, relevant, real-life training and experiences. Without the ability to practice in live-fire simulation, those key responders will not be prepared when the real event happens," said Tom Marsland, Cloud Range VP of Technology and Technical Services.
Evaluating Performance
There are a few facets to consider when establishing evaluation metrics along these seven focus areas:
There should be a consistent structure that evaluates both the maturity of the individuals and the overall team.
A trained and experienced facilitator is needed to guide and assess the team’s performance during the exercise.
The organization should be able to use the measurements for improving its processes and procedures.
Measuring performance is much more complicated than simply gathering data from automated ticketing systems. Instead of hard data such as time stamps, measuring performance in an incident response context requires observing how they handle the various tasks contained within the focus areas and then determining the team’s maturity in handling those tasks.
For example, a new SOC team has been stood up. One challenge that often plagues new teams is communication. That is understandable as new processes, technology, and personnel are all tossed together. An annual tabletop exercise (TTX) is inadequate to address this deficiency. What is needed is a regular cadence of monthly exercises where the team dynamic can mature so that over time, the team is able to communicate effectively and work together in the high-stress environment of an incident.
It’s critical to have a combination of realistic simulation exercises and knowledgeable facilitators that can bring to bear their experience to evaluate the team as objectively as possible. The one distinct advantage to bringing in a trained third party to evaluate a team is that you simply cannot grade your own homework. This is not to say that team managers would deliberately mislead leadership about their team’s capability. However, evaluating your own team can introduce biases, even if they’re subconscious, that will skew the metrics and provide a false picture of the team’s overall capability.
The final piece is how to evaluate the team against the seven focus areas. To evaluate a team’s performance, Cloud Range utilizes a scale from level 1 to level 5. These levels rate each person’s performance and the team as a whole in understanding concepts, completing tasks, and resolving the incident.
At level 1, the low end of the scale, participants are present but add no real impact to the proper handling of the incident. On the opposite end, when the maturity level is level 5, it represents a highly mature team that is able to display expertise in performing key tasks and understanding key concepts. This level can be thought of as the optimal performance level, and a team that can function at this level represents a mature and well-functioning team.
"Organizations have a growing need for an objective knowledge framework in cybersecurity and, ideally, a way for them to measure cyber risk against that framework,” said Marsland. “NIST's NICE Cybersecurity Workforce Framework provides the right foundation. And because Cloud Range maps our training, labs, assessments, team simulation exercises, and more to the NICE Framework and the MITRE ATT&CK Framework, we enable companies to visibly measure and track the progress being made by their teams and reduce their risk."
Putting It All Together
Combining live-fire cyber range training exercises with a rating system provides a clear understanding of the SOC at its current state. From here, organizations can develop and modify processes and procedures and augment their current training to fill the gaps. Over time, a monthly cadence of exercises can increase SOC effectiveness.
For example, the figure below shows the results for a SOC over the period of six months. In the beginning, the organization had some gaps in several areas such as technical proficiency and communication. This could be an indication that the SOC has onboarded new technologies and may still have some rough edges when it comes to intra-team communication.
With these data points in place, the SOC manager and organizational leadership can prioritize which processes or procedures need to be updated, what additional training needs to be conducted, or how to best upskill team members. As the team goes through these improvements, the continued use of incident handling exercises can validate that the changes were effective. At the end of a six-month period, for example, there are clear metrics showing that the SOC team has matured and become more proficient in their cyber defense.
High-performing teams do not always rise to the occasion, but they do sink to the level of their training (or lack thereof). Teams that shun recurring exercises and do not measure performance improvement will often be found lacking when an incident occurs.
Regular simulation training exercises with honest evaluations drive improvement over time. They allow SOC teams to reach a higher level of performance to counter today’s threats. Regular exercises coupled with professional facilitators who provide a structure to rate the team’s performance are a force multiplier when it comes to improving an organization’s response to a cyber incident.