Introduction
The ‘proof of the pudding is in the eating’ is an old English phrase. Commonly understood to mean that it is possible to make an assessment or evaluation of anything only when it has been experienced, culinary experts will tell you that the phrase has its origins in the looks of the fully-cooked pudding. Even when ready it is nigh impossible to gauge its readiness, merely by looking at it. The acid test of tasting is needed.
Though it is a very basic analogy, it can well apply to cybersecurity controls. Just having a system in place, and merely investing in cyber tools is not the end and all. To gauge their effectiveness, they must be tested, or as many would say in the world of cybersecurity benchmarking, challenged.
Why evaluate
In our digital age, reasons abound to argue that cybersecurity needs to be at the top of organizations’ agenda. Cybersecurity incidents have been known to occur with such a high frequency across businesses, and with such devastating effects, that their position as the number one threat to business operations is more than justified. Forbes (1) puts the losses to businesses of all sizes due to cyberattacks at an average of USD 200,000, with almost 60% of businesses failing following a data breach.
Vulnerability aside, simple logic, commercial commonsense, and risk awareness make the case for evaluation of the effectiveness of cybersecurity investments. In our performance-driven world, it’s a given that investments will be looked at closely for their return on investment. The same goes for cybersecurity personnel – are they preventing those crippling data breaches, warding off Denial of Service and ransomware threats, and generally delivering what they have been onboarded for?
There is no denying it. Effectiveness studies give organizations insights into their cybersecurity posture, vindicate their investments in cyber tools, help evaluate their cyber teams’ performance, and facilitate the setting of future goals.
First steps
Before discussing the effectiveness of one’s cybersecurity system, however, it is worthwhile revisiting the fundamentals involved in setting up a cybersecurity plan or reviewing an established one. Forbes (1) suggests a good guideline to evaluate one’s security posture. Key considerations include:
- Accountability matrix clearly defined
- Employee cyber awareness quotient
- Organizational risk culture
- Patch update security practice
- Networked device status and digital identity management
- Data authentication procedures
- Data access rights, and data backups
- Data recovery plan, mitigation and remediation measures
- Adoption of and compliance with regulatory measures
- Overview of threat landscape across platforms
Measuring effectiveness
Verifying how in-built controls of a cybersecurity program work is at the heart of an effectiveness study. Are these controls working as intended? Is their deployment justified? Consulting firm BakerTilly (2) and TechTarget (3) provide parameters for tracking cybersecurity effectiveness that organizations would do well to implement. They include:
- Detected intrusion attempts and volume of security incidents
- Incident severity level analyses
- Recorded incident response times
- Recorded incident remediation times
- Incidence of false positives and negatives
- Patch installation times
- Vulnerability and penetration testing results and assessment
- Review of data access levels and rights
- Data traffic study
- Audit studies and implementation of recommendations, learnings
- Industry best-practice implementation
Defining metrics
Whilst each parameter in TechTarget’s guideline is of crucial importance, organizations are paying special attention in recent times to metrics that measure the time taken for identification/detection of threats, and the time taken to respond and remediate threats. The National Institute of Standards and Technology (NIST) (4) defines metrics as tools that ‘facilitate decision-making and improve performance and accountability through the collection, analysis, and reporting of relevant performance-related data.’ This makes metrics key performance indicators to assess the effectiveness of an organization’s cybersecurity setup and posture.
The metrics everyone is talking about
Thought leader Gartner (6) prefers to express the effectiveness of metrics as a combination of consistency of the system in terms of results, adequacy of the system to meet business needs, reasonableness in terms of their deployment vis a vis the organizational setup, and effectiveness of the system in producing the desired results. Mean Time To Detect/Identify (MTTD), and Mean Time To Repair/Remediate (MTTR) however are the buzzwords in effectiveness testing, widely regarded as the leading metrics for evaluating cybersecurity effectiveness.
MTTD is the average time taken between the onset of a system failure to identify a threat and its detection. The metric records the number of business hours (lapsed) between the moment an alert is triggered and the moment the cybersecurity team begins to investigate that alert. Also known as Mean Time to Identify (MTTI) and Mean Time to Respond (MTTR), this metric tests the agility and alertness of the system when dealing with threats. The metric will throw up possible incidences of alert fatigue (5) or staffing situations that could be contributory factors in the overall time taken to identify and respond to an incident.
The MTTR metric measures the average time it takes to control and remediate a threat after it has been identified. Crowdstrike (7) calls it a metric to ‘assess the performance of DevOps and ITOps, gauge the effectiveness of security processes, evaluate the effectiveness of security solutions, and measure the maintainability of systems.’
MTTR is widely acknowledged as a key factor in reducing disruption time in organizational processes as a consequence of a data breach. Hence reducing MTTR is being increasingly sought out by organizations as it has a direct impact on productivity due to its ability to ensure uptime with the restoration of business operations. TechTarget (8) is convinced that Security Orchestration Automation and Response (SOAR) tools are just the thing organizations need to ensure lower MTTD and MTTR rates.
Conclusion
It has been pointed out that the industry does not have a standard for organizations to benchmark their metrics. A SANS 2019 Incident Response survey (9) taken back in 2019, showed that 52.6% of organizations had an MTTD of less than 24 hours, and 67% of organizations had an MTTR of less than 24 hours. However, the study showed that over a 30-day period, the figures changed dramatically. 81.4% of respondents had an MTTD of 30 days or less, and 95.8% of respondents had an MTTR of 30 days or less.
Going forward, we are presented with a key question : Where does the future of security measurement lie? Many believe that effectiveness studies are headed for digitally-transformed measurement systems that offer the best of descriptive, predictive, and prescriptive analysis capabilities.
Reliability will be the watchword, and analytical metrics will be the go-to tools to reckon the reliability or effectiveness. C-Suites will have their eyes peeled on these metrics, going forward. And CISOs will have their hands full justifying the resilient systems that they have put in place, are producing results in the threat hunting, identifying, responding, and remedying space.
They will not need to be reminded that ‘the proof of the pudding is (indeed) in the eating’.
References:
- Evaluating Your Company’s Cybersecurity Strength: 12 Key Indicators (forbes.com)
- Monitoring and verifying cybersecurity controls effectiveness – Baker Tilly
- 12 key cybersecurity metrics and KPIs for businesses to track | TechTarget
- Metrics of Security (nist.gov)
- The Looming Threat of Fatigue, Stress, and Burnout in Cybersecurity – Aurora Systems Consulting Inc. (aurorait.com)
- 4 Metrics That Prove Your Cybersecurity Program Works (gartner.com)
- Mean Time to Repair Explained – CrowdStrike
- How SOAR helps improve MTTD and MTTR metrics | TechTarget
- MTTD and MTTR: Two Metrics to Improve Your Cybersecurity | Threatpost
