Some thoughts on how we can measure detectors within SOC/MDR. Alerts that ship with SIEM's (canned alerts) are generally of poor quality, but what are some metrics we can use to measure the quality of a detector.
10. LETS LOOK AT SOME SAMPLE ALERTS
PASSWORD SPRAY ATTEMPT DETECTED
▸ Looks at failed logons with multiple usernames.
SIMPLICITY
CERTAINTY
RESILIENCY
RELEVANCY
0 2,5 5 7,5 10
Rarely triggers falsely, Unlikely to not trigger.
No AI required here - AD audit requirements
Great resiliency, but can bypass with time.
100% relevant - common method to guess passwords.
11. LETS LOOK AT SOME SAMPLE ALERTS
SPIKE IN 400 ERRORS ON WEB APPLICATION
▸ Looks for a dramatic increase in HTTP 400 Errors.
SIMPLICITY
CERTAINTY
RESILIENCY
RELEVANCY
0 2,5 5 7,5 10
This one is kinda noisy,
some false positives.
No AI required here. Simple measurement
Web Apps are common targets.
Fairly Resilient
12. LETS LOOK AT SOME SAMPLE ALERTS
MULTIPLE UNIQUE LOGONS FROM SAME SOURCE
▸ Looks for multiple user accounts from same source.
SIMPLICITY
CERTAINTY
RESILIENCY
RELEVANCY
0 2,5 5 7,5 10
NOT!
Not too complicated, but needed tuning.
Nothing to measure against.
Somewhat - if it worked.
13. IN CLOSING
TAKE AWAYS
▸ Alert quality matters.
▸ Tune but don’t tune out.
▸ Chuck bad alerts out.
▸ Relevancy - Important but requires maturity