This presentation discusses how to properly measure the accuracy of Web Application Firewalls. The presentation explain the 4 attributes that must be measured (FP, FN, TP, TN) and how to properly calculate a WAF's accuracy.

1. WAF Accuracy Testing Done Properly
Introducing AWT framework
Ory Segal, Director of Threat Research

2. ©2015 AKAMAI | FASTER FORWARDTM
WAF Accuracy Lingo
• Imagine a WAF that protects against 100% of all possible attack vectors
…by blocking 100% of all HTTP requests
• Accurate WAF testing requires you to measure:
• How many real attacks got blocked (TP)
• How much valid requests were allowed through (TN)
• How much valid traffic was inappropriately blocked (FP)
• How many attacks were allowed through (FN)
• Lets talk about Precision, Recall, Accuracy, MCC…

3. ©2015 AKAMAI | FASTER FORWARDTM
Precision, Recall, Accuracy, MCC
% of blocked requests that were actual attacks
% of attacks that were actually blocked
% of decisions that were good decisions
* MCC: http://en.wikipedia.org/wiki/Matthews_correlation_coefficient
Correlation between WAF decisions
and actual nature of requests
Precision =
tp
tp+ fp
Recall =
tp
tp+ fn
Accuracy =
tp+tn
tp+tn+ fp+ fn
MCC =
tp×tn
(tp+ fp)(tp+ fn)(tn+ fp)(tn+ fn)

4. ©2015 AKAMAI | FASTER FORWARDTM
Lets Look at Some Examples
A WAF’s accuracy needs to be measured both in its ability to block attacks, as
well as it’s ability to allow good traffic through…
WAF Type Requests Valid Attacks Blocked TP TN FP FN P R A MCC
Real 1000 990 10 11 8 987 3 2 0.73 0.8 0.995 0.76
Off 1000 990 10 0 0 990 0 10 N/A 0 0.99 0
Always Block 1000 990 10 1000 10 0 990 0 0.01 1 0.01 0
Noisy 1000 990 10 31 8 967 23 2 0.26 0.8 0.975 0.45
Conservative 1000 990 10 2 2 990 0 8 1.00 0.2 0.992 0.45

5. ©2015 AKAMAI | FASTER FORWARDTM
WAF Testing Framework Requirements
• A tool that will send both valid traffic and real attacks
• Easy addition of test cases (both valid & attacks)
• Accuracy statistics gathering – FP, FN, TP, TN, P, R, A, MCC
• Rich info about each test that was sent – full request, response,
expected behavior, request nature
• Reporting capabilities

6. ©2015 AKAMAI | FASTER FORWARDTM
Introducing:
Akamai WAF Testing Framework

7. ©2015 AKAMAI | FASTER FORWARDTM
Akamai WAF Testing (AWT) Framework
• Written in Python
• Test cases are represented as textual files (.awt)
• Options to create or add new test cases:
• Write text files
• Use a “Burp Extender” to record web interaction (meaningful requests only)
• Transform Wireshark .pcap files (only ports HTTP traffic)
• Multithreaded – can be very fast, or very “considerate”
• Configurable and can work with any WAF
• Intuitive XML & HTML reports
• Easy debugging of FP/FN

8. ©2015 AKAMAI | FASTER FORWARDTM
AWT Built-In Test Cases
In order to accurately assess WAF, we collected test cases from the
following sources:
Retrieved valid traffic from Akamai’s Cloud Security
Intelligence big data platform
Recorded manual interaction with top “problematic”
web sites
Ported known “false positive” test cases from other
tools
Commercial web scanners
Popular SQLi tools
Exploits from the internet
(fuzzers, exploit-db, …
Traffic database is divided to 95% / 5%
Automatic crawling of Alexa Top 100 internet sites
Malicious traffic from
Akamai’s Cloud Security
Intelligence big data
platform

9. ©2015 AKAMAI | FASTER FORWARDTM
AWT Reports - Example