Which metrics should we use? You might expect an “it depends” answer, but there are some metrics that are important for any application security program, regardless of audience or goals. We’ll take a look at a few of them in this post.
3. 3
Why application security metrics?
Sometimes you need:
1. To communicate to your sponsors what you’re doing with the money they
provided for the program.
2. A way to communicate with your development teams that is anchored in
something more than just encouragement.
3. A tool to show yourself how much progress you’re making.
5. 5
It’s a hostile
environment out there
Applications have been a top vector for data breaches over
the last five years because they’re not coded with security
in mind. The software industry’s shift to composing
applications via pre-built—some would say “pre-0wned”—
components has made it more challenging for security
teams by introducing risk via the software supply chain.
So application security is important, but
how do you show progress?
8. You have to have
some way of
measuring the quality
of applications; it
should be aligned with
the needs of the
business.
A lot of your program
measurements are
going to be anchored
in how well your
portfolio does against
a policy.
But what sort of
pass rate should
you expect?
11. 11
When vulnerabilities are
all around you might feel
like your world is on fire.
Let’s try to get our arms
around how common
some of these fatal flaws
really are.
14. You know that guy,
the one who always
insists that the hole
you’re in isn’t as
deep as you think it
is…
15. It turns out that’s true
of AppSec. There are
a lot of people out
there making their
applications safer,
never accepting “no”
for an answer. And it
turns out that tracking
the flaws fixed can be
powerfully
motivational.
17. 17
Source: Veracode State of Software Security vol. 6: https://info.veracode.com/state-of-
software-security-report-volume6.html
How? Empower developers
• Customers in the financial services and manufacturing verticals are
successfully fixing between 65% and 81% of the flaws found in their
applications. Applications undergoing remediation coaching (readouts)
reduce application risk 2.5x more than those that don’t, as measured by
average flaw density per MB
Source: Veracode State of Software Security vol. 6:
https://info.veracode.com/state-of-software-security-report-volume6.html
20. 20
Which One?
It depends. Just as there’s no “one” lineup of a super hero team, you may find you need a different
set of metrics depending on the goals of your program—developer training completion, for instance,
or percent of applications undergoing automated testing. Ultimately it’s up to you, and the needs of
your business.
21. 21
Answers Key Questions for CISOs
• Which industries are doing the best job of reducing
application-layer risk ?
• Do I have more serious vulnerabilities than my peers?
• What percentage of vulnerabilities do my peers remediate?
• How many of our applications should pass the OWASP
Top 10 when initially assessed?
• What are the Top 10 most common vulnerabilities in our
vertical?
• How can I reduce more risk in my organization’s
applications?
Notas del editor
Sometimes it seems like application security programs are a never ending chasm. Why do we need to measure?
For several important reasons—
Sometimes you need to communicate to your sponsors what you’re doing with the money they provided for the program.
Sometimes you need a way to communicate with your development teams that is anchored in something more than just encouragement.
Sometimes you need a tool to show yourself how much progress you’re making.
Sometimes it seems like application security programs are a never ending chasm. Why do we need to measure?
For several important reasons—
Sometimes you need to communicate to your sponsors what you’re doing with the money they provided for the program.
Sometimes you need a way to communicate with your development teams that is anchored in something more than just encouragement.
Sometimes you need a tool to show yourself how much progress you’re making.
Don’t forget – it’s a hostile environment out there. Applications have been a top vector for data breaches over the last five years (see the Verizon Data Breach Reports) because they’re not coded with security in mind. The software industry’s shift to composing applications via pre-built—some would say “pre-0wned”—components has made it more challenging for security teams by introducing risk via the software supply chain. So application security is important, but how do you show progress?
Don’t forget – it’s a hostile environment out there. Applications have been a top vector for data breaches over the last five years (see the Verizon Data Breach Reports) because they’re not coded with security in mind. The software industry’s shift to composing applications via pre-built—some would say “pre-0wned”—components has made it more challenging for security teams by introducing risk via the software supply chain. So application security is important, but how do you show progress?
We’re going to walk through four ways to look at your portfolio of applications and benchmark it against other organizations. Each metric has its strength but they’re definitely better together. Let’s get started…
I think of policy compliance as the bedrock measurement (pardon the pun) of AppSec. You have to have some way of measuring the quality of applications; it should be aligned with the needs of the business. A lot of your program measurements are going to be anchored in how well your portfolio does against a policy.
But what sort of pass rate should you expect? There’s the problem…
Let’s look at the industry perspective for a second—using a kind-of-generic policy, the OWASP Top 10, to look at how your peers are doing.
Not well.
(Highlight relatively high pass rate in FinSvc which is still bad news – more than 50% fail)
OK, so now that we understand how bad the problem is, how do we fix it? One way is to understand the types of issues we’re facing.
When vulnerabilities are all around you might feel like your world is on fire. Let’s try to get our arms around how common some of these fatal flaws really are.
It’s important to note that not every application is subject to the same risks. For instance, SQL Injection, a leading cause of data loss, is only present in about 30% of applications (40% if you’re in the government). Note though that crypto is found in 45 to 80% of applications depending on industry—a problem if your customers or regulators require you to protect sensitive data.
Okay, so we know how good (or bad) our applications are, and we know what the nature of the vulnerabilities are. Now what?
You know that guy, the one who always insists that the hole you’re in isn’t as deep as you think it is…
It turns out that’s true of AppSec. There are a lot of people out there making their applications safer, never accepting “no” for an answer. And it turns out that tracking the flaws fixed can be powerfully motivational.
You know that guy, the one who always insists that the hole you’re in isn’t as deep as you think it is…
It turns out that’s true of AppSec. There are a lot of people out there making their applications safer, never accepting “no” for an answer. And it turns out that tracking the flaws fixed can be powerfully motivational.
Some industries are making huge progress in fixing the vulnerabilities they’ve found. In fact across all Veracode’s customers, they fixed 3 out of every four flaws found by automated scans last year.
So a strategy for reducing risk is “real-time” developer education, enabling developers to go faster.
Policy, top flaw prevalence, fix rate. What’s our fourth metric?