Most application risk managers agree that training software developers to understand security concepts can be an important part of any software security program. Couple that with the Payment Card Industry, who mandate that developers should have training in secure coding techniques as laid out in their Data Security Standard. Yet others call developer training "compliance-ware," a necessary evil and a tax on software development in the enterprise. This presentation shares the results of a yearlong survey of nearly 1,000 software developers that captures their knowledge of application security before and after formal training. The survey queries developers from various backgrounds and industries, to better understand their exposure to secure development concepts and to capture a baseline for post-training improvements. The session also includes the results of a "retest" of a subset of respondents, to identify how much security knowledge they retained after a specific length of time. The results were surprising, and include information every application risk manager should know, particularly those who rely on training as part of an application security strategy.

1. © Copyright 2014 Denim Group - All Rights Reserved
Security Training: Necessary Evil,
Waste of Time or A Genius Move?
"
!
Research From Denim Group !
February 24, 2014!
John B. Dickson, CISSP !
@johnbdickson !

2. © Copyright 2014 Denim Group - All Rights Reserved
“I personally believe that training users in security
is generally a waste of time, and that the money
can be spent better elsewhere.”
Bruce Schneier
2

3. © Copyright 2014 Denim Group - All Rights Reserved
!
• Both trying to change behaviors!
– Target audience has more power to say “no”!
– Deadlines and releases drive training!
• For developers, infrequent, but more disruptive!
– 15-45 minutes vs. 2-day class !
3
How Developer Training is Different

4. © Copyright 2014 Denim Group - All Rights Reserved
Yet Training is Mandated
!
• PCI DSS 3.0
ü Train developers in secure coding techniques, including how to
avoid common coding vulnerabilities, and understanding how
sensitive data is handled in memory
ü Testing Procedures: 6.5.a: Examine software development policies
and procedures to verify that secure coding technique training is
required for developers, based on best practices and guidance
ü Testing Procedures: 6.5.b: Interview a sample of developers to
verify that they are knowledgeable in secure coding techniques
ü Testing Procedures: 6.5.c : Examine training records to verify that
software developers received training on secure coding techniques,
including how to avoid common coding vulnerabilities, and
understanding how sensitive data is handled in memory
4

5. © Copyright 2014 Denim Group - All Rights Reserved
• Harvard Business Review
– Large-scale organization development is rare
– Measurement of results is even rarer
• Workforce analytics rare
– More than 25% of survey respondents use little or no workforce
analytics
– The vast majority (>61%) report their use as tactical, ad hoc, and
disconnected from other key systems and processes
5
But Results Are Not Measured

6. © Copyright 2014 Denim Group - All Rights Reserved
• Software development field growing 30%
• Turnover
– Industry – 14-15%
– General IT – ~20%
– Software Development – ~20 – 30%
!
Sources: Bureau for Labor Statistics and Society of Human Resources
Management!
6
Growth & Turnover Spur
Sense of Urgency

7. © Copyright 2014 Denim Group - All Rights Reserved
Research Overview
• Focus: Assess the software developers depth of
software security knowledge
• Purpose: To measure the impact of software security
training on that level of understanding
• Survey size: 600 software developers surveyed in North
America (US and Canada)
• Vertical markets represented: financial, government,
retail, educational, technology, energy and healthcare
segments
7

8. © Copyright 2014 Denim Group - All Rights Reserved
Respondent Demographics
24
23
148
53
56
128
0
20
40
60
80
100
120
140
160
#ofValidResponses"
Company Size"
233
27
29
143
0
50
100
150
200
250
Software
Developer!
Quality
Assurance!
Architect! Other!#ofValidResponses"
Primary Job Function"
8

9. © Copyright 2014 Denim Group - All Rights Reserved
Respondent Demographics
Less than a Year"
10%"
1-2 Years"
8%"
2-4 Years"
12%"
4-7 Years"
11%"
More than 7 Years"
59%"
So#ware
Development
Experience
9

10. © Copyright 2014 Denim Group - All Rights Reserved
Respondent Demographics
168
86
56
27
95
0
20
40
60
80
100
120
140
160
180
None! Less than a Day! At least 1 day, but less than
2 days!
At least 2 days, but less
than 3 days!
More than 3 days!
#ofValidResponses"
Previous App Sec Training"
10

11. © Copyright 2014 Denim Group - All Rights Reserved
§ 15 Multiple Choice Quiz-Style Questions
§ Targeted at Software Developers
Ø Varied by years of experience, amounts of previous training,
primary job function, company industry and company size
§ Distribution:
Ø Online (before and after)
Ø Hard-copy questionnaires given to instructor-led class
trainees (before and after)
Ø Social media networks (sharing and some paid promotion
with incentives)
11
Methodology

12. © Copyright 2014 Denim Group - All Rights Reserved
Hypotheses
1. Most software developers do not have a basic
understanding of software security concepts.
2. Software security training can improve a developer’s
knowledge of security concepts in the short-term.
3. Certain industries, such as financial services, are
more likely to have software developers that are
already exposed to key software security concepts.
12

13. © Copyright 2014 Denim Group - All Rights Reserved
Sample Questions
If
an
a6acker
were
able
to
view
sensi:ve
customer
records
they
should
not
have
had
access
to,
this
would
be
a(n)_______breach.
___
Confiden3ality
___
Integrity
___
Availability
Authen:ca:on
is...
___
Proving
to
an
applica3on
that
the
user
is
who
they
claim
to
be
___
Confirming
that
the
user
is
allowed
to
access
a
certain
page
or
func3on
___Verifying
that
the
data
displayed
on
a
given
page
is
authen3c
___
Thoroughly
logging
all
of
a
user's
important
ac3vity
13

14. © Copyright 2014 Denim Group - All Rights Reserved
Sample Questions
Marking
a
cookie
as
“secure”
will...
___
Force
all
requests
that
use
the
cookie
to
use
SSL
___
Prevent
an
aPacker
from
guessing
its
value
___
Encrypt
it
when
sent
over
non-­‐SSL
requests
___
Tell
the
browser
not
to
send
it
over
non-­‐SSL
requests
Which
of
the
following
will
help
protect
against
XSS?
___
Only
accep3ng
URL
encoded
GET
parameters
___
Not
using
any
JavaScript
in
the
applica3on
___
Only
using
JavaScript
in
.js
files
stored
on
external
hosts
___
Encoding
special
HTML
characters
in
data
as
it
is
rendered
to
the
page
14

15. © Copyright 2014 Denim Group - All Rights Reserved
Key Survey Results
Architects and software developers had a much higher level of
knowledge than QA, yet in many organizations QA
has a material role in application security
61%
56%
64%
56%
52%
54%
56%
58%
60%
62%
64%
66%
So_ware
Developer
Quality
Assurance
Architect
Other
Average
%
Correct
(Primary
Job
Func:on)
31%
22%
34%
18%
0%
5%
10%
15%
20%
25%
30%
35%
40%
So_ware
Developer
Quality
Assurance
Architect
Other
Group
Passing
Rate
(Primary
Job
Func:on)
15

16. © Copyright 2014 Denim Group - All Rights Reserved
Key Survey Results
Slightly more than half of the respondents correctly answered
basic awareness questions on application but struggled
with ways to operationalize appsec concepts
83%
69%
11%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
#4: Cross Site Scripting
(XSS) causes malicious
scripts to execute on the
user's…
#7: Authentication is… #15: Which of the following
will help protect against
XSS?
Percentage That Answered Correctly
16

17. © Copyright 2014 Denim Group - All Rights Reserved
Key Survey Results
• Almost 100 percent could define input validation, demonstrating
a choppy understanding of advanced secure coding knowledge
• Nearly 90 percent correctly identified proper session IDs which
is reassuring
95%
88%
84%
86%
88%
90%
92%
94%
96%
#1:
Input
valida3on
is…
#11:
What
is
an
example
of
proper
session
IDs?
Percentage That Answered Correctly
17

18. © Copyright 2014 Denim Group - All Rights Reserved
59%
74%
0%
10%
20%
30%
40%
50%
60%
70%
80%
Before
Training
(All)
A_er
Training
(All)
Average
%
correct
Key Survey Results
• Retention rose by more than 25 percent after completing
secure coding training
18

19. © Copyright 2014 Denim Group - All Rights Reserved
Key Survey Results
Enterprises of more than 10,000 personnel had the lowest
secure coding knowledge
61%
64%
58%
60%
62%
58%
55%
56%
57%
58%
59%
60%
61%
62%
63%
64%
65%
1-­‐24
Employees
25-­‐99
Employees
100-­‐499
Employees
500-­‐2499
Employees
2500-­‐9999
Employees
10,000
or
More
Employees
Average
%
Correct
(Company
Size)
33%
39%
26%
32%
32%
19%
0%
5%
10%
15%
20%
25%
30%
35%
40%
45%
1-­‐24
Employees
25-­‐99
Employees
100-­‐499
Employees
500-­‐2499
Employees
2500-­‐9999
Employees
10,000
or
More
Employees
Group
Passing
Rate
(Company
Size)
19

20. © Copyright 2014 Denim Group - All Rights Reserved
Key Survey Results
The majority of the respondents had no prior secure
coding training, which might be surprising
168
86
56
27
95
0
20
40
60
80
100
120
140
160
180
None! Less than a Day! At least 1 day, but
less than 2 days!
At least 2 days, but
less than 3 days!
More than 3 days!
#ofValidResponses"
Previous App Sec Training"
20

21. © Copyright 2014 Denim Group - All Rights Reserved
Key Survey Results
There was no correlation between years of experience and knowledge of secure coding highlighting
the continued need for effective security training
59%
60%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
0
-­‐
7
years
More
than
7
years
experience
Average
%
Correct
Years
of
Development
Experience
Percentage
of
Correct
Answers
(Years
of
Development
Experience)
21

22. © Copyright 2014 Denim Group - All Rights Reserved
Key Survey Results
The respondents that had more than 3 days of app sec
training in the past were able to answer more
than half of the questions correctly
29%
15%
27%
22%
34%
0%
5%
10%
15%
20%
25%
30%
35%
40%
None
Less
than
a
Day
At
least
1
day,
but
less
than
2
days
At
least
2
days,
but
less
than
3
days
More
than
3
days
Percentage
of
group
who
correctly
answered
70%
or
more
ques:ons
Amount
of
Previous
Applica:on
Security
Training
Group
Passing
Rate
(Previous
App
Sec
Training)
59%
57%
60%
59%
63%
54%
55%
56%
57%
58%
59%
60%
61%
62%
63%
64%
None
Less
than
a
Day
At
least
1
day,
but
less
than
2
days
At
least
2
days,
but
less
than
3
days
More
than
3
days
Average
%
Score
Amount
of
Previous
Applica:on
Security
Training
Average
%
Correct
(Previous
App
Sec
Training)
22

23. © Copyright 2014 Denim Group - All Rights Reserved
Key Survey Results
100% correctly identified where cross site scripting
executes after completing training, an increase of almost
20 percentage points
83%
100%
0%
20%
40%
60%
80%
100%
120%
Before Training After Training
Percentage With Correct Answers
#4: Where Cross Site Scripting (XSS) Executes
23

24. © Copyright 2014 Denim Group - All Rights Reserved
Key Survey Results
The number of respondents able to correctly identify
what is application security more than doubled after
training was complete
21%
55%
0%
10%
20%
30%
40%
50%
60%
Before Training After Training
Correctly Identified Application Security Term
24

25. © Copyright 2014 Denim Group - All Rights Reserved
Software Developers Learn Differently
than Companies “Teach”
• Teaching methods are formalized and structured in order to be repeatable
• Type of structures consist of:
– On-site & off-site classroom training
– E-learning for compliance
– Videos, webinars, etc.
!
25

26. © Copyright 2014 Denim Group - All Rights Reserved
Software Developers Learn Differently
than Companies “Teach”
• Teaching methods are formalized and structured in order to be repeatable
• Type of structures consist of:
– On-site & off-site classroom training
– E-learning for compliance
– Videos, webinars, etc.
!
26

27. © Copyright 2014 Denim Group - All Rights Reserved
So How Do Developers Learn?
• Informally and in an unstructured way via:!
• Blogs & RSS feeds !
• Social media with emphasis!
• Developer websites!
• Influential e-mail lists!
• Safarionline!
27

28. © Copyright 2014 Denim Group - All Rights Reserved
Don’t Ignore Basics of Training
• Refresher training is still needed!
• Training must be included in performance plans !
• Managers increasingly want an ROI!
28

29. © Copyright 2014 Denim Group - All Rights Reserved
Incentives Matter!
!
29

30. © Copyright 2014 Denim Group - All Rights Reserved
• Software developers still largely do not understand key software
security concepts
• 73% of respondents “failed” the initial survey
• Average score of 59% before training
• However, software developers’ understanding of key software
security concepts did increase after training
• QA staff struggled to understand software security concept vs.
architects and software developers
30
CONCLUSION

31. © Copyright 2014 Denim Group - All Rights Reserved
Where do we Go from Here?
31

32. © Copyright 2014 Denim Group - All Rights Reserved
" "Questions and Answers?"
!
! !John B. Dickson!
! !@johnbdickson!
! !john@denimgroup.com!
32