Más contenido relacionado La actualidad más candente (20) Similar a Security Training: Necessary Evil, Waste of Time, or Genius Move? (20) Security Training: Necessary Evil, Waste of Time, or Genius Move?1. © Copyright 2014 Denim Group - All Rights Reserved
Security Training: Necessary Evil,
Waste of Time or A Genius Move?
"
!
Research From Denim Group !
February 24, 2014!
John B. Dickson, CISSP !
@johnbdickson !
2. © Copyright 2014 Denim Group - All Rights Reserved
“I personally believe that training users in security
is generally a waste of time, and that the money
can be spent better elsewhere.”
Bruce Schneier
2
3. © Copyright 2014 Denim Group - All Rights Reserved
!
• Both trying to change behaviors!
– Target audience has more power to say “no”!
– Deadlines and releases drive training!
• For developers, infrequent, but more disruptive!
– 15-45 minutes vs. 2-day class !
3
How Developer Training is Different
4. © Copyright 2014 Denim Group - All Rights Reserved
Yet Training is Mandated
!
• PCI DSS 3.0
ü Train developers in secure coding techniques, including how to
avoid common coding vulnerabilities, and understanding how
sensitive data is handled in memory
ü Testing Procedures: 6.5.a: Examine software development policies
and procedures to verify that secure coding technique training is
required for developers, based on best practices and guidance
ü Testing Procedures: 6.5.b: Interview a sample of developers to
verify that they are knowledgeable in secure coding techniques
ü Testing Procedures: 6.5.c : Examine training records to verify that
software developers received training on secure coding techniques,
including how to avoid common coding vulnerabilities, and
understanding how sensitive data is handled in memory
4
5. © Copyright 2014 Denim Group - All Rights Reserved
• Harvard Business Review
– Large-scale organization development is rare
– Measurement of results is even rarer
• Workforce analytics rare
– More than 25% of survey respondents use little or no workforce
analytics
– The vast majority (>61%) report their use as tactical, ad hoc, and
disconnected from other key systems and processes
5
But Results Are Not Measured
6. © Copyright 2014 Denim Group - All Rights Reserved
• Software development field growing 30%
• Turnover
– Industry – 14-15%
– General IT – ~20%
– Software Development – ~20 – 30%
!
Sources: Bureau for Labor Statistics and Society of Human Resources
Management!
6
Growth & Turnover Spur
Sense of Urgency
7. © Copyright 2014 Denim Group - All Rights Reserved
Research Overview
• Focus: Assess the software developers depth of
software security knowledge
• Purpose: To measure the impact of software security
training on that level of understanding
• Survey size: 600 software developers surveyed in North
America (US and Canada)
• Vertical markets represented: financial, government,
retail, educational, technology, energy and healthcare
segments
7
8. © Copyright 2014 Denim Group - All Rights Reserved
Respondent Demographics
24
23
148
53
56
128
0
20
40
60
80
100
120
140
160
#ofValidResponses"
Company Size"
233
27
29
143
0
50
100
150
200
250
Software
Developer!
Quality
Assurance!
Architect! Other!#ofValidResponses"
Primary Job Function"
8
9. © Copyright 2014 Denim Group - All Rights Reserved
Respondent Demographics
Less than a Year"
10%"
1-2 Years"
8%"
2-4 Years"
12%"
4-7 Years"
11%"
More than 7 Years"
59%"
So#ware
Development
Experience
9
10. © Copyright 2014 Denim Group - All Rights Reserved
Respondent Demographics
168
86
56
27
95
0
20
40
60
80
100
120
140
160
180
None! Less than a Day! At least 1 day, but less than
2 days!
At least 2 days, but less
than 3 days!
More than 3 days!
#ofValidResponses"
Previous App Sec Training"
10
11. © Copyright 2014 Denim Group - All Rights Reserved
§ 15 Multiple Choice Quiz-Style Questions
§ Targeted at Software Developers
Ø Varied by years of experience, amounts of previous training,
primary job function, company industry and company size
§ Distribution:
Ø Online (before and after)
Ø Hard-copy questionnaires given to instructor-led class
trainees (before and after)
Ø Social media networks (sharing and some paid promotion
with incentives)
11
Methodology
12. © Copyright 2014 Denim Group - All Rights Reserved
Hypotheses
1. Most software developers do not have a basic
understanding of software security concepts.
2. Software security training can improve a developer’s
knowledge of security concepts in the short-term.
3. Certain industries, such as financial services, are
more likely to have software developers that are
already exposed to key software security concepts.
12
13. © Copyright 2014 Denim Group - All Rights Reserved
Sample Questions
If
an
a6acker
were
able
to
view
sensi:ve
customer
records
they
should
not
have
had
access
to,
this
would
be
a(n)_______breach.
___
Confiden3ality
___
Integrity
___
Availability
Authen:ca:on
is...
___
Proving
to
an
applica3on
that
the
user
is
who
they
claim
to
be
___
Confirming
that
the
user
is
allowed
to
access
a
certain
page
or
func3on
___Verifying
that
the
data
displayed
on
a
given
page
is
authen3c
___
Thoroughly
logging
all
of
a
user's
important
ac3vity
13
14. © Copyright 2014 Denim Group - All Rights Reserved
Sample Questions
Marking
a
cookie
as
“secure”
will...
___
Force
all
requests
that
use
the
cookie
to
use
SSL
___
Prevent
an
aPacker
from
guessing
its
value
___
Encrypt
it
when
sent
over
non-‐SSL
requests
___
Tell
the
browser
not
to
send
it
over
non-‐SSL
requests
Which
of
the
following
will
help
protect
against
XSS?
___
Only
accep3ng
URL
encoded
GET
parameters
___
Not
using
any
JavaScript
in
the
applica3on
___
Only
using
JavaScript
in
.js
files
stored
on
external
hosts
___
Encoding
special
HTML
characters
in
data
as
it
is
rendered
to
the
page
14
15. © Copyright 2014 Denim Group - All Rights Reserved
Key Survey Results
Architects and software developers had a much higher level of
knowledge than QA, yet in many organizations QA
has a material role in application security
61%
56%
64%
56%
52%
54%
56%
58%
60%
62%
64%
66%
So_ware
Developer
Quality
Assurance
Architect
Other
Average
%
Correct
(Primary
Job
Func:on)
31%
22%
34%
18%
0%
5%
10%
15%
20%
25%
30%
35%
40%
So_ware
Developer
Quality
Assurance
Architect
Other
Group
Passing
Rate
(Primary
Job
Func:on)
15
16. © Copyright 2014 Denim Group - All Rights Reserved
Key Survey Results
Slightly more than half of the respondents correctly answered
basic awareness questions on application but struggled
with ways to operationalize appsec concepts
83%
69%
11%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
#4: Cross Site Scripting
(XSS) causes malicious
scripts to execute on the
user's…
#7: Authentication is… #15: Which of the following
will help protect against
XSS?
Percentage That Answered Correctly
16
17. © Copyright 2014 Denim Group - All Rights Reserved
Key Survey Results
• Almost 100 percent could define input validation, demonstrating
a choppy understanding of advanced secure coding knowledge
• Nearly 90 percent correctly identified proper session IDs which
is reassuring
95%
88%
84%
86%
88%
90%
92%
94%
96%
#1:
Input
valida3on
is…
#11:
What
is
an
example
of
proper
session
IDs?
Percentage That Answered Correctly
17
18. © Copyright 2014 Denim Group - All Rights Reserved
59%
74%
0%
10%
20%
30%
40%
50%
60%
70%
80%
Before
Training
(All)
A_er
Training
(All)
Average
%
correct
Key Survey Results
• Retention rose by more than 25 percent after completing
secure coding training
18
19. © Copyright 2014 Denim Group - All Rights Reserved
Key Survey Results
Enterprises of more than 10,000 personnel had the lowest
secure coding knowledge
61%
64%
58%
60%
62%
58%
55%
56%
57%
58%
59%
60%
61%
62%
63%
64%
65%
1-‐24
Employees
25-‐99
Employees
100-‐499
Employees
500-‐2499
Employees
2500-‐9999
Employees
10,000
or
More
Employees
Average
%
Correct
(Company
Size)
33%
39%
26%
32%
32%
19%
0%
5%
10%
15%
20%
25%
30%
35%
40%
45%
1-‐24
Employees
25-‐99
Employees
100-‐499
Employees
500-‐2499
Employees
2500-‐9999
Employees
10,000
or
More
Employees
Group
Passing
Rate
(Company
Size)
19
20. © Copyright 2014 Denim Group - All Rights Reserved
Key Survey Results
The majority of the respondents had no prior secure
coding training, which might be surprising
168
86
56
27
95
0
20
40
60
80
100
120
140
160
180
None! Less than a Day! At least 1 day, but
less than 2 days!
At least 2 days, but
less than 3 days!
More than 3 days!
#ofValidResponses"
Previous App Sec Training"
20
21. © Copyright 2014 Denim Group - All Rights Reserved
Key Survey Results
There was no correlation between years of experience and knowledge of secure coding highlighting
the continued need for effective security training
59%
60%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
0
-‐
7
years
More
than
7
years
experience
Average
%
Correct
Years
of
Development
Experience
Percentage
of
Correct
Answers
(Years
of
Development
Experience)
21
22. © Copyright 2014 Denim Group - All Rights Reserved
Key Survey Results
The respondents that had more than 3 days of app sec
training in the past were able to answer more
than half of the questions correctly
29%
15%
27%
22%
34%
0%
5%
10%
15%
20%
25%
30%
35%
40%
None
Less
than
a
Day
At
least
1
day,
but
less
than
2
days
At
least
2
days,
but
less
than
3
days
More
than
3
days
Percentage
of
group
who
correctly
answered
70%
or
more
ques:ons
Amount
of
Previous
Applica:on
Security
Training
Group
Passing
Rate
(Previous
App
Sec
Training)
59%
57%
60%
59%
63%
54%
55%
56%
57%
58%
59%
60%
61%
62%
63%
64%
None
Less
than
a
Day
At
least
1
day,
but
less
than
2
days
At
least
2
days,
but
less
than
3
days
More
than
3
days
Average
%
Score
Amount
of
Previous
Applica:on
Security
Training
Average
%
Correct
(Previous
App
Sec
Training)
22
23. © Copyright 2014 Denim Group - All Rights Reserved
Key Survey Results
100% correctly identified where cross site scripting
executes after completing training, an increase of almost
20 percentage points
83%
100%
0%
20%
40%
60%
80%
100%
120%
Before Training After Training
Percentage With Correct Answers
#4: Where Cross Site Scripting (XSS) Executes
23
24. © Copyright 2014 Denim Group - All Rights Reserved
Key Survey Results
The number of respondents able to correctly identify
what is application security more than doubled after
training was complete
21%
55%
0%
10%
20%
30%
40%
50%
60%
Before Training After Training
Correctly Identified Application Security Term
24
25. © Copyright 2014 Denim Group - All Rights Reserved
Software Developers Learn Differently
than Companies “Teach”
• Teaching methods are formalized and structured in order to be repeatable
• Type of structures consist of:
– On-site & off-site classroom training
– E-learning for compliance
– Videos, webinars, etc.
!
25
26. © Copyright 2014 Denim Group - All Rights Reserved
Software Developers Learn Differently
than Companies “Teach”
• Teaching methods are formalized and structured in order to be repeatable
• Type of structures consist of:
– On-site & off-site classroom training
– E-learning for compliance
– Videos, webinars, etc.
!
26
27. © Copyright 2014 Denim Group - All Rights Reserved
So How Do Developers Learn?
• Informally and in an unstructured way via:!
• Blogs & RSS feeds !
• Social media with emphasis!
• Developer websites!
• Influential e-mail lists!
• Safarionline!
27
28. © Copyright 2014 Denim Group - All Rights Reserved
Don’t Ignore Basics of Training
• Refresher training is still needed!
• Training must be included in performance plans !
• Managers increasingly want an ROI!
28
30. © Copyright 2014 Denim Group - All Rights Reserved
• Software developers still largely do not understand key software
security concepts
• 73% of respondents “failed” the initial survey
• Average score of 59% before training
• However, software developers’ understanding of key software
security concepts did increase after training
• QA staff struggled to understand software security concept vs.
architects and software developers
30
CONCLUSION
31. © Copyright 2014 Denim Group - All Rights Reserved
Where do we Go from Here?
31
32. © Copyright 2014 Denim Group - All Rights Reserved
" "Questions and Answers?"
!
! !John B. Dickson!
! !@johnbdickson!
! !john@denimgroup.com!
32