Application Assessment Metrics
•
1 like•544 views
Presentation by Yvette du Toit at ISSA in 2011. This presentation is about application assessment metrics and their challenges. Examples of Sensepost metrics are given.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (8)
Similar to Application Assessment Metrics
Similar to Application Assessment Metrics (20)
More from SensePost
More from SensePost (20)
Recently uploaded
Recently uploaded (20)
Application Assessment Metrics
- 1. ASSESSMENTS • VULNERABILITY MANAGEMENT • CONSULTING • TRAINING PRESENTED BY: Yvette du Toit
- 2. Agenda! Agenda! • Background! • Background! • Approach! • Approach! • Examples! • Challenges with • Examples! Application Security Metrics! • Challenges with Application Security Metrics! • Q&A! • Q&A! ASSESSMENTS • VULNERABILITY MANAGEMENT • CONSULTING • TRAINING
- 3. Agenda! Background! • Background! • As Security Consultants we write reports! • Approach! • Examples! – Test, analyse, write up findings, submit to client! • Challenges with • Issues still remain open – why?! Application Security Metrics! – Reports not say enough! • Q&A! – Question value report offer! • Solution – metrics / visualisation! – Graphs, colour, size etc! • First – letʼs take a look at what reports say…! – Qualitative ratings! – Best practice! ASSESSMENTS • VULNERABILITY MANAGEMENT • CONSULTING • TRAINING
- 4. Agenda! What do Reports Say?! • Background! • 2007 - 2011! • Approach! • Many words….! • Examples! • Challenges with • Content (Exec Summary, Technical Summary, Conclusion)! Application Security Metrics! • Are actions effective?! • Q&A! • What would be more valuable – comparison (time & peers)! • How do we use metrics?! Pages Words Assessments 638 224587 Re-Tests 137 28164 Total 775 252751 ASSESSMENTS • VULNERABILITY MANAGEMENT • CONSULTING • TRAINING
- 5. Agenda! Approach! • Background! • Metrics – definition! • Approach! • Definition! – Quantifiable! • Examples! – Characteristics! • Challenges with Application Security • 3 Metric Veterans:! Metrics! • Q&A! – Jacquith - “those that support decision making about risk for the purpose of managing that risk” ! – Marty – “a picture paints a thousand log records”! – Godin: “just because something is easy to measure doesnʼt mean itʼs important”! • NB: To measure what is important & that will yield “useful” information! – Examples of metrics not necessarily useful! ASSESSMENTS • VULNERABILITY MANAGEMENT • CONSULTING • TRAINING
- 6. Agenda! Useful?! • Background! • Metrics can be misleading! • Approach! • Example! • Examples! • Challenges with Application Security Metrics! • Q&A! ASSESSMENTS • VULNERABILITY MANAGEMENT • CONSULTING • TRAINING
- 7. Agenda! Useful?! • Background! • Metrics are not always 100% useful! • Approach! • Example! • Examples! • Challenges with Application Security Metrics! • Q&A! ASSESSMENTS • VULNERABILITY MANAGEMENT • CONSULTING • TRAINING
- 8. Agenda! Approach! • Background! • Why?: illustrate useful information! • Approach! – Recurring issues! • Introduction! • Examples! – Time required to compromise! • Challenges with – Top 10 list! Application Security Metrics! – Effectiveness of remediation! • Q&A! – Benchmarking! • Who? 7 organisations in financial sector! • When? 3 ½ years! • How? Data capture process! – Marco Slaviero (Head of R&D)! – Spreadsheet for data capture! – Report meta-data (project length, frameworks, dates etc.)! – Findings categorised (pre-defined list of vulns)! – Findings ranked (Impact, EoE, Threat metric)! • Normalisation ! – Allows for comparison across time and peers ! ASSESSMENTS • VULNERABILITY MANAGEMENT • CONSULTING • TRAINING
- 9. Agenda! Annual Distribution of Project (Days)! • Background! • Approach! • Examples! • Challenges with Application Security Metrics! • Q&A! ASSESSMENTS • VULNERABILITY MANAGEMENT • CONSULTING • TRAINING
- 10. Agenda! SensePost Metrics Proposal! • Background! • Metrics extracted from report data:! • Approach! • Our Metrics! – Timelines (plotting projects on timeline)! • Examples! – Basic counts and statistics (uncover counts)! • Challenges with • Number of projects! Application Security Metrics! • Number of days! • Q&A! • Number of words and pages in report! – Threat metrics (Findings per threat level)! – Bug class metrics (Findings across categories) ! – Top 10 list ! – Re-Test Metrics! – Benchmarks (comparison to peers)! ASSESSMENTS • VULNERABILITY MANAGEMENT • CONSULTING • TRAINING
- 11. Agenda! SensePost Metrics in Action: Timelines! • Background! • Useful?! • Approach! • Examples! • Challenges with Application Security Metrics! • Q&A! ! "#$%&! '()*&! !"#$%&'()'*++%++#%,-+' ./0' 112304' !"#$%&'()'5%67%+-+' 8/4' 108.2' 7(-9:' 443' 131438' ASSESSMENTS • VULNERABILITY MANAGEMENT • CONSULTING • TRAINING
- 12. Agenda! SensePost Metrics in Action: Threat Metrics! • Background! • Useful?! • Approach! • Examples! • Challenges with Application Security Metrics! • Q&A! ASSESSMENTS • VULNERABILITY MANAGEMENT • CONSULTING • TRAINING
- 13. Agenda! SensePost Metrics in Action: Bug Classes! • Background! • Useful?! • Approach! • Examples! • See 56% of findings occur in Top 11 bug classes! • Challenges with Application Security • 2008 Anomaly (No Re-Tests) ! Metrics! • Q&A! ASSESSMENTS • VULNERABILITY MANAGEMENT • CONSULTING • TRAINING
- 14. Agenda! SensePost Metrics in Action: Top 10! • Background! • Useful? ! • Approach! • Examples! • Challenges with Application Security Metrics! • Q&A! ASSESSMENTS • VULNERABILITY MANAGEMENT • CONSULTING • TRAINING
- 15. Agenda! SensePost Metrics in Action: Re-Test! • Background! • Useful?! • Approach! • Examples! • 29% Critical and 42% High-risk issues remain open ! • Challenges with Application Security Metrics! • Q&A! ASSESSMENTS • VULNERABILITY MANAGEMENT • CONSULTING • TRAINING
- 16. Agenda! SensePost Metrics in Action: Benchmarks! • Background! • Useful?! • Approach! • Examples! • Our client positioned 3rd (not highlighted here)! • Challenges with Application Security Metrics! • Q&A! ASSESSMENTS • VULNERABILITY MANAGEMENT • CONSULTING • TRAINING
- 17. Agenda! Challenges! • Background! • Bug counts vs bug classes! • Approach! – Bug counts – number of findings! • Examples! • Challenges with – Bug classes – categories! Application Security – 2 applications scenario (10 findings 1 bug class vs 1 finding in 10 bug classes)! Metrics! • Q&A! • Depth vs breadth! – Each occurrence – depth! – Each bug class - breadth! ASSESSMENTS • VULNERABILITY MANAGEMENT • CONSULTING • TRAINING
- 18. Agenda! Q&A! • Background! • Thank you! • Approach! • Examples! • Longer paper – mail me! • Challenges with Application Security • Email: yvette@sensepost.com! Metrics! • Q&A! • Contact: +27 79 509 8913! ASSESSMENTS • VULNERABILITY MANAGEMENT • CONSULTING • TRAINING