DevSecCon Boston 2018: My rage quit journey: configuring Netflix tools by Sarah Young

1. BOSTON 10-11 SEPT 2018
My ragequit journey: configuring
Netflix tools
SARAH YOUNG
BOSTON 10-11 SEPT 2018
My ragequit journey: configuring
Netflix tools
SARAH YOUNG

2. BOSTON 10-11 SEPT 2018

3. BOSTON 10-11 SEPT 2018
whoami
• Sarah Young, Security Architect at Versent.
• I’m from Melbourne in Australia.
• I help customers move their stuff into the
cloud securely.
• Worked in tech for the past 9ish years.
• I’ve worked in Europe, New Zealand and
Australia.
• I overuse memes and GIFs.
• Wannabe crazy bird lady.

4. BOSTON 10-11 SEPT 2018
If anyone knows Justin Trudeau, please let me know.

5. BOSTON 10-11 SEPT 2018
I am not a Christian author

6. BOSTON 10-11 SEPT 2018
Firstly…
• This talk is not an attack on Netflix.
• I love Netflix as both an end user of their service and a consumer of their
SecOps tools.
• Alas, I am also not on commission from Netflix.
• The aim of this talk is to demonstrate how everyone struggles with tools
from time-to-time.
• I want to try to reduce “FOFU”, “fear of F!%*ing up”.

7. BOSTON 10-11 SEPT 2018
Intro to Netflix tools
• I don’t have to introduce Netflix… I hope?!
• Netflix have been releasing Open Source tools since 2014.
• They release numerous types of tools:
• Big data
• Content encoding
• Insight, reliability and performance monitoring
• … and much more
• I’m going to focus on some of their security tools.

8. BOSTON 10-11 SEPT 2018
Just one more note…
• I’m aware that there are talks at other conferences and meetups where
companies and individuals talk about successful implementations of these
tools.
• This is not one of those talks.
• I will link to some of the happier Hollywood stories at the end of talk.

9. BOSTON 10-11 SEPT 2018
Tools I’m going to look at
• BLESS (Bastion's Lambda Ephemeral SSH Service)
• Security Monkey
• Repokid

10. BOSTON 10-11 SEPT 2018
The beginning of the journey…
• I was equipped with:
• Git Readmes.
• My work’s sandbox AWS account.
• Google.
• Slightly rusty Linux skills.
• Unlimited cans of fizzy drinks from
the fridge.
• My patience.

11. BOSTON 10-11 SEPT 2018
Don’t test the demo gods

12. BOSTON 10-11 SEPT 2018
BLESS – Qué?
• BLESS stands for Bastion's Lambda Ephemeral SSH Service.
• It’s an Internal Certificate Authority.
• Inside a Lambda function.
• Issues short-lived certificates for EC2 access.
• Certificates have 120 seconds validity by default.

13. BOSTON 10-11 SEPT 2018
BLESS – awscli is not my friend
• Create an AWS role, easy.
• Maybe my Python version is too new for awscli?
• Let’s uninstall Python3.

14. BOSTON 10-11 SEPT 2018
BLESS – saml2aws

15. BOSTON 10-11 SEPT 2018
Firstly…
• Cue lengthy Slack
discussion about how
Brew/Python/awscli suck.
• Let’s just reinstall awscli.

16. BOSTON 10-11 SEPT 2018
BLESS – Virtual-env is additionally not my friend
• False start, let’s go now.
• Have to force install
virtual-env.
• I’m using Docker.
• All goes well here.

17. BOSTON 10-11 SEPT 2018
BLESS – Certificates, KMS and Lambda are dope
• Generate certs just fine.
• Make keys in KMS just fine.
• Make Lambda function just fine.
• Things are going too well… surely?!
Accurate depiction of me at this point

18. BOSTON 10-11 SEPT 2018
BLESS – OSX, you make my life hard
• BLESS should be finished.
• Now to test it.
• I don’t have Boto3…
• … except I do.
• Dammit Python dependencies!

19. BOSTON 10-11 SEPT 2018
BLESS – Dammit Python
Credit: XKCD

20. BOSTON 10-11 SEPT 2018
BLESS – Dammit again Python

21. BOSTON 10-11 SEPT 2018
Sidenote

22. BOSTON 10-11 SEPT 2018
BLESS – What’s the first rule of security…?
• I don’t have creds (apparently).
• Turns out this is a bug in saml2aws.
• I should have updated to 2.7.0 before I started.
I deserve Trump shame for this fail.

23. BOSTON 10-11 SEPT 2018
BLESS – don’t do this

24. BOSTON 10-11 SEPT 2018
BLESS – do this

25. BOSTON 10-11 SEPT 2018
BLESS – Real-life issues
• Very little guidance on how to scale BLESS.
• “Deploy an Amazon Linux AMI” isn’t super
helpful.
• Re-scaling the application takes downtime.
• Debugging BLESS sucks.
• When pen testing BLESS, we had to expose
Unicreds.
• Defeats the object of pen testing somewhat.

26. BOSTON 10-11 SEPT 2018
BLESS – When devs don’t do what they’re told
• Got BLESS running through Jenkins.
• Devs still used our manually deployed bastion.
• ”Make it easy to do the right thing and hard to do the wrong thing”.
• Resources who maintained BLESS rolled off projects.
• Nuances introduced by devs could cause problems.

27. BOSTON 10-11 SEPT 2018
BLESS - scoreboard
• Instructions – 6/10
• Accuracy of instructions – 8/10
• Ease of configuration – 5/10
• Ragequit score – 7/10

28. BOSTON 10-11 SEPT 2018
Security Monkey – Qué?
• Security Monkey is a tool that monitors/alerts/reports one or more AWS
accounts for anomalies.
• Part of a larger suite of tools from Netflix known as the Simian Army.

29. BOSTON 10-11 SEPT 2018
Security Monkey – Deployment structure

30. BOSTON 10-11 SEPT 2018
Security Monkey - Hurrah, instructions!
• Hey, this one looks like it has a
decent walkthrough on Github.
• Let’s give it a go.

31. BOSTON 10-11 SEPT 2018
Security Monkey - Ah, maybe not yay after all.
• Oh wait… it’s kind of out of date…
M1 instances don’t exist any more.
Decide to wing it and pick an
M5. This is not free tier.

32. BOSTON 10-11 SEPT 2018
Security Monkey - When your lab messes things up
• Pro tip: never use a lab your colleague has only half configured.
• Instance was not accessible from external bastion host.
• Bastion host wouldn’t forward SSH keys to the Security Monkey instance.
• Cue numerous error messages and troubleshooting of security groups
and NACLs.

33. BOSTON 10-11 SEPT 2018
Security Monkey – Let’s build this
• Now for the interesting stuff.
• Let’s install this thing.
• Pull all the files from Git…
• Oops, in my enthusiasm I ran the commands for GCP and Openstack.

34. BOSTON 10-11 SEPT 2018
Security Monkey - Why doesn’t my instance recognise loopback?
• All going well until sudo keeps failing.
• My instance does not know it’s own loopback.
• Bad Ubuntu!
• Change to /etc/hosts fixed this.

35. BOSTON 10-11 SEPT 2018
Security Monkey - Python isn’t working
• When a guide posts something like this, you should probably pay
attention to it:
• Because when you don’t, you get this:

36. BOSTON 10-11 SEPT 2018
Security Monkey - Je ne parle pas anglais.
• Running in the virtual environment shell now, my bad.
• Run the commands to compile the web interface.
• Isn’t this installed by default?!
• This makes no sense.
• Rage level getting critical at this point.
• Accurate representation of my face.

37. BOSTON 10-11 SEPT 2018
Security Monkey - Who doesn’t love a 404?
• No idea why, but I had to re-generate the en_US locales.
• Then, success!

38. BOSTON 10-11 SEPT 2018
Security Monkey – I spoke too soon
• Now everything should be running, right?

39. BOSTON 10-11 SEPT 2018
Security Monkey – Mysterious directories
• NGNIX can’t find the UI pages to load.
• Much searching, much raging.
• Transpires that the NGNIX location path was incorrect.
• Files had been copied as
/usr/local/src/security_monkey/security_monkey/static…
There it is

40. BOSTON 10-11 SEPT 2018
Security Monkey – Damn those SSL certs
• Generate self-signed SSL certs.
• Getting an error from Chrome, success!
• STILL GETTING A 404.
• Remove SSL from the config, for now.
• I appreciate the irony as a security professional.

41. BOSTON 10-11 SEPT 2018
Security Monkey – Hello web UI!
• …aaaaand:

42. BOSTON 10-11 SEPT 2018
Security Monkey – Dude, where’s my login server?
• Pretty sure I’m supposed to have a login screen?
• That red error doesn’t look great.
• The Googles reveals that file permissions are a common cause of this
issue.
• Also need to restart the supervisor service.

43. BOSTON 10-11 SEPT 2018
Security Monkey – Success!

44. BOSTON 10-11 SEPT 2018
Security Monkey – Production issues
• Issue lists aren’t very detailed.
• Dashboard scores for the high score view are not update, but show fine
on the summary page.
• Daily summary emails don’t get sent out.

45. BOSTON 10-11 SEPT 2018
Security Monkey - Scoreboard
• Instructions – 8/10
• Accuracy of instructions – 7/10
• Ease of configuration – 5/10
• Ragequit score – 8/10

46. BOSTON 10-11 SEPT 2018
Repokid - Qué?
• Repokid uses Access Advisor provided by Aardvark to remove
permissions granting access to unused services from the inline policies
of IAM roles in an AWS account.
• “When used together, Aardvark and Repokid help us get closer to
the principle of least privilege without sacrificing speed or introducing
heavy process.” - Netflix

47. BOSTON 10-11 SEPT 2018
Repokid - Wow, these instructions are pretty light.
• Even by Netflix standards, these
are pretty light…
• Pull repo from Git.
• Create database.
• Create IAM roles.

48. BOSTON 10-11 SEPT 2018
Repokid – The downsides of using a lab
• Run out of elastic IPs.
• Reassign one, but now my terminal is angry with me.
*sigh*

49. BOSTON 10-11 SEPT 2018
Repokid – More Python woes…
• Instance says virtual env isn’t there (apparently).
• Instance also says there is no Git.
• Fair enough.
• Pull Git package.
• Try to pull repo from Github.
• There’s already a repokid directory?! Me

50. BOSTON 10-11 SEPT 2018
Repokid – Only one thing to do
TERMINATE

51. BOSTON 10-11 SEPT 2018
Repokid – Git and SSH key troubles
• Wash, rinse, repeat the previous slides.
• Accessing Git repo…

52. BOSTON 10-11 SEPT 2018
Repokid – Let’s try that again
• Generate fresh SSH keys.
• Add to my agent.
• Upload to Github.

53. BOSTON 10-11 SEPT 2018
Repokid – Never mentioned I needed a database
• Apparently I need a Dynamo DB.
• Use a small local one for dev purposes.
• Pull Java packages, etc. to run it.
• This seems to be working fine.

54. BOSTON 10-11 SEPT 2018
Repokid – Readmes with footnotes
• The footnotes for Repokid are important.
• Describe what roles need to be set up for the instance to work.
• Might have been useful further up the document…

55. BOSTON 10-11 SEPT 2018
Repokid - Fine tuning JSON
• Fine tune the JSON config file.
• Point at Aardvark, Dynamo DB and IAM role.
• Aaaaand…

56. BOSTON 10-11 SEPT 2018
Repokid – Production issues
• role.policies only checks inline policies. Attached policies are ignored.
• Generates heaps of alerts/errors in Lightsail.
• The advice for the moment is… just put up and shut up.
• (unless you’re going to write your own code to fix)

57. BOSTON 10-11 SEPT 2018
Repokid - Scoreboard
• Instructions – 2/10
• Accuracy of instructions – 2/10
• Ease of configuration – 7/10
• Ragequit score – 7/10

58. BOSTON 10-11 SEPT 2018
And I’m finished!

59. BOSTON 10-11 SEPT 2018
Lessons learned
• Read what instructions you have carefully…
• … but don’t be entirely beholden to them.
• Get your base packages and dependencies in order with your code.
• Have your supporting tools (terminal, Github, etc.) all in order.
• Don’t be afraid to try to run things slightly differently if it works better for
your environment.
• It’s not failing to ask for help if you’re really stuck.

60. BOSTON 10-11 SEPT 2018
What’s next?
• Diffy!
• Diffy is a triage tool to help digital forensics
and quickly identify compromised hosts on
which to focus their response.
• Diffy finds outliers among a group of very
similar hosts and highlights those for a
human investigator, who can then examine
those hosts more closely.
• So far… so little instruction.

61. BOSTON 10-11 SEPT 2018
Documents and links
• Netflix Open Source Software Center - https://netflix.github.io/
• Netflix tech blog - https://medium.com/netflix-techblog
• Netflix Git repository - https://github.com/Netflix
• Lyft’s implementation of BLESS -
https://www.youtube.com/watch?v=PMlT1raRMA0
• Versent’s saml2aws repository - https://github.com/Versent/saml2aws
• Versent’s unicreds repository - https://github.com/Versent/unicreds
• Sethkor’s BLESS repository – https://github.com/sethkor/blesskor
• Risky Business #486 Repokid episode - https://risky.biz/RB486/
• Netflix Security’s YouTube Channel - https://www.youtube.com/channel/UCCic-
LGj5o892PhU_xrWq-g

62. BOSTON 10-11 SEPT 2018
Thanks for not ragequitting
on my talk and going to
happy hour.
Questions?
@_sarahyo