Se ha denunciado esta presentación.
Se está descargando tu SlideShare. ×

Rooted con 2020 - from the heaven to hell in the CI - CD

Anuncio
Anuncio
Anuncio
Anuncio
Anuncio
Anuncio
Anuncio
Anuncio
Anuncio
Anuncio
Anuncio
Anuncio
Cargando en…3
×

Eche un vistazo a continuación

1 de 87 Anuncio
Anuncio

Más Contenido Relacionado

Presentaciones para usted (20)

Similares a Rooted con 2020 - from the heaven to hell in the CI - CD (20)

Anuncio

Más de Daniel Garcia (a.k.a cr0hn) (20)

Más reciente (20)

Anuncio

Rooted con 2020 - from the heaven to hell in the CI - CD

  1. 1. ! From The Heaven to Hell CI/CD Security concerns in the dark world RootedCON 2020
  2. 2. FOLLOW AND COMMENT THE TALK IN REAL TIME http://bit.do/rootedcon2020
  3. 3. WHO WE ARE Security research. Hacking tools developer, DevSecOps. Python developer. Daniel García (cr0hn) Can’t define myself. I go where my curiosity drives to. Most of the time goes bad. I process TeraBytes for breakfast. César Gallego @ggdaniel https://bit.do/cr0hn @CesarGallegoR https://bit.do/cesar-gallego
  4. 4. Disclaimer! Any opinions expressed are personal opinions and don’t represent our employer’s view in any way
  5. 5. Shared vocabulary Core Concepts
  6. 6. In software engineering, continuous integration (CI) is the practice of merging all developers' working copies to a shared mainline several times a day. Grady Booch first proposed the term CI in his 1991 method, although he did not advocate integrating several times a day. Extreme programming (XP) adopted the concept of CI and did advocate integrating more than once per day – perhaps as many as tens of times per day - Wikipedia What CONTINUOUS INTEGRATION is ? “ ”
  7. 7. Continuous delivery is a software engineering approach in which teams produce software in short cycles, ensuring that the software can be reliably released at any time and, when releasing the software, doing so manually. - Wikipedia What CONTINUOUS DEPLOYMENT is ? “ ”
  8. 8. CONTINUOUS DELIVERY VS CONTINUOUS DEPLOYMENT
  9. 9. Legacy Systems
  10. 10. No CD OpsDev
  11. 11. No CI/CD OpsDev
  12. 12. Hell Dev
  13. 13. Some Reasons
  14. 14. WHY CI/CD ? CI/CD It’s all about reaching customers faster and more frequently.
  15. 15. HOW CI/CD ? CI/CD has become so popular that now is a complete software landscape.
  16. 16. WHAT PIPELINE IS ? A pipeline is a chain of processing elements.
  17. 17. WHAT DevOps IS ? DevOps is a set of practices that combines software development (Dev) and information- technology operations (Ops) which aims to shorten the systems development life cycle and provide continuous delivery with high software quality
  18. 18. WHAT DevSecOps IS ? It’s SecDevOps when security is a value and not an obstacle.
  19. 19. Typical Setup
  20. 20. STEPS IN BUILDING SOFTWARE CONSTRUCTION
  21. 21. STEPS IN BUILDING SOFTWARE CONSTRUCTION User Code Building step Deployment step Production
  22. 22. STEPS IN BUILDING SOFTWARE CONSTRUCTION User Code Building step Deployment step Production
  23. 23. STEPS IN BUILDING SOFTWARE CONSTRUCTION User Code Building step Deployment step Production
  24. 24. Follow us down the rabbit hole Starting the journey
  25. 25. In source code
  26. 26. IN THE SOURCE CODE User Code Building step Deployment step Production
  27. 27. ● IDEs are great, but their configuration files shouldn't be in the source code repository. ● Please, use gitignore file. The IDE Leaks!
  28. 28. ● There is not way to set fine grained permissions at common source control versions system. ● Can’t set up permissions for specific files os directories. The BAD ROLE Granularity! Even could be worse... if the repository code is the developer computer
  29. 29. Keep the code safe at Source Code servers. Do backups. Keep the workplace safe. The DEV Machine as only source code server! ● There is a cowboy coder in your team? ● You have very tiny software in production only managed by one developer? ● You a have a guru developer? ● You have the correct security at the workplace? ● Are your developers BYOD?
  30. 30. No all StackOverflow people are good persons (or even humans) In STACK OVERFLOW Works Great! https://trojan-killer.net/the-most-copied-piece-of-java-code-on-stackoverflow-contains-an-error/
  31. 31. ● Are your developers using safe libraries? ● Are you check the libraries they use? ● Even more… they ask you for advice when choice a new library? All Libraries Allowed! https://securityintelligence.com/news/popular-javascript-library-for-node-js-infected-with-malware-to-empty-bitcoin- wallets/ You trust all libraries? so you know that all libraries are malware / vulnerabilidades free?
  32. 32. ● Passwords ● API keys ● Private keys ● …. SECRETS & LEAKS
  33. 33. In the building step
  34. 34. IN THE BUILDING STEP User Code Building step Deployment step Production
  35. 35. ● What if C.I. clone your code in read/write mode? ● What if the build script from user can modify the project source code in building step? Be sure you clone/download your source code as read/only in building step Non read-only source code in building time
  36. 36. Control Artifacts Repository ● What if a developer can publish an arbitrary code into artifacts / libraries repository? ● What if a developer publish a trojanized version of a library? ● What if a developer publish, intentionally, an artifact with known vulnerabilities? You must control who and what publish in the artifact repository
  37. 37. ● Usually C.I. / C.D. system stores sensitive information. ● They need them to access to productive environments, publish artifacts, perform some checks, etc ● They, usually, store as environment vars. The ENV Leak! https://docs.gitlab.com/ee/ci/variables/ Ensure you mark as Protected environment variables in your C.I. system
  38. 38. ● What if an user can execute anything in a Pipeline? ● What if the C.I. has not limited the output traffic? A reverse Shell in the Pipeline https://alionder.net/jenkins-script-console-code-exec-reverse-shell-java-deserialization/ Limit user permissions and output destinations
  39. 39. https://www.youtube.com/watch?v=QDGGPoK4gbk
  40. 40. Keep in mind that the bot user do what developer demands. Do not grant more access permissions that you’ll grant to developers. The mighty CI BOT ● Is your CI/CD executed as root? ● There is only one all mighty bot user? ● Your bot user has admin permissions over production?
  41. 41. ● Do you control what can download a developer when they runs in a pipeline? ● Do you control which command can launch a developer in a C.I. / C.D. configuration file? (Jenkinsfile, gitlab.yaml…) ● Is your C.I / C.D. in different network? Are you sure? The EVIL AGENT (1 / 3)
  42. 42. The EVIL AGENT (3 / 3) ➔ Limit internet access in the pipeline. ➔ Perform a correct hardening of the infrastructure ➔ Fix the execution permissions
  43. 43. ● Is your company using free tier services? ● Has your company GitHub Business account? The Greedy Service consumer! Keep in mind that free tier has limits by IP. Like GitHub, Google Maps… If your deploy rely on this services may be stuck if someone exceed the IP quota.
  44. 44. A git Bomb cannot be cloned. Only a problem with old git versions. Be aware in your older systems. The Git BOMB! ● Are your commits PGP signed? ● You know who can access wrigths? ● Are you using third party repositories?
  45. 45. Allow only some agents to publish images. Check docker layers contents. Check Dockerfile. The DOCKER HUB Leak! ● You have your own container registry? ● Do you check your Dockerfiles? ● Your pipelines has permissions and access to publish in docker hub?
  46. 46. ● Do you use the API? ● Do you control the CI/CD network access? Keep API Safe! Disable access from unnecessary places with a firewall. Do not install vulnerable Plugins.
  47. 47. ● What frequency has your deployment? ● Do you check your dependencies? ● If some deploy software become too old, do you perform again the security analysis? The FOREVER analysis! Monitor your artifacts and check for new vulnerabilities even if your pipeline do not trigger.
  48. 48. ● Do you keep your source code repository safe? ● Do you keep your developers machines safe? ● Do you backup your code? The SOURCE CODE ransomware! If a ransomware exists is because it works. Check permissions and do backups.
  49. 49. Break HISTORY! ● Do you monitor who modify your code? ● Do you have outsourcing in your team? ● You receive third party commits? Are you sure of your source code history. An attacker can break the source code history to cover their trail.
  50. 50. An attacker can convert any program into their own vector. Keep an eye on the software supply chain of your build environments. The evil COMPILER! ● Are your agents ephemeral? ● Are your containers trusted? ● Do you have specialized machine for some exotic language?
  51. 51. A very fat container can spend all free space and avoid new docker builds. A fat container make deploy a slow and error prone process. The Fat DOCKER! ● Do you inspect your Dockerfiles? ● Do you have Docker builds correctly configured? ● Do you control where layers are built?
  52. 52. Without the hash in FROM clause anyone can build a container using the name of popular one. A docker registry can add extra security if is well configured avoiding local cache. The evil DOCKER twin! ● Do you think that a containers registry is a complex overhead? ● You build a small number of containers? ● Your team don’t know how to write good Dockerfiles?
  53. 53. In the deployment step
  54. 54. IN THE DEPLOYMENT STEP User Code Building step Deployment step Production
  55. 55. Use firewall and security proxies to secure the internet access. Run FREE Internet! ● Some people think that CI/CD is “developer stuff” ● Good networking is critical in the place where a lot of security secrets live ● Agents must access internet only were they need to
  56. 56. Outsourcing happens, and an evil user can delay the CI CD action to prevent production fixes. At the same time an outsider attacker can exploit a vulnerable system. The EVIL insider! ● Your company use Outsourcing? ● You think that slow pipelines are harmless? ● You suffer the lack of dependency check?
  57. 57. Modify a Jar to add a trojan is very easy. Hard to detect and can be unbelievable persistent. The Trojan Jar! ● Do you suffer the Jar Hell? ● Your servers has outdated Java stuff? ● Your company has no java artifact repo?
  58. 58. Critical production infrastructure as code must be managed by security aware experts. Secure Infra as Code! ● Do you let your developers decide how to create production machines? ● You rely security on cloud promises?
  59. 59. ● ZIP Bomb is an old attack. ● The attack is very simple but very useful ● Some of system has basic routines to detect these kinds of attacks. The ZIP BOMB (1 / 4)
  60. 60. ● Major of packaged software is packed as a ZIP file: .jar, .war, .docx, .xlsx…. ● Some Application Servers auto deploy them when put files in specific path ● What if we put a ZIP bomb renamed as a valid packed Application for a Tomcat? The ZIP BOMB (2 / 4)
  61. 61. Perform a correct hardening of host and set conservative limits of files, CPU and memory that a processes can get The ZIP BOMB (4/ 4)
  62. 62. ● Memory bomb is type of attack that aims to fill all system memory. ● Not only RAM also SWAP is affected. ● If you don’t have limits in your host it can consume all of your HD space as a SWAP space. Memory BOMB (1 / 5)
  63. 63. ● What if you can run a memory bomb in a C.I. / C.D. system? ● What if the C.I. is deployed as multi-agent? Memory BOMB (2 / 5)
  64. 64. Jenkins agent 1 Jenkins agent 1 Jenkins agent 1 Jenkins behavior: 1 - You put a memory bomb in your Jenkinsfile Memory BOMB (3 / 5) 2 - The Jenkins Master send to the job to an Jenkins agent and it runs the pipeline and the memory bomb. So the Jenkins agent host break down Jenkins master 3 - Jenkins Master detect that the jobs was not finished. So the send the same job to another Jenkins Agent 4 - Jenkins agent runs memory bomb and… break down 5 - Go to step 2
  65. 65. ➔ Less Known but more effective in Docker. ➔ Today powerful computers can die very fast with no clue who pipeline is responsible. ➔ You can lost all your agents berore you find where the problem is. Memory BOMB (5 / 5)
  66. 66. ● Fork bomb is type of attack that aims exhaust a system by creating new processes recursively ● It very difficult to detect if you don’t have a very good log system configured ● Run in a Pipeline is so easy ● In multi-agent system the results are the same that with Memory Bomb Fork BOMB! (1 / 2)
  67. 67. In production
  68. 68. Monitor your CI/CD as the critical system that it is. Must Monitor The system! ● The health of CI/CD is very important ● Even if your pipelines are not executed for a long time you must perform periodic security checks ● Logs are useful information ● Metrics can warn you of misused infrastructure
  69. 69. The API contract must be fulfilled. No less, No more. The more is more problematic. Is your API Honest!? ● Do you use thread model on you APIs? ● How do you know all the endpoints that you have deployed? ● Are debug url opened in production?
  70. 70. Human interactions in production environments can be dangerous. Keep your production inmutable. Use infrastructure as code. No HUMANS Beyond this point! ● Do you know how the current state of your production? ● Do you allow manual “fixes” on production? ● Do you know how many people can access? ● Do you monitor when and how they access?
  71. 71. Security authentication proxies are good tools for this job. Trust the PASSWORD!? ● The passwords are supplied to third party software? ● No human can access but code can? ● Humans can write code to leak passwords
  72. 72. You can mount trojanized libraries like libc as a volume. Keep in mind you deploy volumes. The Docker MOUNTED Trojan libraries!
  73. 73. Containers are just a bunch o deltas on a file storage and a lot of genius around. Don’t forget that layers can be accessed. keep SECRETS safe! ● Do you store secrets in your containers? ● Do you store security configurations on your containers? ● Do you store intellectual property on your containers? ● Where are your containers published?
  74. 74. In the infraestructure
  75. 75. IN THE DEPLOYMENT STEP User Code Building step Deployment step Production
  76. 76. ● Old hack attack but useful ● Alias commands could be the best trojan in a system. ● There are very complicated to detect The Evil Alias! Perform a well hardening of your host systems & be careful with the bot users
  77. 77. ● Do you deploy the C.I. software in your infrastructure? ● Do you have a network isolation from the building software to the production machine? ● Do you remember the scan by using Jenkins? Can you imagine use that with Metasploit to Production machines? The Shared infra! ➔ PLEASE use isolated networks (VPC, VLAN o something applicable to your infrastructure) ➔ If your C.I. system need to access to the production machines use LIMITED access API keys.
  78. 78. ● Agents need a firewall to avoid lateral movements ● Don’t share the production network with any CI/CD software ● Don’t share the internal network with any CI/CD software keep NETWORK safe! If you can keep the CI/CD in their own network. Access control is mandatory.
  79. 79. Keep this in mind Wrap up
  80. 80. ● When your agents go down are less problematic if agents are deploy thru resilient infrastructure (like kubernetes). ● Containers from scratch are blessed simplicity, use often as you can. ● Deploy Critical software thru configurations (and trusted software) over complex code. ● You know about security just apply your knowledge in your CI/CD ● Don’t let that your automation tools become automated INsecurity The TIP Of the iceberg
  81. 81. ➔ Who will watch the watchers? Manage your CI/CD as a critical software (because it is). ➔ Assume that you have a lot of potential insiders attackers. ➔ Protect your C.I. as your production systems. ➔ Monitoring. Always monitoring. Not only in the building step. QUIS CUSTODIET IPSOS CUSTODES?
  82. 82. https://www.99cs.io http://book.99cs.io/

Notas del editor

  • Nuestra opinión es solo nuestra.
  • Las herramientas de integración contínua nacieron como un mecanismo para asegurar el funcionamiento del código cuando un grupo de desarrolladores trabajan en la misma base de código. En un principio para asegurar solamente que el código seguía compilando después de los “merges”; después para asegurar la funcionalidad comenzaron a poner énfasis en más fases, como todos los tipos de test que hay hoy en día.


    https://en.wikipedia.org/wiki/Continuous_delivery
  • La calidad del código creado con las técnicas de integración contínua aumentó tanto que los equipos tenían la confianza de productivizar sin necesidad de una revisión “manual”.

    De repente cada commit tenía muchas posibilidades de llegar a producción sin ningún tipo de intervención. Y para conseguir esto surgieron técnicas alrededor de este concepto, como “A/B testing”, “zero downtime” y muchas otras orientadas a la automatización más extrema del proceso.
  • CI but not CD
    Not CI nor CD
  • CI but not CD
    Not CI nor CD
  • CI but not CD
    Not CI nor CD
  • Por qué el CI/CD?

    CI/CD CONSISTE EN LLEGAR A PRODUCCIÓN MÁS RÁPIDO Y CON MAYOR FRECUENCIA

    Las empresas que dependen de la productivización de software han visto, con envidia, como las grandes tecnológicas hacían cambios en producción decenas de veces al día. Adaptándose al mercado en tiempo record.
    El software tradicionalmente tardaba meses (o años) en llegar a producción.

    En las aproximaciones más agresivas de esta filosofía se persigue que cada commit del código acabe en producción.



  • Hoy en día es rara la empresa que no usa, al menos, una de estas herramientas para automatizar sus procesos.
    Su adopción ha sido un proceso que ha mantenido la aceleración durante los últimos años.


  • Para poder llevar a cabo el despliegue decenas de veces al día hay que minimizar el factor humano, todo lo que sea posible.
    Un pipeline lleva a cabo una de las fases, como compilar, pasar test, crear infraestructura virtual o desplegar el artefacto en un servidor.



  • La forma de acabar con la típica batalla de operaciones vs desarrollo es crear un grupo mixto y una figura nueva (el DevOps).
    De esta forma los problemas que puedan surgir en cualquier fase son tratados sin discusiones sobre en qué tejado está la pelota.



  • Como pasó con desarrolladores y operadores los conflictos con seguridad fueron el siguiente obstáculo a derribar camino a una producción más rápida.
    Si los proyectos incluyen especialistas en seguridad minimizan riesgos y dejaban de ser “parados” por seguridad.

  • Si además estas personas tienen la seguridad comprometida es muy probable que infecten tus sistemas de trabajo. Sin contar que si pierde la máquina pierdes el código.
  • Repositorio de solo lectura
  • Repositorio de artefactos y dependencias controlado y seguro
  • Variables de entorno marcadas seguras
  • Reverse shell en el CI, un atacante puede tener acceso a todo lo que el usuario del CI tenga
  • Reverse shell en el CI, un atacante puede tener acceso a todo lo que el usuario del CI tenga
  • Exceso de privilegios al bot
  • Evil Agent - usar ci/cd para hacking
    Mediante un Pipeline arbitrario o ganando persistencia en el agente
  • Evil Agent - usar ci/cd para hacking
    Mediante un Pipeline arbitrario o ganando persistencia en el agente
  • Evil Agent - usar ci/cd para hacking
    Mediante un Pipeline arbitrario o ganando persistencia en el agente
  • Consumir límites de servicios públicos.
    Parece una tontería pero imagina que tu empresa crece mucho, y hemos dado github por sentado. Y de repente deja de funcionar el pipeline, y github parece caido solo para ti. Si necesitas un hotfix estás en problemas.
  • GitBomb (no en Jenkins)
  • Dockerhub leak mediante docker
  • El Api del CI/CD debe ser segura, y pon especial cuidado con credenciales guardadas en texto plano (enlace a jenkins)
  • Analisis de seguridad One Shot, lo hago en construcción y me olvido para siempre
  • Source Code Ramsonware. Si exsite el softaware de este tipo es que a la gente le funciona.
  • Romper el historico
  • Cuando empujas este tipo de contendores con varias versiones al registry puedes cargartelo
  • Acceso a internet
  • Evil insider - un complice maligno retrasa la salida a producción activamente para que un atacante externo pueda explotar una vulnerabilidad no detectada
  • Zip Bomb
  • Zip Bomb
  • Zip Bomb
  • Zip Bomb
  • Zip Bomb
  • Zip Bomb
  • En algunos casos el pc ha muerto tanto que no se ha grabado ni el log, el único indicio de ejecución es que el contador pierde los números de la ejecución fantasma.
  • En algunos casos el pc ha muerto tanto que no se ha grabado ni el log, el único indicio de ejecución es que el contador pierde los números de la ejecución fantasma.
  • En algunos casos el pc ha muerto tanto que no se ha grabado ni el log, el único indicio de ejecución es que el contador pierde los números de la ejecución fantasma.
  • En algunos casos el pc ha muerto tanto que no se ha grabado ni el log, el único indicio de ejecución es que el contador pierde los números de la ejecución fantasma.
  • En algunos casos el pc ha muerto tanto que no se ha grabado ni el log, el único indicio de ejecución es que el contador pierde los números de la ejecución fantasma.
  • In the same way a typical fork bomb can blown away your agents.
    A fork bomb is also hard to find, sometimes the pipeline ends correctly, and then your agent crash.
  • In the same way a typical fork bomb can blown away your agents.
    A fork bomb is also hard to find, sometimes the pipeline ends correctly, and then your agent crash.
  • Monitorización continua que enlaza con los análisis de seguridad en fases de construcción.
  • Comprobar que las apps cumplen las apis acordadas
  • No humans allowed in production - infraestructura inmutable
  • Exfiltar contraseñas solo accesibles en producción
  • Librería troyanizada como volumen - que parezca culpa de otro
  • Alias de comandos con mala baba
  • Alias de comandos con mala baba
  • Muchas veces el ci y el cd están en la misma
  • Infraestructure as code done by security aware professionals.
  • Infraestructure as code done by security aware professionals.

×