Redis & ZeroMQ: How to scale your applicationrjsmelo
Presented at #PHPLX 11 July 2013
When you need to do some heavy processing how do you scale you application?
You can use Redis and ZeroMQ to leverage the heavy work for you!
With this presentation we will know more about this two technologies and how they can be used to help solve problems with the performance and scalability of your application.
Going from zero to Puppet by Pedro Pessoa, Operations Engineer at Server Density.
Abstract: Using out-of-the-box Puppet for non-sysadmin work - steps from going from no config management to managing 100 nodes and allowing non-sysadmin tasks to be performed.
Speaker Bio: Linux admin for 10+ years. Java/Python/C developer 12+ years. Ops engineer at http://www.serverdensity.com - a hosted server and website monitoring service. Currently processing 12TB+ per month into MongoDB running on dedicated and virtual instances.
www.serverdensity.com/puppetcamp/
Redis is being used as a message queue to asynchronously process image uploads on a website for gaming screenshots. When a user uploads images, the application server adds a message to the Redis queue containing metadata about the upload. A separate process polls the queue and processes each upload by resizing images, creating database entries, and more. This allows upload processing to happen in the background without blocking the user.
A talk I held at perl mongers Wellington about RabbitMQ and Net::AMQP. A brief introduction to RabbitMQ and some basic code samples on how to use it from perl.
A talk I gave at WordCamp Sofa 2016 on measuring and optimizing memory usage, dealing with memory related errors, as well as monitoring server memory health.
The document discusses Nouka, an open source inventory management tool for Linux. Nouka consists of three parts - Nouka data collector, Naya data store, and Yaoya data converter. Nouka data collector runs commands periodically on Linux machines and sends the results to Naya data store. Naya uses Fluentd and MongoDB to store the collected data. Yaoya then converts and outputs the data in various formats like JSON, CSV for analysis. Overall, Nouka provides an automatic and periodic way to collect and centralize inventory data from Linux machines.
agri inventory - nouka data collector / yaoya data convertorToshiaki Baba
This document provides instructions for setting up and using an agri inventory system called nouka and yaoya. Nouka collects data from servers using commands and sends it to the naya data store, which uses fluentd and MongoDB. It explains the components, data formats, and provides steps to get the required software and set up the system.
The document discusses extending Nginx functionalities with Lua. It provides an overview of Nginx architecture and how the lua-nginx-module allows running Lua scripts inside Nginx. This provides a powerful and performant programming environment while leveraging Nginx's event-driven architecture. Examples show how to access Nginx variables and APIs from Lua, issue subrequests, and handle requests non-blockingly using cosockets. Libraries like lua-resty-memcached reuse these extensions to build applications in a scalable manner.
Redis & ZeroMQ: How to scale your applicationrjsmelo
Presented at #PHPLX 11 July 2013
When you need to do some heavy processing how do you scale you application?
You can use Redis and ZeroMQ to leverage the heavy work for you!
With this presentation we will know more about this two technologies and how they can be used to help solve problems with the performance and scalability of your application.
Going from zero to Puppet by Pedro Pessoa, Operations Engineer at Server Density.
Abstract: Using out-of-the-box Puppet for non-sysadmin work - steps from going from no config management to managing 100 nodes and allowing non-sysadmin tasks to be performed.
Speaker Bio: Linux admin for 10+ years. Java/Python/C developer 12+ years. Ops engineer at http://www.serverdensity.com - a hosted server and website monitoring service. Currently processing 12TB+ per month into MongoDB running on dedicated and virtual instances.
www.serverdensity.com/puppetcamp/
Redis is being used as a message queue to asynchronously process image uploads on a website for gaming screenshots. When a user uploads images, the application server adds a message to the Redis queue containing metadata about the upload. A separate process polls the queue and processes each upload by resizing images, creating database entries, and more. This allows upload processing to happen in the background without blocking the user.
A talk I held at perl mongers Wellington about RabbitMQ and Net::AMQP. A brief introduction to RabbitMQ and some basic code samples on how to use it from perl.
A talk I gave at WordCamp Sofa 2016 on measuring and optimizing memory usage, dealing with memory related errors, as well as monitoring server memory health.
The document discusses Nouka, an open source inventory management tool for Linux. Nouka consists of three parts - Nouka data collector, Naya data store, and Yaoya data converter. Nouka data collector runs commands periodically on Linux machines and sends the results to Naya data store. Naya uses Fluentd and MongoDB to store the collected data. Yaoya then converts and outputs the data in various formats like JSON, CSV for analysis. Overall, Nouka provides an automatic and periodic way to collect and centralize inventory data from Linux machines.
agri inventory - nouka data collector / yaoya data convertorToshiaki Baba
This document provides instructions for setting up and using an agri inventory system called nouka and yaoya. Nouka collects data from servers using commands and sends it to the naya data store, which uses fluentd and MongoDB. It explains the components, data formats, and provides steps to get the required software and set up the system.
The document discusses extending Nginx functionalities with Lua. It provides an overview of Nginx architecture and how the lua-nginx-module allows running Lua scripts inside Nginx. This provides a powerful and performant programming environment while leveraging Nginx's event-driven architecture. Examples show how to access Nginx variables and APIs from Lua, issue subrequests, and handle requests non-blockingly using cosockets. Libraries like lua-resty-memcached reuse these extensions to build applications in a scalable manner.
The document contains instructions for creating various prank scripts that can be run on other people's computers without permission to annoy or confuse them. It includes scripts that continuously pop up messages, repeatedly open and close the CD tray, enable caps lock, type text in Notepad, and more. Most of the scripts are simple batches or VBScripts that can be easily created and run. The document encourages modifying the scripts for different purposes and pranking other users.
This document describes using Puppet to launch and maintain EC2 instances with pre-defined software and configurations. It discusses building AMIs with pre-installed but unconfigured software, and using Puppet classes to activate and configure specific packages, services, and monitoring tools on the instances. The classes split package, service, configuration, and monitoring roles, and can be included and parameterized to deploy NTP in different roles on instances.
Czy wiesz co potrafi zrobić twój serwer reverse-proxy? Wydaje Ci się, że żeby zrobić sprytny routing / uwierzytelnianie / autoryzację (niepotrzebne skreślić) między serwisami musisz go napisać w Javie lub jako moduł w C? A co jeżeli odpalanie JVM tylko po to, żeby do każdego żądania http dokleić jeden nagłówek to armata na wróbla? Zwłaszcza, że prawie na pewno gdzieś tam po drodze mijasz nginx... Zapraszam Cię do świata idealnej symbiozy nginx i Lua.
The document lists several dangerous commands that can be run on a Linux or Unix system and cause data loss or system instability. It also provides basic commands for file operations, package management, networking tasks and changing permissions and passwords. The dangerous commands include deleting all files in the root folder, formatting a partition, executing a fork bomb to overload the system, and overwriting the file system on a designated device. The basic commands cover file listing, copying, downloading, creating directories and viewing directory information. It also explains Linux file permissions in terms of read, write and execute attributes.
This document discusses Composer, an open source tool for dependency management in PHP. It describes what Composer is, how to install it, how to define dependencies in a composer.json file and composer.lock file, how Composer generates autoload files and installs vendor libraries, and some common Composer commands. It also provides information on joining the community and lists sources for more documentation on Composer.
Ondřej Šika: Docker, Traefik a CI - Mějte nasazené všeny větve na kterých pra...Develcz
The document describes setting up Docker, Traefik, and CI/CD pipelines. It includes a docker-compose.yml configuration file for Traefik that sets up port forwarding and SSL termination. It also includes a .gitlab-ci.yml file that defines a deploy job that builds a Docker image, pushes it to Docker Hub, and deploys it to a server using Traefik routing.
PSR-7 and PSR-15 are PHP standards for representing HTTP messages and server request handling. PSR-7 defines interfaces for HTTP messages like requests and responses, while PSR-15 defines interfaces for request handlers and middleware. These standards provide a common way for PHP libraries to interact with HTTP messages and requests, improving interoperability. They also allow applications to be built in a way that is compatible with evolving PHP frameworks and tools. Adopting these PSRs helps create a solid foundation for building HTTP applications and middleware in PHP.
Buffer overflow occurs when too much data is inserted into a buffer than it can handle, which may lead to arbitrary code execution if a memory pointer is overwritten. The document provides an example of a buffer overflow happening when user input longer than the buffer size is read using gets() into the buffer without limits. To prevent this, fgets() should be used instead, as it allows specifying the maximum number of characters to read from input.
OlinData Puppet Presentation for MOSC 2012Walter Heck
Walter Heck gave a presentation on Puppet at the 2012 Malaysia Open Source Conference. He introduced OlinData, which provides Puppet consulting and training, and discussed how Puppet allows for scalable, accountable, and versioned infrastructure configuration compared to manual methods. Heck also covered OlinData's Puppet training schedule and consulting services to help companies implement and improve their Puppet usage.
This document provides instructions for setting up a CoreOS cluster with services like MongoDB, SkyDNS, HAProxy, and Datadog monitoring. It describes using systemd unit files to run services like MongoDB and SkyDNS in Docker containers managed by Fleet. Etcd is used for service discovery and configuration management with ConfD. HAProxy provides load balancing of services, and Datadog monitors the cluster. Cloud-init scripts initialize new nodes joining the cluster.
This document provides an overview of basic Linux commands for tasks like file management, installation, networking, compression, searching, SSH, permissions and more. It explains commands for changing directories, listing files, copying/moving files, installing packages, pinging hosts, compressing files and directories, searching for patterns, remote login via SSH, and managing processes and system information.
An (abridged) Ruby Plumber's Guide to *nixEleanor McHugh
This document provides an overview of using Ruby to interface with Unix processes, files, and interprocess communication. It discusses using the Kernel module to spawn and manage processes, reading and writing files using IO methods and fcntl constants, communicating between processes via pipes and sockets, directly calling Unix syscalls via Kernel.syscall and the ruby/dl library, and examples of using POSIX semaphores and memory management with ruby/dl. Further reading links are provided for more in-depth information.
This document provides tips and tricks for using the command line more efficiently. It discusses using SSH configuration files and the screen command to manage terminal sessions. Other topics covered include compressing files with tar, navigating directories more easily, monitoring log files, managing background processes, and using shortcuts to save time typing commands.
This document provides an overview of configuration management with Cfengine. It discusses what Cfengine is and how it can be used to ensure files and processes are configured properly across multiple hosts. It then describes how to get started with Cfengine, including installing it, writing policies to configure a single host, and setting up a basic infrastructure to distribute policies to multiple hosts on a network. Key components like cfagent, cfservd, classes, and policies are defined. Examples of Cfengine configuration files are also provided.
This document provides instructions for installing the Nginx web server from source on Linux. It describes downloading the source code, verifying the signature, compiling it with make, and installing. It also covers configuring Nginx's directories and files, updating to a new version, and configuring a Yum repository to install via package manager.
The document provides an overview of common Linux commands and their functions, such as RMDIR to remove empty directories, CHMOD to change file permissions, GREP for pattern matching, FIND to locate files, ECHO to output text, and MORE to view files page by page. It also covers operators like pipes, logical operators, redirection, and command substitution. The document concludes with descriptions of conditional statements like IF/THEN, FOR, WHILE, UNTIL, and CASE that allow scripts to perform actions conditionally.
RestMQ is a message queue system based on Redis that allows storing and retrieving messages through HTTP requests. It uses Redis' data structures like lists, sets, and hashes to maintain queues and messages. Messages can be added to and received from queues using RESTful endpoints. Additional features include status monitoring, queue control, and support for protocols like JSON, Comet, and WebSockets. The core functionality is language-agnostic but implementations exist in Python and Ruby.
This document discusses the Puppet configuration management tool. It provides an overview of Puppet including its open source nature, supported platforms, file structure, and types of resources it can manage like files, packages, services. It also discusses Facter for collecting system facts. Several examples are shown of how to configure files, packages, services. Finally Amazon EC2 is mentioned as a way to deploy Puppet in a scalable environment.
Presentation on how Puppet has been introduced in Seat Pagine Gialle to automate system administration tasks and easy the cooperation between Ops and Others.
The document contains instructions for creating various prank scripts that can be run on other people's computers without permission to annoy or confuse them. It includes scripts that continuously pop up messages, repeatedly open and close the CD tray, enable caps lock, type text in Notepad, and more. Most of the scripts are simple batches or VBScripts that can be easily created and run. The document encourages modifying the scripts for different purposes and pranking other users.
This document describes using Puppet to launch and maintain EC2 instances with pre-defined software and configurations. It discusses building AMIs with pre-installed but unconfigured software, and using Puppet classes to activate and configure specific packages, services, and monitoring tools on the instances. The classes split package, service, configuration, and monitoring roles, and can be included and parameterized to deploy NTP in different roles on instances.
Czy wiesz co potrafi zrobić twój serwer reverse-proxy? Wydaje Ci się, że żeby zrobić sprytny routing / uwierzytelnianie / autoryzację (niepotrzebne skreślić) między serwisami musisz go napisać w Javie lub jako moduł w C? A co jeżeli odpalanie JVM tylko po to, żeby do każdego żądania http dokleić jeden nagłówek to armata na wróbla? Zwłaszcza, że prawie na pewno gdzieś tam po drodze mijasz nginx... Zapraszam Cię do świata idealnej symbiozy nginx i Lua.
The document lists several dangerous commands that can be run on a Linux or Unix system and cause data loss or system instability. It also provides basic commands for file operations, package management, networking tasks and changing permissions and passwords. The dangerous commands include deleting all files in the root folder, formatting a partition, executing a fork bomb to overload the system, and overwriting the file system on a designated device. The basic commands cover file listing, copying, downloading, creating directories and viewing directory information. It also explains Linux file permissions in terms of read, write and execute attributes.
This document discusses Composer, an open source tool for dependency management in PHP. It describes what Composer is, how to install it, how to define dependencies in a composer.json file and composer.lock file, how Composer generates autoload files and installs vendor libraries, and some common Composer commands. It also provides information on joining the community and lists sources for more documentation on Composer.
Ondřej Šika: Docker, Traefik a CI - Mějte nasazené všeny větve na kterých pra...Develcz
The document describes setting up Docker, Traefik, and CI/CD pipelines. It includes a docker-compose.yml configuration file for Traefik that sets up port forwarding and SSL termination. It also includes a .gitlab-ci.yml file that defines a deploy job that builds a Docker image, pushes it to Docker Hub, and deploys it to a server using Traefik routing.
PSR-7 and PSR-15 are PHP standards for representing HTTP messages and server request handling. PSR-7 defines interfaces for HTTP messages like requests and responses, while PSR-15 defines interfaces for request handlers and middleware. These standards provide a common way for PHP libraries to interact with HTTP messages and requests, improving interoperability. They also allow applications to be built in a way that is compatible with evolving PHP frameworks and tools. Adopting these PSRs helps create a solid foundation for building HTTP applications and middleware in PHP.
Buffer overflow occurs when too much data is inserted into a buffer than it can handle, which may lead to arbitrary code execution if a memory pointer is overwritten. The document provides an example of a buffer overflow happening when user input longer than the buffer size is read using gets() into the buffer without limits. To prevent this, fgets() should be used instead, as it allows specifying the maximum number of characters to read from input.
OlinData Puppet Presentation for MOSC 2012Walter Heck
Walter Heck gave a presentation on Puppet at the 2012 Malaysia Open Source Conference. He introduced OlinData, which provides Puppet consulting and training, and discussed how Puppet allows for scalable, accountable, and versioned infrastructure configuration compared to manual methods. Heck also covered OlinData's Puppet training schedule and consulting services to help companies implement and improve their Puppet usage.
This document provides instructions for setting up a CoreOS cluster with services like MongoDB, SkyDNS, HAProxy, and Datadog monitoring. It describes using systemd unit files to run services like MongoDB and SkyDNS in Docker containers managed by Fleet. Etcd is used for service discovery and configuration management with ConfD. HAProxy provides load balancing of services, and Datadog monitors the cluster. Cloud-init scripts initialize new nodes joining the cluster.
This document provides an overview of basic Linux commands for tasks like file management, installation, networking, compression, searching, SSH, permissions and more. It explains commands for changing directories, listing files, copying/moving files, installing packages, pinging hosts, compressing files and directories, searching for patterns, remote login via SSH, and managing processes and system information.
An (abridged) Ruby Plumber's Guide to *nixEleanor McHugh
This document provides an overview of using Ruby to interface with Unix processes, files, and interprocess communication. It discusses using the Kernel module to spawn and manage processes, reading and writing files using IO methods and fcntl constants, communicating between processes via pipes and sockets, directly calling Unix syscalls via Kernel.syscall and the ruby/dl library, and examples of using POSIX semaphores and memory management with ruby/dl. Further reading links are provided for more in-depth information.
This document provides tips and tricks for using the command line more efficiently. It discusses using SSH configuration files and the screen command to manage terminal sessions. Other topics covered include compressing files with tar, navigating directories more easily, monitoring log files, managing background processes, and using shortcuts to save time typing commands.
This document provides an overview of configuration management with Cfengine. It discusses what Cfengine is and how it can be used to ensure files and processes are configured properly across multiple hosts. It then describes how to get started with Cfengine, including installing it, writing policies to configure a single host, and setting up a basic infrastructure to distribute policies to multiple hosts on a network. Key components like cfagent, cfservd, classes, and policies are defined. Examples of Cfengine configuration files are also provided.
This document provides instructions for installing the Nginx web server from source on Linux. It describes downloading the source code, verifying the signature, compiling it with make, and installing. It also covers configuring Nginx's directories and files, updating to a new version, and configuring a Yum repository to install via package manager.
The document provides an overview of common Linux commands and their functions, such as RMDIR to remove empty directories, CHMOD to change file permissions, GREP for pattern matching, FIND to locate files, ECHO to output text, and MORE to view files page by page. It also covers operators like pipes, logical operators, redirection, and command substitution. The document concludes with descriptions of conditional statements like IF/THEN, FOR, WHILE, UNTIL, and CASE that allow scripts to perform actions conditionally.
RestMQ is a message queue system based on Redis that allows storing and retrieving messages through HTTP requests. It uses Redis' data structures like lists, sets, and hashes to maintain queues and messages. Messages can be added to and received from queues using RESTful endpoints. Additional features include status monitoring, queue control, and support for protocols like JSON, Comet, and WebSockets. The core functionality is language-agnostic but implementations exist in Python and Ruby.
This document discusses the Puppet configuration management tool. It provides an overview of Puppet including its open source nature, supported platforms, file structure, and types of resources it can manage like files, packages, services. It also discusses Facter for collecting system facts. Several examples are shown of how to configure files, packages, services. Finally Amazon EC2 is mentioned as a way to deploy Puppet in a scalable environment.
Presentation on how Puppet has been introduced in Seat Pagine Gialle to automate system administration tasks and easy the cooperation between Ops and Others.
Puppet for Java developers - JavaZone NO 2012Carlos Sanchez
Example code at https://github.com/carlossg/puppet-for-java-devs
More info at http://blog.carlossanchez.eu/tag/devops
Video at http://vimeo.com/49483627
Puppet is an infrastructure-as-code tool that allows easy and automated provisioning of servers, defining the packages, configuration, services,... in code. Enabling DevOps culture, tools like Puppet help drive Agile development all the way to operations and systems administration, and along with continuous integration tools like Jenkins, it is a key piece to accomplish repeatability and continuous delivery, automating the operations side during development, QA or production, and enabling testing of systems configuration.
Traditionally a field for system administrators, Puppet can empower developers, allowing both to collaborate coding the infrastructure needed for their developments, whether it runs in hardware, virtual machines or cloud. Developers and sysadmins can define what JDK version must be installed, application server, version, configuration files, war and jar files,... and easily make changes that propagate across all nodes.
Using Vagrant, a command line automation layer for VirtualBox, they can also spin off virtual machines in their local box, easily from scratch with the same configuration as production servers, do development or testing and tear them down afterwards.
We’ll show how to install and manage Puppet nodes with JDK, multiple application server instances with installed web applications, database, configuration files and all the supporting services. Including getting up and running with Vagrant and VirtualBox for quickstart and Puppet experiments, as well as setting up automated testing of the Puppet code.
The document discusses how immutable infrastructure can be achieved through Puppet by treating systems configuration as code. Puppet allows defining systems in code and enforcing that state through automatic idempotent runs, compensating for inherent system mutability. This brings predictability to infrastructure and allows higher level operations by establishing a foundation of reliable, known states.
More info at http://blog.carlossanchez.eu/tag/devops
Video en español: http://youtu.be/E_OE4l3t5BA
The DevOps movement aims to improve communication between developers and operations teams to solve critical issues such as fear of change and risky deployments. But the same way that Agile development would likely fail without continuous integration tools, the DevOps principles need tools to make them real, and provide the automation required to actually be implemented. Most of the so called DevOps tools focus on the operations side, and there should be more than that, the automation must cover the full process, Dev to QA to Ops and be as automated and agile as possible. Tools in each part of the workflow have evolved in their own silos, and with the support of their own target teams. But a true DevOps mentality requires a seamless process from the start of development to the end in production deployments and maintenance, and for a process to be successful there must be tools that take the burden out of humans.
Apache Maven has arguably been the most successful tool for development, project standardization and automation introduced in the last years. On the operations side we have open source tools like Puppet or Chef that are becoming increasingly popular to automate infrastructure maintenance and server provisioning.
In this presentation we will introduce an end-to-end development-to-production process that will take advantage of Maven and Puppet, each of them at their strong points, and open source tools to automate the handover between them, automating continuous build and deployment, continuous delivery, from source code to any number of application servers managed with Puppet, running either in physical hardware or the cloud, handling new continuous integration builds and releases automatically through several stages and environments such as development, QA, and production.
Continuous Delivery with Maven, Puppet and Tomcat - ApacheCon NA 2013Carlos Sanchez
Continuous Integration, with Apache Continuum or Jenkins, can be extended to fully manage deployments and production environments, running in Tomcat for instance, in a full Continuous Delivery cycle using infrastructure-as-code tools like Puppet, allowing to manage multiple servers and their configurations.
Puppet is an infrastructure-as-code tool that allows easy and automated provisioning of servers, defining the packages, configuration, services,... in code. Enabling DevOps culture, tools like Puppet help drive Agile development all the way to operations and systems administration, and along with continuous integration tools like Apache Continuum or Jenkins, it is a key piece to accomplish repeatability and continuous delivery, automating the operations side during development, QA or production, and enabling testing of systems configuration.
Traditionally a field for system administrators, Puppet can empower developers, allowing both to collaborate coding the infrastructure needed for their developments, whether it runs in hardware, virtual machines or cloud. Developers and sysadmins can define what JDK version must be installed, application server, version, configuration files, war and jar files,... and easily make changes that propagate across all nodes.
Using Vagrant, a command line automation layer for VirtualBox, they can also spin off virtual machines in their local box, easily from scratch with the same configuration as production servers, do development or testing and tear them down afterwards.
We will show how to install and manage Puppet nodes with JDK, multiple Tomcat instances with installed web applications, database, configuration files and all the supporting services. Including getting up and running with Vagrant and VirtualBox for quickstart and Puppet experiments, as well as setting up automated testing of the Puppet code.
Puppet is an open source tool for server configuration management. It allows systems to be configured and maintained in a consistent, automated way. Puppet uses a declarative language to describe system configuration and can manage a variety of operating systems. It provides benefits like reduced entropy, ability to quickly scale machines, change management tracking and repeatable states. Puppet uses a client-server architecture with SSL encryption and supports concepts like templates, defined types and ordering of resources.
Fail2ban is an open source intrusion prevention software developed using the Python programming language. It monitors system logs such as /var/log/pwdfail, /var/log/auth.log, and /var/log/secure for failed login attempts. When the maximum number of failed logins from an IP address within a certain time frame is reached, Fail2ban uses iptables to ban that IP address by adding a DROP rule. It can ban IP addresses for services beyond just SSH, such as SMTP, HTTP, and others. The document then provides instructions on installing and configuring Fail2ban on CentOS 6.3.
More info at http://blog.carlossanchez.eu/2011/11/15/from-dev-to-devops-slides-from-apachecon-na-vancouver-2011/
The DevOps movement aims to improve communication between developers and operations teams to solve critical issues such as fear of change and risky deployments. But the same way that Agile development would likely fail without continuous integration tools, the DevOps principles need tools to make them real, and provide the automation required to actually be implemented. Most of the so called DevOps tools focus on the operations side, and there should be more than that, the automation must cover the full process, Dev to QA to Ops and be as automated and agile as possible. Tools in each part of the workflow have evolved in their own silos, and with the support of their own target teams. But a true DevOps mentality requires a seamless process from the start of development to the end in production deployments and maintenance, and for a process to be successful there must be tools that take the burden out of humans.
Apache Maven has arguably been the most successful tool for development, project standardization and automation introduced in the last years. On the operations side we have open source tools like Puppet or Chef that are becoming increasingly popular to automate infrastructure maintenance and server provisioning.
In this presentation we will introduce an end-to-end development-to-production process that will take advantage of Maven and Puppet, each of them at their strong points, and open source tools to automate the handover between them, automating continuous build and deployment, continuous delivery, from source code to any number of application servers managed with Puppet, running either in physical hardware or the cloud, handling new continuous integration builds and releases automatically through several stages and environments such as development, QA, and production.
Capistrano is an open source tool for running scripts on multiple servers. It’s primary use is for easily deploying applications. While it was built specifically for deploying Rails apps, it’s pretty simple to customize it to deploy other types of applications.
capifony is a deployment recipes collection that works with both symfony and Symfony2 applications.
The document discusses various methods for hardening Linux security, including securing physical and remote access, addressing top vulnerabilities like weak passwords and open ports, implementing security policies, setting BIOS passwords, password protecting GRUB, choosing strong passwords, securing the root account, disabling console programs, using TCP wrappers, protecting against SYN floods, configuring SSH securely, hardening sysctl.conf settings, leveraging open source tools like Mod_Dosevasive, Fail2ban, Shorewall, and implementing security at the policy level with Shorewall.
Puppi is a Puppet modules that drives Puppet's knowledge of the Systems to a command line tool that you can use to check services availability, gather info on the system and deploy application with a single command.
1. The document discusses moving from a Dev to DevOps model by addressing issues like siloization between development and operations teams and embracing concepts like infrastructure as code.
2. It recommends several DevOps tools for infrastructure automation including Puppet, Vagrant, and VeeWee which allow developers to define infrastructure in code and provision environments.
3. The Puppet Domain Specific Language (DSL) is demonstrated for declaring resources like users, files, packages, and services with attributes and relationships between them in a declarative way.
The document discusses deploying a Rails application to Amazon EC2. It explains that the goals are to launch an EC2 instance, connect to it, set up the environment, deploy the application, and profit. It then outlines the plan to launch an instance, connect to it, install necessary packages like Ruby, Rails, and Nginx, configure Nginx and Unicorn, deploy the application using Capistrano, and start the Unicorn process.
Puppet is a tool that allows users to declaratively configure systems. It provides abstraction through defined resources like packages and files, ensures configurations are idempotent, and converges systems to the desired state declaratively rather than imperatively through scripts. Puppet code is organized into reusable modules and managed through version control. Modules should include tests, be validated with tools like puppet-lint, and tested in automated environments like Travis CI to ensure high quality.
PuppetCamp Ghent - What Not to Do with PuppetOlinData
The document discusses common mistakes to avoid when using Puppet, including design mistakes like putting multiple classes in a file, language mistakes like using default options without checking for failures, and resource definition mistakes like recursively declaring ownership on large directories. It provides examples of bad Puppet code patterns and suggests better alternatives to avoid issues like dependency loops. The document is intended to help Puppet users learn from ugly or problematic Puppet code examples.
PuppetCamp Ghent - What Not to Do with PuppetWalter Heck
The document discusses common mistakes to avoid when using Puppet, including design mistakes like poorly structured classes, language mistakes like misusing functionality, and dependency issues. It provides examples of problematic Puppet code and explanations of why they are problematic, such as putting multiple classes in one file, using default options without checking for failures, and creating dependency loops between resources. The goal is to help Puppet users identify and avoid ugly or erroneous Puppet code that could cause problems.
Nmap Scripting Engine and http-enumerationRobert Rowley
Nmap is a network scanning tool that scans hosts and networks for open ports. The Nmap Scripting Engine (NSE) allows Nmap to perform additional checks and functions beyond basic port scanning. NSE uses the Lua programming language to write scripts for tasks like service detection, vulnerability testing, and malware detection. Popular NSE scripts scan for vulnerabilities like SQL injection, fingerprint web servers and applications, perform service/version detection, and more. The NSE community develops and shares new scripts on the Nmap site to continually improve Nmap's scanning abilities.
Writing and Publishing Puppet Modules - PuppetConf 2014Puppet
The document discusses best practices for writing and publishing Puppet modules. It covers module structure, writing manifests and templates, testing modules, publishing to the Forge, and maintaining modules over time. The overall goal is to create reusable modules that are portable, configurable and well-tested.
Similar a Scaling antispam solutions with Puppet (20)
SpamAssassin 4.0 is going to be released soon.
In this version there will be a lot of new features useful to catch new types of spam messages and a lot of improvements.
ACME is a protocol for automating certificate lifecycle management communications between Certificate Authorities (CAs) and a company’s web servers.
The most known implementation is the one made by Let’s Encrypt non-profit CA.
There are many other implementation and one of the most attractive and easy to use is Apache httpd mod_md.
During the talk I will explain why ACME protocol is important to secure web sites and how mod_md could ease the transition to a more secure www.
In this talk I will describe what's new in SpamAssassin 3.4.2, the ideas for further developments and some tips to improve the efficiency of your antispam solution.
seccomp is a computer security facility in the Linux kernel, pledge is a similar security facility in the OpenBSD kernel. In this presentation Giovanni Bechis will review the development story and progress of both kernel interfaces and will analyze the main differences. There will be some examples of implementations of security patches made for some important open source projects.
pf(4) is the OpenBSD packet filter that provides stateful packet filtering and network address translation (NAT). It is used in OpenBSD, FreeBSD, NetBSD, DragonflyBSD, and other systems. Some key features of pf include its flexible rule syntax, atomic ruleset updates, integrated traffic shaping, and ability to divert packets to userspace processes like spamd for inspection. It provides logging in tcpdump format and can integrate with CARP and other services. The pf code was developed for OpenBSD after the previous IPFilter code was removed due to licensing issues.
This document summarizes an overview of the ELK stack presented at LinuxCon Europe 2016. It discusses the components of ELK including Beats, Logstash, Elasticsearch, and Kibana. It provides examples of using these components to collect, parse, store, search, and visualize log data. Specific topics covered include collecting log files using Filebeat and Logstash, parsing logs with Logstash filters, visualizing data in Kibana, programming Elasticsearch with REST APIs and client libraries, and alerting using the open source ESWatcher tool.
OpenSSH is a free SSH implementation used for secure communications and file transfers. It provides strong cryptography, authentication methods, and allows for features like port forwarding, X11 forwarding, and file transfers. OpenSSH uses a modular structure which separates the transport, authentication, and connection protocols. This makes it more secure than the original SSH version 1, especially against man-in-the-middle attacks. The OpenSSH codebase focuses on security through techniques like input validation, sandboxing, and process separation between the monitoring and slave processes.
OpenSMTPD is a FREE implementation of the server-side SMTP protocol as defined by RFC 5321, with some additional standard extensions. It allows ordinary machines to exchange e-mails with other systems speaking the SMTP protocol.
Started out of dissatisfaction with other implementations, OpenSMTPD nowadays is a fairly complete SMTP implementation.
This document discusses the Heartbleed bug in OpenSSL and the creation of LibreSSL as a more secure alternative. It notes that 17% of HTTPS servers were vulnerable to Heartbleed, which allowed attackers to steal passwords, credit cards, and other private data from server memory. LibreSSL was created to have fewer lines of code, modern coding practices, and fewer portability workarounds than OpenSSL to address bugs like Heartbleed. The document emphasizes fixing bugs quickly and not reinventing standard library functions.
LibreSSL is a version of the TLS/crypto stack forked from OpenSSL in 2014, with goals of modernizing the codebase, improving security, and applying best practice development processes.
Primary development occurs inside the OpenBSD source tree with the usual care the project is known for. On a regular basis the code is re-packaged for portable use by other operating systems.
SOGo: sostituire Microsoft Exchange con software Open SourceGiovanni Bechis
Microsoft Exchange è lo standard de-facto in ambito groupware, ora però lo si può sostituire con strumenti open source senza perdere funzionalità e guadagnandoci in efficenza, sicurezze e in possibilità di integrazione con altri software.
Questa è una breve introduzione alle possibilità offerte dalle piattaforme di cloud storage open source come owncloud e seafile; in particolare vengono analizzate le peculiarità del software Seafile.
Npppd is a VPN daemon in OpenBSD that supports PPTP, L2TP and PPPoE. It can authenticate using local files or RADIUS and tunnel IP packets. Npppd first appeared in OpenBSD 5.3 and its configuration file format has changed over time. Example configurations show how to set up basic and advanced npppd VPN tunnels using PPTP or L2TP with IPsec, and how to monitor active sessions. Future work may include better packet filtering and ARP cache integration.
Ssh è lo standard de-facto nelle comunicazioni sicure, questa è una breve ma esaustiva introduzione alle caratteristiche del programma e al suo utilizzo.
Il protocollo ipv6 è la nuova versione del protocollo ipv4, la base di internet così come lo conosciamo ora.
Sul protocollo ipv6 si baserà l'internet del futuro.
Relayd is a daemon to relay and dynamically redirect incoming connections to a target host.
Its main purposes are to run as a load-balancer, application layer gateway, or transparent proxy.
Hand Rolled Applicative User ValidationCode KataPhilip Schwarz
Could you use a simple piece of Scala validation code (granted, a very simplistic one too!) that you can rewrite, now and again, to refresh your basic understanding of Applicative operators <*>, <*, *>?
The goal is not to write perfect code showcasing validation, but rather, to provide a small, rough-and ready exercise to reinforce your muscle-memory.
Despite its grandiose-sounding title, this deck consists of just three slides showing the Scala 3 code to be rewritten whenever the details of the operators begin to fade away.
The code is my rough and ready translation of a Haskell user-validation program found in a book called Finding Success (and Failure) in Haskell - Fall in love with applicative functors.
Using Query Store in Azure PostgreSQL to Understand Query PerformanceGrant Fritchey
Microsoft has added an excellent new extension in PostgreSQL on their Azure Platform. This session, presented at Posette 2024, covers what Query Store is and the types of information you can get out of it.
Measures in SQL (SIGMOD 2024, Santiago, Chile)Julian Hyde
SQL has attained widespread adoption, but Business Intelligence tools still use their own higher level languages based upon a multidimensional paradigm. Composable calculations are what is missing from SQL, and we propose a new kind of column, called a measure, that attaches a calculation to a table. Like regular tables, tables with measures are composable and closed when used in queries.
SQL-with-measures has the power, conciseness and reusability of multidimensional languages but retains SQL semantics. Measure invocations can be expanded in place to simple, clear SQL.
To define the evaluation semantics for measures, we introduce context-sensitive expressions (a way to evaluate multidimensional expressions that is consistent with existing SQL semantics), a concept called evaluation context, and several operations for setting and modifying the evaluation context.
A talk at SIGMOD, June 9–15, 2024, Santiago, Chile
Authors: Julian Hyde (Google) and John Fremlin (Google)
https://doi.org/10.1145/3626246.3653374
WWDC 2024 Keynote Review: For CocoaCoders AustinPatrick Weigel
Overview of WWDC 2024 Keynote Address.
Covers: Apple Intelligence, iOS18, macOS Sequoia, iPadOS, watchOS, visionOS, and Apple TV+.
Understandable dialogue on Apple TV+
On-device app controlling AI.
Access to ChatGPT with a guest appearance by Chief Data Thief Sam Altman!
App Locking! iPhone Mirroring! And a Calculator!!
Transform Your Communication with Cloud-Based IVR SolutionsTheSMSPoint
Discover the power of Cloud-Based IVR Solutions to streamline communication processes. Embrace scalability and cost-efficiency while enhancing customer experiences with features like automated call routing and voice recognition. Accessible from anywhere, these solutions integrate seamlessly with existing systems, providing real-time analytics for continuous improvement. Revolutionize your communication strategy today with Cloud-Based IVR Solutions. Learn more at: https://thesmspoint.com/channel/cloud-telephony
SMS API Integration in Saudi Arabia| Best SMS API ServiceYara Milbes
Discover the benefits and implementation of SMS API integration in the UAE and Middle East. This comprehensive guide covers the importance of SMS messaging APIs, the advantages of bulk SMS APIs, and real-world case studies. Learn how CEQUENS, a leader in communication solutions, can help your business enhance customer engagement and streamline operations with innovative CPaaS, reliable SMS APIs, and omnichannel solutions, including WhatsApp Business. Perfect for businesses seeking to optimize their communication strategies in the digital age.
8 Best Automated Android App Testing Tool and Framework in 2024.pdfkalichargn70th171
Regarding mobile operating systems, two major players dominate our thoughts: Android and iPhone. With Android leading the market, software development companies are focused on delivering apps compatible with this OS. Ensuring an app's functionality across various Android devices, OS versions, and hardware specifications is critical, making Android app testing essential.
How Can Hiring A Mobile App Development Company Help Your Business Grow?ToXSL Technologies
ToXSL Technologies is an award-winning Mobile App Development Company in Dubai that helps businesses reshape their digital possibilities with custom app services. As a top app development company in Dubai, we offer highly engaging iOS & Android app solutions. https://rb.gy/necdnt
Mobile App Development Company In Noida | Drona InfotechDrona Infotech
Drona Infotech is a premier mobile app development company in Noida, providing cutting-edge solutions for businesses.
Visit Us For : https://www.dronainfotech.com/mobile-application-development/
E-commerce Development Services- Hornet DynamicsHornet Dynamics
For any business hoping to succeed in the digital age, having a strong online presence is crucial. We offer Ecommerce Development Services that are customized according to your business requirements and client preferences, enabling you to create a dynamic, safe, and user-friendly online store.
1. Scaling antispam mail servers with Puppet
Giovanni Bechis
<g.bechis@snb.it>
@g_bechis
PuppetCamp 2020, Netherlands
2. About Me
sysadmin and developer @SNB
OpenBSD hacker for ∼ 12 years
Apache SpamAssassin committer
Apache httpd committer
random patches in random open source software (amavisd-new,
courier-imap, cyrus-sasl, memcached, ...)
5. Puppet Facts
Some facts are interesting to scale in a right way all involved softwares
”os.name” to have different configuration for different operating systems
”processors.count” and ”memory.system.total bytes” to correctly
configure Clamav and Apache SpamAssassin
6. Puppet Facts and Erb templates
# Clamav template
# Maximum number of threads running at the same time.
# Default: 10
#MaxThreads 20
<% if @processors[’count’] != 1 -%>
MaxThreads 20
<% end -%>
# Path to a local socket file the daemon will listen on.
# Default: disabled (must be specified by a user)
#LocalSocket /tmp/clamd.socket
<% if @os[’name’] == ’OpenBSD’ -%>
LocalSocket /tmp/clamd
<% end -%>
7. Puppet Facts and Erb templates
# Mail::SpamAssassin::Plugin::ResourceLimits
<% if @memory[’system’][’total_bytes’] < 4142006272 -%>
resource_limit_mem 536870912
<% end -%>
8. Puppet Facts and Erb templates
redis::params { ’redis’:
bind => [’127.0.0.1’, ’172.16.99.134’],
masterauth => ’XXX’,
slaveof => ’172.16.99.133 6379’,
}
bind <%= @bind.join(’ ’) %>
<% if @masterauth -%>masterauth <%= @masterauth %><% end -%>
<% if @slaveof -%>slaveof <%= @slaveof %><% end -%>
9. Config files deploy
Config files deploy
Puppet server does a checkout of our
git repo and pulls updated
configuration files.
Most of the configuration files are the
same for all servers, lot of modules we
are using just copy the correct files in
the correct location.
12. Mimedefang configuration and deploy
Mimedefang deploy
Mimedefang has both code and
configuration files that need to be
deployed.
Mimedefang has several dependencies
like SpamAssassin and Clamav, the
same dependency tree is respected on
Puppet code.