Telescope Inc.'s presentation at AnsibleFest conference. In this talk, discuss how they moved from a baremetal datacenter to AWS, things learned along the way, and how they scale up to meet voting demand with Ansible.
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...
Breaking Up With Your Data Center Presentation
1.
2. WHO IS TELESCOPE?
Industry Leader in Participation
Media
Offices in Los Angeles, New York,
London and Brazil
Real-time Platform – Powering most
demanding high profile audience
participation campaigns for TV,
Sports and Brands
Unique traffic path and unique
requirements/High Capacity/Unique
Traffic Spikes
3. WHO IS TELESCOPE?
Since 2002, leading media companies and brands have entrusted Telescope to power and
optimize results on their most demanding and highest-profile audience engagement initiatives.
Our participation platform enables several billion interactive experiences every year.
4. SHOW NIGHT TRAFFIC
Vote Transactions per minute
5:55 PM 6:55 PM 7:55 PM 8:55 PM 9:55 PM 10:55 PM 11:55 PM 12:55 AM
Total
5. INTRO SLIDES ARE THE BEST!
Why we moved from bare metal to AWS
Why we choose ansible for our automation
How ansible solved our automation challenges
6. OUR FIRST LOVE
6 racks across two buildings at One Wilshire
2 SysAdmins responsible for hardware and operations
No Environment for Developers to play
Our hardware was restricting our ability to grow
Long lead time to build increase total capacity
Research latest hardware
Order hardware
Order power and space
Rack, stack, cable
Install all the things
7. IS THIS RIGHT FOR US?
Hard to justify building up a datacenter
Telescope’s case required significant burst capacity
Business needs changed quickly
Support large architecture with a small team
Wanted to automate, but didn’t know where to start
8. SO MANY OPTIONS
Rackspace
AWS
Azure
Softlayer
Google
Oracle
Digital Ocean
Heroku
Puppet
Chef
Ansible
Bconfig2
Salt
CFEngine
Vagrant
Cobbler
11. WHY AWS?
Leader in cloud hosting
Strong API’s with many supporting
communities across multiple languages
AWS services were a complement to our own
technologies
It (AWS) is the overwhelming market share leader, with
more than five times the cloud IaaS compute capacity in
use than the aggregate total of the other 14 providers in
this Magic Quadrant. It is a thought leader; it is
extraordinarily innovative, exceptionally agile, and very
responsive to the market. It has the richest array of IaaS
features and PaaS-like capabilities, and continues to
rapidly expand its service offerings. It is the provider most
commonly chosen for strategic adoption.
- Gartner, Cloud Infrastructure as a service report
12. TIME TO AUTOMATE
Can you make a build in one
step?
- Joel Spolsky, “The Joel Test: 12 Steps to Better Code”
21. MONITORING
Ansible templates
give us flexibility
for inflexible tools
zabbix_agentd.conf
# {{ ansible_managed }}
# This is a config file for the Zabbix agent
daemon (Unix)
…
Server={% for host in
groups['tag_zabbix_prod'] %} {{
hostvars[host].ec2_private_ip_address }}, {%
endfor %}
…
HostMetadata={{ ec2_security_group_names }}
24. DO YOU EVEN TAG, BRO?
tag_connectapi_{{ env }}
tag_webapp_{{ env }}
tag_hbase{{ cluster }}_{{ env }}:&hadooprole_master
tag_hbase{{ cluster }}_{{ env }}:&hadooprole_node
25. - hosts: localhost
pre_tasks:
- name: Add new instance to host group
add_host:
hostname: “{{ item.1 }}”
groupname: “__zookeeper_temp”
position: “{{ item.0 + 1}}”
with_indexed_items: groups.tag_zookeeper_{{ env }}
- hosts: __zookeeper_temp
roles:
- role: zookeeper
- { role: zabbix, goal: agent }
DO YOU EVEN TAG, BRO?
Add hosts to temp
group
Target temp
group
Install
26. - hosts: localhost
pre_tasks:
- name: Add instances to temporary host group
add_host: hostname={{ item.1 }}
groupname:__hbase_temp
with_indexed_items: groups.tag_hbase{{ cluster
}}_{{ env }}
- name: Pick one from the temporary group
add_host: hostname={{ item }}
groupname: “__the_chosen_one”
with_random_choice: groups.__hbase_temp
- hosts: __the_chosen_one
gather_facts: true
user: ec2-user
sudo: yes
DO YOU EVEN TAG, BRO?
Add hosts to temp
group
Randomly
choose
Install
27. YO DAWG! I HEARD YOU LIKE ANSIBLE!
- name: copy tower tarball
copy:
src: ansible-tower-setup-latest.tar.gz
dest: /tmp/tower-latest.tar.gz
force: yes
- name: set the group vars file
template: src=group_vars_all.j2 dest=/tmp/ansible-tower-
setup-latest/group_vars/all
when: "upgrade is defined and upgrade == 'yes'“
- name: Create a database backup of tower at
/var/log/awx
shell: awx-manage dumpdata > /var/log/awx/backup-{{
ansible_date_time.epoch }}.json
when: "backup is defined and backup != 'no'"
- name: run the tower install
command: ./setup.sh chdir=/tmp/ansible-tower-setup-latest
when: "upgrade is defined and upgrade == 'yes'"
- name: place the license file
template: src=license.j2 dest=/etc/awx/license'
Send the tarball
Set group_vars
Backup
Install
28. WHERE ARE WE NOW?
Most of our major architecture has been
moved to AWS
Still shipping data to our datacenter for
storage and analytics
Coordinating migration of our last pieces
based on client restrictions
29. HOW HAS THIS CHANGED US
Changed our conversations
“Ok, so lets just kill all the queue boxes, Bring up 2 new boxes and new API and start
from scratch”
-11:52pm Thursday Night
“I need your help since you are not doing anything”
“Dude I am busy installing and provisioning 16 hadoop servers”
“Good, so you have time”
-3:42pm Tuesday afternoon
30. WHAT’S LEFT?
Chronos as a replacement for Chron
Zabbix module for better automation
Scaling down
Adding JMX interfaces
Eventually express all items/triggers from a dictionary so changes are tracked in git
Akamai Module
Push cache config changes with web app updates
Remove any embarrassing hacks from Hadoop, Hbase, NDB roles and share them if there
is interest
Operational Tasks to make day to day easier
31. HAPPILY EVER AFTER
Telescope is happy in it’s new relationships
AWS has been a good match for Telescope
Ansible has let us take control of our architecture like never
before
Looking forward to solving bigger problems
32. QUESTIONS?
Iain Wright, Lead Systems Administrator
John Weatherford, VP Engineering
#TeamTelescope
Ps. If this sounds interesting to you, come talk to us!