Recomendados
Más contenido relacionado
La actualidad más candente
La actualidad más candente (19)
Destacado
Destacado (8)
Similar a Weightlifting at SimplySocial
Similar a Weightlifting at SimplySocial (20)
Más de Bogdan Gaza
Último
Último (20)
Weightlifting at SimplySocial
- 2. @bogdan
- 4. + =
- 5. + =
- 6. + =?
- 8. + =
- 9. ARCHITECTURE api.gosimplysocial.com app.gosimplysocial.com API 01 API 02 API N... APP 01 APP 02 APP N... BACKEND 01 BACKEND 02 BACKEND N... DBs
- 10. LIFE OF A REQUEST api.gosimplysocial.com app.gosimplysocial.com API 01 API 01 API N... APP 01 APP 01 APP N... BACKEND 01 BACKEND 02 BACKEND N... DBs API 02 APP 02
- 11. api.gosimplysocial.com app.gosimplysocial.com API 01 API 02 API N... APP 01 APP 02 APP N... BACKEND 01 BACKEND 02 BACKEND N... DBs LIFE OF A REQUEST
- 12. api.gosimplysocial.com app.gosimplysocial.com API 01 API 02 API N... APP 01 APP 02 APP N... BACKEND 01 BACKEND 02 BACKEND N... DBs LIFE OF A REQUEST
- 13. api.gosimplysocial.com app.gosimplysocial.com API 01 API 02 API N... APP 01 APP 02 APP N... BACKEND 01 BACKEND 02 BACKEND N... DBs LIFE OF A REQUEST 2M x
- 14. api.gosimplysocial.com app.gosimplysocial.com API 01 API 02 API N... APP 01 APP 02 APP N... BACKEND 01 BACKEND 02 BACKEND N... DBs LIFE OF A REQUEST
- 15. USE OF PYTHON api.gosimplysocial.com app.gosimplysocial.com API 01 API 02 API N... APP 01 APP 02 APP N... BACKEND 01 BACKEND 02 BACKEND N... DBs
- 16. api.gosimplysocial.com app.gosimplysocial.com API 01 API 02 API N... APP 01 APP 02 APP N... BACKEND 01 BACKEND 02 BACKEND N... DBs USE OF PYTHON
- 17. api.gosimplysocial.com app.gosimplysocial.com API 01 API 02 API N... APP 01 APP 02 APP N... BACKEND 01 BACKEND 02 BACKEND N... DBs USE OF PYTHON
- 18. api.gosimplysocial.com app.gosimplysocial.com API 01 API 02 API N... APP 01 APP 02 APP N... BACKEND 01 BACKEND 02 BACKEND N... DBs USE OF PYTHON
- 19. api.gosimplysocial.com app.gosimplysocial.com API 01 API 02 API N... APP 01 APP 02 APP N... BACKEND 01 BACKEND 02 BACKEND N... DBs USE OF PYTHON
- 20. CAN I HEAR A BOOO FOR ?
- 21. api.gosimplysocial.com app.gosimplysocial.com API 01 API 02 API N... APP 01 APP 02 APP N... BACKEND 01 BACKEND 02 BACKEND N... DBs USE OF PYTHON
- 22. api.gosimplysocial.com app.gosimplysocial.com API 01 API 02 API N... APP 01 APP 02 APP N... BACKEND 01 BACKEND 02 BACKEND N... DBs USE OF PYTHON
- 23. PYTHON ON APPSERVERS AWS EC2 Ubuntu 12.04 Chef NGINX + PHP FPM FABRIC DEPLOYMENT Metrics Logging
- 24. USE OF PYTHON METRICS AWS EC2 Ubuntu 12.04 Chef NGINX + PHP FPM FABRIC Metrics Logging
- 25. INTRODUCING OSSIE METRICS GATHERING REPORTING STATS THROUGH A SIMPLE WEB INTERFACE INTERACTION WITH RUNNING PROcesses BACKEND BY STATSD and GRAPHITE
- 26. INTRODUCING OSSIE import ossie stat = ossie.Client("arcade.random_worker") stat.gauge("a_metric", 256) stat.timer("random.metric", 1.28) with stat.timer("random.metric.with"): # do something that requires an amount of time # ossie client will report the right time
- 28. USE OF PYTHON logging AWS EC2 Ubuntu 12.04 Chef NGINX + PHP FPM FABRIC Metrics Logging
- 29. logginG FULL JSON logging EVEN nginx logs json powered by fluentd + S3 FUTURE: Logs can be searched by elasticsearch + kibana
- 30. api.gosimplysocial.com app.gosimplysocial.com API 01 API 02 API N... APP 01 APP 02 APP N... BACKEND 01 BACKEND 02 BACKEND N... DBs USE OF PYTHON
- 33. INTRODUCING ARCADE queue based workers framework Convention over configuration simple deployment detailed logging and monitoring
- 34. INTRODUCING ARCADE CRAWL INSIGHTS CRAWL PAGES CRAWL EVENTS ARCADE FACEBOOK CRAWL
- 35. INTRODUCING ARCADE CRAWL INSIGHTS CRAWL PAGES CRAWL EVENTS ARCADE FACEBOOK CRAWL queue python workers job types
- 36. INTRODUCING ARCADE facebook/ !"" __init__.py !"" common # !"" __init__.py # !"" crawler/ # !"" exception.py # !"" facebook_model.py # !"" helper.py # !"" indexer/ # !"" ... !"" config # !"" __init__.py # !"" config.yml # $"" env # !"" __init__.py # !"" crawler.py # !"" ... ... !"" tests # !"" __init__.py # !"" fixtures # # !"" crawler_feed.yml # # !"" ... # # $"" stream_routing.yml # !"" test_crawler_feed.py # !"" ... # $"" test_stream_worker.py $"" workers !"" __init__.py !"" crawler.py !"" ...
- 37. from arcade.worker import TaskWorker from arcade.helper import retry, task # [...] class FacebookCrawler(TaskWorker): def __init__(self, dependencies): super(FacebookCrawler, self).__init__(dependencies) @task("default", "crawl") def crawl_feed(self, data): """ Job type: * uid * last_crawl """ feed = CrawlerFeedModel(self) feed.crawl(data) def before_each(self, data): # [...] def run(dependencies): try: FacebookCrawler(dependencies).run() except: dependencies.get("exception").capture_exception()
- 38. AND MANY MORE: continuous integration RELIABLE ALERTING CLOUDWATCH to STATSD REAL USER MONITORING
- 39. Q/A
- 40. Thanks!