Talk at Paris.Py, with captures from demo
•
0 recomendaciones•187 vistas
Bonobo ETL at Paris.Py / Algolia
Recomendados
Recomendados
Más contenido relacionado
Último
Último (20)
Destacado
Destacado (20)
Talk at Paris.Py, with captures from demo
- 2. Introduction
- 3. Extract Transform Load foo bar baz Extract Transform Load
- 4. Extract Transform Load foo bar baz Extract Transform Load Transform more Join DB HTTP POST log?
- 5. Bonobo ETL • It’s just Plain Old Python (3) Objects • Streaming data through them • Write / Configure / Execute / Monitor Code first. Eventually a GUI will come.
- 6. Bonobo is not… • Data Science & Analytics (see pandas) • Workflow Scheduler & Monitor (see airflow) • Big Data (see hadoop, big query, dataflow, spark, …) • A monkey (spoiler : it’s an ape)
- 7. Action
- 8. Quick Demo or brown fox?
- 9. pip install bonobo then import it
- 10. Let’s show what we can do with simple callable (generators, here).
- 11. Adding them to a graph show the data flow
- 12. Run them, see what happens real time.
- 13. Use external services, implementation resolution is done at runtime.
- 15. Provide implementations at runtime, easy to be 12factor-compliant.
- 16. Also easy to read/write from files.
- 17. Simplest cases can be run without any code, from CLI $ bonobo convert orders.csv orders.json - CsvReader in=1 out=29 [done] - JsonWriter in=29 out=29 [done]
- 18. Even with custom transformations in the middle of the chain $ bonobo convert orders.csv -t transform:append_isbn orders.json - CsvReader in=1 out=29 [done] - append_isbn in=29 out=29 [done] - JsonWriter in=29 out=29 [done]
- 19. Epilogue
- 20. Young • First commit : December 2016 • 23 releases, ~420 commits, 4 contributors • 25 releases, ~565 commits, 12 contributors • 34 releases, ~774 commits, 21 contributors • Current : 0.4.3 0.5.1 0.6.1
- 21. … • Open-Source. • Light & Focused. • Very few dependencies. • Comprehensive standard library.
- 22. Integrations • Django • SQLAlchemy • Jupyter • Docker • Google APIs, OpenDataSoft, Cheese Shop :)