Presenter: Peter Wang, Co-Founder & CTO, Continuum Analytics

1. New Capabilities in
the PyData Ecosystem
Peter Wang
Continuum Analytics
@pwang

2. Data Science @ NYT

3. Python & SciPy
• High performance linear algebra, image processing,
optimization via NumPy, optimized C++, FORTRAN
• Large structured data via HDF5, memmap
• Out of core processing, streaming & realtime
• Distributed computing via MPI, IPython Parallel, etc.
• GPU & heterogenous via OpenCL, PyCUDA, others
• Massive adoption in research, national labs, industry
(engineering, finance, etc.)
• IPython Notebook: 2005-2011
• pandas: 2008-2009
• scikit-learn: 2007
• NumPy: 2006
• matplotlib: 2002
• IPython: 2001
• Numarray: 2001
• SciPy: 1999
• Numeric: 1995
Python has >15 year history in scientific computing

4. "Python's Scientific Ecosystem"
@jakevdp

5. "Many More Tools"
@jakevdp

6. Focus On
• Bokeh
• Dask

7. Focus On
• Bokeh
• Dask
• Blaze, odo
• dynd
• xray
• NumPy
• Pandas
• PyTables & h5py
• Beaker Notebook
• IPython widgets, JupyterHub
• conda, Anaconda Cluster
• Docker
• Docker
• Docker
Not Gonna Talk About...

8. Focus On
• Bokeh
• Dask
• Blaze, odo
• dynd
• xray
• NumPy
• Pandas
• PyTables & h5py
• Beaker Notebook
• IPython widgets, JupyterHub
• conda, Anaconda Cluster
• Docker
• Docker
• Docker
Not Gonna Talk About...

9. Bokeh
• Interactive visualization
• Novel graphics
• Streaming, dynamic, large data
• For the browser, with or without a server
• No need to write Javascript
• Support for R, Scala, Julia, Lua
http://bokeh.pydata.org

10. Dashboards & Data Apps

11. Static Notebooks/HTML, Interactive Plots
http://nbviewer.ipython.org/github/bokeh/bokeh-notebooks/blob/master/tutorial/00%20-%20intro.ipynb#Interaction

12. Extensible Architecture
server.py BrowserApp Model
BokehJS
object graph
bokeh-server
bokeh.py
object graph
JSON

14. rBokeh
http://hafen.github.io/rbokeh

15. Dask

16. Example: Ocean Temp Data
• http://www.esrl.noaa.gov/psd/data/gridded/
data.noaa.oisst.v2.highres.html
• Every 1/4 degree, 720x1440 array each day

17. Example: Ocean Temp Data
• http://www.esrl.noaa.gov/psd/data/gridded/
data.noaa.oisst.v2.highres.html
• Every 1/4 degree, 720x1440 array each day

18. Bigger Data
36 years: 720 x 1440 x 12341 x 4 = 51 GB uncompressed
If you don't have this much RAM...

19. Bigger Data
36 years: 720 x 1440 x 12341 x 4 = 51 GB uncompressed
If you don't have this much RAM...
... better start chunking.

20. DAG of Computation

21. Dask: Out of Core Scheduler for Python

22. Dask: Out of Core Scheduler for Python
• A parallel computing framework

23. Dask: Out of Core Scheduler for Python
• A parallel computing framework
• That leverages the excellent Python ecosystem

24. Dask: Out of Core Scheduler for Python
• A parallel computing framework
• That leverages the excellent Python ecosystem
• Using blocked algorithms and task scheduling

25. Dask: Out of Core Scheduler for Python
• A parallel computing framework
• That leverages the excellent Python ecosystem
• Using blocked algorithms and task scheduling
• Written in pure Python

26. Dask: Out of Core Scheduler for Python
• A parallel computing framework
• That leverages the excellent Python ecosystem
• Using blocked algorithms and task scheduling
• Written in pure Python

27. Dask: Out of Core Scheduler for Python
• A parallel computing framework
• That leverages the excellent Python ecosystem
• Using blocked algorithms and task scheduling
• Written in pure Python
Core Ideas

28. Dask: Out of Core Scheduler for Python
• A parallel computing framework
• That leverages the excellent Python ecosystem
• Using blocked algorithms and task scheduling
• Written in pure Python
Core Ideas
• Dynamic task scheduling yields sane parallelism

29. Dask: Out of Core Scheduler for Python
• A parallel computing framework
• That leverages the excellent Python ecosystem
• Using blocked algorithms and task scheduling
• Written in pure Python
Core Ideas
• Dynamic task scheduling yields sane parallelism
• Simple library to enable parallelism

30. Dask: Out of Core Scheduler for Python
• A parallel computing framework
• That leverages the excellent Python ecosystem
• Using blocked algorithms and task scheduling
• Written in pure Python
Core Ideas
• Dynamic task scheduling yields sane parallelism
• Simple library to enable parallelism
• Dask.array/dataframe to encapsulate the functionality

31. Dask: Out of Core Scheduler for Python
• A parallel computing framework
• That leverages the excellent Python ecosystem
• Using blocked algorithms and task scheduling
• Written in pure Python
Core Ideas
• Dynamic task scheduling yields sane parallelism
• Simple library to enable parallelism
• Dask.array/dataframe to encapsulate the functionality
• Distributed scheduler coming

32. Simple Architecture

33. Core Concepts

34. dask.array: OOC, parallel, ND array
Arithmetic: +, *, ...
Reductions: mean, max, ...
Slicing: x[10:, 100:50:-2]
Fancy indexing: x[:, [3, 1, 2]]
Some linear algebra: tensordot, qr, svd
Parallel algorithms (approximate quantiles, topk, ...)
Slightly overlapping arrays
Integration with HDF5

35. dask.dataframe: OOC, parallel, ND array
Elementwise operations: df.x + df.y
Row-wise selections: df[df.x > 0]
Aggregations: df.x.max()
groupby-aggregate: df.groupby(df.x).y.max()
Value counts: df.x.value_counts()
Drop duplicates: df.x.drop_duplicates()
Join on index: dd.merge(df1, df2, left_index=True,
right_index=True)

36. More Complex Graphs
cross validation

37. http://continuum.io/blog/xray-dask

38. PyData's Future
• Dozens of international meetup groups
• Intl conferences each year, including collab
with EuroPython, Strata, and others
• More companies investing in the ecosystem
• Dato - SFrame, SGraph, ...
• Cloudera - Impyla, Ibis, ...
• Microsoft - Python in AzureML
• Databricks - PySpark
• Continuum - *.*