Se ha denunciado esta presentación.
Utilizamos tu perfil de LinkedIn y tus datos de actividad para personalizar los anuncios y mostrarte publicidad más relevante. Puedes cambiar tus preferencias de publicidad en cualquier momento.
/ (+ )
lablup.com
,
,
.
/ (+ )
lablup.com
,
,
.
▪
▪
▪ /
▪
▪
/
▪ TNF/ Needlworks
▪ …
▪
▪
▪
▪ NBA
▪
80 Gbps
▪
▪ …
http://www.lablup.com .
▪ 21 , 20 , 19
▪
▪ ,
21
?
▪
.
▪
.
.
..
▪
▪
▪ / / /
▪
▪ /
▪ Three keyboardists but…
▪
▪
▪
▪ resource consolidation
▪
▪
!
Python!
…
…
Python!
▪
▪ :
▪ PEP-8
▪ :
▪
▪
Python T_T
▪
▪
▪ PyPy / Pyston
▪ :
▪ C
▪ Cython
▪
Python /
▪
▪ CS101
▪ “ ”
▪
▪ indent
…
▪
▪ MATLAB numpy
:
▪
▪ MATLAB : matplotlib, ipython notebook
▪ Ultra-complex figures
▪
▪
▪
▪ /
:
▪
▪ NBA
▪
▪ configuration Python
▪ I/O
stdout asyncio
It's time to go Python 3
▪ Python 3
▪ Django, numpy/scipy, …
▪ Python 3.3
▪ PEP-393: str
unicode codepoint
▪ narrow build ...
It's time to go Python 3
▪ Python 3.4
▪ PEP-3156: asyncio coroutine
▪ I/O
▪ GIL (global interpreter lock)
▪
▪ pip virtuale...
▪ RabbitMQ, Kafka, Celery, ...
▪
▪ (broker) ,
▪ redis …
– ZeroMQ
▪
▪ Request-Reply
▪ 1
( : HTTP-like server/client)
▪ Push-Pull
▪ N-to-1
( : logging system, task partitioning)
▪ ...
– ZeroMQ
▪ :
▪ zero broker, zero latency, zero administration, ...
▪ : connection
▪ TCP / UDP / UNIX domain / local "in-pr...
– LogStash
▪ LogStash
▪ input / filter / output
▪ ZeroMQ, S3
▪ ( !)
Django
User Kernel
Logstash
Server
Database
ZeroMQ (pu...
On-premise vs. Hosting vs. Cloud
▪
▪ " " ...
▪ , ,
▪ AWS
▪
▪
▪ …
▪ 2012 R2 / /
.
Docker containers
▪ docker container
▪
▪ VM deploy/destroy
▪ AWS EC2 Instance: ( )
▪ pooling
▪ Amazon ECS?
▪ container (ru...
Docker containers
▪
▪ : instance
▪ Microsoft docker
▪ Azure Windows
Polymer library
▪ HTML5
▪ HTML imports / Custom elements / Shadow DOM
▪ Web components
▪ “Polyfill”
▪
▪ DOMelements
Polymer library
▪ ...
▪ !
▪ Bootstrap + theme?
▪
▪ HTML5 !
HTML imports!
Reusable web
components!
Shadow DOM!
Polyfill!
Polymer
▪
▪ 7 ? ?
Electron: Cross-platform Polymer
▪ : cross-platform webapp container
▪ Electron
▪ Github Node.js
▪ Node Chromium wrapping
...
▪
▪
▪
▪ Django UnitTest + Selenium
▪ Selenium webdriver:
▪
▪
▪
▪ sphinx
▪ reST
▪
▪
▪ !
▪
▪
▪
▪ sphinx
▪ reST
▪
▪
▪ !
▪
.
.
, ,
+
,
▪ +
.
▪ :
▪ –
▪ , , scaling, ...
➞ ++
▪ :
▪ –
▪
▪ .
▪ !
▪
▪
CodeON!
//codeonweb.com
!
…
, …
“ .”
▪ React / Flux / AngularJS / Polymer / ...
▪ +UI
▪ Polymer 0.9 API ....
▪ (?) Bootstrap …
▪ I/O 2015 Polymer 1.0
...
Polymer: it is too google to be true
▪
▪ 0.5 - 0.8rc2 - 0.9 - …
▪
▪ ( )
▪ Google I/O - 1.0 :
Polymer: vulcanize
▪ Vulcanize
▪ :
▪ IE – +
▪ CSS – +
▪ Crisper javascript map shadow DOM
root - +
Polymer: vulcanize
▪ Vulcanize
▪ :
▪ IE –
▪ CSS –
▪ Crisper javascript map shadow DOM
root -
Polymer: polyfill
▪
▪
▪
(html import
shadow DOM)
polyfill
▪
▪
Polymer + Django
▪
▪ {{ … }}
▪ Polymer –
▪ Django –
▪
▪
▪ …
Polymer + Django + Security
▪ CORS (cross-origin resource scripting)
▪ Django – CORS header ( )
▪ Polymer – vulcanize Java...
Polymer
▪ Production ready?!
▪
▪ “Do not swim in sandocean. Swim in the ocean.”
▪ HTML5
▪
▪ X-tag, react, polymer…
▪ ?
Polymer
▪ Production ready?!
▪
▪ “Do not swim in sandocean. Swim in the ocean.”
▪ HTML5
▪
▪ X-tag, react, polymer…
▪ ?
Electron + web
▪ ... !
▪
▪
(Noto )
▪
▪ CSS:
.
▪
?
: Django + RDS
▪ UTF-8 Python ?
▪ (cf., str / bytes / io / codecs)
▪ MySQL SQLite
▪ MySQL
▪ ForeignKey Django
▪ MySQL + Dj...
Python 3.4 – asyncio / coroutine
▪ .
▪ yield	from	 ...
▪ AST	 static	analysis coroutine
yield	from
▪ , 100% static analysi...
Python 3.4 – asyncio loop
▪ loop.close()
..
import asyncio
import asyncio_redis
loop = asyncio.get_event_loop()
conn = loo...
Python 3.4 – asyncio loop
▪ asyncio graceful shutdown ,
1. loop._run_once()
2. loop.run_until_complete(asyncio.sleep(0))
3...
Python 3.4 – asyncio loop
#! /usr/bin/env python3
import asyncio
loop = asyncio.get_event_loop()
@asyncio.coroutine
def my...
Python 3.4 – asyncio loop
#! /usr/bin/env python3
import asyncio
loop = asyncio.get_event_loop()
@asyncio.coroutine
def my...
asyncio_redis
▪ Redis asyncio
▪ Heisenbug ! 🚨
▪ SCAN key ...
▪ while if ➞
▪ https://github.com/jonathanslenders/asyncio-
r...
: – ZeroMQ
▪ Asynchronous ? block ??
▪ socket connect
. ( ...)
▪ send/recv timeout
▪ aiozmq , blocking call
asyncio.wait_f...
: – LogStash
▪ zmq input plugin s3 output plugin
zmq recv_string
▪ 1: s3 output plugin “write”
, PutObject
DeleteObject .
...
: Docker
▪
▪ docker registry "latest"
latest
▪ 2.0 2.0.1 tag
▪
▪
▪ docker 1.8 , CoreOS (rkt)
▪ /API
:
▪ Jupyter / ipython
▪ scalable
▪ Jupyter
▪ Unix	
▪
▪ :
▪
▪ – Sorna
▪ .
CodeON
▪ Container Resource Consolidation
▪ Kubernetes
▪ paxos raft
agreement key-value
store .
▪ Ingen REPL (read-evaluate-print...
▪
▪
▪ User profiler + backtracer + ML with back propagation
▪
▪ d3.js fluid nonlinear navigation UI
▪
▪
▪
▪
▪
▪
.
▪ ... + +
…
▪ .
▪ ?
Lablup Inc.
http://www.lablup.com
연구자 및 교육자를 위한 계산 및 분석 플랫폼 설계 - PyCon KR 2015
연구자 및 교육자를 위한 계산 및 분석 플랫폼 설계 - PyCon KR 2015
연구자 및 교육자를 위한 계산 및 분석 플랫폼 설계 - PyCon KR 2015
연구자 및 교육자를 위한 계산 및 분석 플랫폼 설계 - PyCon KR 2015
연구자 및 교육자를 위한 계산 및 분석 플랫폼 설계 - PyCon KR 2015
연구자 및 교육자를 위한 계산 및 분석 플랫폼 설계 - PyCon KR 2015
연구자 및 교육자를 위한 계산 및 분석 플랫폼 설계 - PyCon KR 2015
연구자 및 교육자를 위한 계산 및 분석 플랫폼 설계 - PyCon KR 2015
연구자 및 교육자를 위한 계산 및 분석 플랫폼 설계 - PyCon KR 2015
연구자 및 교육자를 위한 계산 및 분석 플랫폼 설계 - PyCon KR 2015
Próxima SlideShare
Cargando en…5
×

연구자 및 교육자를 위한 계산 및 분석 플랫폼 설계 - PyCon KR 2015

3.368 visualizaciones

Publicado el

현대 과학 연구에는 컴퓨터를 이용한 계산 및 분석 작업이 필수적입니다. 그러나 거대 스케일의 계산 및 분석 작업을 수행할 경우 컴퓨팅 리소스의 적절한 관리 및 확장 용이성을 확보하는 것은 많은 리소스를 필요로 합니다. 우리는 컴퓨터 계산 작업 및 분석 작업을 표준화하고 클라우드에서 처리하는 파이썬3 기반의 오픈소스 플랫폼을 설계 및 개발하고 있습니다. 또한 이 플랫폼 위에서 돌아가는 교육 / 연구 플랫폼을 함께 설계하고 있습니다.

새로운 서비스를 변화하는 환경에 맞추어 개발하는 일은 즐거운 경험인 동시에, "무엇을" "어떻게" "왜" 로 이어지는 지뢰밭을 거니는 일이기도 합니다. "무엇을" 만들지 고민하고 토론하며 결정하고, 설계하고, 토론하고, 목표가 바뀌는 과정이 일어납니다. "어떻게" 만드느냐의 지뢰들로는 python 2에서 python 3 로의 전환, 웹 프레임웍인 Django와 프론트엔드 프레임웍들과의 충돌, 아마존 elastic computing cloud와 docker를 사용한 디플로이 시나리오 등 삽질 중에 발생하는 일들이 있습니다. "왜"에 대한 질문들은 무겁지만 피해갈 수 없습니다. "왜 하필 파이썬인가?" "왜 하필 그런 서비스를 만드려 하는가" 등의 질문은, 무엇인가를 만들기로 결심한 사람들에게 주어지는 가장 중요한 질문이자 보상이기도 합니다.

저희는 지난 2개월동안 이 과정을 통해 우리가 배운 것들을 공유하고자 합니다. 구체적으로는 개발 중인 플랫폼 구조, 설계 과정의 경험 및 python 3 기반의 플랫폼 개발시 주의할 점들에 대해 이야기하고, 그와 함께 지뢰밭을 걷게 만든 '원동력' 에 대해 함께 이야기할 수 있는 자리가 되었으면 합니다.

Publicado en: Software

연구자 및 교육자를 위한 계산 및 분석 플랫폼 설계 - PyCon KR 2015

  1. 1. / (+ ) lablup.com , , .
  2. 2. / (+ ) lablup.com , , .
  3. 3. ▪ ▪ ▪ / ▪ ▪ / ▪ TNF/ Needlworks ▪ … ▪ ▪ ▪ ▪ NBA ▪ 80 Gbps ▪ ▪ … http://www.lablup.com .
  4. 4. ▪ 21 , 20 , 19 ▪ ▪ ,
  5. 5. 21 ?
  6. 6. ▪ . ▪ .
  7. 7. . ..
  8. 8. ▪ ▪ ▪ / / / ▪ ▪ /
  9. 9. ▪ Three keyboardists but…
  10. 10. ▪ ▪ ▪ ▪ resource consolidation ▪ ▪
  11. 11. ! Python! … …
  12. 12. Python! ▪ ▪ : ▪ PEP-8 ▪ : ▪ ▪
  13. 13. Python T_T ▪ ▪ ▪ PyPy / Pyston ▪ : ▪ C ▪ Cython ▪
  14. 14. Python / ▪ ▪ CS101 ▪ “ ” ▪ ▪ indent … ▪ ▪ MATLAB numpy
  15. 15. : ▪ ▪ MATLAB : matplotlib, ipython notebook ▪ Ultra-complex figures ▪ ▪ ▪ ▪ /
  16. 16. : ▪ ▪ NBA ▪ ▪ configuration Python ▪ I/O stdout asyncio
  17. 17. It's time to go Python 3 ▪ Python 3 ▪ Django, numpy/scipy, … ▪ Python 3.3 ▪ PEP-393: str unicode codepoint ▪ narrow build UTF-16, wide build UTF-32 ▪ UTF-8/16/32 ▪ ASCII/Latin-1 ▪ / ·
  18. 18. It's time to go Python 3 ▪ Python 3.4 ▪ PEP-3156: asyncio coroutine ▪ I/O ▪ GIL (global interpreter lock) ▪ ▪ pip virtualenv ▪
  19. 19. ▪ RabbitMQ, Kafka, Celery, ... ▪ ▪ (broker) , ▪ redis …
  20. 20. – ZeroMQ ▪ ▪ Request-Reply ▪ 1 ( : HTTP-like server/client) ▪ Push-Pull ▪ N-to-1 ( : logging system, task partitioning) ▪ Publish-Subscribe ▪ 1-to-N / N-to-M ( : broadcast)
  21. 21. – ZeroMQ ▪ : ▪ zero broker, zero latency, zero administration, ... ▪ : connection ▪ TCP / UDP / UNIX domain / local "in-process" ▪ zeromq transport header (ZMTP) ▪ payload (64+ ) ▪ : persistent queue ▪ TCP reliability
  22. 22. – LogStash ▪ LogStash ▪ input / filter / output ▪ ZeroMQ, S3 ▪ ( !) Django User Kernel Logstash Server Database ZeroMQ (push-pull) + JSON AWS S3 msgpack
  23. 23. On-premise vs. Hosting vs. Cloud ▪ ▪ " " ... ▪ , , ▪ AWS ▪ ▪ ▪ … ▪ 2012 R2 / / .
  24. 24. Docker containers ▪ docker container ▪ ▪ VM deploy/destroy ▪ AWS EC2 Instance: ( ) ▪ pooling ▪ Amazon ECS? ▪ container (run task) latency ▪ …
  25. 25. Docker containers ▪ ▪ : instance ▪ Microsoft docker ▪ Azure Windows
  26. 26. Polymer library ▪ HTML5 ▪ HTML imports / Custom elements / Shadow DOM ▪ Web components ▪ “Polyfill” ▪ ▪ DOMelements
  27. 27. Polymer library ▪ ... ▪ ! ▪ Bootstrap + theme? ▪ ▪ HTML5 !
  28. 28. HTML imports! Reusable web components! Shadow DOM! Polyfill!
  29. 29. Polymer ▪ ▪ 7 ? ?
  30. 30. Electron: Cross-platform Polymer ▪ : cross-platform webapp container ▪ Electron ▪ Github Node.js ▪ Node Chromium wrapping ▪ / / ▪ ATOM ▪ ? ▪ Apache Cordova
  31. 31. ▪ ▪ ▪ ▪ Django UnitTest + Selenium ▪ Selenium webdriver: ▪
  32. 32. ▪ ▪ ▪ sphinx ▪ reST ▪ ▪ ▪ ! ▪
  33. 33. ▪ ▪ ▪ sphinx ▪ reST ▪ ▪ ▪ ! ▪
  34. 34. . .
  35. 35. , , + ,
  36. 36. ▪ + . ▪ : ▪ – ▪ , , scaling, ... ➞ ++ ▪ : ▪ – ▪
  37. 37. ▪ . ▪ ! ▪ ▪
  38. 38. CodeON! //codeonweb.com !
  39. 39.
  40. 40. , …
  41. 41. “ .”
  42. 42. ▪ React / Flux / AngularJS / Polymer / ... ▪ +UI ▪ Polymer 0.9 API .... ▪ (?) Bootstrap … ▪ I/O 2015 Polymer 1.0 ...
  43. 43. Polymer: it is too google to be true ▪ ▪ 0.5 - 0.8rc2 - 0.9 - … ▪ ▪ ( ) ▪ Google I/O - 1.0 :
  44. 44. Polymer: vulcanize ▪ Vulcanize ▪ : ▪ IE – + ▪ CSS – + ▪ Crisper javascript map shadow DOM root - +
  45. 45. Polymer: vulcanize ▪ Vulcanize ▪ : ▪ IE – ▪ CSS – ▪ Crisper javascript map shadow DOM root -
  46. 46. Polymer: polyfill ▪ ▪ ▪ (html import shadow DOM) polyfill ▪ ▪
  47. 47. Polymer + Django ▪ ▪ {{ … }} ▪ Polymer – ▪ Django – ▪ ▪ ▪ …
  48. 48. Polymer + Django + Security ▪ CORS (cross-origin resource scripting) ▪ Django – CORS header ( ) ▪ Polymer – vulcanize Javascript CORS ▪ Crisper CORS Javascript ▪ - ▪ XSS ▪ Django - CSRF token ▪ Polymer – header iron-ajax ▪
  49. 49. Polymer ▪ Production ready?! ▪ ▪ “Do not swim in sandocean. Swim in the ocean.” ▪ HTML5 ▪ ▪ X-tag, react, polymer… ▪ ?
  50. 50. Polymer ▪ Production ready?! ▪ ▪ “Do not swim in sandocean. Swim in the ocean.” ▪ HTML5 ▪ ▪ X-tag, react, polymer… ▪ ?
  51. 51. Electron + web ▪ ... ! ▪ ▪ (Noto ) ▪ ▪ CSS: . ▪ ?
  52. 52. : Django + RDS ▪ UTF-8 Python ? ▪ (cf., str / bytes / io / codecs) ▪ MySQL SQLite ▪ MySQL ▪ ForeignKey Django ▪ MySQL + Django + Korean = EPIC FAIL ▪ ( ) Postgres ... ▪ postgres ( / ) .
  53. 53. Python 3.4 – asyncio / coroutine ▪ . ▪ yield from ... ▪ AST static analysis coroutine yield from ▪ , 100% static analysis ▪ duck typing + dynamic method generation + method proxy pattern ... ▪ Python 3.5 await/async !
  54. 54. Python 3.4 – asyncio loop ▪ loop.close() .. import asyncio import asyncio_redis loop = asyncio.get_event_loop() conn = loop.run_until_complete( asyncio_redis.Connection.create('localhost', 6379)) conn.close() loop.close() Task was destroyed but it is pending! task: <Task pending coro=<_reader_coroutine() running at /... /lib/python3.4/site-packages/asyncio_redis/protocol.py:919> wait_for=<Future pending cb=[Task._wakeup()]>>
  55. 55. Python 3.4 – asyncio loop ▪ asyncio graceful shutdown , 1. loop._run_once() 2. loop.run_until_complete(asyncio.sleep(0)) 3. loop.run_until_complete(server.wait_closed()) ▪ , wait_closed() coroutine . ▪ event loop .
  56. 56. Python 3.4 – asyncio loop #! /usr/bin/env python3 import asyncio loop = asyncio.get_event_loop() @asyncio.coroutine def my_timer(): i = 0 while True: yield from asyncio.sleep(1) print(i) i += 1 try: asyncio.async(my_timer(), loop=loop) loop.run_forever() except KeyboardInterrupt: pass finally: loop.close() ^CTask was destroyed but it is pending! task: <Task pending coro=<my_timer() done, defined at test.py:4> wait_for=<Future pending cb=[Task._wakeup()]>>
  57. 57. Python 3.4 – asyncio loop #! /usr/bin/env python3 import asyncio loop = asyncio.get_event_loop() @asyncio.coroutine def my_timer(): i = 0 while True: yield from asyncio.sleep(1) print(i) i += 1 try: asyncio.async(my_timer(), loop=loop) loop.run_forever() except KeyboardInterrupt: for t in asyncio.Task.all_tasks(): t.cancel() try: loop._run_once() except asyncio.CancelledError: pass finally: loop.close()
  58. 58. asyncio_redis ▪ Redis asyncio ▪ Heisenbug ! 🚨 ▪ SCAN key ... ▪ while if ➞ ▪ https://github.com/jonathanslenders/asyncio- redis/issues/65 ▪ Connection pool API ▪ API call connection stateful API ( : SELECT) ▪ (aioredis)
  59. 59. : – ZeroMQ ▪ Asynchronous ? block ?? ▪ socket connect . ( ...) ▪ send/recv timeout ▪ aiozmq , blocking call asyncio.wait_for timeout . ▪ pyzmq , socket.poll(msec) timeout .
  60. 60. : – LogStash ▪ zmq input plugin s3 output plugin zmq recv_string ▪ 1: s3 output plugin “write” , PutObject DeleteObject . ▪ 2: config , output plugin log level · . ▪ 3: output plugin input plugin receive timeout .
  61. 61. : Docker ▪ ▪ docker registry "latest" latest ▪ 2.0 2.0.1 tag ▪ ▪ ▪ docker 1.8 , CoreOS (rkt) ▪ /API
  62. 62. : ▪ Jupyter / ipython ▪ scalable ▪ Jupyter ▪ Unix ▪ ▪ : ▪ ▪ – Sorna ▪ .
  63. 63. CodeON
  64. 64. ▪ Container Resource Consolidation ▪ Kubernetes ▪ paxos raft agreement key-value store . ▪ Ingen REPL (read-evaluate-print loop) ▪ stdout/stderr streaming ▪ interactive plot ▪ image, sound multimedia
  65. 65. ▪ ▪ ▪ User profiler + backtracer + ML with back propagation ▪ ▪ d3.js fluid nonlinear navigation UI ▪ ▪ ▪ ▪ ▪
  66. 66. ▪ . ▪ ... + + … ▪ . ▪ ?
  67. 67. Lablup Inc. http://www.lablup.com

×