Se ha denunciado esta presentación.
Utilizamos tu perfil de LinkedIn y tus datos de actividad para personalizar los anuncios y mostrarte publicidad más relevante. Puedes cambiar tus preferencias de publicidad en cualquier momento.

Dmitry Tiagulskyi, Yaroslav Yermilov "It Scales Until It Doesn’t"

336 visualizaciones

Publicado el

We are used to thinking that “high-load” means distributed systems, computing power, application, and kernel profiling. But sometimes you can’t simply scale your cluster. Maybe your hashmaps don’t fit in the server memory. Maybe you need single-digit millisecond latency. Maybe the cost is too high. Or your server is a … mobile phone.

In this talk, we will show how popular and lesser-known algorithms, data structures, and systems tuning helped us to overcome these blockers. Who said you don’t need to know algorithms nowadays?

Publicado en: Tecnología
  • Sé el primero en comentar

  • Sé el primero en recomendar esto

Dmitry Tiagulskyi, Yaroslav Yermilov "It Scales Until It Doesn’t"

  1. 1. It Scales Until It Doesn’t yaroslav.yermilov@grammarly.com - Software Engineer @ Core Services dima.tiagulskyi@grammarly.com - Software Engineer @ Core Services
  2. 2. ● Algorithms & Data Structures for Language Models (Ngrams) ○ Thomas H. Cormen, Charles E. Leiserson, Ronald L. Rivest, Clifford Stein “Introduction to Algorithms” (Chapter 11 “Hash Tables” (253) specifically) ○ F.C. Botelho, R. Pagh, and N. Ziviani. Simple and space-efficient minimal perfect hash functions. In Proc. of the 10th Workshop on Algorithms and Data Structures (WADs’07), pages 139–150. Springer LNCS vol. 4619, 2007. ○ Djamal Belazzougui, Fabiano Botelho, and Martin Dietzfelbinger. 2009. Hash, displace, and compress. Algorithms - ESA 2009, pages 682–693. ○ Kenneth Heafield. 2011. KenLM: Faster and smaller language model queries. In Proceedings of the Sixth Workshop on Statistical Machine Translation, Edinburgh, UK, July. Association for Computational Linguistics. ○ Adam Pauls and Dan Klein. 2011. Faster and smaller ngram language models. In Proceedings of ACL, Portland, Oregon. ○ David Talbot and Miles Osborne. 2007. Randomised language modelling for statistical machine translation. In Proceedings of ACL, pages 512–519, Prague, Czech Republic. ○ D. Guthrie, M. Hepple, and W. Liu, “Efficient minimal perfect hash language models,” in Proceedings of LREC’10. Valletta, Malta: European Language Resources Association (ELRA), May 2010. References
  3. 3. ● AWS Virtualization, ENA ○ https://en.wikipedia.org/wiki/X86_virtualization ○ https://aws.amazon.com/ru/blogs/aws/elastic-network-adapter-high-performance-network-interface-for-amazon-ec2/ ○ https://medium.com/@paccattam/aws-enhanced-networking-an-overview-aee8a852cf5c ○ AWS re:Invent 2017: Optimizing Network Performance for Amazon EC2 Instances: https://youtu.be/-dWgqtGKPfc ○ https://www.kernel.org/doc/Documentation/networking/scaling.txt ○ AWS re:Invent 2017: C5 Instances and the Evolution of Amazon EC2 Virtualization: https://youtu.be/LabltEXk0VQ ○ http://www.brendangregg.com/blog/2017-11-29/aws-ec2-virtualization-2017.html ○ https://blog.ubuntu.com/2017/04/05/ubuntu-on-aws-gets-serious-performance-boost-with-aws-tuned-kernel References
  4. 4. ● NUMA ○ https://www.redhat.com/files/summit/2014/summit2014_riel_chegu_w_0340_automatic_numa_balancing.pdf ○ https://www.cmg.org/wp-content/uploads/2015/10/numa.pdf ● Network Servers, IO Multiplexing & Epoll ○ https://eli.thegreenplace.net/2018/measuring-context-switching-and-memory-overheads-for-linux-threads/ ○ Asynchronous IO with Boost.Asio: https://youtu.be/rwOv_tw2eA4 ○ https://eklitzke.org/blocking-io-nonblocking-io-and-epoll ○ https://habr.com/post/416669/ ● Netty ○ One Framework to rule them all by Norman Maurer: https://youtu.be/DKJ0w30M0vg ○ https://dou.ua/lenta/columns/netty-optimization/ References
  5. 5. ● Benchmarking ○ http://www.brendangregg.com/blog/2018-06-30/benchmarking-checklist.html ○ http://www.brendangregg.com/activebenchmarking.html ○ http://www.brendangregg.com/usemethod.html ○ http://www.brendangregg.com/linuxperf.html ● Java Profiling ○ http://psy-lob-saw.blogspot.com/2016/02/why-most-sampling-java-profilers-are.html ○ https://github.com/jvm-profiling-tools/async-profiler ○ https://medium.com/netflix-techblog/java-in-flames-e763b3d32166 References

×