Redis for Data Science and Engineering by Dmitry Polyakovsky of Oracle

•Descargar como PPTX, PDF•

0 recomendaciones•356 vistas

This document contains a presentation about using Redis for data science and engineering. It introduces the presenter and provides an agenda that covers using Redis for data science and data engineering. The presentation notes that Redis can be used as both a data store and job queue, has flexible data structures and is fast, though it uses RAM and cannot query by value. It also lists Python Pandas and includes a demo and links for further information.

Software

PRESENTED
BY
Redis for Data Science / Engineering
Dmitry Polyakovsky

PRESENTED
BY
• Cache
• Background job queue
• API throttling
• Feature flags
• ...
Common Redis use-cases

PRESENTED
BY
1 Data Science
2 Data Engineering
Agenda:

PRESENTED
BY
import pandas as pd
df1 = pd.read_csv('file.csv')
df2 = pd.read_json('http://.../something.json')
df3 = pd.read_sql_query('select * from …', connection)
df1.aggregate(...)
Python Pandas

PRESENTED
BY
• Both data store and job queue
• Flexible data structures
• Speed
Pros

PRESENTED
BY
• RAM usage
• Cannot query by value
Cons

PRESENTED
BY
• SQL DB
• NoSQL DB
• Large flat files
• RabbitMQ / ActiveMQ / …
Alternatives

PRESENTED
BY
• http://dmitrypol.github.io/categories.html#redis
• https://twitter.com/dmitrypol
• https://pandas.pydata.org/
• https://pypi.org/project/redis/
Links

Más contenido relacionado

La actualidad más candente

SQL, Redis and Kubernetes by Paul Stanton of Windocks - Redis Day Seattle 2020

Redis Labs

RedisConf18 - Designing a Redis Client for Humans

Redis Labs

Leveraging Redis for System Monitoring by Adam McCormick of SBG - Redis Day S...

Redis Labs

On this webinar, EnterpriseDB's Jamie Watt, VP of Global Support Services and Ajay Patel, Cloud Services Delivery Manager, will demonstrate how to launch on-demand Oracle compatible Postgres Databases in minutes on EDB Cloud Database Service (CDS). Specifically, the webinar covers the following: - Instant Provisioning, Scaling and Managing Postgres - Built-in Monitoring, High Availability, and Backup tools - Using Pre-defined templates for standard configurations

EPAS + Cloud = Oracle Compatible Postgres in Minutes

EDB

RediSearch 1.6 by Pieter Cailliau - Redis Day Bangalore 2020

Redis Labs

MongoDB seminar

mahdi dousti

RedisConf17 - IoT Backend with Redis and Node.js

Redis Labs

The CARFAX vehicle history database contains over twelve billion documents in a twelve shard cluster that replicates to multiple data centers. This will be a step by step walk through of how we deploy our servers, manage high volume reads and writes, and our configuration for high availability. By automating everything from the operating system install up we are able deploy complete replica clusters quickly and efficiently. Using distributed processing and message queuing we load millions of new documents each day with a projected growth over a billion records per year. Through the use of tagging, server configuration, and read settings we deliver content with high consistency and availability.

Building a Scalable and Modern Infrastructure at CARFAX

MongoDB

RedisConf17 - Amadeus - Redis-Cluster operator

Redis Labs

MongoDB Administration 101

MongoDB

Afs manager

Manfred Furuholmen

HBaseConAsia2018 Track3-7: The application of HBase in New Energy Vehicle Mon...

Michael Stack

MongoDB is a leading database technology that combines the foundations of RDBMS with the innovations of NoSQL, allowing organizations to simultaneously boost productivity and lower TCO. MongoDB Enterprise Advanced is a finely-tuned package of advanced software, enterprise-grade support, and other services designed to accelerate your success with MongoDB in every stage of your app lifecycle, from early development to the scale-out of mission-critical production environments. With the release of 3.2, MongoDB Enterprise Advanced now includes: MongoDB Ops Manager 2.0 MongoDB Compass, the MongoDB GUI MongoDB Connector for Business Intelligence Encrypted Storage Engine In-Memory Storage Engine (beta) Attend this webinar to learn how MongoDB Enterprise Advanced can help you get to market faster and de-risk your mission critical deployments.

Introduction to MongoDB Enterprise

MongoDB

Webinar: Technical Introduction to Native Encryption on MongoDB

MongoDB

RedisConf17 - Smartwaiver - Using Redis for Kiosk Registration Command and Co...

Redis Labs

Need to break free from Oracle? An open source-based DBMS like Postgres is the answer. So, what is stopping you? Discover how to migrate from Oracle to Postgres, quickly and without risk. This talk will explain how to leverage tools and technologies to convert your Oracle database to EDB Postgres with ease and offer helpful best practices, including: What challenges to look for, like complex schemas, database stored procedures and more, where to start, like self-contained databases that help free up Oracle licenses, and, when to deploy your workload on-premises or in the cloud

No Time to Waste: Migrate from Oracle to Postgres in Minutes

EDB

Redis TimeSeries

Redis Labs

RedisConf18 - The Intelligent Database Proxy

Redis Labs

Solving Complex Scaling Problems by Prashant Kumar and Abhishek Jain of Myntr...

Redis Labs

Postgres expert Saurabh Shelar goes through the features and a live demo of EDB Postgres Failover Manager (EFM). You don’t want to miss this! EDB Postgres Failover Manager (EFM) is a high-availability module from EnterpriseDB that enables a Postgres Master node to automatically failover to a Standby node in the event of a software or hardware failure on the Master. Saurabh will cover key features of EDB Postgres Failover Manager and Demo Failover scenarios. Highlights Include: - Tuneable Properties for User Environment - Supported Failover Scenarios - A step by step demo

EDB Failover Manager - Features and Demo

EDB

La actualidad más candente (20)

SQL, Redis and Kubernetes by Paul Stanton of Windocks - Redis Day Seattle 2020

RedisConf18 - Designing a Redis Client for Humans

Leveraging Redis for System Monitoring by Adam McCormick of SBG - Redis Day S...

EPAS + Cloud = Oracle Compatible Postgres in Minutes

RediSearch 1.6 by Pieter Cailliau - Redis Day Bangalore 2020

MongoDB seminar

RedisConf17 - IoT Backend with Redis and Node.js

Building a Scalable and Modern Infrastructure at CARFAX

RedisConf17 - Amadeus - Redis-Cluster operator

MongoDB Administration 101

Afs manager

HBaseConAsia2018 Track3-7: The application of HBase in New Energy Vehicle Mon...

Introduction to MongoDB Enterprise

Webinar: Technical Introduction to Native Encryption on MongoDB

RedisConf17 - Smartwaiver - Using Redis for Kiosk Registration Command and Co...

No Time to Waste: Migrate from Oracle to Postgres in Minutes

Redis TimeSeries

RedisConf18 - The Intelligent Database Proxy

Solving Complex Scaling Problems by Prashant Kumar and Abhishek Jain of Myntr...

EDB Failover Manager - Features and Demo

Similar a Redis for Data Science and Engineering by Dmitry Polyakovsky of Oracle

http://www.agildata.com/agildata-hosts-big-data-meetup-featuring-apache-spark/ Slides for talks given at the Denver Java Users Group, Boulder Java Users Group, Denver/Boulder Big Data Users Group. Dan and Andy will spend an evening rolling up our sleeves with you to try out some real-world use cases for Apache Spark. We’ll cover Spark’s RDD API, the DataFrame API, as well as the brand new Dataset API.

Hands on with Apache Spark

Dan Lynn

data science toolkit 101: set up Python, Spark, & Jupyter

Raj Singh

App auto crud

Laurent Dami

Serving Deep Learning Models At Scale With RedisAI: Luca Antiga

Redis Labs

Introduction to Apache Spark

Juan Pedro Moreno

5 docker data_etl and visualization_hands_on

FEG

Paris Data Geek - Spark Streaming

Djamel Zouaoui

Presented by Udo Seidel, Chief Architect and Digital Evangelist, Amadeus In the recent past there were quite some discussions about security in the context of introducing or using Docker. It is true that there are some gaps to be closed but the whole story does not start from square one either. At Amadeus we are using Docker to build our future-oriented services and to introduce devops culture. Due to the nature of our business we have to deal with Security certifications like PCI-DSS, SSAE 16 and ISO 27001. This talks described the challenges we were facing in that context and how we mastered them. The story has technical and non-technical aspects.

DockerCon EU 2015: Docker and PCI-DSS - Lessons learned in a security sensiti...

Docker, Inc.

Container Monitoring with Sysdig

Sreenivas Makam

This presentation will be useful to those who would like to get acquainted with Apache Spark architecture, top features and see some of them in action, e.g. RDD transformations and actions, Spark SQL, etc. Also it covers real life use cases related to one of ours commercial projects and recall roadmap how we’ve integrated Apache Spark into it. Was presented on Morning@Lohika tech talks in Lviv. Design by Yarko Filevych: http://www.filevych.com/

Introduction to real time big data with Apache Spark

Taras Matyashovsky

Big data analysis in python @ PyCon.tw 2013

Jimmy Lai

The Actor Model applied to the Raspberry Pi and the Embedded Domain

Omer Kilic

Apache Spark 2.0 and subsequent releases of Spark 2.1 and 2.2 have laid the foundation for many new features and functionality. Its main three themes—easier, faster, and smarter—are pervasive in its unified and simplified high-level APIs for Structured data. In this introductory part lecture and part hands-on workshop, you’ll learn how to apply some of these new APIs using Databricks Community Edition. In particular, we will cover the following areas: Agenda: • Overview of Spark Fundamentals & Architecture • What’s new in Spark 2.x • Unified APIs: SparkSessions, SQL, DataFrames, Datasets • Introduction to DataFrames, Datasets and Spark SQL • Introduction to Structured Streaming Concepts • Four Hands-On Labs

Jump Start on Apache Spark 2.2 with Databricks

Anyscale

Python and Bigdata - An Introduction to Spark (PySpark)

hiteshnd

Fundamentals of performance tuning PHP on IBM i

Zend by Rogue Wave Software

In the last few years, the DevOps movement has introduced ground breaking approaches to the way we manage the lifecycle of software development and deployment. Today organisations aspire to fully automate the deployment of microservices and web applications with tools such as Chef, Puppet and Ansible. However, the deployment of data-processing pipelines remains a relic from the dark-ages of software development. Processing large-scale data pipelines is the main engineering task of the Big Data era, and it should be treated with the same respect and craftsmanship as any other piece of software. That is why we created Apache Amaterasu (Incubating) - an open source framework that takes care of the specific needs of Big Data applications in the world of continuous delivery. In this session, we will take a close look at Apache Amaterasu (Incubating) a simple and powerful framework to build and dispense pipelines. Amaterasu aims to help data engineers and data scientists to compose, configure, test, package, deploy and execute data pipelines written using multiple tools, languages and frameworks. We will see what Amaterasu provides today, and how it can help existing Big Data application and demo some of the new bits that are coming in the near future. Speaker: Yaniv Rodenski, Senior Solutions Architect, Couchbase

Introduction to Apache Amaterasu (Incubating): CD Framework For Your Big Data...

DataWorks Summit

Getting started with Apache Spark in Python - PyLadies Toronto 2016

Holden Karau

Presto @ Zalando - Big Data Tech Warsaw 2020

Piotr Findeisen

A lot of data scientists use the python library pandas for quick exploration of data. The most useful construct in pandas (based on R, I think) is the dataframe, which is a 2D array(aka matrix) with the option to “name” the columns (and rows). But pandas is not distributed, so there is a limit on the data size that can be explored. Spark is a great map-reduce like framework that can handle very big data by using a shared nothing cluster of machines. This work is an attempt to provide a pandas-like DSL on top of spark, so that data scientists familiar with pandas have a very gradual learning curve.

Introduction to df

Mohit Jaggi

df: Dataframe on Spark

Alpine Data

Similar a Redis for Data Science and Engineering by Dmitry Polyakovsky of Oracle (20)

Hands on with Apache Spark

data science toolkit 101: set up Python, Spark, & Jupyter

App auto crud

Serving Deep Learning Models At Scale With RedisAI: Luca Antiga

Introduction to Apache Spark

5 docker data_etl and visualization_hands_on

Paris Data Geek - Spark Streaming

DockerCon EU 2015: Docker and PCI-DSS - Lessons learned in a security sensiti...

Container Monitoring with Sysdig

Introduction to real time big data with Apache Spark

Big data analysis in python @ PyCon.tw 2013

The Actor Model applied to the Raspberry Pi and the Embedded Domain

Jump Start on Apache Spark 2.2 with Databricks

Python and Bigdata - An Introduction to Spark (PySpark)

Fundamentals of performance tuning PHP on IBM i

Introduction to Apache Amaterasu (Incubating): CD Framework For Your Big Data...

Getting started with Apache Spark in Python - PyLadies Toronto 2016

Presto @ Zalando - Big Data Tech Warsaw 2020

Introduction to df

df: Dataframe on Spark

Más de Redis Labs

Redis Day Bangalore 2020 - Session state caching with redis

Redis Labs

Protecting Your API with Redis by Jane Paek - Redis Day Seattle 2020

Redis Labs

Rust and Redis - Solving Problems for Kubernetes by Ravi Jagannathan of VMwar...

Redis Labs

Highly Available Persistent Session Management Service by Mohamed Elmergawi o...

Redis Labs

Building a Multi-dimensional Analytics Engine with RedisGraph by Matthew Goos...

Redis Labs

RedisGraph 2.0 by Pieter Cailliau - Redis Day Bangalore 2020

Redis Labs

RedisTimeSeries 1.2 by Pieter Cailliau - Redis Day Bangalore 2020

Redis Labs

RedisAI 0.9 by Sherin Thomas of Tensorwerk - Redis Day Bangalore 2020

Redis Labs

Rate-Limiting 30 Million requests by Vijay Lakshminarayanan and Girish Koundi...

Redis Labs

Three Pillars of Observability by Rajalakshmi Raji Srinivasan of Site24x7 Zoh...

Redis Labs

Redis as a High Scale Swiss Army Knife by Rahul Dagar and Abhishek Gupta of G...

Redis Labs

Deploying Redis as a Sidecar in Kubernetes by Janakiram MSV - Redis Day Banga...

Redis Labs

Moving Beyond Cache by Yiftach Shoolman - Redis Day Bangalore 2020

Redis Labs

Real-time GeoSearching at Scale with RediSearch by Apoorva Gaurav and Ronil M...

Redis Labs

Build a High-performance Partner Analytics Platform by Ashish Jadhav and Neer...

Redis Labs

Build a Deep Learning App with Tensorflow & Redis by Jayesh Ahire and Sherin ...

Redis Labs

Accelerating Recommendations at Viu by Amarendra Kumar and Kulbhushan Pachaur...

Redis Labs

A Low-latency Logging Framework by Rajat Panwar of HolidayMe - Redis Day Bang...

Redis Labs

Why We Love Redis by Amit Agrawal CTO of sRide - Redis Day Bangalore 2020

Redis Labs

Redis Streams

Redis Labs

Más de Redis Labs (20)