PLNOG 3: Tomasz Mikołajczyk - Data scalability. Why you should care?

•

0 recomendaciones•12 vistas

PLNOG 3: Tomasz Mikołajczyk - Data scalability. Why you should care?

Data scalability. Why you
should care?
Tomasz Mikolajczyk
& the GridwiseTech team
www.gridwisetech.com
tomek@gridwisetech.com

GridwiseTech www.gridwisetech.com
Presentation survey
Presentation plan
1) About GridwiseTech
2) Building efficient database oriented, IT systems
3) Momentum + case study
Presentation duration
40 minutes

GridwiseTech www.gridwisetech.com
About GridwiseTech
Leading vendor-
independent
scalable technology
expert.
Solve scalability and
performance problems
with virtualization,
Grid, distributed data
processing, caching.
Implement vision for
distributed
infrastructures.
Mission: turn avant-garde technology into customer’s business benefits
About us

GridwiseTech www.gridwisetech.com
Chosen customers
Chosen partners
Our references
About us

GridwiseTech www.gridwisetech.com
Building efficient IT system
About us
Application
WWW
SOA
Middleware
Processing
Data provisioning
Data persistence

GridwiseTech www.gridwisetech.com
Vendors puzzle
About us
Application
WWW
SOA
Platform Computing, Sun
Microsystems, Altair,
UnivaUD, Cluster Resources
Gigaspaces, Gemstone,
Coherence, eXludus,
Terracotta
Oracle, IBM, Microsoft
PostreSQL, MySQL
Citrix, Vmvare,Novell, Virtual
Iron, RedHat

GridwiseTech www.gridwisetech.com
Improving the system efficiency

GridwiseTech www.gridwisetech.com
Simple data oriented IT system
Internal or
external
data sources
applications
working on data

GridwiseTech www.gridwisetech.com
IT systems are constantly growing
Increased number
of users
Increased number
of applications
Increased amount
of data

GridwiseTech www.gridwisetech.com
IT systems are constantly growing
Infrastructure
bottleneck

GridwiseTech www.gridwisetech.com
Hardware scale up – the first idea
Typical solution:
buy bigger and faster
hardware

GridwiseTech www.gridwisetech.com
Hardware scale up – the first idea
Recurring software and
hardware migrations will
always generate substantial
costs

GridwiseTech www.gridwisetech.com
Introduction of scalability
Simultaneous
calculation
Load balancing

GridwiseTech www.gridwisetech.com
Introduction of scalability
Simultaneous
calculation
Cloud
Load balancing

GridwiseTech www.gridwisetech.com
Introduction of distributed processing
Cloud
I/O bottleneck

GridwiseTech www.gridwisetech.com
Problem with data access?
Database / configuration / storage optimization
In-memory databases
Database clustering
Sharding
Data grids
Distributed caching

GridwiseTech www.gridwisetech.com
Fully scalable system
Scaling out data

GridwiseTech www.gridwisetech.com
Project for electronic manufacturer
Technical scope of the problem:
2,5 million transactions per day
many databases, different vendors and/or
versions
4TB of data to maintain
constant environment modifications
400,000 lines of PL/SQL code to analyze
Improve report generation time by 70% allowing for increase in daily production.
Project main goal
Business problem:
lack of possibility to increase production
limited process manageability due to report
availability delay
Repetitive infrastructure down time
resulting in data inconstancy
Data normalization problem due to many
different data sources

GridwiseTech www.gridwisetech.com
Initial customer architecture
1
2

GridwiseTech www.gridwisetech.com
Designed architecture
Distributed caching

GridwiseTech www.gridwisetech.com
Momentum

GridwiseTech www.gridwisetech.com
Project for electronic manufacturer
Results of scaling manufacturer infrastructure:
Scalability ensured on each functional layer
Real-time report generation
10x times data processing speed-up
100,000 of reads/writes per second (7500 before)
Fully manageable production process
Improve report generation time by 70% allowing for increase in daily production.
Project main goal

GridwiseTech www.gridwisetech.com
GridwiseTech's value add
GridwiseTech's value add:
 Application for automatic mapping database tables into
objects.
 Scalable Feeder application that provide fast data replication
between multiple data sources and data grid.
 Data normalization from multiple data sources.
 Mechanism, that prevent from RAM overloading.
 Mechanism, that provide synchronisation possibility between
services and applications.
 Technique of building thread enabled and fully scalable
external services with batch processing support.

Thank you
www.gridwisetech.com
contact@gridwisetech.com

Más contenido relacionado

La actualidad más candente

Yellowbrick Webcast with DBTA for Real-Time Analytics

Yellowbrick Webcast with DBTA for Real-Time Analytics

Yellowbrick Webcast with DBTA for Real-Time Analytics

Yellowbrick Data

Watch full webinar here:https://buff.ly/2Edqlpo Hybrid cloud computing is slowing becoming the standard for businesses. The transition to hybrid can be challenging depending on the environment and the needs of the business. A successful move will involve using the right technology and seeking the right help. At the same time, multi-cloud strategies are on the rise. More enterprise organizations than ever before are analyzing their current technology portfolio and defining a cloud strategy that encompasses multiple cloud platforms to suit specific app workloads, and move those workloads as they see fit. In this session, you will learn: *Key challenges of migration to the cloud in a complex data landscape *How data virtualization can help build a data driven, multi-location cloud architecture for real time integration *How customers are taking advantage of data virtualization to save time and costs with limited resources

Data Virtualization to Survive a Multi and Hybrid Cloud World

Data Virtualization to Survive a Multi and Hybrid Cloud World

Data Virtualization to Survive a Multi and Hybrid Cloud World

Watch full webinar here: https://bit.ly/3aIofv9 The best of breed big data fabrics should deliver actionable insights to the business users with minimal effort, provide end-to-end security to the entire enterprise data platform and provide real-time data integration, while delivering self-service data platform to business users. While big data initiatives have become necessary for any business to generate actionable insights, big data fabric has become a necessity for any successful big data initiative. The best of breed big data fabrics should deliver actionable insights to the business users with minimal effort, provide end-to-end security to the entire enterprise data platform and provide real-time data integration, while delivering self-service data platform to business users. Attend this session to learn how big data fabric enabled by data virtualization: - Provides lightning fast self-service data access to business users - Centralizes data security, governance and data privacy - Fulfills the promise of data lakes to provide actionable insights

Data Virtualization enabled Data Fabric: Operationalize the Data Lake (APAC)

Data Virtualization enabled Data Fabric: Operationalize the Data Lake (APAC)

Data Virtualization enabled Data Fabric: Operationalize the Data Lake (APAC)

End to End Supply Chain Control Tower

End to End Supply Chain Control Tower

End to End Supply Chain Control Tower

Watch full webinar here: https://bit.ly/3x7xVuR Organizations worldwide are adopting a variety of the public cloud service providers (i.e. AWS, Google, Microsoft) and each have a portfolio of storage, compute, network, and security options. All of which create significant challenges in managing a hybrid and multi-cloud enterprise architecture. Even worse is the impact to the governance and integration of data from the clouds and physical infrastructure to support the broad array of analytics and operational requirements. Can one public cloud provider meet all your needs today and in the future? How do you manage across multiple public and private clouds you have today and where your data exists? And, how would you manage and operate your multi-cloud and on-premises systems to gain value from your data in any of them? The Chief Research Officer at Ventana Research, Mark Smith, will expound the challenges and path ahead for virtualization and integration of your data and the clouds, setting an architectural path for best success.

The Virtualization of Clouds - The New Enterprise Data Architecture Opportunity

The Virtualization of Clouds - The New Enterprise Data Architecture Opportunity

The Virtualization of Clouds - The New Enterprise Data Architecture Opportunity

Partner Keynote: How Logical Data Fabric Knits Together Data Visualization wi...

Partner Keynote: How Logical Data Fabric Knits Together Data Visualization wi...

Partner Keynote: How Logical Data Fabric Knits Together Data Visualization wi...

What's New In Neo4j 3.4 & Bloom Update

What's New In Neo4j 3.4 & Bloom Update

What's New In Neo4j 3.4 & Bloom Update

ironSource Atom BigData Berlin

ironSource Atom BigData Berlin

ironSource Atom BigData Berlin

Gianluigi Vigano, Senior Architect and Fouad Teban, Regional Presales Manager...

Gianluigi Vigano, Senior Architect and Fouad Teban, Regional Presales Manager...

Gianluigi Vigano, Senior Architect and Fouad Teban, Regional Presales Manager...

Dataconomy Media

Scaling Multi-Cloud Deployments with Denodo: Automated Infrastructure Management

Scaling Multi-Cloud Deployments with Denodo: Automated Infrastructure Management

Scaling Multi-Cloud Deployments with Denodo: Automated Infrastructure Management

Quick Intro to Google Cloud Technologies

Quick Intro to Google Cloud Technologies

Quick Intro to Google Cloud Technologies

Clinical genomic analytics pipelines using Databricks and the Delta Lake for the benefit of loading individual reads from raw sequencing or base-call files have significant advantages over more traditional methods. Analysis pipelines that perform genomic mapping to purpose-built reference data artifacts persisted to tables allows for enhanced performance that is magnitudes greater than previous mapping methods. These scalable, reproducible, and potentially open sourced methods have the ability to transform bioinformatics and R&D data management / governance.

Managing R&D Data on Parallel Compute Infrastructure

Managing R&D Data on Parallel Compute Infrastructure

Managing R&D Data on Parallel Compute Infrastructure

Bad data leads to bad decisions and broken customer experiences. Organizations depend on complete and accurate data to power their business, maintain efficiency, and uphold customer trust. With thousands of datasets and pipelines running, how do we ensure that all data meets quality standards, and that expectations are clear between producers and consumers? Investing in shared, flexible components and practices for monitoring data health is crucial for a complex data organization to rapidly and effectively scale. At Zillow, we built a centralized platform to meet our data quality needs across stakeholders. The platform is accessible to engineers, scientists, and analysts, and seamlessly integrates with existing data pipelines and data discovery tools. In this presentation, we will provide an overview of our platform’s capabilities, including: Giving producers and consumers the ability to define and view data quality expectations using a self-service onboarding portal Performing data quality validations using libraries built to work with spark Dynamically generating pipelines that can be abstracted away from users Flagging data that doesn’t meet quality standards at the earliest stage and giving producers the opportunity to resolve issues before use by downstream consumers Exposing data quality metrics alongside each dataset to provide producers and consumers with a comprehensive picture of health over time

Democratizing Data Quality Through a Centralized Platform

Democratizing Data Quality Through a Centralized Platform

Democratizing Data Quality Through a Centralized Platform

Watch full webinar here: [https://buff.ly/2sLhFAc] TransAlta is an electric power generator company headquartered in Calgary, Alberta. TransAlta's IT department initiated "Zero Data Center" project to move their entire data layer to the cloud for flexibility, agility and lower TCO. Data virtualization technology played a central role in TransAlta's real-time data integration, while helping them move to the cloud with zero down-time Attend this Denodo DataFest 2018 session to learn: Who is TransAlta and why TransAlta wanted to move their entire enterprise data layer to the cloud Why data virtualization played a critical role in TransAlta's cloud modernization effort How TransAlta uses DV in their energy trading, wind icing forecast and HR fuctions

Cloud Modernization with Data Virtualization

Cloud Modernization with Data Virtualization

Cloud Modernization with Data Virtualization

In textile industry, fabric defect relies on human inspection traditionally, which is inaccurate, inconsistent, inefficient and expensive. There were automatic systems developed on the defect detection by identifying the faults in fabric surface using the image and video processing techniques. However, the existing solution has insufficiencies in defect data sharing, backhaul interconnect, maintenance and etc. By evolving to an edge-optimized architecture, we can help textile industry improve fabric quality, reduce operation cost and increase production efficiency. In this session, I’ll share: What’s edge computing and why it’s important to intelligence manufacturing What’s the characteristics, strengths and weaknesses of traditional fabric defect detection method Why textile industry can benefit from edge computing infrastructure How to design and implement an edge-enabled application for fabric defect detection in real-time Insights, synergy and future research directions

KEYNOTE: Edge optimized architecture for fabric defect detection in real-time

KEYNOTE: Edge optimized architecture for fabric defect detection in real-time

KEYNOTE: Edge optimized architecture for fabric defect detection in real-time

At the heart of this DataBench webinar is the goal to share a benchmarking process helping European organisations developing Big Data Technologies to reach for excellence and constantly improve their performance, by measuring their technology development activity against parameters of high business relevance. The webinar aims to provide the audience with a framework and tools to assess the performance and impact of Big Data and AI technologies, by providing real insights coming from DataBench. In addition, representatives from other projects part of the BDV PPP such as DeepHealth and They-Buy-for-You will participate to share the challenges and opportunities they have identified on the use of Big Data, Analytics, AI. The perspective of other projects that also have looked into benchmarking, such as Track&Now and I-BiDaaS will be introduced.

Virtual BenchLearning - Data Bench Framework

Virtual BenchLearning - Data Bench Framework

Virtual BenchLearning - Data Bench Framework

Big Data Value Association

Watch full webinar here: https://goo.gl/rSfYKV Gartner states in its Predicts 2018: Data Management Strategies Continue to Shift Toward Distributed, “As data management activities are becoming more widespread in both distributed processing use cases, like IoT, and demands for new types of data, emerging roles such as data scientists or data engineers are expected to be driving the new data management requirements in the coming two years. These trends indicate that both the collection of data as well as the need to connect to data are rapidly becoming the new normal, and that the days of a single data store with all the data of interest — the enterprise data warehouse — are long gone.” Data management solutions are becoming distributed, heterogeneous and extremely diverse. Attend this session to learn: • How to evolve architecture patterns in the cloud using data virtualization. • How data virtualization accelerates cloud migration and modernization. • Successful cloud implementation case studies.

Evolving From Monolithic to Distributed Architecture Patterns in the Cloud

Evolving From Monolithic to Distributed Architecture Patterns in the Cloud

Evolving From Monolithic to Distributed Architecture Patterns in the Cloud

A Modern Data Architecture in Azure, presented by Armand van Oijen (Kadenza) ...

A Modern Data Architecture in Azure, presented by Armand van Oijen (Kadenza) ...

A Modern Data Architecture in Azure, presented by Armand van Oijen (Kadenza) ...

Patrick Van Renterghem

Perth Microsoft Data & Analytics User Group - Building Solutions with Azure D...

Perth Microsoft Data & Analytics User Group - Building Solutions with Azure D...

Perth Microsoft Data & Analytics User Group - Building Solutions with Azure D...

Sergio Zenatti Filho

Machine learning development brings many new complexities beyond the traditional software development lifecycle. Unlike traditional software development, ML developers want to try multiple algorithms, tools and parameters to get the best results, and they need to track this information to reproduce work. In addition, developers need to use many distinct systems to productionize models. In this talk, learn how to operationalize ML across the full lifecycle with Databricks Machine Learning.

Learn to Use Databricks for the Full ML Lifecycle

Learn to Use Databricks for the Full ML Lifecycle

Learn to Use Databricks for the Full ML Lifecycle

La actualidad más candente (20)

Yellowbrick Webcast with DBTA for Real-Time Analytics

Yellowbrick Webcast with DBTA for Real-Time Analytics

Yellowbrick Webcast with DBTA for Real-Time Analytics

Data Virtualization to Survive a Multi and Hybrid Cloud World

Data Virtualization to Survive a Multi and Hybrid Cloud World

Data Virtualization to Survive a Multi and Hybrid Cloud World

Data Virtualization enabled Data Fabric: Operationalize the Data Lake (APAC)

Data Virtualization enabled Data Fabric: Operationalize the Data Lake (APAC)

Data Virtualization enabled Data Fabric: Operationalize the Data Lake (APAC)

End to End Supply Chain Control Tower

End to End Supply Chain Control Tower

End to End Supply Chain Control Tower

The Virtualization of Clouds - The New Enterprise Data Architecture Opportunity

The Virtualization of Clouds - The New Enterprise Data Architecture Opportunity

The Virtualization of Clouds - The New Enterprise Data Architecture Opportunity

Partner Keynote: How Logical Data Fabric Knits Together Data Visualization wi...

Partner Keynote: How Logical Data Fabric Knits Together Data Visualization wi...

Partner Keynote: How Logical Data Fabric Knits Together Data Visualization wi...

What's New In Neo4j 3.4 & Bloom Update

What's New In Neo4j 3.4 & Bloom Update

What's New In Neo4j 3.4 & Bloom Update

ironSource Atom BigData Berlin

ironSource Atom BigData Berlin

ironSource Atom BigData Berlin

Gianluigi Vigano, Senior Architect and Fouad Teban, Regional Presales Manager...

Gianluigi Vigano, Senior Architect and Fouad Teban, Regional Presales Manager...

Gianluigi Vigano, Senior Architect and Fouad Teban, Regional Presales Manager...

Scaling Multi-Cloud Deployments with Denodo: Automated Infrastructure Management

Scaling Multi-Cloud Deployments with Denodo: Automated Infrastructure Management

Scaling Multi-Cloud Deployments with Denodo: Automated Infrastructure Management

Quick Intro to Google Cloud Technologies

Quick Intro to Google Cloud Technologies

Quick Intro to Google Cloud Technologies

Managing R&D Data on Parallel Compute Infrastructure

Managing R&D Data on Parallel Compute Infrastructure

Managing R&D Data on Parallel Compute Infrastructure

Democratizing Data Quality Through a Centralized Platform

Democratizing Data Quality Through a Centralized Platform

Democratizing Data Quality Through a Centralized Platform

Cloud Modernization with Data Virtualization

Cloud Modernization with Data Virtualization

Cloud Modernization with Data Virtualization

KEYNOTE: Edge optimized architecture for fabric defect detection in real-time

KEYNOTE: Edge optimized architecture for fabric defect detection in real-time

KEYNOTE: Edge optimized architecture for fabric defect detection in real-time

Virtual BenchLearning - Data Bench Framework

Virtual BenchLearning - Data Bench Framework

Virtual BenchLearning - Data Bench Framework

Evolving From Monolithic to Distributed Architecture Patterns in the Cloud

Evolving From Monolithic to Distributed Architecture Patterns in the Cloud

Evolving From Monolithic to Distributed Architecture Patterns in the Cloud

A Modern Data Architecture in Azure, presented by Armand van Oijen (Kadenza) ...

A Modern Data Architecture in Azure, presented by Armand van Oijen (Kadenza) ...

A Modern Data Architecture in Azure, presented by Armand van Oijen (Kadenza) ...

Perth Microsoft Data & Analytics User Group - Building Solutions with Azure D...

Perth Microsoft Data & Analytics User Group - Building Solutions with Azure D...

Perth Microsoft Data & Analytics User Group - Building Solutions with Azure D...

Learn to Use Databricks for the Full ML Lifecycle

Learn to Use Databricks for the Full ML Lifecycle

Learn to Use Databricks for the Full ML Lifecycle

Similar a PLNOG 3: Tomasz Mikołajczyk - Data scalability. Why you should care?

Analyzing petabytes of smartmeter data using Cloud Bigtable, Cloud Dataflow, ...

Analyzing petabytes of smartmeter data using Cloud Bigtable, Cloud Dataflow, ...

Analyzing petabytes of smartmeter data using Cloud Bigtable, Cloud Dataflow, ...

By 2020, 50% of all new software will process machine-generated data of some sort (Gartner). Historically, machine data use cases have required non-SQL data stores like Splunk, Elasticsearch, or InfluxDB. Today, new SQL DB architectures rival the non-SQL solutions in ease of use, scalability, cost, and performance. Please join this webinar for a detailed comparison of machine data management approaches.

Webinar: SQL for Machine Data?

Webinar: SQL for Machine Data?

Webinar: SQL for Machine Data?

ข้อมูล IoT จากอุปกรณ์มีปริมาณมาก และมีความไดนามิกต่อเนื่องตลอดเวลา มีหลายมิติ ทำอย่างไร จึงจะประมวผลได้เร็วและตอบสนองแบบทันทีทันได้ โดยใช้เครื่องมือ Streaming processing and bigdata analytics

IoT meets Big Data

IoT meets Big Data

IoT meets Big Data

ratthaslip ranokphanuwat

"Cindy Xing is a Principal Software Dev Lead at Microsoft, with over 15 years of working experience in building and delivering large scale software distributed systems. Her talk during the Data Science Conference will be focused on Edge Computing. Edge computing is the practice of processing data near the edge of your network, where the data is being generated, instead of in a centralized data-processing warehouse."

Digital transformation and AI @Edge

Digital transformation and AI @Edge

Digital transformation and AI @Edge

Institute of Contemporary Sciences

With all the hype around Cloud and SDN, business decision makers are finding themselves trying to navigate through many new concepts and consequently needing to change the way they have traditionally selected their IT infrastructure. Technologies are now becoming more integrated and it is more important than ever to help your business be agile enough to keep up with the demands of your users and your customers. Come hear from Lisa Guess to learn how organizations can embrace Cloud technologies such as automation, SDN and Orchestration platforms to help you build next-generation networks.

Lisa Guess - Embracing the Cloud

Lisa Guess - Embracing the Cloud

Lisa Guess - Embracing the Cloud

centralohioissa

The Evolution of Data Architecture

The Evolution of Data Architecture

The Evolution of Data Architecture

Wei-Chiu Chuang

Enterprises are rapidly adopting stream computing backbones, in-memory data stores, change data capture, and other low-latency approaches for end-to-end applications. As businesses modernize their data architectures over the next several years, they will begin to evolve toward all-streaming architectures. In this webcast, Wikibon, Attunity, and MemSQL will discuss how enterprise data professionals should migrate their legacy architectures in this direction. They will provide guidance for migrating data lakes, data warehouses, data governance, and transactional databases to support all-streaming architectures for complex cloud and edge applications. They will discuss how this new architecture will drive enterprise strategies for operationalizing artificial intelligence, mobile computing, the Internet of Things, and cloud-native microservices. Link to the Wikibon report - wikibon.com/wikibons-2018-big-data-analytics-trends-forecast Link to Attunity Streaming CDC Book Download - http://www.bit.ly/cdcbook Link to MemSQL's Free Data Pipeline Book - http://go.memsql.com/oreilly-data-pipelines

Digital Business Transformation in the Streaming Era

Digital Business Transformation in the Streaming Era

Digital Business Transformation in the Streaming Era

abiquo

Watch full webinar here: https://bit.ly/34iCruM Many organizations are embarking on strategically important journeys to embrace data and analytics. The goal can be to improve internal efficiencies, improve the customer experience, drive new business models and revenue streams, or – in the public sector – provide better services. All of these goals require empowering employees to act on data and analytics and to make data-driven decisions. However, getting data – the right data at the right time – to these employees is a huge challenge and traditional technologies and data architectures are simply not up to this task. This webinar will look at how organizations are using Data Virtualization to quickly and efficiently get data to the people that need it. Attend this session to learn: - The challenges organizations face when trying to get data to the business users in a timely manner - How Data Virtualization can accelerate time-to-value for an organization’s data assets - Examples of leading companies that used data virtualization to get the right data to the users at the right time

Bridging the Last Mile: Getting Data to the People Who Need It (APAC)

Bridging the Last Mile: Getting Data to the People Who Need It (APAC)

Bridging the Last Mile: Getting Data to the People Who Need It (APAC)

Choosing the Right Database: Exploring MySQL Alternatives for Modern Applications by Bhanu Jamwal, Head of Solution Engineering, PingCAP at the Mydbops Opensource Database Meetup 14. This presentation discusses the challenges in choosing the right database for modern applications, focusing on MySQL alternatives. It highlights the growth of new applications, the need to improve infrastructure, and the rise of cloud-native architecture. The presentation explores alternatives to MySQL, such as MySQL forks, database clustering, and distributed SQL. It introduces TiDB as a distributed SQL database for modern applications, highlighting its features and top use cases. Case studies of companies benefiting from TiDB are included. The presentation also outlines TiDB's product roadmap, detailing upcoming features and enhancements.

Choosing the Right Database: Exploring MySQL Alternatives for Modern Applicat...

Choosing the Right Database: Exploring MySQL Alternatives for Modern Applicat...

Choosing the Right Database: Exploring MySQL Alternatives for Modern Applicat...

Big Data and Semantic Web in Manufacturing

Big Data and Semantic Web in Manufacturing

Big Data and Semantic Web in Manufacturing

Nitesh Khilwani

Flash session -streaming--ses1243-lon

Flash session -streaming--ses1243-lon

Flash session -streaming--ses1243-lon

Jeffrey T. Pollock

Predictive Analytics: Why (I)IoT Is Different

Predictive Analytics: Why (I)IoT Is Different

Predictive Analytics: Why (I)IoT Is Different

I will share the vision and the production journey of how we build enterprise shared AI As A Service platforms with distributed deep learning technologies. Including those topics: 1) The vision of Enterprise Shared AI As A Service and typical AI services use cases at FinTech industry 2) The high level architecture design principles for AI As A Service 3) The technical evaluation journey to choose an enterprise deep learning framework with comparisons, such as why we choose Deep learning framework based on Spark ecosystem 4) Share some production AI use cases, such as how we implemented new Users-Items Propensity Models with deep learning algorithms with Spark,improve the quality , performance and accuracy of offer and campaigns design, targeting offer matching and linking etc. 5) Share some experiences and tips of using deep learning technologies on top of Spark , such as how we conduct Intel BigDL into a real production.

AI as a Service, Build Shared AI Service Platforms Based on Deep Learning Tec...

AI as a Service, Build Shared AI Service Platforms Based on Deep Learning Tec...

AI as a Service, Build Shared AI Service Platforms Based on Deep Learning Tec...

Watch full webinar here: https://bit.ly/35FUn32 Presented at CDAO New Zealand Advanced data science techniques, like machine learning, have proven an extremely useful tool to derive valuable insights from existing data. Platforms like Spark, and complex libraries for R, Python, and Scala put advanced techniques at the fingertips of the data scientists. However, most architecture laid out to enable data scientists miss two key challenges: - Data scientists spend most of their time looking for the right data and massaging it into a usable format - Results and algorithms created by data scientists often stay out of the reach of regular data analysts and business users Watch this session on-demand to understand how data virtualization offers an alternative to address these issues and can accelerate data acquisition and massaging. And a customer story on the use of Machine Learning with data virtualization.

Why Your Data Science Architecture Should Include a Data Virtualization Tool ...

Why Your Data Science Architecture Should Include a Data Virtualization Tool ...

Why Your Data Science Architecture Should Include a Data Virtualization Tool ...

Watch full webinar here: https://bit.ly/3cUA0Qi Many organizations are embarking on strategically important journeys to embrace data and analytics. The goal can be to improve internal efficiencies, improve the customer experience, drive new business models and revenue streams, or – in the public sector – provide better services. All of these goals require empowering employees to act on data and analytics and to make data-driven decisions. However, getting data – the right data at the right time – to these employees is a huge challenge and traditional technologies and data architectures are simply not up to this task. This webinar will look at how organizations are using Data Virtualization to quickly and efficiently get data to the people that need it. Attend this session to learn: - The challenges organizations face when trying to get data to the business users in a timely manner - How Data Virtualization can accelerate time-to-value for an organization’s data assets - Examples of leading companies that used data virtualization to get the right data to the users at the right time

Bridging the Last Mile: Getting Data to the People Who Need It

Bridging the Last Mile: Getting Data to the People Who Need It

Bridging the Last Mile: Getting Data to the People Who Need It

Learn how to leverage MongoDB and Big Data technologies to derive rich business insight and build high performance business intelligence platforms. This presentation includes: - Uncovering Opportunities with Big Data analytics - Challenges of real-time data processing - Best practices for performance optimization - Real world case study This presentation was given in partnership with CIGNEX Datamatics.

Webinar: Faster Big Data Analytics with MongoDB

Webinar: Faster Big Data Analytics with MongoDB

Webinar: Faster Big Data Analytics with MongoDB

Data integration and processing is a huge challenge in Industrial IoT (IIoT, aka Industry 4.0 or Automation Industry) due to monolithic systems and proprietary protocols. Apache Kafka, its ecosystem (Kafka Connect, KSQL) and Apache PLC4X are a great open source choice to implement this integration end to end in a scalable, reliable and flexible way. This blog post covers a high level overview about the challenges and a good, flexible architecture. At the end, I share a video recording and the corresponding slide deck. These provide many more details and insights. Apache Kafka is the De-facto Standard for Real-Time Event Streaming. It provides Open Source (Apache 2.0 License) Global-scale Real-time Persistent Storage Stream Processing PCL4X allows vertical integration and to write software independent of PLCs using JDBC-like adapters for various protocols like Siemens S7, Modbus, Allen Bradley, Beckhoff ADS, OPC-UA, Emerson, Profinet, BACnet, Ethernet. Github example: https://github.com/kaiwaehner/iiot-integration-apache-plc4x-kafka-connect-ksql-opc-ua-modbus-siemens-s7 More details: http://www.kai-waehner.de/blog/2019/09/02/iiot-data-integr…and-apache-plc4x/ ‎ Video Recording: https://youtu.be/RWKggid25ds

IIoT / Industry 4.0 with Apache Kafka, Connect, KSQL, Apache PLC4X

IIoT / Industry 4.0 with Apache Kafka, Connect, KSQL, Apache PLC4X

IIoT / Industry 4.0 with Apache Kafka, Connect, KSQL, Apache PLC4X

Flexible and Scalable Integration in the Automation Industry/Industrial IoT

Flexible and Scalable Integration in the Automation Industry/Industrial IoT

Flexible and Scalable Integration in the Automation Industry/Industrial IoT

"An introduction to Kx Technology - a Big Data solution", Kyra Coyne, Data Sc...

"An introduction to Kx Technology - a Big Data solution", Kyra Coyne, Data Sc...

"An introduction to Kx Technology - a Big Data solution", Kyra Coyne, Data Sc...

Similar a PLNOG 3: Tomasz Mikołajczyk - Data scalability. Why you should care? (20)

Analyzing petabytes of smartmeter data using Cloud Bigtable, Cloud Dataflow, ...

Analyzing petabytes of smartmeter data using Cloud Bigtable, Cloud Dataflow, ...

Analyzing petabytes of smartmeter data using Cloud Bigtable, Cloud Dataflow, ...

Webinar: SQL for Machine Data?

Webinar: SQL for Machine Data?

Webinar: SQL for Machine Data?

IoT meets Big Data

IoT meets Big Data

IoT meets Big Data

Digital transformation and AI @Edge

Digital transformation and AI @Edge

Digital transformation and AI @Edge

Lisa Guess - Embracing the Cloud

Lisa Guess - Embracing the Cloud

Lisa Guess - Embracing the Cloud

The Evolution of Data Architecture

The Evolution of Data Architecture

The Evolution of Data Architecture

Digital Business Transformation in the Streaming Era

Digital Business Transformation in the Streaming Era

Digital Business Transformation in the Streaming Era

abiquo

Bridging the Last Mile: Getting Data to the People Who Need It (APAC)

Bridging the Last Mile: Getting Data to the People Who Need It (APAC)

Bridging the Last Mile: Getting Data to the People Who Need It (APAC)

Choosing the Right Database: Exploring MySQL Alternatives for Modern Applicat...

Choosing the Right Database: Exploring MySQL Alternatives for Modern Applicat...

Choosing the Right Database: Exploring MySQL Alternatives for Modern Applicat...

Big Data and Semantic Web in Manufacturing

Big Data and Semantic Web in Manufacturing

Big Data and Semantic Web in Manufacturing

Flash session -streaming--ses1243-lon

Flash session -streaming--ses1243-lon

Flash session -streaming--ses1243-lon

Predictive Analytics: Why (I)IoT Is Different

Predictive Analytics: Why (I)IoT Is Different

Predictive Analytics: Why (I)IoT Is Different

AI as a Service, Build Shared AI Service Platforms Based on Deep Learning Tec...

AI as a Service, Build Shared AI Service Platforms Based on Deep Learning Tec...

AI as a Service, Build Shared AI Service Platforms Based on Deep Learning Tec...

Why Your Data Science Architecture Should Include a Data Virtualization Tool ...

Why Your Data Science Architecture Should Include a Data Virtualization Tool ...

Why Your Data Science Architecture Should Include a Data Virtualization Tool ...

Bridging the Last Mile: Getting Data to the People Who Need It

Bridging the Last Mile: Getting Data to the People Who Need It

Bridging the Last Mile: Getting Data to the People Who Need It

Webinar: Faster Big Data Analytics with MongoDB

Webinar: Faster Big Data Analytics with MongoDB

Webinar: Faster Big Data Analytics with MongoDB

IIoT / Industry 4.0 with Apache Kafka, Connect, KSQL, Apache PLC4X

IIoT / Industry 4.0 with Apache Kafka, Connect, KSQL, Apache PLC4X

IIoT / Industry 4.0 with Apache Kafka, Connect, KSQL, Apache PLC4X

Flexible and Scalable Integration in the Automation Industry/Industrial IoT

Flexible and Scalable Integration in the Automation Industry/Industrial IoT

Flexible and Scalable Integration in the Automation Industry/Industrial IoT

"An introduction to Kx Technology - a Big Data solution", Kyra Coyne, Data Sc...

"An introduction to Kx Technology - a Big Data solution", Kyra Coyne, Data Sc...

"An introduction to Kx Technology - a Big Data solution", Kyra Coyne, Data Sc...

Último

FWD Group - Insurer Innovation Award 2024

FWD Group - Insurer Innovation Award 2024

FWD Group - Insurer Innovation Award 2024

The Digital Insurer

Sidekick Solutions uses Bonterra Impact Management (fka Social Solutions Apricot) and automation solutions to integrate data for business workflows. We believe integration and automation are essential to user experience and the promise of efficient work through technology. Automation is the critical ingredient to realizing that full vision. We develop integration products and services for Bonterra Case Management software to support the deployment of automations for a variety of use cases. This video focuses on the deployment of external web forms using Jotform for Bonterra Impact Management. This solution can be customized to your organization’s needs and deployed to support the common use cases below: - Intake and consent - Assessments - Surveys - Applications - Program registration Interested in deploying web form automations for Bonterra Impact Management? Contact us at sales@sidekicksolutionsllc.com to discuss next steps.

Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...

Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...

Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...

Jeffrey Haguewood

DBX First Quarter 2024 Investor Presentation

DBX First Quarter 2024 Investor Presentation

DBX First Quarter 2024 Investor Presentation

EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER

EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER

EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER

In the thrilling conclusion to 2023, ransomware groups had a banner year, really outdoing themselves in the "make everyone's life miserable" department. LockBit 3.0 took gold in the hacking olympics, followed by the plucky upstarts Clop and ALPHV/BlackCat. Apparently, 48% of organizations were feeling left out and decided to get in on the cyber attack action. Business services won the "most likely to get digitally mugged" award, with education and retail nipping at their heels. Hackers expanded their repertoire beyond boring old encryption to the much more exciting world of extortion. The US, UK and Canada took top honors in the "countries most likely to pay up" category. Bitcoins were the currency of choice for discerning hackers, because who doesn't love untraceable money?

Ransomware_Q4_2023. The report. [EN].pdf

Ransomware_Q4_2023. The report. [EN].pdf

Ransomware_Q4_2023. The report. [EN].pdf

Overkill Security

Scalable LLM APIs for AI and Generative AI Application Development Ettikan Karuppiah, Director/Technologist - NVIDIA Apidays Singapore 2024: Connecting Customers, Business and Technology (April 17 & 18, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays Singapore 2024 - Scalable LLM APIs for AI and Generative AI Applicati...

Apidays Singapore 2024 - Scalable LLM APIs for AI and Generative AI Applicati...

Apidays Singapore 2024 - Scalable LLM APIs for AI and Generative AI Applicati...

How to Troubleshoot Apps for the Modern Connected Worker

How to Troubleshoot Apps for the Modern Connected Worker

How to Troubleshoot Apps for the Modern Connected Worker

GenAI Risks & Security Meetup 01052024.pdf

GenAI Risks & Security Meetup 01052024.pdf

GenAI Risks & Security Meetup 01052024.pdf

presentation ICT roal in 21st century education

presentation ICT roal in 21st century education

presentation ICT roal in 21st century education

AXA XL - Insurer Innovation Award Americas 2024

AXA XL - Insurer Innovation Award Americas 2024

AXA XL - Insurer Innovation Award Americas 2024

The Digital Insurer

In this session, we will delve into strategic approaches for optimizing knowledge management within Microsoft 365, amidst the evolving landscape of Copilot. From leveraging automatic metadata classification and permission governance with SharePoint Premium, to unlocking Viva Engage for the cultivation of knowledge and communities, you will gain actionable insights to bolster your organization's knowledge-sharing initiatives. In this session, we will also explore how to facilitate solutions to enable your employees to find answers and expertise within Microsoft 365. You will leave equipped with practical techniques and a deeper understanding of how there is more to effective knowledge management than just enabling Copilot, but building actual solutions to prepare the knowledge that Copilot and your employees can use.

Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...

Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...

Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...

Manulife - Insurer Transformation Award 2024

Manulife - Insurer Transformation Award 2024

Manulife - Insurer Transformation Award 2024

The Digital Insurer

Following the popularity of "Cloud Revolution: Exploring the New Wave of Serverless Spatial Data," we're thrilled to announce this much-anticipated encore webinar. In this sequel, we'll dive deeper into the Cloud-Native realm by uncovering practical applications and FME support for these new formats, including COGs, COPC, FlatGeoBuf, GeoParquet, STAC, and ZARR. Building on the foundation laid by industry leaders Michelle Roby of Radiant Earth and Chris Holmes of Planet in the first webinar, this second part offers an in-depth look at the real-world application and behind-the-scenes dynamics of these cutting-edge formats. We will spotlight specific use-cases and workflows, showcasing their efficiency and relevance in practical scenarios. Discover the vast possibilities each format holds, highlighted through detailed discussions and demonstrations. Our expert speakers will dissect the key aspects and provide critical takeaways for effective use, ensuring attendees leave with a thorough understanding of how to apply these formats in their own projects. Elevate your understanding of how FME supports these cutting-edge technologies, enhancing your ability to manage, share, and analyze spatial data. Whether you're building on knowledge from our initial session or are new to the serverless spatial data landscape, this webinar is your gateway to mastering cloud-native formats in your workflows.

Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME

Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME

Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME

The value of a flexible API Management solution for Open Banking Steve Melan, Manager for IT Innovation and Architecture - State's and Saving's Bank of Luxembourg Apidays New York 2024: The API Economy in the AI Era (April 30 & May 1, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays New York 2024 - The value of a flexible API Management solution for O...

Apidays New York 2024 - The value of a flexible API Management solution for O...

Apidays New York 2024 - The value of a flexible API Management solution for O...

Architecting Cloud Native Applications

Architecting Cloud Native Applications

Architecting Cloud Native Applications

Corporate and higher education. Two industries that, in the past, have had a clear divide with very little crossover. The difference in goals, learning styles and objectives paved the way for differing learning technologies platforms to evolve. Now, those stark lines are blurring as both sides are discovering they have content that’s relevant to the other. Join Tammy Rutherford as she walks through the pros and cons of corporate and higher ed collaborating. And the challenges of these different technology platforms working together for a brighter future.

Corporate and higher education May webinar.pptx

Corporate and higher education May webinar.pptx

Corporate and higher education May webinar.pptx

Rustici Software

Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood

Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood

Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood

Juan lago vázquez

Automating Google Workspace (GWS) & more with Apps Script

Automating Google Workspace (GWS) & more with Apps Script

Automating Google Workspace (GWS) & more with Apps Script

Strategies for Landing an Oracle DBA Job as a Fresher

Strategies for Landing an Oracle DBA Job as a Fresher

Strategies for Landing an Oracle DBA Job as a Fresher

Remote DBA Services

"I see eyes in my soup": How Delivery Hero implemented the safety system for ...

"I see eyes in my soup": How Delivery Hero implemented the safety system for ...

"I see eyes in my soup": How Delivery Hero implemented the safety system for ...

Último (20)

FWD Group - Insurer Innovation Award 2024

FWD Group - Insurer Innovation Award 2024

FWD Group - Insurer Innovation Award 2024

Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...

Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...

Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...

DBX First Quarter 2024 Investor Presentation

DBX First Quarter 2024 Investor Presentation

DBX First Quarter 2024 Investor Presentation

EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER

EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER

EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER

Ransomware_Q4_2023. The report. [EN].pdf

Ransomware_Q4_2023. The report. [EN].pdf

Ransomware_Q4_2023. The report. [EN].pdf

Apidays Singapore 2024 - Scalable LLM APIs for AI and Generative AI Applicati...

Apidays Singapore 2024 - Scalable LLM APIs for AI and Generative AI Applicati...

Apidays Singapore 2024 - Scalable LLM APIs for AI and Generative AI Applicati...

How to Troubleshoot Apps for the Modern Connected Worker

How to Troubleshoot Apps for the Modern Connected Worker

How to Troubleshoot Apps for the Modern Connected Worker

GenAI Risks & Security Meetup 01052024.pdf

GenAI Risks & Security Meetup 01052024.pdf

GenAI Risks & Security Meetup 01052024.pdf

presentation ICT roal in 21st century education

presentation ICT roal in 21st century education

presentation ICT roal in 21st century education

AXA XL - Insurer Innovation Award Americas 2024

AXA XL - Insurer Innovation Award Americas 2024

AXA XL - Insurer Innovation Award Americas 2024

Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...

Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...

Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...

Manulife - Insurer Transformation Award 2024

Manulife - Insurer Transformation Award 2024

Manulife - Insurer Transformation Award 2024

Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME

Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME

Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME

Apidays New York 2024 - The value of a flexible API Management solution for O...

Apidays New York 2024 - The value of a flexible API Management solution for O...

Apidays New York 2024 - The value of a flexible API Management solution for O...

Architecting Cloud Native Applications

Architecting Cloud Native Applications

Architecting Cloud Native Applications

Corporate and higher education May webinar.pptx

Corporate and higher education May webinar.pptx

Corporate and higher education May webinar.pptx

Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood

Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood

Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood

Automating Google Workspace (GWS) & more with Apps Script

Automating Google Workspace (GWS) & more with Apps Script

Automating Google Workspace (GWS) & more with Apps Script

Strategies for Landing an Oracle DBA Job as a Fresher

Strategies for Landing an Oracle DBA Job as a Fresher

Strategies for Landing an Oracle DBA Job as a Fresher

"I see eyes in my soup": How Delivery Hero implemented the safety system for ...

"I see eyes in my soup": How Delivery Hero implemented the safety system for ...

"I see eyes in my soup": How Delivery Hero implemented the safety system for ...

PLNOG 3: Tomasz Mikołajczyk - Data scalability. Why you should care?

1. Data scalability. Why you should care? Tomasz Mikolajczyk & the GridwiseTech team www.gridwisetech.com tomek@gridwisetech.com

2. GridwiseTech www.gridwisetech.com Presentation survey Presentation plan 1) About GridwiseTech 2) Building efficient database oriented, IT systems 3) Momentum + case study Presentation duration 40 minutes

3. GridwiseTech www.gridwisetech.com About GridwiseTech Leading vendor- independent scalable technology expert. Solve scalability and performance problems with virtualization, Grid, distributed data processing, caching. Implement vision for distributed infrastructures. Mission: turn avant-garde technology into customer’s business benefits About us

4. GridwiseTech www.gridwisetech.com Chosen customers Chosen partners Our references About us

5. GridwiseTech www.gridwisetech.com Building efficient IT system About us Application WWW SOA Middleware Processing Data provisioning Data persistence

6. GridwiseTech www.gridwisetech.com Vendors puzzle About us Application WWW SOA Platform Computing, Sun Microsystems, Altair, UnivaUD, Cluster Resources Gigaspaces, Gemstone, Coherence, eXludus, Terracotta Oracle, IBM, Microsoft PostreSQL, MySQL Citrix, Vmvare,Novell, Virtual Iron, RedHat

7. GridwiseTech www.gridwisetech.com Improving the system efficiency

8. GridwiseTech www.gridwisetech.com Simple data oriented IT system Internal or external data sources applications working on data

9. GridwiseTech www.gridwisetech.com IT systems are constantly growing Increased number of users Increased number of applications Increased amount of data

10. GridwiseTech www.gridwisetech.com IT systems are constantly growing Infrastructure bottleneck

11. GridwiseTech www.gridwisetech.com Hardware scale up – the first idea Typical solution: buy bigger and faster hardware

12. GridwiseTech www.gridwisetech.com Hardware scale up – the first idea Recurring software and hardware migrations will always generate substantial costs

13. GridwiseTech www.gridwisetech.com Introduction of scalability Simultaneous calculation Load balancing

14. GridwiseTech www.gridwisetech.com Introduction of scalability Simultaneous calculation Cloud Load balancing

15. GridwiseTech www.gridwisetech.com Introduction of distributed processing Cloud I/O bottleneck

16. GridwiseTech www.gridwisetech.com Problem with data access? Database / configuration / storage optimization In-memory databases Database clustering Sharding Data grids Distributed caching

17. GridwiseTech www.gridwisetech.com Fully scalable system Scaling out data

18. GridwiseTech www.gridwisetech.com Project for electronic manufacturer Technical scope of the problem: 2,5 million transactions per day many databases, different vendors and/or versions 4TB of data to maintain constant environment modifications 400,000 lines of PL/SQL code to analyze Improve report generation time by 70% allowing for increase in daily production. Project main goal Business problem: lack of possibility to increase production limited process manageability due to report availability delay Repetitive infrastructure down time resulting in data inconstancy Data normalization problem due to many different data sources

19. GridwiseTech www.gridwisetech.com Initial customer architecture 1 2

20. GridwiseTech www.gridwisetech.com Designed architecture Distributed caching

21. GridwiseTech www.gridwisetech.com Momentum

22. GridwiseTech www.gridwisetech.com Project for electronic manufacturer Results of scaling manufacturer infrastructure: Scalability ensured on each functional layer Real-time report generation 10x times data processing speed-up 100,000 of reads/writes per second (7500 before) Fully manageable production process Improve report generation time by 70% allowing for increase in daily production. Project main goal

23. GridwiseTech www.gridwisetech.com GridwiseTech's value add GridwiseTech's value add:  Application for automatic mapping database tables into objects.  Scalable Feeder application that provide fast data replication between multiple data sources and data grid.  Data normalization from multiple data sources.  Mechanism, that prevent from RAM overloading.  Mechanism, that provide synchronisation possibility between services and applications.  Technique of building thread enabled and fully scalable external services with batch processing support.

24. Thank you www.gridwisetech.com contact@gridwisetech.com