Lessons learned building a big data analytics engine, from proprietary to open source

•

0 recomendaciones•276 vistas

Lessons learned building a big data analytics engine, from proprietary to open source by Álvaro Santamaria & Joel Brunger After spending four years building a proprietary all-in-one streaming analytics engine for financial services, it became clear that open-source was starting to pull ahead. Alvaro will talk about the challenges of creating an IT operations solution for financial services; what to build, what not to build, and how to use open source tools to get past the infrastructure and focus on the business problems that matter.

Tecnología

Álvaro Santamaría
Data Scientist – ITRS
@dofideas
Joel Brunger
System Engineer - MapR
@joelbrunger
Lessons Learned with Visualisation
and Machine Learning for Big Data

Make real-time decisions
based on historical wisdom

Jay Krepps – 2013
“The Log: What every software engineer should know about real-time data's unifying abstraction”
The uber-system Lego-like, OS-based

1. Big-data viz is not a matter of “front-end”

3. Information extraction
GROUP BY country,
timestamp window of 10 minutes
SELECT count(),
average(temperature),
median(temperature),
max(temperature),
...
tdigest(temperature)

J on the Beach
All Data, One Platform, Every Cloud
Limitless Possibilities

What is MapR?
MapR is the industry’s leading data platform for AI and Analytics.

Rendezvous Architecture
MapR is the industry’s leading data platform for AI and Analytics.

Rendezvous Architecture
The Decoy is design to just collect data inputs

Rendezvous Architecture
Introducing the Canary

Why did ITRS choose MapR for ‘Gateway Hub’
MapR is the industry’s leading data platform for AI and Analytics.
Ø Simplicity (integrated platform)
Ø Real-time
Ø Processing must be performed in the cluster
Ø Enterprise features

MapR enables ITRS ‘Gateway Hub’ to provide the following benefits
MapR is the industry’s leading data platform for AI and Analytics.
Ø Smarter monitoring
Ø Additional features, application and Services
Ø Global Data Fabric
Ø Support ML in real-time

Más contenido relacionado

La actualidad más candente

The annual GPU Technology Conference focused on the promising field of deep learning in 2015. And we made four major announcements that will fuel its advancement: Titan X, the world's fastest GPU; DIGITS DevBox, GPU deep learning platform; Pascal GPU architecture; NVIDIA DRIVE PX, deep learning platform for self-driving cars. The press responded to these announcements with quotes, featured in this presentation, including ones from Mashable, Forbes, re/code, and The Wall Street Journal. The week-long event was shared in astounding numbers with many blog posts and streaming keynotes.

GTC 2015 Highlights

NVIDIA

From Data to Insights to Action: When Transactions and Analytics Converge

Ali Hodroj

10×6 Fintech: un nouveau pilier? Pascal Laffineur, Altran Benelux

Thierry Raizer

This paper proposes a framework for synthesizing infrastructure configurations for evaluating edge computing systems under different conditions. There are a number of tools to sim- ulate or emulate edge systems, and while they typically provide ways of modeling infrastructure and network topologies, they lack reusable building blocks common to edge scenarios. Consequently, most edge computing systems evaluations to this date rely on either highly application-specific testbeds, or abstract scenarios and abstract infrastructure configurations. We analyze four existing or emerging edge infrastructure scenarios, from which we elicit common concepts. The scenarios serve as input to synthesize plausible infrastructure configurations, that are parameterizable in cluster density, device heterogeneity, and network topology. We demonstrate how our tool can generate synthetic infrastructure configurations for the reference scenarios, and how these configurations can be used to evaluate aspects of edge computing systems.

Synthesizing Plausible Infrastructure Configurations for Evaluating Edge Comp...

Thomas Rausch

GIS - Asset Management - Sea Ports

Prashanth Kantheti

Big Data Analytics @ Munich Re - VIII. International Istanbul Insurance Confe...

SigortaTatbikatcilariDernegi

Future of data management

CTRLS

Jim Novack (Talent Swarm) Anand Gupta (Bigdatastrategy) Collaborative Work Environments (CWE) combined with Telepresence and Mixed Reality technologies offer new ways to improve the outcomes of engineering and building large construction, petrochemical, industrial, aeronautical and defense industry projects. This presentation will describe how in the near future, design, implementation and control processes in these projects will be performed more safely and accurately at lower cost by proposing a framework of existing, open and already adopted standards. Augmented World Expo (AWE) is back for its seventh year in our largest conference and expo featuring technologies giving us superpowers: augmented reality (AR), virtual reality (VR) and wearable tech. Join over 4,000 attendees from all over the world including a mix of CEOs, CTOs, designers, developers, creative agencies, futurists, analysts, investors, and top press in a fantastic opportunity to learn, inspire, partner, and experience first hand the most exciting industry of our times. See more at http://AugmentedWorldExpo.com

AR and Big Data: Interoperable Data Repositories for Collaborative Work Envir...

AugmentedWorldExpo

Digital Transformation; Digital Twins for Delivering Business Value in IIoT

The Hive

Data Science in the Enterprise

The Hive

Top 5 Data Science Sessions from GTC 2019

NVIDIA

Top 5 Deep Learning and AI Stories - September 14, 2018

NVIDIA

OpenPOWER partner presentation - GTS Data

Ganesan Narayanasamy

The Need for Speed: How the Auto Industry Accelerates Machine Learning with Visual Analytics by Aaron Williams, VP of Global Community, MapD Technologies, Inc. While GPU-accelerated analytics have already radically accelerated the speed of training machine learning models, data scientists and analysts still grapple with deriving insights from these complex models to better inform decision-making. The key: Visualizing and interrogating black box models with a GPU-enabled architecture. Volkswagen and MapD will discuss how interactive, visual analytics are helping the automotive brand interactively explore the output of their ML models to interrogate them in real time, for greater accuracy and reduced biases. They'll also examine how applying the GPU Data Frame to their efforts has enabled them to accelerate data science by minimizing data transfers and made it possible for their complex, multi-platform machine learning workflows to run entirely on GPUs.

Data Con LA 2018 - How the Auto Industry Accelerates ML with Analytics by Aa...

Data Con LA

How Vnomics built a "Digital Twin" for Commercial Trucking

Lloyd Palum

SPARK SUMMIT SESSION - A majority of the electricity in the U.S. is traded in independent system operator (ISO) based wholesale markets. ISO-based markets typically function in a two-step settlement process with day-ahead (DA) financial settlements followed by physical real-time (spot) market settlements for electricity. In this work, we focus on obtaining equilibrium bidding strategies for electricity generators in DA markets. Electricity prices in DA markets are determined by the ISO, which matches competing supply offers from power generators with demand bids from load serving entities. Since there are multiple generators competing with one another to supply power, this can be modeled as a competitive Markov decision problem, which we solve using a reinforcement learning approach. For power networks of realistic sizes, the state-action space could explode, making the RL procedure computationally intensive. This has motivated us to solve the above problem over Spark. The talk provides the following takeaways: 1. Modeling the day-ahead market as a Markov decision process 2. Code sketches to show the markov decision process solution over Spark and Mahout over Apache Tez 3. Performance results comparing Mahout over Apache Tez and Spark.

SPARK USE CASE- Distributed Reinforcement Learning for Electricity Market Bi...

Impetus Technologies

IMGS Presentation: AGI Belfast - Geo Big 5 - Open Geospatial - Transforming G...

IMGS

Expect More from Hadoop

MapR Technologies

NVIDIA Corporation Brochure: Who We Are

NVIDIA

Dsdt meetup-january2018

JDA Labs MTL

La actualidad más candente (20)

GTC 2015 Highlights

From Data to Insights to Action: When Transactions and Analytics Converge

10×6 Fintech: un nouveau pilier? Pascal Laffineur, Altran Benelux

Synthesizing Plausible Infrastructure Configurations for Evaluating Edge Comp...

GIS - Asset Management - Sea Ports

Big Data Analytics @ Munich Re - VIII. International Istanbul Insurance Confe...

Future of data management

AR and Big Data: Interoperable Data Repositories for Collaborative Work Envir...

Digital Transformation; Digital Twins for Delivering Business Value in IIoT

Data Science in the Enterprise

Top 5 Data Science Sessions from GTC 2019

Top 5 Deep Learning and AI Stories - September 14, 2018

OpenPOWER partner presentation - GTS Data

Data Con LA 2018 - How the Auto Industry Accelerates ML with Analytics by Aa...

How Vnomics built a "Digital Twin" for Commercial Trucking

SPARK USE CASE- Distributed Reinforcement Learning for Electricity Market Bi...

IMGS Presentation: AGI Belfast - Geo Big 5 - Open Geospatial - Transforming G...

Expect More from Hadoop

NVIDIA Corporation Brochure: Who We Are

Dsdt meetup-january2018

Similar a Lessons learned building a big data analytics engine, from proprietary to open source

Machine learning has made it possible for technologists to do amazing things with data. Its arrival coincides with the evolution of networked manufacturing systems driven by IoT. In this presentation we’ll examine the rise of IoT and ML from a practitioners perspective to better understand how applications of AI can be built in industrial settings. We'll walk through a case study that combines multiple IoT and ML technologies to monitor and optimize an industrial heating and cooling HVAC system. Through this instructive example you'll see how the following components can be put into action: 1. A StreamSets data pipeline that sources from MQTT and persists to OpenTSDB 2. A TensorFlow model that predicts anomalies in streaming sensor data 3. A Spark application that derives new event streams for real-time alerts 4. A Grafana dashboard that displays factory sensors and alerts in an interactive view By walking through this solution step-by-step, you'll learn how to build the fundamental capabilities needed in order to handle endless streams of IoT data and derive ML insights from that data: 1. How to transport IoT data through scalable publish/subscribe event streams 2. How to process data streams with transformations and filters 3. How to persist data streams with the timeliness required for interactive dashboards 4. How to collect labeled datasets for training machine learning models At the end of this presentation you will have learned how a variety of tools can be used together to build ML enhanced applications and data products for instrumented manufacturing systems. Speakers Ian Downard, Sr. Developer Evangelist, MapR William Ochandarena, Senior Director of Product Management, MapR

Designing data pipelines for analytics and machine learning in industrial set...

DataWorks Summit

Dell NVIDIA AI Roadshow - South Western Ontario

Bill Wong

FIWARE Global Summit - Advanced ML/AI Techniques with FIWARE and Connected Io...

FIWARE

Artificial Intelligence: Context of application of AI in Chemicals

accenture

SoftElegance Services: Data Science, Data Engineering, Big Data Architecture

Daryna Dubitska

Platform for Big Data Analytics and Visual Analytics: CSIRO use cases. Februa...

Tomasz Bednarz

How to design ai functions to the cloud native infra

Chun Myung Kyu

Industrial IoT and the emergence of Edge Computing Navigating the Technologic...

Roberto Siagri

Azure and Predix

Altoros

SAP Leonardo

Mahmoud Farag

Enterprises in every industry are gearing up for AI’s integration with IoT at the edge. Analytics and cloud-based applications are crucial foundations for the AIoT infrastructure. But even more importantly, AIoT requires complete, real-time access to the data in fulfill the needs of highly responsive edge computing applications. In our experience, many customers are facing the same difficulties with regards to cyber level and physical level device integration in the new AI era. As the world's leading industrial storage and memory provider, Innodisk has a solid track record with more than 2000 customers, and expertise built on more than a decade of integration of hardware, firmware and software solutions. Attend this webinar to learn about: - Preparing your business for the new Internet of Things (IoT) an AI era - How do we Overcome the Current Architectural Issues? - Increasing process efficiency and delivering a better customer experience - Facilitating new platforms that enable rapid development of next generation intelligent IoT systems - Trends and technology in AIoT intelligent storage/ data optimization

Overcoming the AIoT Obstacles through Smart Component Integration

Innodisk Corporation

Arpan pal roboticsensing_sw2015

Arpan Pal

Dell AI and HPC University Roadshow

Bill Wong

Dell AI Oil and Gas Webinar

Bill Wong

AIoT: Intelligence on Microcontroller

Andri Yadi

Teradata and Cisco integrated journey to IoT and Smart city

Artur Borycki

Introduction to IIOT Course & Benefits: This is an Experiential Learning Program in the Industrial Internet of Things Area. This course covers below topics: - Fundamentals of Electronics for IOT Engineering - Using Arduino & RaspberryPi_Programming for IOT - Manufacturing PCB & Component Assembly of IOT Devices - Embedded Systems Coding & Firmware Injection - Fundamentals of Python Programming - Fundamentals of SQLite & MySQL - Advances IIOT Programming using Python - Data Storage & Accessing Mechanism Using MySQL & Cloud - Data Sciences - Deep Learning with TensorFlow for IIOT - Data Sciences - Machine Learning using IBM Watson for IIOT - Data Visualization - Tableau for IOT

InfoRepos Academy Introduction v1.1 - IIOT Experiential Learning Program

InfoRepos Technologies

Watch full webinar here: https://bit.ly/2Y0vudM What is Data Virtualization and why do I care? In this webinar we intend to help you understand not only what Data Virtualization is but why it's a critical component of any organization's data fabric and how it fits. How data virtualization liberates and empowers your business users via data discovery, data wrangling to generation of reusable reporting objects and data services. Digital transformation demands that we empower all consumers of data within the organization, it also demands agility too. Data Virtualization gives you meaningful access to information that can be shared by a myriad of consumers. Register to attend this session to learn: - What is Data Virtualization? - Why do I need Data Virtualization in my organization? - How do I implement Data Virtualization in my enterprise?

Data Virtualization: An Introduction

Denodo

The acquisition of labeled, unbiased, high quality remote sensing information for training AI systems is expensive, error prone, and sometimes impossible or dangerous. The efficacy of Remote Sensing and Imagery Analysis tools that use AI depends directly on the data used for training and validation, meaning that the cost and availability of data limits the application of AI for imagery exploitation. Synthetic Computer Vision (CV) data has become a strategy to reduce the cost and limitations of using real-world data in detection problems in data sparse domains. Focusing on remote sensing data including visible and invisible electromagnetic spectra, attendees will learn about the expanding options for generating synthetic data that are being used in commercial and academic domains, the technology options available for users who want to create CV content of a variety of types, and patterns of creating synthetic data to support Learning Objectives - Describe synthetic data including different types such as Generative AI and physics-based data - Identify the opportunities for applying synthetic data in place of real sensor data Will be able to describe the steps required to generate synthetic data for computer vision workflows from concept to production for training and validating AI. - The intent of this class is to introduce the concepts and mechanisms behind the creation of synthetic data and to expose students to approaches for generating synthetic data using tools currently on the market. - Familiarity with concepts around AI training and validation using remotely sensed data will be helpful for attendees.

2023 GEOINT Tutorial - Synthetic Data Tools for Computer Vision-Based AI - Re...

Chris Andrews

In this workshop, learn how you can integrate Xilinx edge machine learning with massive scale AWS Cloud analytics, machine learning model building, and dashboards. Based on an Oil & Gas scenario, you will learn how to combine AWS Cloud services with AWS Greengrass on Zynq Ultrascale+ and Amazon FreeRTOS on Xilinx Zynq-7000. After this workshop, you will have a concrete understanding of Machine Learning applicability at the edge and its relationship with the AWS Cloud.

Integrate the AWS Cloud with Responsive Xilinx Machine Learning at the Edge (...

Amazon Web Services

Similar a Lessons learned building a big data analytics engine, from proprietary to open source (20)

Designing data pipelines for analytics and machine learning in industrial set...

Dell NVIDIA AI Roadshow - South Western Ontario

FIWARE Global Summit - Advanced ML/AI Techniques with FIWARE and Connected Io...

Artificial Intelligence: Context of application of AI in Chemicals

SoftElegance Services: Data Science, Data Engineering, Big Data Architecture

Platform for Big Data Analytics and Visual Analytics: CSIRO use cases. Februa...

How to design ai functions to the cloud native infra

Industrial IoT and the emergence of Edge Computing Navigating the Technologic...

Azure and Predix

SAP Leonardo

Overcoming the AIoT Obstacles through Smart Component Integration

Arpan pal roboticsensing_sw2015

Dell AI and HPC University Roadshow

Dell AI Oil and Gas Webinar

AIoT: Intelligence on Microcontroller

Teradata and Cisco integrated journey to IoT and Smart city

InfoRepos Academy Introduction v1.1 - IIOT Experiential Learning Program

Data Virtualization: An Introduction

2023 GEOINT Tutorial - Synthetic Data Tools for Computer Vision-Based AI - Re...

Integrate the AWS Cloud with Responsive Xilinx Machine Learning at the Edge (...

Más de J On The Beach

Big Data examples always give the correct answers. However, in the real world, Big Data might be corrupt, contradictory or consist of so many small files it becomes extremely hard to keep track - let alone scale. A solid architecture will help to overcome many of the difficulties. Floris will talk about a real-world implementation of a massively scalable ETL architecture. Two years ago, at the time of the implementation, Airflow just became part of Apache and still left many features to be desired for. However, requirements from the start were thousands of ETL tasks per day on average, but on occasion, this could become hundreds of thousands. The script-based method that was in place was already not capable to meet the requirements on a day to day basis and needed to be replaced as soon as possible. So this custom framework was rolled out in just 8 weeks of development time.

Massively scalable ETL in real world applications: the hard way

J On The Beach

Traditional Big Data is done on Data you have. You load the data into a repository and perform map reduce or other style calculations on the data. However, certain industries need to perform complex operations on data you might not have. Data you can acquire, Data that can be shared with you, and Data that you can model are all types of data you may not have but may need to integrate instantly into a complex data analysis. Problem is: you may not even know you need this data until deep into the execution stack at runtime. This talk discusses a new functional language paradigm for dealing naturally with data you don’t have and about how to make all data first-class citizens, regardless of whether you have it or you don’t, and we will give a demo of a project written in Scala to deal exactly with this issue.

Big Data On Data You Don’t Have

J On The Beach

Industry 4.0, aka the "Fourth Industrial Revolution," refers to the computerization of manufacturing. One important aspect of Industry 4.0 is the ability to monitor the health and reliability of a physical manufacturing plant using low-cost IoT sensors. For example, machine learning models can be trained to predict the physical degradation of a manufacturing system as a function of acoustic measurements obtained from strategically placed microphones; however, the same acoustic measurements can be used to reverse engineer proprietary information about the manufacturing process and/or precisely what is being manufactured at the time of recording. Thus, improved reliability and fault tolerance is achieved at the cost of what appears to be an unprecedented new class of security vulnerabilities related to the acoustic side channel. As a case study, we report a novel acoustic side channel attack against a commercial DNA synthesizer, a commonly used instrument in fields such as synthetic biology. Using a smart phone-quality microphone placed on or in the near vicinity of a DNA synthesizer, we were able to determine with 88.07% accuracy the sequence of DNA being produced; using a database of biologically relevant known-sequences, we increased the accuracy of our model to 100%. An academic or industrial research project may use the synthetic DNA to engineer an organism with desired traits or functions; however, while the organism is still under development, prior to publication, patent, and/or copyright, the research remains vulnerable to academic intellectual property theft and/or industrial espionage. On the other hand, this attack could also be used for benevolent purposes, for example, to determine whether a suspected criminal or terrorist is engineering a harmful pathogen. Thus, it is essential to recognize both the benefits and risks inherent to the cyber-physical systems that will inevitably control Industry 4.0 manufacturing processes and to take steps to mitigate them whenever possible.

Acoustic Time Series in Industry 4.0: Improved Reliability and Cyber-Security...

J On The Beach

Where is the edge in IoT and how much can you do there? Data collection? Analytics? I’ll show you how to build and deploy an embedded IoT edge platform that can do data collection, analytics, dashboarding and much more. All using Open Source. As IoT deployments move forward, the need to collect, analyze, and respond to data further out on the edge becomes a critical factor in the success – or failure – of any IoT project. Network bandwidth costs may be dropping, and storage is cheaper than ever, but at IoT scale, these costs can still quickly overrun a project’s budget and ultimately doom it to failure. The more you centralize your data collection and storage, the higher these costs become. Edge data collection and analysis can dramatically lower these costs, plus decrease the time to react to critical sensor data. With most data platforms, it simply isn’t practical, or even possible, to push collection AND analytics to the edge. In this talk I’ll show how I’ve done exactly this with a combination of open source hardware – Pine64 – and open source software – InfluxDB – to build a practical, efficient and scalable data collection and analysis gateway device for IoT deployments. The edge is where the data is, so the edge is where the data collection and analytics needs to be.

Pushing it to the edge in IoT

J On The Beach

Event Stream Processing is a popular paradigm for building robust and performant systems in many different domains, from IoT to fraud detection to high-frequency trading. Because of the wide range of scenarios and requirements, it is difficult to conceptualize a unified programming model that would be equally applicable to all of them. Another tough challenge is how to build streaming systems with cardinalities of topics ranging from hundreds to billions while delivering good performance and scalability. In this session, Sergey Bykov will talk about the journey of building Orleans Streams that originated in gaming and monitoring scenarios, and quickly expanded beyond them. He will cover the programming model of virtual streams that emerged as a natural extension of the virtual actor model of Orleans, the architecture of the underlying runtime system, the compromises and hard choices made in the process. Sergey will share the lessons learned from the experience of running the system in production, and future ideas and opportunities that remain to be explored.

Drinking from the firehose, with virtual streams and virtual actors

J On The Beach

How do we deploy? From Punched cards to Immutable server pattern

J On The Beach

Over the last twenty years, there has been a paradigm shift in software development: from meticulously planned release cycles to an experimental way of working in which lead times are becoming shorter and shorter. How can Java ever keep up with this trend when we have Docker containers that are several hundred megabytes in size, with warm-up times of ten minutes or longer? In this talk, I'll demonstrate how we can use Quarkus so that we can create super small, super fast Java containers! This will give us better possibilities for scaling up and down - which can be a game-changer, especially in a serverless environment. It will also provide the shortest possible lead times, as well as a much better use of cloud performance with the added bonus of lower costs.

Java, Turbocharged

J On The Beach

When Cloud Native meets the Financial Sector

J On The Beach

The advancement of technology in the last decade or so has allowed astronomy to see exponential growth in data volumes. ESA's space telescope Euclid will gather high-resolution images of a third of the sky, ~850GB of data downloaded daily for 6 years, by 2032 ground-based telescope LSST will have generated 500PB of data and the radio telescope SKA will be producing more data per second than the entire internet worldwide. This talk will address the questions of what current techniques exist to address big data volumes, how the astronomical community will prepare for this big data wave, and what other challenges lie ahead?

The big data Universe. Literally.

J On The Beach

The world is moving from a model where data sits at rest, waiting for people to make requests of it, to where data is constantly moving and streams of data flow to and from devices with or without human interaction. Decisions need to be made based on these streams of data in real-time, models need to be updated, and intelligence needs to be gathered. In this context, our old-fashioned approach of CRUD REST APIs serving CRUD database calls just doesn't cut it. It's time we moved to a stream-centric view of the world.

Streaming to a New Jakarta EE

J On The Beach

Our increasingly connected world leveraging the Internet of Things (IoT) creates great value, in connected healthcare, smart cities, and more. The increasing use of IoT also creates great risk. We will discuss the challenges and risks we need to address as developers in TIPPSS - Trust, Identity, Privacy, Protection, Safety, and Security - for devices, systems and solutions we deliver and use. Florence leads IEEE workstreams on clinical IoT and data interoperability with blockchain addressing TIPPSS issues. She is an author of IEEE articles on "Enabling Trust and Security - TIPPSS for IoT" and "Wearables and Medical Interoperability - the Evolving Frontier", "TIPPSS for Smart Cities" in the 2017 book "Creating, Analysing and Sustaining Smarter Cities: A Systems Perspective" , and Editor in Chief for an upcoming book on "Women Securing the Future with TIPPSS for IoT."

The TIPPSS Imperative for IoT - Ensuring Trust, Identity, Privacy, Protection...

J On The Beach

Want to run your AI algorithms directly in the browser on the client-side? Now you can with WebAssembly and Blazor. Join us as we write code directly in WebAssembly. Then, we’ll look at Blazor and how you can use it, along with WebAssembly to run your tooling client side in the browser. Want to run your AI algorithms directly in the browser on the client-side without the need for transpilers or browser plug-ins? Well, now you can with WebAssembly and Blazor. WebAssembly (WASM) is the W3C specification that will be used to provide the next generation of development tools for the web and beyond. Blazor is Microsoft’s experiment that allows ASP.Net developers to create web pages that do much of the scripting work in C# using WASM. Come join us as we learn to write code directly in WebAssembly’s human-readable format. Then, we’ll look at the current state of Blazor and how you can use it, along with WebAssembly to run your tooling client side in the browser.

Pushing AI to the Client with WebAssembly and Blazor

J On The Beach

RAFT protocol is a well-known protocol for consensus in Distributed Systems. Want to learn how consensus is achieved in a system with a large amount of data such as Axon Server’s Event Store? Join this talk to hear about all specifics regarding data replication in highly available Event Store! Axon is a free and open source Java framework for writing Java applications following DDD, event sourcing, and CQRS principles. While especially useful in a microservices context, Axon provides great value in building structured monoliths that can be broken down into microservices when needed. Axon Server is a messaging platform specifically built to support distributed Axon applications. One of its key benefits is storing events published by Axon applications. In not so rare cases, the number of these events is over millions, even billions. Availability of Axon Server plays a significant role in the product portfolio. To keep event replication reliable we chose RAFT protocol for consensus implementation of our clustering features. In short, consensus involves multiple servers agreeing on values. Once they reach a decision on a value, that decision is final. Typical consensus algorithms make progress when any majority of their servers is available; for example, a cluster of 5 servers can continue to operate even if 2 servers fail. If more servers fail, they stop making progress (but will never return an incorrect result). Join this talk to learn why we chose RAFT; what were our findings during the design, the implementation, and testing phase; and what does it mean to replicate an event store holding billions of events!

Axon Server went RAFTing

J On The Beach

Thinking of moving to Microservices? Watch out! That quest is full of traps, social traps. If you are not able to handle it, you may be blocked by meetings, frustration, endless challenges that will make you miss the monolith. In this talk, I share my experience and mistakes, so you can avoid them. Creating or migrating to a Microservices architecture might easily become a big mess, not only due to technical challenges but mostly because of human factors: it’s a major change in the software culture of a company. In this talk, I’ll share my past experience as the technical lead of an ambitious Microservices-based product, I’ll go through the parts we struggled with, and give you some advice on how to deal with what I call the Six Pitfalls: The Common Patterns Phobia The Book Club Cult The Never-Decoupled Story The Buzz Words Syndrome The Agile Trap The Conway’s Law Hackers

The Six Pitfalls of building a Microservices Architecture (and how to avoid t...

J On The Beach

Instead of randomly injecting faults ( i.e. Chaos Monkey), what if we could order our experiments to perform min number of experiments for maximum yield? We present a solution(& results) to the problem of experiment selection using Lineage Driven Fault Injection to reduce the search space of faults. Lineage Driven Fault Injection (LDFI) is a state of the art technique in chaos engineering experiment selection. LDFI since its inception has used an SAT solver under the hood which presents solutions to the decision problem (which faults to inject) in no particular order. As SRE’s we would like to perform experiments that reveal the bugs that the customers are most likely to hit first. In this talk, we present new improvements to LDFI that orders the experiment suggestions. In the first the half of the talk we will show LDFI is a technique that can be widely used within an enterprise. We present the motivation for ordering the chaos experiments along with some prioritization we utilized while conducting the experiments. We also highlight how ordering is a general purpose technique that we can use to encode the peculiarities of a heterogeneous microservices architecture. LDFI can work in an enterprise by harnessing the observability infrastructure to model the redundancy of the system. Next, we present experiments conducted within our organization using ordered LDFI and some preliminary results. We show examples of services where we discovered bugs, and how carefully controlling the order of experiments allowed LDFI to avoid running unnecessary experiments. We also present an example of an application where we declared the service shippable under crash stop model. We also present a comparison with Chaos Monkey and show how LDFI found the known bugs in a given application using orders of magnitude fewer experiments than a random fault injection tool like Chaos Monkey. Finally, we discuss how we plan to take LDFI forward. We discuss open problems and possible solutions for scalarizing probabilities of failure, latency injection, integration with service mesh technologies like envoy for fine-grained fault injection, fault injection for stateful systems. Key takeaways: 1) Understand how LDFI can be integrated in the enterprise by harnessing the observability infrastructure. 2) Limitations of LDFI w.r.t unordered solutions and why ordering matters for chaos engineering experiments. 3) Preliminary results of prioritized LDFI and a future direction for the community.

Madaari : Ordering For The Monkeys

J On The Beach

Servers are doomed to fail

J On The Beach

Distributed systems collaborate to achieve collective goals via a system of rules. Rules that affords good hygiene, fault tolerance, effective communication and trusted feedback. These rules form protocols which enable the system to achieve its goals. Distributed and concurrent systems can be considered a social group that collaborates to achieve collective goals. In order to collaborate a system of rules must be applied, that affords good hygiene, fault tolerance, and effective communication to coordinate, share knowledge, and provide feedback in a polite trusted manner. These rules form a number of protocols which enable the group to act as a system which is greater than the sum of the individual components. In this talk, we will explore the history of protocols and their application when building distributed systems.

Interaction Protocols: It's all about good manners

J On The Beach

Do you want to check the efficiency of the new, state of the art, GraalVM JIT Compiler in comparison to the old but mostly used JIT C2? Let’s have a side by side comparison from a performance standpoint on the same source code. The talk reveals how traditional Just In Time Compiler (e.g. JIT C2) from HotSpot/OpenJDK internally manages runtime optimizations for hot methods in comparison to the new, state of the art, GraalVM JIT Compiler on the same source code, emphasizing all of the internals and strategies used by each Compiler to achieve better performance in most common situations (or code patterns). For each optimization, there is Java source code and corresponding generated assembly code in order to prove what really happens under the hood. Each test is covered by a dedicated benchmark (JMH), timings and conclusions. Main topics of the agenda: - Scalar replacement - Null Checks - Virtual calls - Lock coarsening - Lock elision - Virtual calls - Scalar replacement - Lambdas - Vectorization (few cases) The tools used during my research study are JITWatch, Java Measurement Harness, and perf. All test scenarios will be launched against the latest official Java release (e.g. version 11).

A race of two compilers: GraalVM JIT versus HotSpot JIT C2. Which one offers ...

J On The Beach

Leadership is easy when you're a manager, or an expert in a field, or a conference speaker! In a Kanban organisation, though, we "encourage acts of leadership at every level". In this talk, we look at what it means to be a leader in the uncertain, changing and high-learning environment of software development. We learn about the importance of safety in encouraging others to lead and follow, and how to get that safety using both technical and human practices; the necessity of a clear, compelling vision and provision of information on how we're achieving it; and the need to be able to ask awkward and difficult questions... especially the ones without easy answers.

Leadership at every level

J On The Beach

During this presentation, we will answer how much you’ll need to invest in a superhero costume to be as popular as Superman. We will generate a unique logo which will stand against the ever popular Batman and create new superhero teams. We shall achieve it using linear regression and neural networks. Machine learning is one of the hottest buzzwords in technology today as well as one of the most innovative fields in computer science – yet people use libraries as black boxes without basic knowledge of the field. In this session, we will strip them to bare math, so next time you use a machine learning library, you’ll have a deeper understanding of what lies underneath. During this session, we will first provide a short history of machine learning and an overview of two basic teaching techniques: supervised and unsupervised learning. We will start by defining what machine learning is and equip you with an intuition of how it works. We will then explain the gradient descent algorithm with the use of simple linear regression to give you an even deeper understanding of this learning method. Then we will project it to supervised neural networks training. Within unsupervised learning, you will become familiar with Hebb’s learning and learning with concurrency (winner takes all and winner takes most algorithms). We will use Octave for examples in this session; however, you can use your favourite technology to implement presented ideas. Our aim is to show the mathematical basics of neural networks for those who want to start using machine learning in their day-to-day work or use it already but find it difficult to understand the underlying processes. After viewing our presentation, you should find it easier to select parameters for your networks and feel more confident in your selection of network type, as well as be encouraged to dive into more complex and powerful deep learning methods.

Machine Learning: The Bare Math Behind Libraries

J On The Beach

Más de J On The Beach (20)

Massively scalable ETL in real world applications: the hard way

Big Data On Data You Don’t Have

Acoustic Time Series in Industry 4.0: Improved Reliability and Cyber-Security...

Pushing it to the edge in IoT

Drinking from the firehose, with virtual streams and virtual actors

How do we deploy? From Punched cards to Immutable server pattern

Java, Turbocharged

When Cloud Native meets the Financial Sector

The big data Universe. Literally.

Streaming to a New Jakarta EE

The TIPPSS Imperative for IoT - Ensuring Trust, Identity, Privacy, Protection...

Pushing AI to the Client with WebAssembly and Blazor

Axon Server went RAFTing

The Six Pitfalls of building a Microservices Architecture (and how to avoid t...

Madaari : Ordering For The Monkeys

Servers are doomed to fail

Interaction Protocols: It's all about good manners

A race of two compilers: GraalVM JIT versus HotSpot JIT C2. Which one offers ...

Leadership at every level

Machine Learning: The Bare Math Behind Libraries

Último

This presentations targets students or working professionals. You may know Google for search, YouTube, Android, Chrome, and Gmail, but did you know Google has many developer tools, platforms & APIs? This comprehensive yet still high-level overview outlines the most impactful tools for where to run your code, store & analyze your data. It will also inspire you as to what's possible. This talk is 50 minutes in length.

Powerful Google developer tools for immediate impact! (2023-24 C)

wesley chun

Three things you will take away from the session: • How to run an effective tenant-to-tenant migration • Best practices for before, during, and after migration • Tips for using migration as a springboard to prepare for Copilot in Microsoft 365 Main ideas: Migration Overview: The presentation covers the current reality of cross-tenant migrations, the triggers, phases, best practices, and benefits of a successful tenant migration Considerations: When considering a migration, it is important to consider the migration scope, performance, customization, flexibility, user-friendly interface, automation, monitoring, support, training, scalability, data integrity, data security, cost, and licensing structure Next Wave: The next wave of change includes the launch of Copilot, which requires businesses to be prepared for upcoming changes related to Copilot and the cloud, and to consolidate data and tighten governance ShareGate: ShareGate can help with pre-migration analysis, configurable migration tool, and automated, end-user driven collaborative governance

Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff

sammart93

Webinar Recording: https://www.panagenda.com/webinars/why-teams-call-analytics-is-critical-to-your-entire-business Nothing is as frustrating and noticeable as being in an important call and being unable to see or hear the other person. Not surprising then, that issues with Teams calls are among the most common problems users call their helpdesk for. Having in depth insight into everything relevant going on at the user’s device, local network, ISP and Microsoft itself during the call is crucial for good Microsoft Teams Call quality support. To ensure a quick and adequate solution and to ensure your users get the most out of their Microsoft 365. But did you know that ‘bad calls’ are also an excellent indicator of other problems arising? Precisely because it is so noticeable!? Like the canary in the mine, bad calls can be early indicators of problems. Problems that might otherwise not have been noticed for a while but can have a big impact on productivity and satisfaction. Join this session by Christoph Adler to learn how true Microsoft Teams call quality analytics helped other organizations troubleshoot bad calls and identify and fix problems that impacted Teams calls or the use of Microsoft365 in general. See what it can do to keep your users happy and productive! In this session we will cover - Why CQD data alone is not enough to troubleshoot call problems - The importance of attributing call problems to the right call participant - What call quality analytics can do to help you quickly find, fix-, and prevent problems - Why having retrospective detailed insights matters - Real life examples of how others have used Microsoft Teams call quality monitoring to problem shoot problems with their ISP, network, device health and more.

Why Teams call analytics are critical to your entire business

panagenda

2024: Domino Containers - The Next Step. News from the Domino Container commu...

Martijn de Jong

The 7 Things I Know About Cyber Security After 25 Years | April 2024

Rafal Los

A Principled Technologies deployment guide Conclusion Deploying VMware Cloud Foundation 5.1 on next gen Dell PowerEdge servers brings together critical virtualization capabilities and high-performing hardware infrastructure. Relying on our hands-on experience, this deployment guide offers a comprehensive roadmap that can guide your organization through the seamless integration of advanced VMware cloud solutions with the performance and reliability of Dell PowerEdge servers. In addition to the deployment efficiency, the Cloud Foundation 5.1 and PowerEdge solution delivered strong performance while running a MySQL database workload. By leveraging VMware Cloud Foundation 5.1 and PowerEdge servers, you could help your organization embrace cloud computing with confidence, potentially unlocking a new level of agility, scalability, and efficiency in your data center operations.

Deploy with confidence: VMware Cloud Foundation 5.1 on next gen Dell PowerEdg...

Principled Technologies

Real Time Object Detection Using Open CV

Khem

Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...

Neo4j

Scaling API-first – The story of a global engineering organization Ian Reasor, Senior Computer Scientist - Adobe Radu Cotescu, Senior Computer Scientist - Adobe Apidays New York 2024: The API Economy in the AI Era (April 30 & May 1, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe

apidays

Manulife - Insurer Innovation Award 2024

The Digital Insurer

Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024

The Digital Insurer

Strategies for Landing an Oracle DBA Job as a Fresher

Remote DBA Services

With more memory available, system performance of three Dell devices increased, which can translate to a better user experience Conclusion When your system has plenty of RAM to meet your needs, you can efficiently access the applications and data you need to finish projects and to-do lists without sacrificing time and focus. Our test results show that with more memory available, three Dell PCs delivered better performance and took less time to complete the Procyon Office Productivity benchmark. These advantages translate to users being able to complete workflows more quickly and multitask more easily. Whether you need the mobility of the Latitude 5440, the creative capabilities of the Precision 3470, or the high performance of the OptiPlex Tower Plus 7010, configuring your system with more RAM can help keep processes running smoothly, enabling you to do more without compromising performance.

Boost PC performance: How more available memory can improve productivity

Principled Technologies

Data Cloud, More than a CDP by Matt Robison

Anna Loughnan Colquhoun

Tata AIG General Insurance Company - Insurer Innovation Award 2024

The Digital Insurer

Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving

Edi Saputra

Discord is a free app offering voice, video, and text chat functionalities, primarily catering to the gaming community. It serves as a hub for users to create and join servers tailored to their interests. Discord’s ecosystem comprises servers, each functioning as a distinct online community with its own channels dedicated to specific topics or activities. Users can engage in text-based discussions, voice calls, or video chats within these channels. Understanding Discord Servers Discord servers are virtual spaces where users congregate to interact, share content, and build communities. Servers may revolve around gaming, hobbies, interests, or fandoms, providing a platform for like-minded individuals to connect. Communication Features Discord offers a range of communication tools, including text channels for messaging, voice channels for real-time audio conversations, and video channels for face-to-face interactions. These features facilitate seamless communication and collaboration. What Does NSFW Mean? The acronym NSFW stands for “Not Safe For Work,” indicating content that may be inappropriate for professional or public settings. NSFW Content NSFW content encompasses material that is sexually explicit, violent, or otherwise graphic in nature. It often includes nudity, profanity, or depictions of sensitive topics.

Understanding Discord NSFW Servers A Guide for Responsible Users.pdf

UK Journal

Artificial Intelligence Chap.5 : Uncertainty

Khushali Kathiriya

Effective data discovery is crucial for maintaining compliance and mitigating risks in today's rapidly evolving privacy landscape. However, traditional manual approaches often struggle to keep pace with the growing volume and complexity of data. Join us for an insightful webinar where industry leaders from TrustArc and Privya will share their expertise on leveraging AI-powered solutions to revolutionize data discovery. You'll learn how to: - Effortlessly maintain a comprehensive, up-to-date data inventory - Harness code scanning insights to gain complete visibility into data flows leveraging the advantages of code scanning over DB scanning - Simplify compliance by leveraging Privya's integration with TrustArc - Implement proven strategies to mitigate third-party risks Our panel of experts will discuss real-world case studies and share practical strategies for overcoming common data discovery challenges. They'll also explore the latest trends and innovations in AI-driven data management, and how these technologies can help organizations stay ahead of the curve in an ever-changing privacy landscape.

TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery

TrustArc

The presentation explores the development and application of artificial intelligence (AI) from its inception to its current status in the modern world. The term "artificial intelligence" was first coined by John McCarthy in 1956 to describe efforts to develop computer programs capable of performing tasks that typically require human intelligence. This concept was first introduced at a conference held at Dartmouth College, where programs demonstrated capabilities such as playing chess, proving theorems, and interpreting texts. In the early stages, Alan Turing contributed to the field by defining intelligence as the ability of a being to respond to certain questions intelligently, proposing what is now known as the Turing Test to evaluate the presence of intelligent behavior in machines. As the decades progressed, AI evolved significantly. The 1980s focused on machine learning, teaching computers to learn from data, leading to the development of models that could improve their performance based on their experiences. The 1990s and 2000s saw further advances in algorithms and computational power, which allowed for more sophisticated data analysis techniques, including data mining. By the 2010s, the proliferation of big data and the refinement of deep learning techniques enabled AI to become mainstream. Notable milestones included the success of Google's AlphaGo and advancements in autonomous vehicles by companies like Tesla and Waymo. A major theme of the presentation is the application of generative AI, which has been used for tasks such as natural language text generation, translation, and question answering. Generative AI uses large datasets to train models that can then produce new, coherent pieces of text or other media. The presentation also discusses the ethical implications and the need for regulation in AI, highlighting issues such as privacy, bias, and the potential for misuse. These concerns have prompted calls for comprehensive regulations to ensure the safe and equitable use of AI technologies. Artificial intelligence has also played a significant role in healthcare, particularly highlighted during the COVID-19 pandemic, where it was used in drug discovery, vaccine development, and analyzing the spread of the virus. The capabilities of AI in healthcare are vast, ranging from medical diagnostics to personalized medicine, demonstrating the technology's potential to revolutionize fields beyond just technical or consumer applications. In conclusion, AI continues to be a rapidly evolving field with significant implications for various aspects of society. The development from theoretical concepts to real-world applications illustrates both the potential benefits and the challenges that come with integrating advanced technologies into everyday life. The ongoing discussion about AI ethics and regulation underscores the importance of managing these technologies responsibly to maximize their their benefits while minimizing potential harms.

Artificial Intelligence: Facts and Myths

Joaquim Jorge

Lessons learned building a big data analytics engine, from proprietary to open source

1. Álvaro Santamaría Data Scientist – ITRS @dofideas Joel Brunger System Engineer - MapR @joelbrunger Lessons Learned with Visualisation and Machine Learning for Big Data

3. Make real-time decisions based on historical wisdom

5. Jay Krepps – 2013 “The Log: What every software engineer should know about real-time data's unifying abstraction” The uber-system Lego-like, OS-based

7. 1. Big-data viz is not a matter of “front-end”

8. 2. 𝜅arao 𝜅e visualisations

9. 3. Information extraction

10. 3. Information extraction

11. 3. Information extraction

12. 3. Information extraction

13. 3. Information extraction GROUP BY country, timestamp window of 10 minutes SELECT count(), average(temperature), median(temperature), max(temperature), ... tdigest(temperature)

14. 3. Information extraction

15. 4. Hold state…

16. 4. Hold state…

17. 5. … and deliver (state).

18. 6. Build your pipeline

19.

20. J on the Beach All Data, One Platform, Every Cloud Limitless Possibilities

21. What is MapR? MapR is the industry’s leading data platform for AI and Analytics.

22. Rendezvous Architecture MapR is the industry’s leading data platform for AI and Analytics.

23. Rendezvous Architecture MapR is the industry’s leading data platform for AI and Analytics.

24. Rendezvous Architecture MapR is the industry’s leading data platform for AI and Analytics.

25. Rendezvous Architecture MapR is the industry’s leading data platform for AI and Analytics.

26. Rendezvous Architecture The Decoy is design to just collect data inputs

27. Rendezvous Architecture MapR is the industry’s leading data platform for AI and Analytics.

28. Rendezvous Architecture Introducing the Canary

29. Rendezvous Architecture MapR is the industry’s leading data platform for AI and Analytics.

30. Why did ITRS choose MapR for ‘Gateway Hub’ MapR is the industry’s leading data platform for AI and Analytics. Ø Simplicity (integrated platform) Ø Real-time Ø Processing must be performed in the cluster Ø Enterprise features

31. MapR enables ITRS ‘Gateway Hub’ to provide the following benefits MapR is the industry’s leading data platform for AI and Analytics. Ø Smarter monitoring Ø Additional features, application and Services Ø Global Data Fabric Ø Support ML in real-time

32. Thank you JOnTheBeach 2018

33. Álvaro Santamaría Data Scientist – ITRS @dofideas Joel Brunger System Engineer - MapR @joelbrunger Lessons Learned with Visualisation and Machine Learning for Big Data

Lessons learned building a big data analytics engine, from proprietary to open source

Recomendados

Recomendados

Más contenido relacionado

La actualidad más candente

La actualidad más candente (20)

Similar a Lessons learned building a big data analytics engine, from proprietary to open source

Similar a Lessons learned building a big data analytics engine, from proprietary to open source (20)

Más de J On The Beach

Más de J On The Beach (20)

Último

Último (20)

Lessons learned building a big data analytics engine, from proprietary to open source