2013.10.24 big datavisualization

•

29 recomendaciones•11,123 vistas

1. The document discusses techniques for visualizing and interacting with billion record databases in real-time. 2. It describes using binning, aggregation, and sampling to summarize large datasets and make them perceptually and interactively scalable. 3. Key techniques discussed include binning data into discrete buckets, aggregating statistics within bins, and using GPU processing and WebGL to enable fast querying and linking of visual summaries across multiple plots of billion record datasets.

Tecnología Empresariales

Visualizing
“Big” Data
Sean Kandel & Jeﬀrey Heer
Trifacta Inc. @trifacta

How can we visualize and
interact with billion+ record
databases in real-time?

Two Challenges:
1. Eﬀective visual encoding
2. Real-time interaction

Perceptual and interactive
scalability should be limited
by the chosen resolution of
the visualized data, not the
number of records.

Bin > Aggregate (> Smooth) > Plot
1. Bin Divide data domain into discrete “buckets”
Categories: Already discrete (but check cardinality)
Numbers: Choose bin intervals (uniform, quantile, ...)
Time: Choose time unit: Hour, Day, Month, etc.
Geo: Bin x, y coordinates after cartographic projection

Hexagonal or Rectangular Bins?

100,000 Data Points

Hexagonal Bins

Rectangular Bins

Hex bins better estimate density for 2D plots,
but the improvement is marginal [Scott 92], while
rectangles support reuse and query processing.

Plot: Visual Encoding
Choose Most Eﬀective Encoding [Cleveland & McGill ’84]
1D Plot -> Position or Length Encoding
Histograms, line charts, etc.

2D Plot -> Area or Color Encoding
Spatial dimensions (x, y) already allocated.
While less eﬀective than area for magnitude
estimation, color can be used at the per-pixel level
and provides an overall “gestalt”

Standard Color Ramp
Counts near zero are white.
-> Outliers are missed

Add Discontinuity after Zero
Counts near zero remain visible.
-> Outliers can be seen

Linear Alpha Interpolation
is not perceptually linear.

Cube-Root Alpha Interpolation
approximates perceptual linearity.

Color Encoding
Min. Non-Zero Intensity (α=0.15) [1]

Perceptual Scaling (γ=1/3) [2]

Luminance (in range 0-1) User-Adjustable Min/Max Values [3]

[1] Keep small non-zero values visible (outliers!)
[2] Match color ramp to perceptual distances
[3] Enable exploration across value ranges

Interaction Techniques?
1. Select
Detail-on-Demand
2. Navigate Pan & Zoom
3. Query Brush & Link

Y
512

…

1023

5-D Data Cube
Month, Day, Hour, X, Y
767

…
X
Month …11

11
…

0
23
…

0
23
…

11
…
0
23
…

1

1

1

0

0

0

Hour

0 1

… 30

0 1

…

30

0 1

256

… 30

Day

12 x 31 x 24 x 512 x 512 = ~2.3 billion cells

Y
512

…

1023

Brushing January
Month, Day, Hour, X, Y
767

…
X
Month …11

11
…

0
23
…

0
23
…

11
…
0
23
…

1

1

1

0

0

0

Hour

0 1

… 30

0 1

…

30

0 1

256

… 30

Day

31 x 24 x 512 x 512 = ~195 million cells

Multivariate Data Tiles
1. Send data, not pixels
2. Embed multi-dim data

Full 5-D Cube

Σ

Σ

Σ

Σ

For any pair of 1D or 2D binned plots, the
maximum number of dimensions needed
to support brushing & linking is four.

~2.3B bins

Full 5-D Cube

Σ

Σ

Σ

13 3-D Data Tiles

Σ

~17.6M bins
(in 352KB!)

Query & Render on GPU via WebGL

Pack data tiles as PNG image ﬁles,
bind to WebGL as image textures.

Query & Render on GPU via WebGL

Σ
Invoke program for each output bin.
Executes in parallel on GPU.

Performance Benchmarks
Simulate interaction:
brushing & linking
across binned plots.
- imMens vs. Proﬁler
- 4x4 and 5x5 plots
- 10 to 50 bins
Measure time from
selection to render.
Test setup:
2.3 GHz MacBook Pro (4-core)
NVIDIA GeForce GT 650M
Google Chrome v.23.0

5 dimensions x 50 bins/dim x 25 plots

imMens

~50fps querying of visual
summaries of 1B data points.

In-Memory Data Cube

Number of Data Points

NanoCubes

[1] Lins et. al. Infovis 2013
[2] Sismanis et. al. SIGMOD 2002

NanoCubes

[1] Lins et. al. Infovis 2013

Resources
imMens
Tableau Public
BigVis (R)
Nanocubes
BlinkDB
MapD

vis.stanford.edu/projects/immens
tableausoftware.com/public
github.com/hadley/bigvis
nanocubes.net
blinkdb.org
geops.csail.mit.edu/docs/

Acknowledgments
Zhicheng “Leo” Liu
Biye Jiang

Más contenido relacionado

La actualidad más candente

Chapter - 4 Data Mining Concepts and Techniques 2nd Ed slides Han & Kamber

error007

02 Geographic scripting in uDig - halfway between user and developer

Andrea Antonello

Blur Filter - Hanpo

Hanpo Cheng

Gaussian Image Blurring in CUDA C++

Darshan Parsana

Lec02 03 rasterization

Maaz Rizwan

Tutor 41 Big Numbers

Max Kleiner

Drawing Fonts

Budditha Hettige

http://www.Media4MathPlus.com In this issue of Math in the News we look at 3D printing technology. In particular, we look at Nike's recent unveiling of a new running shoe developed using 3D printing. Mathematically, we look at recursive functions, since the 3D printing technology was used for rapid, iterative prototyping of the new shoe. We look at an example of a recursive function, using the Babylonian method to calculate the square root of a number.

Math in the News: Issue 87

Media4math

Deep learning

DrBaljitSinghKhehra

신뢰 전파 기법을 이용한 스테레오 정합(Stereo matching using belief propagation algorithm)

Hansol Kang

Opensource gis development - part 4

Andrea Antonello

代表的なオープンソース空間データサーバの1つであるGeoServerは、多くの強力な機能を提供します。特に、さまざまなデータソースからの空間データへの接続とパブリッシングをサポートします。 GeoServerはOpen Geospatial Consortiumによって地理空間フィーチャデータを要求するために設定された標準プロトコルであるWeb Feature Service（WFS）もサポートしています。しかしながら、GeoServerは2次元ジオメトリのための関数しか提供しないため、3D空間データを処理する関数はほとんどありません。 GeoServerの重要なコンポーネントであるJTS Topology Suiteは3D空間操作をサポートしていないため、ソリッドジオメトリもサポートしていません。この講演では、3D空間データを扱うために私たちが実装したGeoServerの拡張モジュールを紹介します。 GeoServer, one of the representative open source spatial data servers, provides many powerful features. In particular, it supports connecting to and publishing spatial data from a variety of data sources. GeoServer also supports Web Feature Service (WFS), which is a standard protocol established by the Open Geospatial Consortium to request geospatial feature data. However, GeoServer provides functions only for two-dimensional geometry, so it provides few functions for handling 3D spatial data. Because JTS Topology Suite, which is an important component of GeoServer, does not support 3D spatial operations, it also does not support solid geometries. In this talk, I will introduce extension modules of GeoServer that we have implemented to handle 3D spatial data.

Foss4 g 2017-kansai-ryoo-kim

OSgeo Japan

ข้อมูลและสารสนเทศ

Surapong Jakang

Object recognition from RGB-D sensors has recently emerged as a renowned and challenging research topic. The current systems often require large amounts of time to train the models and to classify new data. We proposed an effective and fast object recognition approach from 3D data acquired from depth sensors such as Structure or Kinect sensors. Our contribution in this work} is to present a novel fast and effective approach for real-time object recognition from 3D depth data: - First, we extract simple but effective frame-level features, which we name as differential frames, from the raw depth data. - Second, we build a recognition system based on Extreme Learning Machine classifier with a Local Receptive Field (ELM-LRF).

Fast Object Recognition from 3D Depth Data with Extreme Learning Machine

Soma Boubou

Y Tiles

Christopher Yunker

One of the important application in image processing is to be able to judge the different edges in an image and be able to distinguish various parts of it. Clearly, this is an application dependent task and results and analysis will vary as per the situation. These slides show a general description on how to calculate the number of cells in a microscopic tissue. There are two version one being the over-stained tissue and the other being the better one. We see how the algorithm is able to calculate the number of cells.

Cell calculation

Munnangi Anirudh

Spatial Analysis with R - the Good, the Bad, and the Pretty

Noam Ross

Genome Browser based on Google Maps API

Hong ChangBum

Image sampling and quantization

BCET, Balasore

Image transforms

BCET, Balasore

La actualidad más candente (20)

Chapter - 4 Data Mining Concepts and Techniques 2nd Ed slides Han & Kamber

02 Geographic scripting in uDig - halfway between user and developer

Blur Filter - Hanpo

Gaussian Image Blurring in CUDA C++

Lec02 03 rasterization

Tutor 41 Big Numbers

Drawing Fonts

Math in the News: Issue 87

Deep learning

신뢰 전파 기법을 이용한 스테레오 정합(Stereo matching using belief propagation algorithm)

Opensource gis development - part 4

Foss4 g 2017-kansai-ryoo-kim

ข้อมูลและสารสนเทศ

Fast Object Recognition from 3D Depth Data with Extreme Learning Machine

Y Tiles

Cell calculation

Spatial Analysis with R - the Good, the Bad, and the Pretty

Genome Browser based on Google Maps API

Image sampling and quantization

Image transforms

Similar a 2013.10.24 big datavisualization

Interactive Latency in Big Data Visualization Zhicheng "Leo" Liu, Research Scientist at the Creative Technologies Lab at Adobe Research January 22nd, 2014 Reducing interactive latency is a central problem in visualizing large datasets. I discuss two inter-related projects in this problem space. First, I present the imMens system and show how we can achieve real-time interaction at 50 frames per second for billions of data points by combining techniques such as data tiling and parallel processing. Second, I discuss an ongoing user study that aims to understand the effect of interactive latency on human cognitive behavior in exploratory visual analysis. Big Data Visualization Meetup - South Bay http://www.meetup.com/Big-Data-Visualisation-South-Bay/

Interactive Latency in Big Data Visualization

bigdataviz_bay

Data Mining: Concepts and Techniques (3rd ed.)— Chapter 5

Salah Amean

Ics2311 l02 Graphics fundamentals

bridgekloud

2 olap operaciones

Claudia Gomez

Datascape Introduction

Daden Limited

Making BIG DATA smaller

Tony Tran

TDC2017 | São Paulo - Trilha Java EE How we figured out we had a SRE team at ...

tdc-globalcode

Time-Evolving Graph Processing On Commodity Clusters

Jen Aman

Object Detection using Deep Neural Networks

Usman Qayyum

How should data be preprocessed for use in machine learning algorithms? How to identify the most predictive attributes of a dataset? What features can generate to improve the accuracy of a model? Feature Engineering is the process of extracting and selecting, from raw data, features that can be used effectively in predictive models. As the quality of the features greatly influences the quality of the results, knowing the main techniques and pitfalls will help you to succeed in the use of machine learning in your projects. In this talk, we will present methods and techniques that allow us to extract the maximum potential of the features of a dataset, increasing flexibility, simplicity and accuracy of the models. The analysis of the distribution of features and their correlations, the transformation of numeric attributes (such as scaling, normalization, log-based transformation, binning), categorical attributes (such as one-hot encoding, feature hashing, Temporal (date / time), and free-text attributes (text vectorization, topic modeling). Python, Python, Scikit-learn, and Spark SQL examples will be presented and how to use domain knowledge and intuition to select and generate features relevant to predictive models.

Feature Engineering - Getting most out of data for predictive models - TDC 2017

Gabriel Moreira

Introduction to Applied Machine Learning

SheilaJimenezMorejon

Talk Abstract Aggregation has long been a use case of Accumulo Iterators. Iterators' ability to reduce data during compaction and scanning can greatly simplify an aggregation system built on Accumulo. This talk will first review how Accumulo's Iterators/Combiners work in the context of aggregating values. I'll then step back and look at the abstraction of aggregation functions as commutative operations and the several benefits that result by making this abstraction. We will see how it becomes no harder to introduce powerful operations such as cardinality estimation and approximate top-k than it is to sum integers. I will show how to integrate these ideas into Accumulo with an example schema and Iterator. Finally, a practical aggregation use case will be discussed to highlight the concepts from the talk. Speakers Gadalia O'Bryan Senior Solutions Architect, Koverse Gadalia O'Bryan is a Sr. Solutions Architect at Koverse, where she leads customer projects and contributes to key feature and algorithm design, such as Koverse's Aggregation Framework. Prior to Koverse, Gadalia was a mathematician for the National Security Agency. She has an M.A. in mathematics from UCLA and has been working with Accumulo for the past 6 years. Bill Slacum Software Engineer, Koverse Bill is an Accumulo committer and PMC member who has been working on large scale query and analytic frameworks since 2010. He holds BS's in computer science and financial economics from UMBC. Having never used his passport to leave the United States, he is currently a national man of mystery.

Accumulo Summit 2015: Building Aggregation Systems on Accumulo [Leveraging Ac...

Accumulo Summit

05 cubetech

JoonyoungJayGwak

Specialized indexing for NoSQL Databases like Accumulo and HBase

Jim Klucar

Beginning direct3d gameprogramming01_thehistoryofdirect3dgraphics_20160407_ji...

JinTaek Seo

Geo & capped collections with MongoDB

Rainforest QA

Cahall Final Intern Presentation

Daniel Cahall

An optimal data partitioning in parallel/distributed implementation of clustering algorithms is a necessary computation as it ensures independent task completion, fair distribution, less number of affected points and better & faster merging. Though partitioning using Kd-Tree is being conventionally used in academia, it suffers from performance drenches and bias (non equal distribution) as dimensionality of data increases and hence is not suitable for practical use in industry where dimensionality can be of order of 100’s to 1000’s. To address these issues we propose two new partitioning techniques using existing mathematical models & study their feasibility, performance (bias and partitioning speed) & possible variants in choosing initial seeds. First method uses an n-dimensional hashed grid based approach which is based on mapping the points in space to a set of cubes which hashes the points. Second method uses a tree of voronoi planes where each plane corresponds to a partition. We found that grid based approach was computationally impractical, while using a tree of voronoi planes (using scalable K-Means++ initial seeds) drastically outperformed the Kd-tree tree method as dimensionality increased.

An Efficient Method of Partitioning High Volumes of Multidimensional Data for...

IJERA Editor

In this paper we propose Regularised Cross-Modal Hashing (RCMH) a new cross-modal hashing model that projects annotation and visual feature descriptors into a common Hamming space. RCMH optimises the hashcode similarity of related data-points in the annotation modality using an iterative three-step hashing algorithm: in the first step each training image is assigned a K-bit hashcode based on hyperplanes learnt at the previous iteration; in the second step the binary bits are smoothed by a formulation of graph regularisation so that similar data-points have similar bits; in the third step a set of binary classifiers are trained to predict the regularised bits with maximum margin. Visual descriptors are projected into the annotation Hamming space by a set of binary classifiers learnt using the bits of the corresponding annotations as labels. RCMH is shown to consistently improve retrieval effectiveness over state-of-the-art baselines.

Regularised Cross-Modal Hashing (SIGIR'15 Poster)

Sean Moran

Processing of large data requires new approaches to data mining: low, close to linear, complexity and stream processing. While in the traditional data mining the practitioner is usually presented with a static dataset, which might have just a timestamp attached to it, to infer a model for predicting future/takeout observations, in stream processing the problem is often posed as extracting as much information as possible on the current data to convert them to an actionable model within a limited time window. In this talk I present an approach based on HBase counters for mining over streams of data, which allows for massively distributed processing and data mining. I will consider overall design goals as well as HBase schema design dilemmas to speed up knowledge extraction process. I will also demo efficient implementations of Naive Bayes, Nearest Neighbor and Bayesian Learning on top of Bayesian Counters.

Hadoop Summit 2012 | Bayesian Counters AKA In Memory Data Mining for Large Da...

Cloudera, Inc.

Similar a 2013.10.24 big datavisualization (20)

Interactive Latency in Big Data Visualization

Data Mining: Concepts and Techniques (3rd ed.)— Chapter 5

Ics2311 l02 Graphics fundamentals

2 olap operaciones

Datascape Introduction

Making BIG DATA smaller

TDC2017 | São Paulo - Trilha Java EE How we figured out we had a SRE team at ...

Time-Evolving Graph Processing On Commodity Clusters

Object Detection using Deep Neural Networks

Feature Engineering - Getting most out of data for predictive models - TDC 2017

Introduction to Applied Machine Learning

Accumulo Summit 2015: Building Aggregation Systems on Accumulo [Leveraging Ac...

05 cubetech

Specialized indexing for NoSQL Databases like Accumulo and HBase

Beginning direct3d gameprogramming01_thehistoryofdirect3dgraphics_20160407_ji...

Geo & capped collections with MongoDB

Cahall Final Intern Presentation

An Efficient Method of Partitioning High Volumes of Multidimensional Data for...

Regularised Cross-Modal Hashing (SIGIR'15 Poster)

Hadoop Summit 2012 | Bayesian Counters AKA In Memory Data Mining for Large Da...

Último

Three things you will take away from the session: • How to run an effective tenant-to-tenant migration • Best practices for before, during, and after migration • Tips for using migration as a springboard to prepare for Copilot in Microsoft 365 Main ideas: Migration Overview: The presentation covers the current reality of cross-tenant migrations, the triggers, phases, best practices, and benefits of a successful tenant migration Considerations: When considering a migration, it is important to consider the migration scope, performance, customization, flexibility, user-friendly interface, automation, monitoring, support, training, scalability, data integrity, data security, cost, and licensing structure Next Wave: The next wave of change includes the launch of Copilot, which requires businesses to be prepared for upcoming changes related to Copilot and the cloud, and to consolidate data and tighten governance ShareGate: ShareGate can help with pre-migration analysis, configurable migration tool, and automated, end-user driven collaborative governance

Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff

sammart93

Building Digital Trust in a Digital Economy Veronica Tan, Director - Cyber Security Agency of Singapore Apidays Singapore 2024: Connecting Customers, Business and Technology (April 17 & 18, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...

apidays

🐬 The future of MySQL is Postgres 🐘

RTylerCroy

BooK Now Call us at +918448380779 to hire a gorgeous and seductive call girl for sex. Take a Delhi Escort Service. The help of our escort agency is mostly meant for men who want sexual Indian Escorts In Delhi NCR. It should be noted that any impersonator will get 100 attention from our Young Girls Escorts in Delhi. They will assume the position of reliable allies. VIP Call Girl With Original Photos Book Tonight +918448380779 Our Cheap Price 1 Hour not available 2 Hours 5000 Full Night 8000 TAG: Call Girls in Delhi, Noida, Gurgaon, Ghaziabad, Connaught Place, Greater Kailash Delhi, Lajpat Nagar Delhi, Mayur Vihar Delhi, Chanakyapuri Delhi, New Friends Colony Delhi, Majnu Ka Tilla, Karol Bagh, Malviya Nagar, Saket, Khan Market, Noida Sector 18, Noida Sector 76, Noida Sector 51, Gurgaon Mg Road, Iffco Chowk Gurgaon, Rajiv Chowk Gurgaon All Delhi Ncr Free Home Deliver

08448380779 Call Girls In Greater Kailash - I Women Seeking Men

Delhi Call girls

Tech Trends Report 2024 Future Today Institute.pdf

hans926745

Sara Mae O’Brien Scott and Tatiana Baquero Cakici, Senior Consultants at Enterprise Knowledge (EK), presented “AI Fast Track to Search-Focused AI Solutions” at the Information Architecture Conference (IAC24) that took place on April 11, 2024 in Seattle, WA. In their presentation, O’Brien-Scott and Cakici focused on what Enterprise AI is, why it is important, and what it takes to empower organizations to get started on a search-based AI journey and stay on track. The presentation explored the complexities of enterprise search challenges and how IA principles can be leveraged to provide AI solutions through the use of a semantic layer. O’Brien-Scott and Cakici showcased a case study where a taxonomy, an ontology, and a knowledge graph were used to structure content at a healthcare workforce solutions organization, providing personalized content recommendations and increasing content findability. In this session, participants gained insights about the following: Most common types of AI categories and use cases; Recommended steps to design and implement taxonomies and ontologies, ensuring they evolve effectively and support the organization’s search objectives; Taxonomy and ontology design considerations and best practices; Real-world AI applications that illustrated the value of taxonomies, ontologies, and knowledge graphs; and Tools, roles, and skills to design and implement AI-powered search solutions.

IAC 2024 - IA Fast Track to Search Focused AI Solutions

Enterprise Knowledge

A Domino Admins Adventures (Engage 2024)

Gabriella Davis

In this session, we will delve into strategic approaches for optimizing knowledge management within Microsoft 365, amidst the evolving landscape of Copilot. From leveraging automatic metadata classification and permission governance with SharePoint Premium, to unlocking Viva Engage for the cultivation of knowledge and communities, you will gain actionable insights to bolster your organization's knowledge-sharing initiatives. In this session, we will also explore how to facilitate solutions to enable your employees to find answers and expertise within Microsoft 365. You will leave equipped with practical techniques and a deeper understanding of how there is more to effective knowledge management than just enabling Copilot, but building actual solutions to prepare the knowledge that Copilot and your employees can use.

Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...

Drew Madelung

CNv6 Instructor Chapter 6 Quality of Service

giselly40

Axa Assurance Maroc - Insurer Innovation Award 2024

The Digital Insurer

Automating Google Workspace (GWS) & more with Apps Script

wesley chun

With more memory available, system performance of three Dell devices increased, which can translate to a better user experience Conclusion When your system has plenty of RAM to meet your needs, you can efficiently access the applications and data you need to finish projects and to-do lists without sacrificing time and focus. Our test results show that with more memory available, three Dell PCs delivered better performance and took less time to complete the Procyon Office Productivity benchmark. These advantages translate to users being able to complete workflows more quickly and multitask more easily. Whether you need the mobility of the Latitude 5440, the creative capabilities of the Precision 3470, or the high performance of the OptiPlex Tower Plus 7010, configuring your system with more RAM can help keep processes running smoothly, enabling you to do more without compromising performance.

Boost PC performance: How more available memory can improve productivity

Principled Technologies

08448380779 Call Girls In Civil Lines Women Seeking Men

Delhi Call girls

Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...

Neo4j

How to Troubleshoot Apps for the Modern Connected Worker

ThousandEyes

Abhishek Deb(1), Mr Abdul Kalam(2) M. Des (UX) , School of Design, DIT University , Dehradun. This paper explores the future potential of AI-enabled smartphone processors, aiming to investigate the advancements, capabilities, and implications of integrating artificial intelligence (AI) into smartphone technology. The research study goals consist of evaluating the development of AI in mobile phone processors, analyzing the existing state as well as abilities of AI-enabled cpus determining future patterns as well as chances together with reviewing obstacles as well as factors to consider for more growth.

Exploring the Future Potential of AI-Enabled Smartphone Processors

debabhi2

Partners Life - Insurer Innovation Award 2024

The Digital Insurer

The Raspberry Pi 5 was announced on October 2023. This new version of the popular embedded device comes with a new iteration of Broadcom’s VideoCore GPU platform, and was released with a fully open source driver stack, developed by Igalia. The presentation will discuss some of the major changes required to support this new Video Core iteration, the challenges we faced in the process and the solutions we provided in order to deliver conformant OpenGL ES and Vulkan drivers. The talk will also cover the next steps for the open source Raspberry Pi 5 graphics stack. (c) Embedded Open Source Summit 2024 April 16-18, 2024 Seattle, Washington (US) https://events.linuxfoundation.org/embedded-open-source-summit/ https://eoss24.sched.com/event/1aBEx

Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...

Igalia

2024: Domino Containers - The Next Step. News from the Domino Container commu...

Martijn de Jong

08448380779 Call Girls In Friends Colony Women Seeking Men

Delhi Call girls

2013.10.24 big datavisualization

1. Visualizing “Big” Data Sean Kandel & Jeﬀrey Heer Trifacta Inc. @trifacta

2. How can we visualize and interact with billion+ record databases in real-time?

3. Two Challenges: 1. Eﬀective visual encoding 2. Real-time interaction

4. Perceptual and interactive scalability should be limited by the chosen resolution of the visualized data, not the number of records.

5. Perception

6. Data Sampling Binning Modeling

7. Google Fusion Tables (Sampling)

8. imMens (Binned Aggregation)

9. Bin > Aggregate (> Smooth) > Plot 1. Bin Divide data domain into discrete “buckets” Categories: Already discrete (but check cardinality) Numbers: Choose bin intervals (uniform, quantile, ...) Time: Choose time unit: Hour, Day, Month, etc. Geo: Bin x, y coordinates after cartographic projection

10. Number of Bins?

11. Hexagonal or Rectangular Bins? 100,000 Data Points Hexagonal Bins Rectangular Bins Hex bins better estimate density for 2D plots, but the improvement is marginal [Scott 92], while rectangles support reuse and query processing.

12. Bin > Aggregate (> Smooth) > Plot 1. Bin Divide data domain into discrete “buckets” Categories: Already discrete (but check cardinality) Numbers: Choose bin intervals (uniform, quantile, ...) Time: Choose time unit: Hour, Day, Month, etc. Geo: Bin x, y coordinates after cartographic projection 2. Aggregate Count, Sum, Average, Min, Max, ...

13. Bin > Aggregate (> Smooth) > Plot 1. Bin Divide data domain into discrete “buckets” Categories: Already discrete (but check cardinality) Numbers: Choose bin intervals (uniform, quantile, ...) Time: Choose time unit: Hour, Day, Month, etc. Geo: Bin x, y coordinates after cartographic projection 2. Aggregate Count, Sum, Average, Min, Max, ... (3. Smooth Optional: smooth aggregates [Wickham ’13])

14. [1] Wickham 2013

15. Bin > Aggregate (> Smooth) > Plot 1. Bin Divide data domain into discrete “buckets” Categories: Already discrete (but check cardinality) Numbers: Choose bin intervals (uniform, quantile, ...) Time: Choose time unit: Hour, Day, Month, etc. Geo: Bin x, y coordinates after cartographic projection 2. Aggregate Count, Sum, Average, Min, Max, ... (3. Smooth Optional: smooth aggregates [Wickham ’13]) 4. Plot Visualize the aggregate summary values

16. Plot: Visual Encoding Choose Most Eﬀective Encoding [Cleveland & McGill ’84] 1D Plot -> Position or Length Encoding Histograms, line charts, etc. 2D Plot -> Area or Color Encoding Spatial dimensions (x, y) already allocated. While less eﬀective than area for magnitude estimation, color can be used at the per-pixel level and provides an overall “gestalt”

17. Standard Color Ramp Counts near zero are white. -> Outliers are missed Add Discontinuity after Zero Counts near zero remain visible. -> Outliers can be seen

18. Linear Alpha Interpolation is not perceptually linear. Cube-Root Alpha Interpolation approximates perceptual linearity.

19. Color Encoding Min. Non-Zero Intensity (α=0.15) [1] Perceptual Scaling (γ=1/3) [2] Luminance (in range 0-1) User-Adjustable Min/Max Values [3] [1] Keep small non-zero values visible (outliers!) [2] Match color ramp to perceptual distances [3] Enable exploration across value ranges

20. Design Space of Binned Plots

21. Interaction

22. Interaction Techniques? 1. Select Detail-on-Demand 2. Navigate Pan & Zoom 3. Query Brush & Link

23.

24.

25. Y 512 … 1023 5-D Data Cube Month, Day, Hour, X, Y 767 … X Month …11 11 … 0 23 … 0 23 … 11 … 0 23 … 1 1 1 0 0 0 Hour 0 1 … 30 0 1 … 30 0 1 256 … 30 Day 12 x 31 x 24 x 512 x 512 = ~2.3 billion cells

26. Y 512 … 1023 Brushing January Month, Day, Hour, X, Y 767 … X Month …11 11 … 0 23 … 0 23 … 11 … 0 23 … 1 1 1 0 0 0 Hour 0 1 … 30 0 1 … 30 0 1 256 … 30 Day 31 x 24 x 512 x 512 = ~195 million cells

27.

28.

29. Multivariate Data Tiles 1. Send data, not pixels 2. Embed multi-dim data

30. Full 5-D Cube Σ Σ Σ Σ For any pair of 1D or 2D binned plots, the maximum number of dimensions needed to support brushing & linking is four.

31. Y : 512 bins X : 512 bins

32.

33.

34.

35.

36.

37.

38.

39.

40.

41.

42.

43.

44.

45.

46.

47.

48. ~2.3B bins Full 5-D Cube Σ Σ Σ 13 3-D Data Tiles Σ ~17.6M bins (in 352KB!)

49. Query & Render on GPU via WebGL Pack data tiles as PNG image ﬁles, bind to WebGL as image textures.

50. Query & Render on GPU via WebGL Σ Invoke program for each output bin. Executes in parallel on GPU.

51. Query & Render on GPU via WebGL Σ

52. Performance Benchmarks Simulate interaction: brushing & linking across binned plots. - imMens vs. Proﬁler - 4x4 and 5x5 plots - 10 to 50 bins Measure time from selection to render. Test setup: 2.3 GHz MacBook Pro (4-core) NVIDIA GeForce GT 650M Google Chrome v.23.0

53. 5 dimensions x 50 bins/dim x 25 plots imMens ~50fps querying of visual summaries of 1B data points. In-Memory Data Cube Number of Data Points

54. NanoCubes [1] Lins et. al. Infovis 2013 [2] Sismanis et. al. SIGMOD 2002

55. NanoCubes [1] Lins et. al. Infovis 2013

56. Resources imMens Tableau Public BigVis (R) Nanocubes BlinkDB MapD vis.stanford.edu/projects/immens tableausoftware.com/public github.com/hadley/bigvis nanocubes.net blinkdb.org geops.csail.mit.edu/docs/

57. Acknowledgments Zhicheng “Leo” Liu Biye Jiang

58. Visualizing “Big” Data Sean Kandel & Jeﬀrey Heer Trifacta Inc. @trifacta