Natural Language to Visualization by Neural Machine Translation

•Descargar como PPTX, PDF•

0 recomendaciones•170 vistas

ivaderivader

Tecnología

Contents
• Overview
• Related Work
• Design Requirements
• Background and Problem Formulation
• Model – ncNet
• Evaluation
• Limitation and Future Work

2
What is neural machine translation (NMT)?
• Given input sequence, translate them into output sequence (seq2seq)
 e.g., English to Spanish
 (input, output) can be any pair of (sequence, sequence)!

4
Contribution
• Proposing ncNet: Transformer-based seq2seq model for NL2VIS
• Presenting visualization-grammar, Vega-Zero for Vega-Lite
• Attention forcing and Visualization-aware translation
A. Satyanarayan, D. Moritz, K. Wongsuphasawat, and J. Heer. Vega-lite: A grammar of interactive graphics. IEEE TVCG, 23(1):341–350, 2017

5
Related Work
NL4DV
FlowSense
• Utilize semantic parsers, instead of using deep-learning based approach
B. Yu and C. T. Silva. Flowsense: A natural language interface for visual data exploration within a dataflow system. IEEE TVCG, pp. 1–11, 2020.
A. Narechania, A. Srinivasan, and J. T. Stasko. NL4DV: A toolkit for generating analytic specifications for data visualization from natural language queries. In VIS, 2020.

6
Design Requirements
• Easy-to-use
 Simple query from novice(to create visualization) user  Suitable visualization
• End-to-End
 Semantic parser  hard coded, should be tuned separately
 Need to be end-to-end manner
• Language-agnostic (Language-independent)
 The output sequence of ncNet should be utilized in various visualization languages
• e.g., Vega-Lite, D3, ggplot2…

7
Background – Data, NL Query
• Data D: tabular dataset from JSON, CSV, or related table from DB
• NL Query: user query for corresponding data D
• Objective: given NL query with dataset D, predict Vega-Zero specification
(Input Sequence, Output Sequence)

8
Language-Agnostic query: Vega-Zero
• A visualization grammar
 Simplified version of Vega-Lite
 Model friendly
• Direct use of Vega-Lite?
 User friendly rather than model friendly
 hierarchical grammar  hard to train
• Language-agnostic
 By simple formatting
 Vega-Lite, ggplot2
: unit

9
Background – Chart Templates
• User input
• Constrain output sequence

11
Methods: Input Handling
• Input Tokenization
 Tokenize three types of input data
• Input Embedding

12
What is Transformer?
• One of the most famous NLP model
• The model only utilize attention mechanism
• Parallelly handle input sequence
 better than RNN, but need positional encoding
• Well-motivated

14
Methods: visualization-aware translation
• Modified version of Beam Search
• Tokens in specific steps have appropriate tokens (e.g., scatter only in [T])
• Top-k with heuristic rules
 If there is no appropriate tokens, select top-1 token

15
Evaluation
• Dataset
 nvBench: 25,750 data in format (NL query, VIS) with ~780 tables from 105 domains

16
Evaluation: User Study
• 4 experts, 2 non-experts
• Four failure cases – fail on predicting 1) columns; 2) chart type;
3) top-1 selection; 4) Vega-Zero fails but vis match

17
Evaluation: Comparison
• Compare sota NL2SQL, ncNet w/o C, ncNet w/ C
• ncNet performs better than sota NL2SQL model

18
Limitation and Future work
• (Limitation) Benchmark data are relatively small and limited
• (Limitation) Supporting only one-shot NL queries
• (Future work) supporting conversational NL queries  dialog system?
• (Future work) Chart embedding methods

Natural Language to Visualization by Neural Machine Translation

Más contenido relacionado

La actualidad más candente

On using monolingual corpora in neural machine translation

NAIST Machine Translation Study Group

https://telecombcn-dl.github.io/2017-dlsl/ Winter School on Deep Learning for Speech and Language. UPC BarcelonaTech ETSETB TelecomBCN. The aim of this course is to train students in methods of deep learning for speech and language. Recurrent Neural Networks (RNN) will be presented and analyzed in detail to understand the potential of these state of the art tools for time series processing. Engineering tips and scalability issues will be addressed to solve tasks such as machine translation, speech recognition, speech synthesis or question answering. Hands-on sessions will provide development skills so that attendees can become competent in contemporary data analytics tools.

Neural Machine Translation (D3L4 Deep Learning for Speech and Language UPC 2017)

Universitat Politècnica de Catalunya

Main Talk: Google's Neural Machine Translation System and Research progress Abstract: Neural Machine Translation (NMT) is an end-to-end learning approach for automated translation, with the potential to overcome many of the weaknesses of conventional phrase-based translation systems. In this talk, I will talk about the model architecture, word-pieces design, training algorithm and how to make training/serving faster. Possibly I will mention about the zero-shot for Multilingual model as well. Also, I will cover what/how translation research makes continuous progress from last year. Speaker:Xiaobing Liu Xiaobing Liu is Google Brain Staff software engineer and machine learning researcher. In his work, Xiaobing focuses on Tensorflow and some key applications where Tensorflow could be applied to improve Google products, such as Google Search, Play recommendation and Google translation and Medical Brain. His research interests span from system to the practice of machine learning. His research contributions have been successfully implemented into various commercial products at Tencent, Yahoo. and Google He has served in the program committee for ACL 2017 and session chair for AAAI 2017, including publications at top conference such as Recsys, NIPS, ACL.

Moving to neural machine translation at google - gopro-meetup

Chester Chen

Machine Translation Introduction

nlab_utokyo

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Minh Pham

Notes on attention mechanism

Khang Pham

Advanced Neural Machine Translation (D4L2 Deep Learning for Speech and Langua...

Universitat Politècnica de Catalunya

Julia language: inside the corporation

Andre Pemmelaar

BERT: Bidirectional Encoder Representations from Transformers

Liangqun Lu

[Paper review] BERT

JEE HYUN PARK

Bert

Abdallah Bashir

Attention Mechanism in Language Understanding and its Applications

Artifacia

BERT

Khang Pham

BERT

Sang Hyun Jeon

Learning New Semi-Supervised Deep Auto-encoder Features for Statistical Machi...

Vimukthi Wickramasinghe

Master Thesis of Computer Engineering: OpenTranslator

Giuseppe D'Onofrio

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Young Seok Kim

A Feature-Based Model for Nested Named-Entity Recognition at VLSP-2018 NER Ev...

Minh Pham

Bert pre_training_of_deep_bidirectional_transformers_for_language_understanding

ThyrixYang1

A neural conversational_model

sotanemoto

La actualidad más candente (20)

On using monolingual corpora in neural machine translation

Neural Machine Translation (D3L4 Deep Learning for Speech and Language UPC 2017)

Moving to neural machine translation at google - gopro-meetup

Machine Translation Introduction

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Notes on attention mechanism

Advanced Neural Machine Translation (D4L2 Deep Learning for Speech and Langua...

Julia language: inside the corporation

BERT: Bidirectional Encoder Representations from Transformers

[Paper review] BERT

Bert

Attention Mechanism in Language Understanding and its Applications

BERT

Learning New Semi-Supervised Deep Auto-encoder Features for Statistical Machi...

Master Thesis of Computer Engineering: OpenTranslator

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

A Feature-Based Model for Nested Named-Entity Recognition at VLSP-2018 NER Ev...

Bert pre_training_of_deep_bidirectional_transformers_for_language_understanding

A neural conversational_model

Similar a Natural Language to Visualization by Neural Machine Translation

[論文紹介]A syntactic neural model for general purpose code generation

OgataTomoya

Recurrent Neural Networks have shown to be very powerful models as they can propagate context over several time steps. Due to this they can be applied effectively for addressing several problems in Natural Language Processing, such as Language Modelling, Tagging problems, Speech Recognition etc. In this presentation we introduce the basic RNN model and discuss the vanishing gradient problem. We describe LSTM (Long Short Term Memory) and Gated Recurrent Units (GRU). We also discuss Bidirectional RNN with an example. RNN architectures can be considered as deep learning systems where the number of time steps can be considered as the depth of the network. It is also possible to build the RNN with multiple hidden layers, each having recurrent connections from the previous time steps that represent the abstraction both in time and space.

Recurrent Neural Networks, LSTM and GRU

ananth

Nlp and transformer (v3s)

H K Yoon

230922 Semantic Exploration from Language Abstractions and Pretrained Represe...

Seungjoon1

Deep Learning Architectures for NLP (Hungarian NLP Meetup 2016-09-07)

Márton Miháltz

The Transformer architecture is responsible for many state of the art results in Natural Language Processing. A central feature behind its superior performance over Recurrent Neural Networks is its multi-headed self-attention mechanism. However, the superior performance comes at a cost, an O(n2) time and memory complexity, where n is the size of the input sequence. Because of this, it is computationally infeasible to feed large documents to the standard transformer. To overcome this limitation, a number of approaches have been proposed, which involve modifying the self-attention mechanism in interesting ways. In this presentation, I will describe the transformer architecture, and specifically the self-attention mechanism, and then describe some of the approaches proposed to address the O(n2) complexity. Some of these approaches have also been implemented in the HuggingFace transformers library, and I will demonstrate some code for doing document level operations using one of these approaches.

Transformer Mods for Document Length Inputs

Sujit Pal

node2vec: Scalable Feature Learning for Networks.pptx

ssuser2624f71

Natural language interfaces (NLI) offer end-users an easy and convenient way to query ontology-based knowledge graphs. They automatically generate database queries based on their natural language inputs, avoiding the need for the end user to learn different query languages. NLIs can be used with REST APIs to facilitate and enrich the interactions with knowledge graphs, in domains such as interactive root cause analysis (RCA), dynamic dashboard generation, and Online Transactional Processing (OLTP). In this talk, you'll learn about a natural language interface built with a TypeDB server running on Raspberry Pi4. This application offers a conversational bot assistant with Cisco Webex for an efficient and flexible way to facilitate human-machine interactions. In particular, this talk will demonstrate how natural language inputs are translated into TypeQL queries using Abstract Syntax Trees that represent the syntactic structure discovered during the Named Entity Recognition (NER) analysis of the textual inputs provided by Rasa 2.X running on an Intel Celeron J3455 miniPC.

Natural Language Interface to Knowledge Graph

Vaticle

ECCV2022 paper reading - MultiMAE: Multi-modal Multi-task Masked Autoencoders...

Antonio Tejero de Pablos

When machine learning models are productionized, they are commonly formed as workflows with multiple tasks, managed by a task scheduler such as Airflow, Prefect. Traditionally each task within the same workflow uses similar computing frameworks (e.g. Python, Spark, and PyTorch) in the same backend computing environment (e.g. AWS EMR, Google DataProc) with globally fixed settings (e.g. instances, cores, memory). In complicated use cases, such traditional workflows create large resource and runtime inefficiency, hence it is highly desired to use different computing frameworks in the same workflow in different computing environments. Such workflows can be named as superworkflows. Fugue is an open-sourced abstraction layer on top of different computing frameworks and creates uniform interfaces to use these frameworks without dealing with the complexities associated with them. To this end, Fugue can be viewed as a superframework. In addition, Kubernetes (K8S) is a container orchestration system, and it is easy to create different computing environments (e.g. Spark, PyTorch) with different docker images as everything is containerized in K8S. It is natural to combine K8S and Fugue to create superworkflows for complicated machine learning problems. In this talk, we use a popular graph neural network named Node2Vec as an example to illustrate how to create an efficient superworkflow using Fugue and K8S on very large graphs with hundreds of millions of vertices and edges. We also demonstrate how to partition the whole Node2Vec process into multiple tasks based on their complexities and parallelism. Benchmark testing is conducted for comparing performance and resource efficiency. Finally, it is easy to generalize this superworkflow concept to other deep learning problems.

Superworkflow of Graph Neural Networks with K8S and Fugue

Databricks

Gnn overview

Louis (Yufeng) Wang

ResNet, short for "Residual Network," is a type of deep neural network architecture that was introduced by Microsoft researchers in 2015. ResNet is designed to address the problem of vanishing gradients, which can occur in deep neural networks that are many layers deep. The main innovation in ResNet is the use of residual connections, also known as skip connections. These connections allow information from earlier layers of the network to bypass some of the later layers and be directly fed into the later layers. This helps to ensure that the gradient signal from the output can propagate back through the network during training, which can help to prevent the vanishing gradient problem. ResNet has been shown to be very effective at image recognition and other computer vision tasks. It has achieved state-of-the-art performance on a number of benchmark datasets, such as ImageNet. Since its introduction, many variations and improvements to the original ResNet architecture have been proposed, including ResNeXt, Wide ResNet, and Residual Attention Network (RANet).

Resnet.pdf

YanhuaSi

5_RNN_LSTM.pdf

FEG

Introduction to deep learning

Vishwas Lele

A Generalization of Transformer Networks to Graphs.pptx

ssuser2624f71

Graph neural networks (GNN) is a tool that brings great predictive power to graph machine learning tasks such as link prediction and node classification. However, GNN architectures are typically very compute heavy and as such are not feasible to run at massive scale. In this talk, we will leverage the graph sampling features of the Neo4j Graph Data Science (GDS) library as well as the inductive power of GNNs to bring GNNs to scale. We will also show how the GDS Python Client can, with great performance, be used to integrate the GDS workflow with other GNN Python libraries.

008 GNNs at Scale With Graph Data Science Sampling and Python Client Integrat...

Neo4j

Transfer Learning for Low Resource Languages and Domains

Efsun Kayi

Presentation of the survey published in the IEEE COMMUNICATIONS SURVEYS & TUTORIALS 2015. Software-Defined Networking (SDN) has received a great deal of attention from both academia and industry in recent years. Studies on SDN have brought a number of interesting technical discussions on network architecture design, along with scientific contributions. Researchers, network operators, and vendors are trying to establish new standards and provide guidelines for proper implementation and deployment of such novel approach. It is clear that many of these research efforts have been made in the southbound of the SDN architecture, while the northbound interface still needs improvements. By focusing in the SDN northbound, this paper surveys the body of knowledge and discusses the challenges for developing SDN software. We investigate the existing solutions and identify trends and challenges on programming for SDN environments. We also discuss future developments on techniques, specifications, and methodologies for programmable networks, with the orthogonal view from the Software Engineering discipline.

A Software Engineering Perspective on SDN Programmability

Felipe Alencar

240325_JW_labseminar[node2vec: Scalable Feature Learning for Networks].pptx

thanhdowork

Talk from NVidia Developer Connect

Anuj Gupta

Similar a Natural Language to Visualization by Neural Machine Translation (20)

[論文紹介]A syntactic neural model for general purpose code generation

Recurrent Neural Networks, LSTM and GRU

Nlp and transformer (v3s)

230922 Semantic Exploration from Language Abstractions and Pretrained Represe...

Deep Learning Architectures for NLP (Hungarian NLP Meetup 2016-09-07)

Transformer Mods for Document Length Inputs

node2vec: Scalable Feature Learning for Networks.pptx

Natural Language Interface to Knowledge Graph

ECCV2022 paper reading - MultiMAE: Multi-modal Multi-task Masked Autoencoders...

Superworkflow of Graph Neural Networks with K8S and Fugue

Gnn overview

Resnet.pdf

5_RNN_LSTM.pdf

Introduction to deep learning

A Generalization of Transformer Networks to Graphs.pptx

008 GNNs at Scale With Graph Data Science Sampling and Python Client Integrat...

Transfer Learning for Low Resource Languages and Domains

A Software Engineering Perspective on SDN Programmability

240325_JW_labseminar[node2vec: Scalable Feature Learning for Networks].pptx

Talk from NVidia Developer Connect

Más de ivaderivader

Argument Mining

ivaderivader

Papers at CHI23

ivaderivader

DDGK: Learning Graph Representations for Deep Divergence Graph Kernels

ivaderivader

So Predictable! Continuous 3D Hand Trajectory Prediction in Virtual Reality

ivaderivader

Reinforcement Learning-based Placement of Charging Stations in Urban Road Net...

ivaderivader

Prediction for Retrospection: Integrating Algorithmic Stress Prediction into ...

ivaderivader

Mem2Seq: Effectively Incorporating Knowledge Bases into End-to-End Task-Orien...

ivaderivader

A Style-Based Generator Architecture for Generative Adversarial Networks

ivaderivader

CatchLIve: Real-time Summarization of Live Streams with Stream Content and In...

ivaderivader

Perception! Immersion! Empowerment! Superpowers as Inspiration for Visualization

ivaderivader

Learning to Remember Patterns: Pattern Matching Memory Networks for Traffic F...

ivaderivader

Neural Approximate Dynamic Programming for On-Demand Ride-Pooling

ivaderivader

StoryMap: Using Social Modeling and Self-Modeling to Support Physical Activit...

ivaderivader

Bad Breakdowns, Useful Seams, and Face Slapping: Analysis of VR Fails on YouTube

ivaderivader

Invertible Denoising Network: A Light Solution for Real Noise Removal

ivaderivader

Traffic Demand Prediction Based Dynamic Transition Convolutional Neural Network

ivaderivader

MusicBERT: Symbolic Music Understanding with Large-Scale Pre-Training

ivaderivader

Screen2Vec: Semantic Embedding of GUI Screens and GUI Components

ivaderivader

Augmenting Decisions of Taxi Drivers through Reinforcement Learning for Impro...

ivaderivader

Recommending What Video to Watch Next: A Multitask Ranking System

ivaderivader

Más de ivaderivader (20)