Más contenido relacionado dtGateway: Making Apache Apex Operable1. © 2016 DataTorrent
David Yan
PMC Member, Apache Apex
Architect, DataTorrent
June 1, 2016
dtGateway
Making Apache Apex operable
2. © 2016 DataTorrent
What is dtGateway
2
• Web server communicating in JSON via REST API
• You can use the REST API to:
Get performance metrics of Apex application instances, operators,
containers
Retrieve container logs, dynamically change operator properties,
and change the DAG of a running Apex application
Manage Apex application packages
Launch and kill applications
Query and visualize application data in real time
• Powers the DataTorrent RTS Web UI
3. © 2016 DataTorrent
Links
3
• dtGateway Overview:
http://docs.datatorrent.com/dtgateway/
• dtGateway REST API Specification:
http://docs.datatorrent.com/dtgateway_api/
4. © 2016 DataTorrent
Internal Dynamics
4
HDFS
Apex STRAMs
Node
Managers
Resource Manager
dtGateway
REST over
HTTP
REST over
HTTP
Hadoop Java
API
Hadoop Java
API
HTTP
Apex Worker
Containers
RP
C
This is awesome!
5. © 2016 DataTorrent
Security and Access Control
5
• dtGateway by default runs under a dedicated UNIX user “dtadmin”
• Supports front-end authentication: password, Kerberos, LDAP, PAM,
ActiveDirectory and generic JAAS
• Supports back-end Kerberos handshake with Hadoop components
when Hadoop secure mode is turned on.
• When authentication is enabled, dtGateway will impersonate the
logged-in user when launching applications
• RBAC: Users are assigned to roles, and each role has a set of
permissions.
• For more info on security in dtGateway:
http://docs.datatorrent.com/dtgateway_security/
7. © 2016 DataTorrent
Use Case for Monitoring: Ad Tech
7
Metrics captured using REST API
● Number of Planned Containers and Allocated Containers
● Total memory allocated
● Latency, both end-to-end and at individual operator level
● Number of failed containers
● Application Status (Fire alert if FAILED, KILLED)
● Operator status - the # operators in PENDING_DEPLOY state
Open Source Example:
https://github.com/DataTorrent/examples/tree/master/tools
8. © 2016 DataTorrent
Resources
8
• Apache Apex website - http://apex.apache.org/
• Subscribe - http://apex.apache.org/community.html
• Download - http://apex.apache.org/downloads.html
• Twitter - @ApacheApex; Follow - https://twitter.com/apacheapex
• Facebook - https://www.facebook.com/ApacheApex/
• Meetup - http://www.meetup.com/topics/apache-apex
• Startup Program – Free Enterprise License for Startups, Educational Institutions,
Non-Profits - https://www.datatorrent.com/startups/
• Cloud Trial - https://www.datatorrent.com/download/cloud-trial/
9. © 2016 DataTorrent
We Are Hiring
9
• jobs@datatorrent.com
• Back-End Engineers
• Front-End Engineers
• QA Automation Engineers
• Solutions Engineers
• Build and Release Engineers