Hadoop Summit 2013 : Continuous Integration on top of hadoop

•

4 recomendaciones•3,653 vistas

Wisely chen

This topic is on Hadoop Summit 2013 San Jose.

Tecnología

Continuous Integration
on top of hadoop
Wisely Chen & Neal Lee
Tuesday, June 11, 13

Agenda
• Who I am
• Problem
• Solution
• Demo
• Q&A
Tuesday, June 11, 13

Who I am
• Wisely Chen ( thegiive@gmail.com )
• Release manager of Yahoo![Taiwan] shopping and data team
• Love to promote open source tech at Taiwan
• Ruby and Rails : Coscup 2006, Ubisunrise 2007, OSDC 2007
• Puppet : PHPConf 2012 , RubyConf 2012
• Release Practice :Webconf 2013, Coscup 2012
Tuesday, June 11, 13

Who I am
• Neal Lee (@neal_lee)
• Data Engineer at Yahoo![Taiwan]
• Aiming to build up an easy use of self-service BI
platform connecting to Hadoop.
Tuesday, June 11, 13

One click
• Manual commit code to SCM
• And DONE
• Auto unit testing
• Auto push beta for performance testing
• Auto push to production grid
• Auto trigger code
Tuesday, June 11, 13

Continuous Integration
Tuesday, June 11, 13

Continuous Integration
• A software engineering practice
• Maintain code repos
• Automate the build
• Make the build self-testing
• Everyone commit to the baseline everyday
• Every commit should be a build
• Test in a clone of production environment
• Make it easy to get the latest deliverables
• Everyone can see the result of latest build
• Automate deployment
Tuesday, June 11, 13

We focus on
• A software engineering practice
• Maintain code repos
• Automate the build
• Make the build self-testing
• Everyone commit to the baseline everyday
• Every commit should be a build
• Test in a clone of production environment
• Make it easy to get the latest deliverables
• Everyone can see the result of latest build
• Automate deployment
Tuesday, June 11, 13

CI ﬂow
4. CI slave
exec local
UnitTest
7. CI slave
exec
Performanc
11.
CI exec pig
People
DEV Alpha Beta Grid Prod Grid
2. notify CI
5. deploy 8. deploy
CI
Master
1. Commit
Code
SCM
3. Call 6. Call
10. Call
9. git tag
12. notify
user
Tuesday, June 11, 13

CI ﬂow
4. CI slave
exec local
UnitTest
CI slave
exec
Performanc
CI exec pig
People
DEV Alpha Beta Grid Prod Grid
2. notify
CI
CI
Master
1. Commit
Code
SCM
3. Call
5. Notify
user
Tuesday, June 11, 13

CI ﬂow
4. CI slave
exec local
UnitTest
7. CI slave
exec
Performanc
CI exec pig
People
DEV Alpha Beta Grid Prod Grid
2. notify
CI
5. deploy
CI
Master
1. Commit
Code
SCM
3. Call 6. Call
8.Notify
user
Tuesday, June 11, 13

CI ﬂow
4. CI slave
exec local
UnitTest
7. CI slave
exec
Performanc
CI exec pig
People
DEV Alpha Beta Grid Prod Grid
2. notify
CI
5. deploy 8. deploy
CI
Master
1. Commit
Code
SCM
3. Call 6. Call
9. Notify
user
Tuesday, June 11, 13

Unit Test
4. CI slave
exec local
UnitTest
7. CI slave
exec
Performanc
11.
CI exec pig
People
DEV Alpha Beta Grid Prod Grid
2. notify CI
5. deploy 8. deploy
CI
Master
1. Commit
Code
SCM
3. Call 6. Call
10. Call
9. git tag
12. notify
user
Tuesday, June 11, 13

PigUnit
• A simple xUnit framework
• No cluster set up is required in local mode
• Unit testing, regression testing, and rapid
prototyping on the ﬂy
Tuesday, June 11, 13

Using PigUnit
• Coding
• Write PigUnit test case
• Run local PigUnit test
• Push to grid
• Run Pig on grid
• Get right result !
Tuesday, June 11, 13

Unit test is live doc
• Unit test is runnable live doc
• Pass test case and meet previous
requirement
Tuesday, June 11, 13

Performance Test
4. CI slave
exec local
UnitTest
7. CI slave
exec
Performanc
11.
CI exec pig
People
DEV Alpha Beta Grid Prod Grid
2. notify CI
5. deploy 8. deploy
CI
Master
1. Commit
Code
SCM
3. Call 6. Call
10. Call
9. git tag
12. notify
user
Tuesday, June 11, 13

Vaidya
• Rule based performance diagnosis of M/R jobs
• Extensible framework
• You can add your own rules
• Write complex rules using existing rules
Tuesday, June 11, 13

CI toolset
CI slave
exec local
UnitTest
CI slave
exec
Performanc
CI exec pig
People
DEV Alpha Beta Grid Prod Grid
notify CI
deploy deploy
CI
Commit
Code
SCM
Call
Vaidya
BASH
Tuesday, June 11, 13

CI is ﬂexible
• MapReduce can use MapUnit
• Hive can use hive_test
• Pig can use PigUnit
Tuesday, June 11, 13

CI testing build pipeline
Tuesday, June 11, 13

Conclusion
• Auto testing will save your life
• CI will boost your productivity
• This process can feed in any platform
Tuesday, June 11, 13

Más contenido relacionado

La actualidad más candente

Karim Fanadka

CodeFest

Smarter deployments with octopus deploy

Thibaud Gravrand

Building Web Apps in Ratpack

Daniel Woods

Intro to Ratpack (CDJDN 2015-01-22)

David Carr

A commonly used version control system in the ColdFusion community is Subversion -- a centralized system that relies on being connected to a central server. The next generation version control systems are “decentralized”, in that version control tasks do not rely on a central server. Decentralized version control systems are more efficient and offer a more practical way of software development. In this session, Indy takes you through the considerations in moving from Subversion to Git, a decentralized version control system. You also get to understand the pros and cons of each and hear of the practical experience of migrating projects to decentralized version control. Version control is often used in conjunction with a testing framework and continuous integration. To complete the picture, Indy walks you through how to integrate Git with a testing framework, MXUnit, and a continuous integration server, Hudson.

Make It Cooler: Using Decentralized Version Control

indiver

There's more to Ratpack than non-blocking

Marcin Erdmann

Automated acceptance test

Bryan Liu

Releasing open-source libraries is more than sharing your GitHub URL with the world. There are many considerations and steps involved especially for successful and long-lived projects. In this talk, we’ll cover the principles behind creating, releasing, and maintaining high-quality libraries. Topics will include structuring the repository, implementing modern PHP standards, maintaining changelogs, using CI tests, releasing new versions, and more.

Releasing High Quality Packages - Longhorn PHP 2021

Colin O'Dell

Continuous delivery - tools and techniques

Mike McGarr

Continuous Delivery - Devoxx Morocco 2016

Rafał Leszko

Continuous Delivery - Voxxed Days Thessaloniki 21.10.2016

Rafał Leszko

Every customer-facing project aims to achieve high uptime. We, as developers, write thorough unit and integration tests to catch as many bugs as possible. We even set up staging environments to simulate production as well as we can. Yet, downtime still happens and not only once we had to revert a change. Blue-Green deployment is a technique that allows us to ship code with high confidence, minimizing the risk of errors. We will go through the why and what of Blue-Green,as well as show how you can start doing it using Spinnaker. BDEVOPS @ Adobe 11.04.2017

Safe deployments with Blue-Green and Spinnaker

Mihnea Dobrescu-Balaur

Speed up your regression and reduce cost load with Selenoid + K8s + ReportPortal

Danylo Kuvshynov

dotnetsheff: Continuous delivery with Team City and Octopus Deploy

Kevin Kuszyk

Developing in the Fastlane -> How LookLive uses Fastlane to automate and spee...

Donny Wals

DevOps 及 TDD 開發流程哲學

謝宗穎

Ratpack Web Framework

Daniel Woods

Continuous delivery in Qbon

Jaric Kuo

Perl Continous Integration

Michael Peters

Using Docker for Testing

Mukta Aphale

La actualidad más candente (20)

Karim Fanadka

Smarter deployments with octopus deploy

Building Web Apps in Ratpack

Intro to Ratpack (CDJDN 2015-01-22)

Make It Cooler: Using Decentralized Version Control

There's more to Ratpack than non-blocking

Automated acceptance test

Releasing High Quality Packages - Longhorn PHP 2021

Continuous delivery - tools and techniques

Continuous Delivery - Devoxx Morocco 2016

Continuous Delivery - Voxxed Days Thessaloniki 21.10.2016

Safe deployments with Blue-Green and Spinnaker

Speed up your regression and reduce cost load with Selenoid + K8s + ReportPortal

dotnetsheff: Continuous delivery with Team City and Octopus Deploy

Developing in the Fastlane -> How LookLive uses Fastlane to automate and spee...

DevOps 及 TDD 開發流程哲學

Ratpack Web Framework

Continuous delivery in Qbon

Perl Continous Integration

Using Docker for Testing

Destacado

Road to sbt 1.0 paved with server

Eugene Yokota

Road to sbt 1.0: Paved with server (2015 Amsterdam)

Eugene Yokota

SBT Crash Course

Michal Bigos

FiloDB - Breakthrough OLAP Performance with Cassandra and Spark

Evan Chan

You won't find this in many places - an overview of deploying, configuring, and running Apache Spark, including Mesos vs YARN vs Standalone clustering modes, useful config tuning parameters, and other tips from years of using Spark in production. Also, learn about the Spark Job Server and how it can help your organization deploy Spark as a RESTful service, track Spark jobs, and enable fast queries (including SQL!) of cached RDDs.

Productionizing Spark and the Spark Job Server

Evan Chan

Big Data Architecture and Deployment

Cisco Canada

Devoxx 2016 Using Jenkins, Gerrit and Spark for Continuous Delivery Analytics

Luca Milanesio

MLLeap, or How to Productionize Data Science Workflows Using Spark by Mikha...

Spark Summit

Understanding Data Consistency in Apache Cassandra

DataStax

Strata NYC 2015: Sketching Big Data with Spark: randomized algorithms for lar...

Databricks

Hadoop on Docker

Rakesh Saha

MongoDB as a Data Warehouse: Time Series and Device History Data (Medtronic)

MongoDB

Kafka Reliability - When it absolutely, positively has to be there

Gwen (Chen) Shapira

Dealing with Changed Data in Hadoop

DataWorks Summit

An Introduction to Sparkling Water by Michal Malohlava

Spark Summit

Organizations need to perform increasingly complex analysis on their data — streaming analytics, ad-hoc querying and predictive analytics — in order to get better customer insights and actionable business intelligence. However, the growing data volume, speed, and complexity of diverse data formats make current tools inadequate or difficult to use. Apache Spark has recently emerged as the framework of choice to address these challenges. Spark is a general-purpose processing framework that follows a DAG model and also provides high-level APIs, making it more flexible and easier to use than MapReduce. Thanks to its use of in-memory datasets (RDDs), embedded libraries, fault-tolerance, and support for a variety of programming languages, Apache Spark enables developers to implement and scale far more complex big data use cases, including real-time data processing, interactive querying, graph computations and predictive analytics. In this session, we present a technical deep dive on Spark running on Amazon EMR. You learn why Spark is great for ad-hoc interactive analysis and real-time stream processing, how to deploy and tune scalable clusters running Spark on Amazon EMR, how to use EMRFS with Spark to query data directly in Amazon S3, and best practices and patterns for Spark on Amazon EMR.

(BDT309) Data Science & Best Practices for Apache Spark on Amazon EMR

Amazon Web Services

Companies in every industry look for ways to explore new data types and large data sets that were previously too big to capture, store and process. They need to unlock insights from data such as clickstream, geo-location, sensor, server log, social, text and video data. However, becoming a data-first enterprise comes with many challenges. Join this webinar organized by three leaders in their respective fields and learn from our experts how you can accelerate the implementation of a scalable, cost-efficient and robust Big Data solution. Cisco, Hortonworks and Red Hat will explore how new data sets can enrich existing analytic applications with new perspectives and insights and how they can help you drive the creation of innovative new apps that provide new value to your business.

A Comprehensive Approach to Building your Big Data - with Cisco, Hortonworks ...

Hortonworks

Not Your Father's Database by Vida Ha

Spark Summit

Running Spark in Production

DataWorks Summit/Hadoop Summit

Testing of Hadoop, NoSQL and Data Warehouses Visually ----------------------------------------------------------------------------- We just made automated data testing really easy. Automate your Big Data testing visually, with no programming needed. See how to automate Hadoop, No SQL and Data Warehouse testing visually, without writing any SQL or HQL. See how QuerySurge, the leading Big Data testing solution, provides novices and non-technical team members with a fast & easy way to be productive immediately while speeding up testing for team members skilled in SQL/HQL. This webinar is geared towards: - Big Data & Data Warehouse Architects, ETL Developers - ETL Testers, Big Data Testers - Data Analysts - Operations teams - Business Intelligence (BI) Architects - Data Management Officers & Directors You will learn how to: • Improve your Data Quality • Accelerate your data testing cycles • Reduce your costs & risks • Realize a huge ROI

Big Data Testing : Automate theTesting of Hadoop, NoSQL & DWH without Writing...

RTTS

Destacado (20)

Road to sbt 1.0 paved with server

Road to sbt 1.0: Paved with server (2015 Amsterdam)

SBT Crash Course

FiloDB - Breakthrough OLAP Performance with Cassandra and Spark

Productionizing Spark and the Spark Job Server

Big Data Architecture and Deployment

Devoxx 2016 Using Jenkins, Gerrit and Spark for Continuous Delivery Analytics

MLLeap, or How to Productionize Data Science Workflows Using Spark by Mikha...

Understanding Data Consistency in Apache Cassandra

Strata NYC 2015: Sketching Big Data with Spark: randomized algorithms for lar...

Hadoop on Docker

MongoDB as a Data Warehouse: Time Series and Device History Data (Medtronic)

Kafka Reliability - When it absolutely, positively has to be there

Dealing with Changed Data in Hadoop

An Introduction to Sparkling Water by Michal Malohlava

(BDT309) Data Science & Best Practices for Apache Spark on Amazon EMR

A Comprehensive Approach to Building your Big Data - with Cisco, Hortonworks ...

Not Your Father's Database by Vida Ha

Running Spark in Production

Big Data Testing : Automate theTesting of Hadoop, NoSQL & DWH without Writing...

Similar a Hadoop Summit 2013 : Continuous Integration on top of hadoop

Continuous integration

Basma Alkerm

Test Driven Development & CI/CD

Shanmuga S Muthu

Introduction to Continuous Integration

Zahra Golmirzaei

Presented at STPCon 2016. With the extensive amount of testing performed nightly on large software projects, test and verification teams often experience lengthy wait times for the availability of test results of the latest build. As we strive to identify and resolve issues as fast as possible, alternative methods of test execution have to be found. Learn how to use Jenkins to launch tests in parallel across a number of Virtual Machines, monitor execution health, and process results. Learn about various Jenkins plugins and how they contributed to the solution. Learn how to trigger downstream jobs, even if they are on separate Jenkins instances.

Test parallelization using Jenkins

Rogue Wave Software

TYPO3 Camp Stuttgart 2015 - Continuous Delivery with Open Source Tools

Michael Lihs

A Continuous Integration system is often considered one of the key elements involved in supporting an agile software development and testing environment. As a traditional software tester transitioning to an agile development environment it became clear to me that I would need to put this essential infrastructure in place and promote improved development practices in order to make the transition to agile testing possible. This experience report discusses a continuous integration implementation I led last year. The initial motivations for implementing continuous integration are discussed and a pre and post-assessment using Martin Fowler's "Practices of Continuous Integration" is provided along with the technical specifics of the implementation. The report concludes with a retrospective of my experiences implementing and promoting continuous integration within the context of agile testing.

Enabling Agile Testing Through Continuous Integration Agile2009

sstolberg

Automated Visual Regression Testing by Dave Sadlon

QA or the Highway

AgileLINC Continous Slides by Daniel Harp

Barry Gavril

Continuous integration & deployment

Alan Harper

Atagg2015 Continuous delivery by building environment using docker

Agile Testing Alliance

Continuous Integration and Delivery

Brandon Cornett

Towards Continuous Delivery in Mobile Apps

Phillip Wheatley

Continuous integration

Dennis van der Stelt

From Renamer Plugin to Polyglot IDE

intelliyole

Continuous Delivery Using Jenkins

Cliffano Subagio

DevOps / Agile Tools Seminar 2013

Ethan Ram

In the world of fast changing technologies and business requirements there is no way to stay with old-school approaches to develop software. Common, leave manual interaction behind, let machine do what it's best capable for and spend your time to solve real problems. During this talk I'm going to demonstrate how to build a sophisticated continuous development pipeline for Java based project to cover things like build, test, code quality, deploy and delivery using Gradle, Jenkins, SonarQube, Docker and Ansible. As an outcome of this presentation there will be an open source project with source code.

Continuous Development Pipeline

Izzet Mustafaiev

What does it mean to 'do' Continuous Integration? It used to be enough to execute your unit tests in CI. But the bar is steadily raising for engineering practices. In the last decade we've seen tremendous improvements inacceptance testing. JavaScript is now a platform in it's own right. Cloudcomputing is now vital. There's growing interest in deployment to prod.So Continuous Integration is under more pressure than ever. As the bar slowly raises for engineering practices, we ll present 2011's minimum viable feature set for Continuous Integration

Continuous Integration, the minimum viable product

Julian Simpson

Intro to PHP Testing

Ran Mizrahi

How we realized SOA by Python at PyCon JP 2015

hirokiky

Similar a Hadoop Summit 2013 : Continuous Integration on top of hadoop (20)

Continuous integration

Test Driven Development & CI/CD

Introduction to Continuous Integration

Test parallelization using Jenkins

TYPO3 Camp Stuttgart 2015 - Continuous Delivery with Open Source Tools

Enabling Agile Testing Through Continuous Integration Agile2009

Automated Visual Regression Testing by Dave Sadlon

AgileLINC Continous Slides by Daniel Harp

Continuous integration & deployment

Atagg2015 Continuous delivery by building environment using docker

Continuous Integration and Delivery

Towards Continuous Delivery in Mobile Apps

Continuous integration

From Renamer Plugin to Polyglot IDE

Continuous Delivery Using Jenkins

DevOps / Agile Tools Seminar 2013

Continuous Development Pipeline

Continuous Integration, the minimum viable product

Intro to PHP Testing

How we realized SOA by Python at PyCon JP 2015

Último

presentation ICT roal in 21st century education

jfdjdjcjdnsjd

Architecting Cloud Native Applications

WSO2

The value of a flexible API Management solution for Open Banking Steve Melan, Manager for IT Innovation and Architecture - State's and Saving's Bank of Luxembourg Apidays New York 2024: The API Economy in the AI Era (April 30 & May 1, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays New York 2024 - The value of a flexible API Management solution for O...

apidays

DBX First Quarter 2024 Investor Presentation

Dropbox

Strategies for Landing an Oracle DBA Job as a Fresher

Remote DBA Services

The action of the next cyber saga takes place in the mystical lands of the Asia-Pacific region, where the main characters began their digital activities in the middle of 2021 and qualitatively strengthened it in 2022. Corporate espionage, document theft, audio recordings, and data leaks from messaging platforms were all a matter of one day for Dark Pink. Their geographical focus may have started in the Asia-Pacific region, but their ambitions knew no bounds, targeting a European government ministry in a bold move to expand their portfolio. Their victim profile was as diverse as a UN meeting, targeting military organizations, government agencies, and even a religious organization. Because discrimination is not a fashionable agenda. In the world of cybercrime, they serve as a reminder that sometimes the most serious threats come in the most unassuming packages with a pink bow.

Cyberprint. Dark Pink Apt Group [EN].pdf

Overkill Security

2024: Domino Containers - The Next Step. News from the Domino Container commu...

Martijn de Jong

"I see eyes in my soup": How Delivery Hero implemented the safety system for ...

Zilliz

EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER

MadyBayot

Tracing the root cause of a performance issue requires a lot of patience, experience, and focus. It’s so hard that we sometimes attempt to guess by trying out tentative fixes, but that usually results in frustration, messy code, and a considerable waste of time and money. This talk explains how to correctly zoom in on a performance bottleneck using three levels of profiling: distributed tracing, metrics, and method profiling. After we learn to read the JVM profiler output as a flame graph, we explore a series of bottlenecks typical for backend systems, like connection/thread pool starvation, invisible aspects, blocking code, hot CPU methods, lock contention, and Virtual Thread pinning, and we learn to trace them even if they occur in library code you are not familiar with. Attend this talk and prepare for the performance issues that will eventually hit any successful system. About authorWith two decades of experience, Victor is a Java Champion working as a trainer for top companies in Europe. Five thousands developers in 120 companies attended his workshops, so he gets to debate every week the challenges that various projects struggle with. In return, Victor summarizes key points from these workshops in conference talks and online meetups for the European Software Crafters, the world’s largest developer community around architecture, refactoring, and testing. Discover how Victor can help you on victorrentea.ro : company training catalog, consultancy and YouTube playlists.

Finding Java's Hidden Performance Traps @ DevoxxUK 2024

Victor Rentea

ICT role in 21st century education and its challenges

rafiqahmad00786416

Webinar Recording: https://www.panagenda.com/webinars/why-teams-call-analytics-is-critical-to-your-entire-business Nothing is as frustrating and noticeable as being in an important call and being unable to see or hear the other person. Not surprising then, that issues with Teams calls are among the most common problems users call their helpdesk for. Having in depth insight into everything relevant going on at the user’s device, local network, ISP and Microsoft itself during the call is crucial for good Microsoft Teams Call quality support. To ensure a quick and adequate solution and to ensure your users get the most out of their Microsoft 365. But did you know that ‘bad calls’ are also an excellent indicator of other problems arising? Precisely because it is so noticeable!? Like the canary in the mine, bad calls can be early indicators of problems. Problems that might otherwise not have been noticed for a while but can have a big impact on productivity and satisfaction. Join this session by Christoph Adler to learn how true Microsoft Teams call quality analytics helped other organizations troubleshoot bad calls and identify and fix problems that impacted Teams calls or the use of Microsoft365 in general. See what it can do to keep your users happy and productive! In this session we will cover - Why CQD data alone is not enough to troubleshoot call problems - The importance of attributing call problems to the right call participant - What call quality analytics can do to help you quickly find, fix-, and prevent problems - Why having retrospective detailed insights matters - Real life examples of how others have used Microsoft Teams call quality monitoring to problem shoot problems with their ISP, network, device health and more.

Why Teams call analytics are critical to your entire business

panagenda

Keynote 2: APIs in 2030: The Risk of Technological Sleepwalk Paolo Malinverno, Growth Advisor - The Business of Technology Apidays New York 2024: The API Economy in the AI Era (April 30 & May 1, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays New York 2024 - APIs in 2030: The Risk of Technological Sleepwalk by ...

apidays

Scaling API-first – The story of a global engineering organization Ian Reasor, Senior Computer Scientist - Adobe Radu Cotescu, Senior Computer Scientist - Adobe Apidays New York 2024: The API Economy in the AI Era (April 30 & May 1, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe

apidays

ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke

Product Anonymous

[BuildWithAI] Introduction to Gemini.pdf

Sandro Moreira

💥 You’re lucky! We’ve found two different (lead) developers that are willing to share their valuable lessons learned about using UiPath Document Understanding! Based on recent implementations in appealing use cases at Partou and SPIE. Don’t expect fancy videos or slide decks, but real and practical experiences that will help you with your own implementations. 📕 Topics that will be addressed: • Training the ML-model by humans: do or don't? • Rule-based versus AI extractors • Tips for finding use cases • How to start 👨‍🏫👨‍💻 Speakers: o Dion Morskieft, RPA Product Owner @Partou o Jack Klein-Schiphorst, Automation Developer @Tacstone Technology

DEV meet-up UiPath Document Understanding May 7 2024 Amsterdam

UiPathCommunity

Join our latest Connector Corner webinar to discover how UiPath Integration Service revolutionizes API-centric automation in a 'Quote to Cash' process—and how that automation empowers businesses to accelerate revenue generation. A comprehensive demo will explore connecting systems, GenAI, and people, through powerful pre-built connectors designed to speed process cycle times. Speakers: James Dickson, Senior Software Engineer Charlie Greenberg, Host, Product Marketing Manager

Connector Corner: Accelerate revenue generation using UiPath API-centric busi...

DianaGray10

MINDCTI Revenue Release Quarter One 2024

MIND CTI

Sidekick Solutions uses Bonterra Impact Management (fka Social Solutions Apricot) and automation solutions to integrate data for business workflows. We believe integration and automation are essential to user experience and the promise of efficient work through technology. Automation is the critical ingredient to realizing that full vision. We develop integration products and services for Bonterra Case Management software to support the deployment of automations for a variety of use cases. This video focuses on the deployment of external web forms using Jotform for Bonterra Impact Management. This solution can be customized to your organization’s needs and deployed to support the common use cases below: - Intake and consent - Assessments - Surveys - Applications - Program registration Interested in deploying web form automations for Bonterra Impact Management? Contact us at sales@sidekicksolutionsllc.com to discuss next steps.

Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...

Jeffrey Haguewood

Hadoop Summit 2013 : Continuous Integration on top of hadoop

1. Continuous Integration on top of hadoop Wisely Chen & Neal Lee Tuesday, June 11, 13

2. Agenda • Who I am • Problem • Solution • Demo • Q&A Tuesday, June 11, 13

3. Who I am • Wisely Chen ( thegiive@gmail.com ) • Release manager of Yahoo![Taiwan] shopping and data team • Love to promote open source tech at Taiwan • Ruby and Rails : Coscup 2006, Ubisunrise 2007, OSDC 2007 • Puppet : PHPConf 2012 , RubyConf 2012 • Release Practice :Webconf 2013, Coscup 2012 Tuesday, June 11, 13

4. Who I am • Neal Lee (@neal_lee) • Data Engineer at Yahoo![Taiwan] • Aiming to build up an easy use of self-service BI platform connecting to Hadoop. Tuesday, June 11, 13

5. Story 1 Tuesday, June 11, 13

6. Another Story Tuesday, June 11, 13

7. Yet Another Story Tuesday, June 11, 13

8. Solution Tuesday, June 11, 13

9. One click • Manual commit code to SCM • And DONE • Auto unit testing • Auto push beta for performance testing • Auto push to production grid • Auto trigger code Tuesday, June 11, 13

10. This feeling is 爽! Tuesday, June 11, 13

11. Continuous Integration Tuesday, June 11, 13

12. Continuous Integration • A software engineering practice • Maintain code repos • Automate the build • Make the build self-testing • Everyone commit to the baseline everyday • Every commit should be a build • Test in a clone of production environment • Make it easy to get the latest deliverables • Everyone can see the result of latest build • Automate deployment Tuesday, June 11, 13

13. We focus on • A software engineering practice • Maintain code repos • Automate the build • Make the build self-testing • Everyone commit to the baseline everyday • Every commit should be a build • Test in a clone of production environment • Make it easy to get the latest deliverables • Everyone can see the result of latest build • Automate deployment Tuesday, June 11, 13

14. CI ﬂow 4. CI slave exec local UnitTest 7. CI slave exec Performanc 11. CI exec pig People DEV Alpha Beta Grid Prod Grid 2. notify CI 5. deploy 8. deploy CI Master 1. Commit Code SCM 3. Call 6. Call 10. Call 9. git tag 12. notify user Tuesday, June 11, 13

15. CI ﬂow 4. CI slave exec local UnitTest CI slave exec Performanc CI exec pig People DEV Alpha Beta Grid Prod Grid 2. notify CI CI Master 1. Commit Code SCM 3. Call 5. Notify user Tuesday, June 11, 13

16. CI ﬂow 4. CI slave exec local UnitTest 7. CI slave exec Performanc CI exec pig People DEV Alpha Beta Grid Prod Grid 2. notify CI 5. deploy CI Master 1. Commit Code SCM 3. Call 6. Call 8.Notify user Tuesday, June 11, 13

17. CI ﬂow 4. CI slave exec local UnitTest 7. CI slave exec Performanc CI exec pig People DEV Alpha Beta Grid Prod Grid 2. notify CI 5. deploy 8. deploy CI Master 1. Commit Code SCM 3. Call 6. Call 9. Notify user Tuesday, June 11, 13

18. Unit Test 4. CI slave exec local UnitTest 7. CI slave exec Performanc 11. CI exec pig People DEV Alpha Beta Grid Prod Grid 2. notify CI 5. deploy 8. deploy CI Master 1. Commit Code SCM 3. Call 6. Call 10. Call 9. git tag 12. notify user Tuesday, June 11, 13

19. PigUnit • A simple xUnit framework • No cluster set up is required in local mode • Unit testing, regression testing, and rapid prototyping on the ﬂy Tuesday, June 11, 13

20. Using PigUnit • Coding • Write PigUnit test case • Run local PigUnit test • Push to grid • Run Pig on grid • Get right result ! Tuesday, June 11, 13

21. Unit test is live doc • Unit test is runnable live doc • Pass test case and meet previous requirement Tuesday, June 11, 13

22. Performance Test 4. CI slave exec local UnitTest 7. CI slave exec Performanc 11. CI exec pig People DEV Alpha Beta Grid Prod Grid 2. notify CI 5. deploy 8. deploy CI Master 1. Commit Code SCM 3. Call 6. Call 10. Call 9. git tag 12. notify user Tuesday, June 11, 13

23. Vaidya • Rule based performance diagnosis of M/R jobs • Extensible framework • You can add your own rules • Write complex rules using existing rules Tuesday, June 11, 13

24. CI toolset CI slave exec local UnitTest CI slave exec Performanc CI exec pig People DEV Alpha Beta Grid Prod Grid notify CI deploy deploy CI Commit Code SCM Call Vaidya BASH Tuesday, June 11, 13

25. CI is ﬂexible • MapReduce can use MapUnit • Hive can use hive_test • Pig can use PigUnit Tuesday, June 11, 13

26. Github trigger CI Tuesday, June 11, 13

27. CI testing build pipeline Tuesday, June 11, 13

28. Testing Trend Tuesday, June 11, 13

29. DEMO Tuesday, June 11, 13

30. Conclusion • Auto testing will save your life • CI will boost your productivity • This process can feed in any platform Tuesday, June 11, 13

31. 謝謝大家 Tuesday, June 11, 13