As Hadoop and Big Data technologies take root in the enterprise, they bring new challenges for data orchestration and interaction with traditional environments. The journey of transitioning to a modern data platform involves not only ingesting and offloading data from multiple sources, but also transforming the data, making it instantly available to the business, for on demand analytics. Manual efforts and scripting are no longer practical and projects seem to come to a grinding halt due to failures and delays.
Join this session by Cloudwick and BMC and learn through real-world examples of a proven approach that meet the required standards of ease of use, scalability, security, data governance and service level management.
Introduce myself and a few words about BMC
Big Data project is a journey
This is not a simple process - right tools, the right skills and even the right culture
Enterprise-grade implementations = complex (Data ingestion, Processing, Analytics
In this session : Cloudwick and BMC sharing experience
WE have been in this business of Digital Enterprise Management for over 30 years now
Workload Automation or Scheduling – providing end to end automation of business process across the enterprise
MF, Unix, Linux, ERP, File Transfers, Java, Informatica, Cognos, Hadoop and so on and so on
Analysts recognition
A personal story – Vietnam
Digital Transformation
Technology adoption
Customer expectations
It may be cliche, we are truly living in a digital world
Every speaker nowadays talk about the use of the mobile (flights, check-in, hotel, etc)
Online doctor appointment via video on a mobile app
Oh, and yes occasionally I will use it also for phone calls
Going digital is not an option – it’s a must to maintain and grow the business
You can see the numbers
Davos World Economic Forum – The chairman of BBVA Talking about the Digital disruption …
20,000 banks in the world and many will disappear – those that will not adopt new technologies as he called it – the Uber moment
So, let’s drop all the technologies we have been using for decades and go digital? Yeah? Really? Hmm… not quite
Anyone recognizes the city in the photo? New York, 1904 , what happened in 1904? The first subway was operated ..
But the original subway cars have been replaced, other things like plumbing, lighting, air conditioning, security have been improved as well
Today, the subway is still the heartbeat of the city, moving over 4 million people every day, and tying buildings together
New York City is a great example of building upon layers, rather than leaving them behind
Modernizing the legacy subway infrastructure and at the same time adding the newest modern architectures - like an enterprise IT environment
When going digital or specifically implementing Big Data and IOT initiatives, companies cannot drop existing technology and replace with new
Like the subway in New York, the strong foundation, is what really allows the modernization to success and flourish
So, when we talked to companies implementing Big Data we realized that the challenges they are facing …
How do I schedule my Big Data workflow? how can I identify something went wrong and fix it on time?
How do I assure my SLAs are met? ready for changes? Integrate? secure? scale?
I will talk more about it and about our scheduling solution that helps you address all of these challenges, in a couple of minutes but first, let me turn over to our partner Tina Diamond from Cloudwick who will share from her experience and of how they have been helping customers in these areas
How will Enterprises manage big data -
It is about Building a Data Platform for efficiently, optimised for scale
Enterprises Require: Security, Access and Control with optimum Performance
Scalable architecture in the platform and software
Built on a flexible infrastructure
Advanced analytics and Apps to grow the business
*I would like to introduce C
I am the MD, I joined Cloudwick, a US based OS PS in Jan 2016
Focused on OS, Big Data& Advanced Analytics, delivering On prem, hybrid and cloud environments to Enterprises
*Share old world challenges with new world solutions:
What is our Differentiator: Cloudwick was founded on BD, it is our DNA
Core/Foundation: Network & Consultants
These are not new challenges
New world challenges with Old World Problems still includes:
Building a platform in order to access their data
Security
Governance: with access and control
Predictive / Prescriptive Analytics
Example of a UK customer that Transformed these challenges : UKDA
UKDA – They used old existing way to base the new security… what happened, completely locked down, couldn’t get data in or out
UKDA – Largest Government Research Agency for Social Sciences
Security – 0 Data Loss – DR strategy took time
Hybrid environment accessing data – security –
Synergy between platform and analytics
Strategy: share their architecture and data with other research compn
Going a bit deeper, sharing where The most time and focus was spent understanding and building this solution
Just like in Maslow’s model, each lower stage is a necessary foundation for the stages above it.
If any foundation is weak, the higher stages may crumble and leak even if they initially work. Resources for building each stage involve a combination of human talent and the right set of tools for your needs.
Collect: For a robust data pipeline, you want access to clean, reliable data then manage for - Govern, Quality, Platform, Multi-tenancy & Operations
Manage: Once you have a consistent stream of clean data, you need to have a clear understanding of the data’s composition. Descriptive statistics is where you look into the past and ask questions that have concrete answers to them.
TR: The other question I’m always interested in ,… who holds the key to building?
Both IT & Business hold the key to unlock the value – the goal is the same, to secure, govern allow the individuals that have signing authority to access – once that is there, the individual can access as often as they like –rely on their valuable assets being in the safest and most secure location for them, with back up strategies and insurance in case of a robbery
Business understands what data they want to get to
IT unlocks and makes sure it’s secure and accessible to the business users
Transition: Once the keys are established, how do you build alignment?
IT builds from the bottom up to enable business to drive from the top down
IT Builds for growth, Business demands growth
Automated technology is allowing the Gap to close
Though still about understand the old problems with new solutions
Business is becoming more savvy with Tech
IT are becoming aware of the business
AZ…. Pharma that has both
Once the synergy and conversations unfold, use cases start to emerge
How to you…. Determine the Business value
I see it by prioritsing Use Cases, and calculating 4 areas:
Metrics to improve: application, activation, retention, revenue, referral, operation
Expected improvement: 10x or more, 2x or more, 1-2x, 1x, <1x
% of users affected: 100%, 80-99%, 50-79%, 20-49%, 10-19%, <10%
Frequency of Use Case: Daily, few times/week, weekly, monthly, few times a year, once a year or less
Calculate:
As the previous slide showed, it’s a staged process, building upon each other, going from….
Once you implement batch use cases, streaming will follow and accelerate the value for your compeletive edge
Global Copyright infringment: Use Case / Batch – got them down from 4 hours / match to 1 second
Next implementing Kafka use case
Disrupting revenue streaming –
Streaming is 50 days?
TR: At this time, I’d like to hand over to Alon to share the BMC offering he helps his customers with
We talked the challenges organizations implementing Big Data have and the many of these challenges have been around for a long time
business keeps throwing things at us and it boils down to the needs of the business day
With Control-M we are uniquely positioned to help our customers in reconciling these challenges
Being consistent across all the technologies
EDW offload job or Hadoop Pig or Hive or Scoop or Impala or any other tool to load and manipulate data
Beyond Hadoop
Self Service
Cloud
The process for building the workflows - Simple drag and drop
For example - transfer data, running a Hadoop process like scoop or MapReduce, a few in parallel, creating data and then perhaps another file transfer + apply SLA
That’s it - simple as that to put a workflow together
No need of scripting or the headache to integrate all apps
Automation is a must in order to support today’s modern application delivery
Scripting is just not good enough
The more I talk to orgs implementing Big Data, the more I hear about time wasted on scripting and failures - it doesn’t need 2 b this way
What we offer - automate each in every step of your big data journey, from the data ingestion process, through data processing and analytics
Sky, a British telecommunications – using Big Data to cost-effectively deliver their media services and content
Converted from Jenkins to Control-M (management, speed, reliability)
Uses Control-M for Hadoop to run customer analytics (make realistic estimations of what people are watching)
Estimates a 40% reduction in time from Dev to Prod by replacing Jenkins with Control-M in Development phase
Navistar, a $10B company who is on the latest frontier with big data and the Internet of Things (IoT)
OnCommand Connection system is used to monitor the performance of more than 160,000 trucks
It captures millions of data points from vehicles, including fuel economy, geo location, idle time, and potential failures
They can diagnose issues as they develop and provide guidance to fleet owners—helping them route trucks to dealer service centers
This means less time in the garage and more time on the road, and that translates into a higher return on the investment in the vehicle