AWS re:Invent 2018 - Machine Learning recap (December 2018)
•Download as PPTX, PDF•
3 likes•1,153 views
Talk @ AWS Builders Day London / Dublin
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to AWS re:Invent 2018 - Machine Learning recap (December 2018)
Similar to AWS re:Invent 2018 - Machine Learning recap (December 2018) (20)
More from Julien SIMON
More from Julien SIMON (20)
Recently uploaded
Recently uploaded (20)
AWS re:Invent 2018 - Machine Learning recap (December 2018)
- 1. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Julien Simon Principal Technical Evangelist, AI & Machine Learning, AWS @julsimon * Based on a deck by Dan Mbanga, Global Lead Business Dev. Manager, ML Services AWS re:Invent 2018 New Machine Learning Services
- 2. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. M L F R A M E W O R K S & I N F R A S T R U C T U R E A I S E R V I C E S R E K O G N I T I O N I M A G E P O L L Y T R A N S C R I B E T R A N S L A T E C O M P R E H E N D L E XR E K O G N I T I O N V I D E O Vision Speech Language Chatbots A M A Z O N S A G E M A K E R B U I L D T R A I N F O R E C A S T Forecasting T E X T R A C T P E R S O N A L I Z E Recommendations D E P L O Y Pre-built algorithms & notebooks Data labeling (G R O U N D T R U T H ) One-click model training & tuning Optimization (N E O ) One-click deployment & hosting M L S E R V I C E S F r a m e w o r k s I n t e r f a c e s I n f r a s t r u c t u r e E C 2 P 3 & P 3 N E C 2 C 5 F P G A s G R E E N G R A S S E L A S T I C I N F E R E N C E Reinforcement learningAlgorithms & models ( A W S M A R K E T P L A C E F O R M A C H I N E L E A R N I N G )
- 3. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Threeareas weareimproving for ML developers CostData Cost We’re improving both training and inference speed & cost Data Preparing data for ML is major expensive, complex, and time consuming Ease of use We continue to want to reduce the barrier of entry to ML for all developers
- 4. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. ImprovingTraining& InferenceCost
- 5. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon EC2P3dn instance ThelargestP3instance,optimizedfordistributedtraining https://aws.amazon.com/blogs/aws/new-ec2-p3dn-gpu-instances-with-100-gbps-networking-local-nvme-storage-for-faster-machine-learning-p3-price-reduction/ Reduce machine learning training time Better GPU utilization Support larger, more complex models K E Y F E AT U R E S 100Gbps of networking bandwidth 8 NVIDIA Tesla V100 GPUs 32GB of memory per GPU (2x more P3) 96 Intel Skylake vCPUs (50% more than P3) with AVX-512
- 6. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. MostcostefficientplatformforTensorFlow Stock TensorFlow 65% scaling efficiency with 256 GPUs
- 7. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. MostcostefficientplatformforTensorFlow Stock TensorFlow AWS-Optimized TensorFlow 65% 90% scaling efficiency with 256 GPUs scaling efficiency with 256 GPUs Available with Amazon SageMaker and the AWS Deep Learning AMIs
- 8. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. MostcostefficientplatformforTensorFlow https://aws.amazon.com/about-aws/whats-new/2018/11/tensorflow-scalability-to-256-gpus/ Fastest time for TensorFlow Stock TensorFlow AWS-Optimized TensorFlow 65% 90% scaling efficiency with 256 GPUs scaling efficiency with 256 GPUs 30m 14m training time training time Available with Amazon SageMaker and the AWS Deep Learning AMIs
- 9. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Dynamic training withApache MXNetand RIs https://aws.amazon.com/blogs/machine-learning/introducing-dynamic-training-for-deep-learning-with-amazon-ec2/ Use a variable number of instances for distributed training No loss of accuracy Coming soon spot instances, additional frameworks
- 10. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Training getsalotof attention, but whatabout inference?
- 11. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Prediction Training Inference (Prediction) 90% Training 10% Predictions drive complexity and cost inproduction
- 12. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Thechallengesof prediction in production One size does not fit all Elasticity is important
- 13. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon EC2C5ninstance https://aws.amazon.com/blogs/aws/new-c5n-instances-with-100-gbps-networking/ Intel Xeon Platinum 8000 Up to 3.5GHz single core speed Up to 100Gbit networking Based on Nitro hypervisor for bare metal-like performance
- 14. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon ElasticInference https://aws.amazon.com/blogs/aws/amazon-elastic-inference-gpu-powered-deep-learning-inference-acceleration/ Match capacity to demand Available between 1 to 32 TFLOPS K E Y F E AT U R E S Integrated with Amazon EC2, Amazon SageMaker, and Amazon DL AMIs Support for TensorFlow, Apache MXNet, and ONNX with PyTorch coming soon Single and mixed-precision operations Lower inference costs up to 75%
- 15. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Making iteasiertoobtain highquality labeled data
- 16. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Successfulmodels require high-qualitydata
- 17. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Successfulmodels require high-qualitydata
- 18. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. AmazonSageMakerGroundTruth https://aws.amazon.com/blogs/aws/amazon-sagemaker-ground-truth-build-highly-accurate-datasets-and-reduce-labeling-costs-by-up-to-70
- 19. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Howitworks Raw Data
- 20. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Howitworks
- 21. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Howitworks
- 22. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Howitworks
- 23. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Howitworks
- 24. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Creatingtraining data
- 25. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
- 26. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Driving EaseofUse
- 27. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. AmazonSageMaker: build,train,and deploy ML 1 2 3 1 2 3 Recommendation with Factorization Machines Time-series with Deep AR A lot of expertise is still required Can we make it simpler?
- 28. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Personalize https://aws.amazon.com/blogs/aws/amazon-personalize-real-time-personalization-and-recommendation-for-everyone
- 29. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Forecast https://aws.amazon.com/blogs/aws/amazon-forecast-time-series-forecasting-made-easy/
- 30. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. AWS Marketplacefor MachineLearning MLalgorithmsandmodelsavailableinstantly S E L L E R S B U Y E R S
- 31. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Over 150models and algorithmsavailable Natural Language Processing Grammar & Parsing Text OCR Computer Vision Named Entity Recognition Video Classification Speech Recognition Text-to-Speech Speaker Identification Text Classification 3D Images Anomaly Detection Text Generation Object Detection Regression Text Clustering Handwriting Recognition Ranking S O M E O F T H E A V A I L A B L E A L G O R I T H M S A N D M O D E L S S E L E C T E D V E N D O R S
- 32. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Model optimization isextremelycomplex
- 33. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Trainonce,run anywhere
- 34. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. AmazonSageMaker Neo https://aws.amazon.com/blogs/aws/amazon-sagemaker-neo-train-your-machine-learning-models-once-run-them-anywhere/
- 35. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. What’snext for MachineLearning?
- 36. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. AmazonSageMakerRL Reinforcementlearningforeverydeveloperanddatascientist Broad support for frameworks Broad support for simulation environments including SimuLink and MatLab K E Y F E AT U R E S TensorFlow, Apache MXNet, Intel Coach, and Ray RL support 2D & 3D physics environments and OpenAI Gym support Supports Amazon Sumerian and Amazon RoboMaker Fully managed Example notebooks and tutorials
- 37. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. IntroducingAWS DeepRacer Fullyautonomous1/18thscaleracecar,drivenbyreinforcementlearning HD video camera Dual-core Intel processorFour-wheel drive Dual power for compute and drive AccelerometerGyroscope © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
- 38. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Getting started
- 39. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. MachineLearningUniversity Uses the same materials used to train Amazon developers Foundational knowledge with real-world application Structured courses and specialist certification https://aws.training/machinelearning
- 40. Thank you! © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Julien Simon Principal Technical Evangelist, AI & Machine Learning, AWS @julsimon
Editor's Notes
- 1/ The new Amazon EC2 P3dn instance 2/ With four-times the networking bandwidth and twice the GPU memory of the largest P3 instance, P3dn is ideal for large scale distributed training. No one else has anything close. 3/ P3dn.24xlarge instances offer 96vCPUs of Intel Skylake processors to reduce preprocessing time of data required for machine learning training. 3/ The enhanced networking of the P3n instance allows GPUs to be used more efficiently in multi-node configurations so training jobs complete faster. 4/ Finally, the extra GPU memory allows developers to easily handle more advanced machine learning models such as holding and processing multiple batches of 4k images for image classification and object detection systems
- 1/ in order to take advantage of all their data to train more accurate, more sophisticated models, or if they want to train their existing models quickly – customers usually need to scale their training across multiple GPUs, not just in a single instances, but across multiple instances. 2/ this can often involve hundreds of GPUs. 3/ Unfortunately, the inner workings of TensorFlow makes scaling across GPUs on different instances very inefficient. 4/ For example, at 256 GPUs, TensorFlow is only able to use 65% of the total capacity; that’s incredibly wasteful and expensive. [NOTE: a neural network is made up of hundreds of thousands of weighted network connections; these weights are adjusted over and over again - potentially billions of times during training; when training is distributed across the GPUs you also need to be able to efficiently share and update these weights with all the GPUs; relatively easy on one instance - you can just store them in memory - super hard when you get to tens of hundreds of GPUs). Hard problem.] TRANSITION: Customers told us this was becoming a major problem when working with TF, so we…
- 1/ Today I’m pleased to announce that we have been able to overcome this limitation, significantly improving TensorFlow's scaling efficiency up to 90% across 256 GPUs. 2/ To achieve close to linear scalability across hundreds of GPUs. 3/ We did this by improving the way in which TensorFlow shares the model parameters across multiple instances, making the sharing faster and more efficient between instances. 4/ We got a further improvement through the 100 gig networking available on the new P3dn instances, but the majority of the benefit came from the internal architectural changes we made to the framework itself (nine tenths of the benefits came from TF improvements). 3/ This means that customers can use more GPUs to train more data, in less time. TRANSITION: to give you an idea of the impact of this……
- 1/ Let’s take a look at a common computer vision model for image classification, a deep neural network called ResNet-50 <rez net fifty>, trained on hundreds of thousands of images. 2/ The fastest time to train this model, by a team in Mountain View, was 30 minutes, using a specially built training algorithm which was optimized just for this single neural network, and for specialized hardware which is only available in beta (and not available to most developers). These improvements are locked away from most models, and out of reach of the vast majority of developers. 4/ With the improvements we made in TF, we reduced training time by over 50% to just 14 minutes. This is the fastest time for training ResNet using TensorFlow, anywhere. 5/ But even more importantly, our optimizations can be applied to multiple different models - including convolutional neural networks (images), and recurrent neural networks (language, recommendation). 6/ It also runs on P3 instances, which are globally available to all developers in 14 regions. 7/ Available in SageMaker and the Deep Learning AMI
- 1/ Successful models are built on high-quality training data, and collecting and labeling the training dataset at the start of this workflow still involves a lot of time and effort. 2/ For example, building a computer vision system that is reliable enough to identify objects - such as traffic lights, stop signs, or pedestrians - requires thousands of hours of video recordings, consisting of hundreds of millions of video frames.
- 1/ And each one of these frames must be labeled to build a dataset that is can be used for training 2/This means human labelers first need to evaluate each frame and label objects, such as traffic signals, pedestrians, other vehicles, and even the road, so that the model can learn to identify these objects on its own. 3/ Today, customers distribute the labeling tasks across as many as thousands of human labelers, adding significant overhead and cost because of the sheer scale and complexity of managing so many people, and even then the process takes months. 4/ Further, if the labelers incorrectly label objects, the system will learn from the bad information and make inaccurate predictions, leading to real-world consequences, such as a car’s inability to detect a stop sign. 5/ Customers try to filter out errors with audits and redundant reviews, but this increases the time and cost required. TRANSITION: Increasingly, the time, expense, and complexity required for accurate labeling of large datasets have become so prohibitive that customers abandon their efforts to train new types of models that solve sophisticated problems.
- 1/ It all starts with data - raw data, which as yet, does not have the labels needed; it’s just raw text, or raw images, or speech, without details of what is inside the text, what items are inside the images, or what words and context the speech contains.
- 1/ to get started, Ground Truth will select a small number of diverse data, and send them to humans for annotation; 2/ It can do this through Mechanical Turk, or your own workforce, or a crowdsourcing company. 3/ Ground Truth collects the human labeled data, and builds a special, custom machine learning model. It then starts to run the rest of the raw data through the model.
- 1/ Where the model has a high confidence in the results based on what it has learned so far, it will apply the automatic annotations to the training data.
- 1/ where the model is less confident in the results, it will pass the new data to human annotators to provide labels.
- 1/ these annotations are validated across multiple human annotators, and then contribute to the training data set. 2/ Additionally, those new labels are passed back to the custom model to improve automated labels. 3/ This means that over time, Ground Truth can label more data automatically - only a small percentage of new data will need to be sent to human annotators; the rest can be done automatically.
- 1/ Amazon Mechanical Turk to access either a crowdsourced workforce of over 500,000 workers 2/for data which requires confidentiality, service guarantees, or special skills of pre-authorized workers a private workforce of your own employees 3/ or specific third-party vendors (such as iMerit, Vivitec, Cogito Tech, CapeStart and iVision), which offer a range of prices and geographic availability. This lowers the overall cost of creating new datasets, keeping datasets up to date with newly generated data, and provides a better return on the cost of human annotation, since those annotations automatically contribute to the overall efficiency of the system. We think Ground Truth will significantly change the economics of generating training data so that more data becomes available for machine learning. Really exciting.
- 1/ Once deployed in production SM manages the compute infrastructure on your behalf 2/ Not only does auto scaling, performs health checks, handles node failures under the covers, applies security patches, performs other routine maintenance 3/ ALL with C/W Monitoring and Logging 4/ FINALLY, one other really cool thing w SM is that it’s built in a MODULAR way (can build and train and deploy elsewhere like the edge OR can host models that were trained elsewhere)…Your Choice 5/ We couldn’t be more excited about SM and think it’s a huge playing field leveler for everyday developers and scientists
- 1/From today developers can access over a hundred algorithms and models, covering a remarkable breadth of capabilities from our Marketplace sellers. This is just a subset, but it gives you a sense of the capabilities which are now available; from speaker identification and speech recognition, to video classification and handwriting recognition. As a developer, these are all just a few clicks away. 2/You can just browse or search the AWS Marketplace as normal, select the algorithm or model you want to use 3/ And subscribe in a single click. 4/ new algorithm or model is the available in Amazon SageMaker console, and you can train models using the algorithms, or start running predictions on the pre-trained models immediately.
- These performance and accuracy trade offs are felt most acutely at the edge. 1/ IoT applications are usually running on devices, out there in the real world. This means that the accuracy of models can be felt quickly, and immediately. Consumer IoT applications have a high expectation of accuracy - such as Alexa detecting the wake word reliably - the accuracy of that model really matters to the overall experience. In industrial IoT, devices are often responsible for monitoring and maintaining and core manufacturing processes, or safety. The accuracy of a model here is critical. 2/ Applications running on IoT devices at the edge are commonly very sensitive to latency; it’s part of the reason why customers are running the workload there in the first place, because they can’t afford the round trip to the cloud and back. So any increase in that latency can have a meaningful impact on the success of the device itself. 3/ IoT applications are often incredibly resource constrained, in a way which is much more acute than in the cloud. The devices are smaller, and have less memory and processing power, which is a real problem for machine learning models. 4/ In many cases, IoT applications need to run on very diverse hardware platforms, with a dizzying myriad of processor architectures. To get any sort of performance, developers have to optimize 5/ Finally, one of the key benefits of machine learning can get lost; the ability to continually improve the model. IoT applications are great data generators, and once that data is “ground-truthed”, it can be used to build more sophisticated models. However, if the effort to optimize those improved models for the constraints and diverse hardware at the edge is high, then it’s less likely to happen, and developers are leaving money on the table. A real missed opportunity. 6/ We don’t think that customers should have to choose between accuracy and performance. It’s a false choice, with a high cost. So I’m excited to announce a new feature of SageMaker…
- 1/ There are a lot of demands placed on organizations when dealing with documents. What they typically want to be able to do sounds straightforward… 2/ They want to be able to identify documents in any format; 3/ and then extract text from those documents, accurately. 4/ But there are a whole ton of challenges which make this difficult; such as the variety of forms and formats, and the quality. 5/ The way customers try to overcome this complexity today is by either by manual review (which is accurate, but time consuming and expensive), or 6/ with simple OCR and/or.. 7/ template based data extraction (which is fast, but tends not to be accurate enough, so they end up sending the documents to manual review or verification anyway). TRANSITION: we think there is a better way, and that instead of manual reviews, simplistic OCR, and templates, we can replace that heavy lifting with smart, cheap, powerful machine learning…