"Using Deep Learning for Video Event Detection on a Compute Budget," a Presentation from PathPartner Technology
•
1 recomendación•818 vistas
For the full video of this presentation, please visit: https://www.embedded-vision.com/platinum-members/pathpartner/embedded-vision-training/videos/pages/may-2019-embedded-vision-summit For more information about embedded vision, please visit: http://www.embedded-vision.com Praveen Nayak, Tech Lead at PathPartner Technology, presents the "Using Deep Learning for Video Event Detection on a Compute Budget" tutorial at the May 2019 Embedded Vision Summit. Convolutional neural networks (CNNs) have made tremendous strides in object detection and recognition in recent years. However, extending the CNN approach to understanding of video or volumetric data poses tough challenges, including trade-offs between representation quality and computational complexity, which is of particular concern on embedded platforms with tight computational budgets. This presentation explores the use of CNNs for video understanding. Nayak reviews the evolution of deep representation learning methods involving spatio- temporal fusion from C3D to Conv-LSTMs for vision-based human activity detection. He proposes a decoupled alternative to this fusion, describing an approach that combines a low-complexity predictive temporal segment proposal model and a fine-grained (perhaps high- complexity) inference model. PathPartner Technology finds that this hybrid approach, in addition to reducing computational load with minimal loss of accuracy, enables effective solutions to these high complexity inference tasks.
Recomendados
Recomendados
Más contenido relacionado
La actualidad más candente
La actualidad más candente (20)
Similar a "Using Deep Learning for Video Event Detection on a Compute Budget," a Presentation from PathPartner Technology
Similar a "Using Deep Learning for Video Event Detection on a Compute Budget," a Presentation from PathPartner Technology (20)
Más de Edge AI and Vision Alliance
Más de Edge AI and Vision Alliance (20)
Último
Último (20)
"Using Deep Learning for Video Event Detection on a Compute Budget," a Presentation from PathPartner Technology
- 1. © 2019 Pathpartner Technology Using Deep Learning for Video Event Detection on a Compute Budget Praveen Nayak Pathpartner Technology May 2019
- 2. © 2019 Pathpartner Technology Outline • Introduction to Video Event Detection • Learning Representations from Video • From Video Representation to Event Detection • Decoupling the “When” and “What” • Results, case study on UCF101-Thumos2015 challenge • Conclusion 2
- 3. © 2019 Pathpartner Technology Introduction to Video Event Detection 3
- 4. © 2019 Pathpartner Technology Video data as viewed by ML • Video is a 3D signal • Spatial Coordinates x,y (limited by WxH) • Temporal Coordinates t (limited by T) • If we fix t, we obtain an image/frame • We can understand videos as sequence of images 4 Introduction
- 5. © 2019 Pathpartner Technology Event Detection • Retrieve start (tstart) and end (tend)points of “event” from temporally “untrimmed” video • Evaluation metric: mAP and Recall for a given temporal IoU (tIoU) 5 Introduction to Event Detection nI nU 𝑡𝐼𝑂𝑈 = 𝑛𝐼 𝑛𝑈
- 6. © 2019 Pathpartner Technology Learning Representations from Video 6
- 7. © 2019 Pathpartner Technology Spatiotemporal fusion networks 7 Learning representations from Video Image: Kim et. al, Weighing classes and streams: toward better methods for two-stream convolutional networks
- 8. © 2019 Pathpartner Technology Convolutions for spatiotemporal data 8 Learning representations from Video • C3D model: All convs are 3D, Fewer parameters than 2D convolutions over multiple frames Image: D.Tran et. al, Learning Spatiotemporal Features with 3D Convolutional Networks C3D feature vector 3D Convolutions
- 9. © 2019 Pathpartner Technology Convolutions for spatiotemporal data 9 Learning representations from Video • C3D model: All convs are 3D, Fewer parameters than 2D convolutions over multiple frames. Image: D.Tran et. al, Learning Spatiotemporal Features with 3D Convolutional Networks C3D feature vector 3D Convolutions Joint Appearance and Motion features at every layer
- 10. © 2019 Pathpartner Technology State-of-the-art video descriptors 10 Learning representations from Video Image: J Carriera et al, Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset
- 11. © 2019 Pathpartner Technology Action Classification Datasets 11 Learning representations from Video Dataset Action Classes # Clips Temporal trimming HMDB-51[1] 51 ~7k Yes UCF-101[2] 101 ~13k Yes Kinetics[3] 400 ~160k No • The classification metric: mAP, similar to image based classification metric, extended in temporal domain • Classifier to make one decision per clip UCF-101 HMDB-51 Kinetics [1] UCF-101, University of Central Florida, https://www.crcv.ucf.edu/data/UCF101.php [2] HMDB-51, Brown university (http://serre-lab.clps.brown.edu/resource/hmdb-a-large- human-motion-database/) [3] Kinetics dataset, Deepmind (https://deepmind.com/research/open-source/open- source-datasets/kinetics/)
- 12. © 2019 Pathpartner Technology State-of-the-art video descriptors 12 Learning representations from Video Image and Table: J Carriera et al, Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset
- 13. © 2019 Pathpartner Technology From Video Representation to Event Detection 13
- 14. © 2019 Pathpartner Technology From RCNN to Segment-CNN 14 From Video Representation to Event Detection Image: Z. Shou, Temporal Action Localization in Untrimmed Videos via Multi-stage CNNs
- 15. © 2019 Pathpartner Technology From RCNN to Segment-CNN 15 From Video Representation to Event Detection Image: Z. Shou, Temporal Action Localization in Untrimmed Videos via Multi-stage CNNs Computationally very intensive!
- 16. © 2019 Pathpartner Technology TAL-NET: Parallels from Faster R-CNN • Problem split : class-sensitive proposal generation, followed by inference 16 From Video representations to Video event detection Faster-RCNN TAL-NET Image: Y. Chao et. Al, Rethinking the Faster R-CNN Architecture for Temporal Action Localization
- 17. © 2019 Pathpartner Technology TAL-NET: Parallels from Faster R-CNN • Problem split : class-sensitive proposal generation, followed by inference 17 From Video representations to Video event detection Faster-RCNN TAL-NET Image: Y. Chao et. Al, Rethinking the Faster R-CNN Architecture for Temporal Action Localization End-to-end-Trainable
- 18. © 2019 Pathpartner Technology Computational cost of Detection 18 From Video Representations to Video event detection Model GMAC/ inference #params (Million) GMAC/ video VGG 15 138 46.5k C3D- SCNN[1] 79 80 237k C3D-LSTM[2] 24 86 72k TAL-NET[3] 29 98 87k SSAD[4] 61 356 183k • Template size: 1. C3D, TAL-NET:171x128x16 2. VGG: 224x224x3 • GMAC/inference: could be for frame-level inference (VGG) or clip- level inference (C3D) • GMAC/video: assumes average video length of 3000 frames GMACs per video of event detectors [1] Temporal Action Localization in Untrimmed Videos via Multi-stage CNNs http://dvmmweb.cs.columbia.edu/files/dvmm_scnn_paper.pdf [2] Temporal Activity Detection in Untrimmed Videos with Recurrent Neural Networks https://imatge-upc.github.io/activitynet-2016-cvprw/ [3] Rethinking the Faster R-CNN Architecture for Temporal Action Localization https://arxiv.org/pdf/1804.07667.pdf [4] Single Shot action detection https://arxiv.org/abs/1710.06236 Accuracy vs GMACs / video of event detectors
- 19. © 2019 Pathpartner Technology Decoupling the “When” and “What” 19
- 20. © 2019 Pathpartner Technology Proposed event detection system • Break down problem into two parts: 1. Class agnostic segment proposal (low complexity) 2. Video segment inference (can be high complexity depending on nature of video) 20 Decoupling the “when” and “what”
- 21. © 2019 Pathpartner Technology Proposed event detection system • Break down problem into two parts: 1. Class agnostic segment proposal (low complexity) 2. Video segment inference (can be high complexity depending on nature of video) • Inference model may be long or short term temporal, called on-demand 21 Decoupling the “when” and “what”
- 22. © 2019 Pathpartner Technology Proposed event detection system • Break down problem into two parts: 1. Class agnostic segment proposal (low complexity) 2. Video segment inference (can be high complexity depending on nature of video) • Inference model may be long or short term temporal, called on-demand. • Characteristics of a good segment proposal: • Cater to arbitrary event lengths • Discriminate event from background, irrespective of event • needs to run for every frame, so low in complexity 22 Decoupling the “when” and “what”
- 23. © 2019 Pathpartner Technology Class agnostic segment proposal • Formulate problem as unsupervised “anomaly detection” • Train a model to learn anomalies against background • At deployment, predict a binary label, i.e., anomaly has occurred or not 23 Decoupling the “when” and “what” Anomaly Detector Video clip Anomaly DetectorAnomaly DetectorAnomaly DetectorAnomaly DetectorAnomaly Detector Yes/No Yes/No Yes/No Yes/No Yes/No Yes/No Result t
- 24. © 2019 Pathpartner Technology Training an anomaly detector 24 Decoupling the “when” and “what” Yt-1 Yt Yt+1 E LSTM Encoder Decoder LSTM D Space Time Memory Memory Convolutional Encoder Convolutional Decoder Y’t+1 Error 𝑒𝑡 = 𝑌′ 𝑡 + 1 − 𝑌𝑡 + 1 2 Video Autoencoder Framework
- 25. © 2019 Pathpartner Technology Training an anomaly detector 25 Decoupling the “when” and “what” Video Autoencoder Framework Yt-1 Yt Yt+1 E LSTM Encoder Decoder LSTM D Space Time Memory Memory Convolutional Encoder Convolutional Decoder Y’t+1 Error 𝑒𝑡 = 𝑌′ 𝑡 + 1 − 𝑌𝑡 + 1 2 2D-CNN 2D-CNN Conv- LSTM Conv- LSTM
- 26. © 2019 Pathpartner Technology Training an anomaly detector 26 Decoupling the “when” and “what” Video Autoencoder Framework Yt-1 Yt Yt+1 E LSTM Encoder Decoder LSTM D Space Time Memory Memory Convolutional Encoder Convolutional Decoder Y’t+1 Error 𝑒𝑡 = 𝑌′ 𝑡 + 1 − 𝑌𝑡 + 1 2 2D-CNN 2D-CNN Conv- LSTM Conv- LSTM Sparse, low-dimensional encoding
- 27. © 2019 Pathpartner Technology Training an anomaly detector 27 Decoupling the “when” and “what” Yt-1 Yt Yt+1 E LSTM Encoder Decoder LSTM D Space Time Memory Memory Convolutional Encoder Convolutional Decoder Y’t+1 Error 𝑒𝑡 = 𝑌′ 𝑡 + 1 − 𝑌𝑡 + 1 2 Video Autoencoder Framework “Learn to Represent Background”
- 28. © 2019 Pathpartner Technology Deployment of anomaly detector • When event occurs, reconstruction is poor • Anomaly decision based on “Regularity score” • rmin and rmax are derived on validation set. 28 Decoupling the “when” and “what” Yt-1 Yt Yt+1 E LSTM Encoder Decoder LSTM D Space Time Memory Memory Convolutional Encoder Convolutional Decoder Y’t+1 Error Regularity ScoreRt = 1 − ( 𝑟𝑡 −𝑟𝑚𝑖 𝑛 𝑟 𝑚𝑎𝑥 ) Rt ≈ 1, higher likelihood of background
- 29. © 2019 Pathpartner Technology Deployment of anomaly detector • When event occurs, reconstruction is poor • Anomaly decision based on “Regularity score” • rmin and rmax are derived on validation set. • 0.8 GMACs / frame, template: 171x128 29 Decoupling the “when” and “what” Yt-1 Yt Yt+1 E LSTM Encoder Decoder LSTM D Space Time Memory Memory Convolutional Encoder Convolutional Decoder Y’t+1 Error Regularity ScoreRt = 1 − ( 𝑟𝑡 −𝑟𝑚𝑖 𝑛 𝑟 𝑚𝑎𝑥 ) Rt ≈ 1, higher likelihood of background
- 30. © 2019 Pathpartner Technology Event detection pipeline 30 Decoupling the “when” and “what” Regularity Score t C3D C3D CleanAndJerk CleanAndJerk CleanAndJerk C3D
- 31. © 2019 Pathpartner Technology Choosing threshold for characterizing anomalies • Choosing threshold R critical for accuracy and complexity • Large values of R ≈ 1 → more false positives , better recall • Small values of R ≈ rmin/rmax → better recall with large number of false positives 31 Decoupling the “when” and “what” R R R Pred GT Pred GT Pred GT False detection Event Missed
- 32. © 2019 Pathpartner Technology Results – F1 score vs complexity on Thumos ’15 • Thumos ‘15 challenge, conducted as a CVPR ’15 workshop • Subset of UCF101, Only 20 human action classes have event labels in untrimmed videos. • Evaluation metric: Average recall and mAP for given tIOU 32 Results Threshold (R) #frame proposals mAP (0.4 tIoU) Recall (0.4 tIOU) GMAC/ video (avg) 0.3 10978 0.43 0.19 15.5k 0.4 16505 0.33 0.21 22.2k 0.5 18048 0.31 0.39 24.1k 0.6 23486 0.17 0.40 30.5k 0.7 30107 0.11 0.41 38.5k Model #Event Proposals (avg) Recall (0.2 tIoU) GMAC/video (avg) TAL-NET 200 0.51 87k TORNADO[1] 30 0.63 46.8k Ours 84 0.421 13.9k TORNADO is a better event proposal, at the cost of additional compute Effect of Threshold R on Accuracy/Compute Comparison with Event Proposal methods [1] TORNADO: A Spatio-Temporal Convolutional Regression Network for Video Action Proposal http://www.ntu.edu.sg/home/shijian.lu/Publicationss
- 33. © 2019 Pathpartner Technology Analysis of effect of Segment proposals 33 Results No proposals, inference at single temporal scale Missed detections for a class likely to be false positive for another!
- 34. © 2019 Pathpartner Technology Analysis of effect of Segment proposals 34 Results With proposals, inference at single temporal scale Reduced Confusion between events → reduced false positives, decrease in recall for some events. Also, Increase in confusion against background
- 35. © 2019 Pathpartner Technology Computational Cost of Detection 35 Results Model GMAC/ inference (max) #params (Million) GMAC/ video VGG 15 138 46.5k C3D- SCNN[1] 79 80 237k C3D-LSTM[2] 24 86 72k TAL-NET[3] 29 98 87k SSAD[4] 61 356 183k TORNADO[5] 32 90.5 46.8k Ours (ConvLSTM- AD + C3D) 24 81 13.9k Accuracy vs GMACs/video of event detectors [1] Temporal Action Localization in Untrimmed Videos via Multi-stage CNNs http://dvmmweb.cs.columbia.edu/files/dvmm_scnn_paper.pdf [2] Temporal Activity Detection in Untrimmed Videos with Recurrent Neural Networks https://imatge-upc.github.io/activitynet-2016-cvprw/ [3] Rethinking the Faster R-CNN Architecture for Temporal Action Localization https://arxiv.org/pdf/1804.07667.pdf [4] Single Shot action detection https://arxiv.org/abs/1710.06236 [5] TORNADO: A Spatio-Temporal Convolutional Regression Network for Video Action Proposal http://www.ntu.edu.sg/home/shijian.lu/Publicationss GMACs per video of event detectors Segment Proposals
- 36. © 2019 Pathpartner Technology Summary • Event Detection in Video with joint spatiotemporal features is computationally expensive • Decoupling models for making inferences in the spatial and long-term temporal modalities can effectively reduce the overall GMACs/Video • On systems with multiple compute units, decoupling provides logical separation of algorithm into a part that needs to run at a high rate and a part that is only called on-demand, enables heterogenous compute • Accuracy – Complexity trade-off controlled by segment proposals 36
- 37. © 2019 Pathpartner Technology Thank You 37
- 38. © 2019 Pathpartner Technology Example of Resource Slide 38 Technical Papers [1] Learning Spatiotemporal Features with 3D Convolutional Networks https://arxiv.org/pdf/1412.0767.pdf [2] Temporal Activity Detection in Untrimmed Videos with Recurrent Neural Networks https://imatge-upc.github.io/activitynet- 2016-cvprw/ [3] Rethinking the Faster R-CNN Architecture for Temporal Action Localization https://arxiv.org/pdf/1804.07667.pdf Embedded Vision Summit “Using Deep Learning for Video Event Detection on a Compute Budget”