AI Systems Validation - ATA Pune 18th Meetup
•
1 like•725 views
A session on "AI System Validation" was presented by Devesh Rajadhyax at the ATA Pune 18th Meetup in Cognizant in June, 2018.
Recommended
Recommended
More Related Content
Similar to AI Systems Validation - ATA Pune 18th Meetup
Similar to AI Systems Validation - ATA Pune 18th Meetup (20)
More from Agile Testing Alliance
More from Agile Testing Alliance (20)
Recently uploaded
Recently uploaded (20)
AI Systems Validation - ATA Pune 18th Meetup
- 1. AI system validation Devesh Rajadhyax Founder and CEO, Cere Labs
- 2. Contents • Introduction to AI • Machine Learning • Model Validation • Challenges © Cere Labs Pvt. Ltd. 2
- 3. Introduction to AI © Cere Labs Pvt. Ltd.3
- 4. About Cere Labs © Cere Labs Pvt. Ltd. 4 • Three year old privately held company from Mumbai, India • Creator of AI platform ‘Cerescope’ used for unstructured data processing • Applications in banking, healthcare/pharma, manufacturing, agriculture, retail • 20+ people – mostly AI engineers, expertise in Machine Learning, Deep Learning, Cognitive Computing • Part of SAP Co-innovation Labs • Research interests – GA, GAN, Speech processing,
- 5. What is AI? • AI is the pursuit of imitating human capabilities © Cere Labs Pvt. Ltd. 5 Human capabilities Perception Action Computation and memory • Speech recognition • Computer vision • Natural language processing • Writing recognition • Robotics • Natural language generation • Text to speech • Conversation • Predictive Analytics • Anomaly Detection • Planning and reasoning • Decisions • Pattern recognition
- 6. AI based computation © Cere Labs Pvt. Ltd. 6 Technologies • Machine Learning • Deep Learning • Natural Language Processing • Logic programming Buddha and the child Features of AI computing • Learning not algorithmic • Uncertain and not well-defined problems • Inaccurate Examples • Prediction • Sales forecast • Demand forecast • Anomaly • Fraud detection • Disease detection • Reasoning • Chess programs • Expert systems
- 7. Well known AI systems © Cere Labs Pvt. Ltd. 7 Recommendation systems Demand forecasting Failure prediction Customer behaviour
- 8. Machine Learning © Cere Labs Pvt. Ltd.8
- 9. Introduction © Cere Labs Pvt. Ltd. 9 Classification • Separating given data in classes Regression • Coming up with a number f(x)x y ML algorithm (x1,y1), (x2,y2), … f(x) How ML works Algorithms • Supervised • Linear regression • Support Vector Machines • Random Forest • Unsupervised • Clustering • PCA
- 10. ML project cycle © Cere Labs Pvt. Ltd. 10 Credit Card Fraud Detection • Features • Member data • Transaction data • Trend data • Outcome • Fraud/Normal • Data set • 1 million records • Data use • 80% training • 10% testing • 10% validation
- 11. Validation © Cere Labs Pvt. Ltd.11
- 12. Testing model performance • Accuracy measurement • Expressed as percentage • In case of regression • Root Mean Error (RME) © Cere Labs Pvt. Ltd. 12 Actual Predicted Remarks Normal Normal Correct Fraud Fraud Correct Normal Fraud Incorrect Fraud Normal Incorrect • Why the training error is of no use? • What is the difference between validation and test data set? • Choosing the test data - why a tester will have to be data scientist?
- 13. Challenges • Accuracy – not quite that simple • Data bias • Over-fitting • Concept drift and model re-training © Cere Labs Pvt. Ltd. 13 Actual Predicted Remarks Normal Normal Model worked well Fraud Fraud Model worked well Normal Fraud Mistake but less sensitive Fraud Normal Serious error
- 14. Thanks Devesh Rajadhyax CEO, Cere Labs Pvt. Ltd. devesh.rajadhyax@cerelabs.com