Se ha denunciado esta presentación.
Utilizamos tu perfil de LinkedIn y tus datos de actividad para personalizar los anuncios y mostrarte publicidad más relevante. Puedes cambiar tus preferencias de publicidad en cualquier momento.
Error-proof Machine Learning with explainability and human-centric design
WE ARE AN END-TO-END ARTIFICIAL
INTELLIGENCE SOFTWARE FACTORY
Human-in-the-loop
AI systems inevitably lose accuracy
over t...
WE ARE PART OF
THE RACCOONS GROUP
Software
prototyping
AI end-to-end
Chatbot framework
Blockchain services
Full stack Java...
AI is...
NOT new (but has changed)
Early AI (1950’s – 2000’s)
• Deterministic, human-made algorithms
• Centered around planning or ...
Probabilistic
(Modern) AI makes mistakes!
• ~95-99% accuracy is typical (depending on usecase)
• Requires strategy for 5% ...
LESSON 1
Think about mistakes
NOT Self-Learning!
Machine Learning learns from VERIFIED data
• Many tasks require human verification for the
system to le...
NOT Self-Learning!
support question central support domain experts support question ML algorithm domain experts support qu...
LESSON 2
Learn from your users
Human-Centered Machine Learning
Cooperation benefits all parties
• AI eliminates tedium
• Humans focus on edge cases
• Two...
LESSON 3
Learn your users
Our work
CASE 1
ORDER ADMINSTRATION
AT TVH
TVH Group is a Belgian company active in the market of forklift trucks, aerial work plat...
A R T I F I C I A L I N T E L L I G E N C E
Determine if the email is a rental
order or not & extracts all entities
in the...
Customers of TVH send orders via email to the order administration. A typical example is shown below.
CONNECT TO MAILBOX
V E R I F I C A T I O N P L A C E O R D E RC O N N E C T T O M A I L B O X
A R T I F I C I A L
I N T E L L I G E N C E
SOL...
Automatically determines if the email is a rental order & extracts all entities in the email.
ARTIFICIAL INTELLIGENCE
P L A C E O R D E R
V E R I F I C AT I O N
SOLUTION
Determine if the email is a rental
order or not & extracts all entitie...
P r e s e n t a n i n t u i t i v e v e r i f i c a t i o n s t e p f o r u s e r s w i t h a u t o m a t i c r e t r a i ...
P L A C E O R D E R
S t r u c t u r e d o r d e r i s p l a c e d
i n t h e E R P s y s t e m
P L A C E O R D E R
SOLUTION...
BUSINESS CASE
The current order administration in each branch of TVH has 7 full time equivalents handling over 400 incomin...
THE KLASSIF.AI SOLUTION
K l a s s i f . a i h a s i d e n t i f i e d f o u r k e y c o m p o n e n t s t h a t a l l o w ...
KLASSIF.AI
Platform features
OCR USER MANAGEMENT
AUTOMATIC
RETRAINING
SECURITY NOTIFICATIONS
ADMIN PANEL
MULTILINGUAL
SUPP...
PROJECT
APPROACH
K L A S S I F . A I
We map and visualise your business
processes
1 . I N TA K E
Your domain experts gener...
CASE 2
HEALTHCARE DATA EXTRACTION
AT RIZIV
RIZIV is the Belgian federal institute for health and disability insurance. The...
O C R A R T I F I C I A L I N T E L L I G E N C E V E R I F I C A T I O N
B Y D O C T O R
Convert scanned document to
comp...
U P L O A D D O C U M E N T S
O C R
SOLUTION
Convert scanned document to
computer readable text
Extract all entities such ...
A R T I F I C I A L
I N T E L L I G E N C EU P L O A D D O C U M E N T S O C R
SOLUTION
Convert scanned document to
comput...
P L A C E O R D E R
S t r u c t u r e d o r d e r i s p l a c e d
i n t h e E R P s y s t e m
V E R I F I C AT I O N
B Y D...
Present an intuitive verification step for doctors with automatic retraining
VERIFICATION BY DOCTOR
CASE 3
DIGITALISING UNEMPLOYMENT BENEFITS
AT HVW
HVW, the Belgian relief fund for unemployment benefits, is a public insti...
H A N D W R I T I N G
R E C O G N I T I O N
A R T I F I C I A L I N T E L L I G E N C E V E R I F I C A T I O N
C o n v e ...
U s e r s c a n u p l o a d a p h o t o o f t h e i r u n e m p l o y m e n t c a r d
UPLOAD PHOTO
U P L O A D P H O T O
U s e r s c a n u p l o a d a p h o t o
o f t h e i r u n e m p l o y m e n t
c a r d
H A N D W R I ...
A R T I F I C I A L
I N T E L L I G E N C E
E x t r a c t c a l e n d a r i n f o r m a t i o n
f r o m p h o t o
U P L O ...
P L A C E O R D E R
S t r u c t u r e d o r d e r i s p l a c e d
i n t h e E R P s y s t e m
V E R I F I C AT I O N
P r e...
P r e s e n t a n i n t u i t i v e v e r i f i c a t i o n s t e p f o r u s e r s w i t h a u t o m a t i c r e t r a i ...
CASE 4
AI PRESENTATION ANALIST
In an international organisation a lot of
effort is expended in keeping the
personnel up-to...
A R C H I T E C T U R E A R T I F I C I A L I N T E L L I G E N C E U S E R I N T E R F A C E
Our clever architecture only...
U s e r s c a n v i e w t h e f e e d b a c k a b o u t t h e i r p r e s e n t a t i o n
USER INTERFACE
U s e r s c a n v i e w t h e f e e d b a c k a b o u t t h e i r p r e s e n t a t i o n
USER INTERFACE
U s e r s c a n v i e w t h e f e e d b a c k a b o u t t h e i r p r e s e n t a t i o n
USER INTERFACE
And what about explainability?
Explainability is mainly relevant for decision making
AI (which we haven’t been asked to bu...
The choice among algorithm categories can partially be made based on the user persona's
ability to intervene at different ...
NEED FOR A HUMAN IN THE LINK?
LET’S GET STARTED!
CONTACT US STAY IN TOUCH
info@brainjar.ai
+32 (0) 473 36 15 55
Gaston Gee...
Implementing error-proof, business-critical Machine Learning, presentation by Deevid De Meyer, co-founder of Brainjar, pre...
Implementing error-proof, business-critical Machine Learning, presentation by Deevid De Meyer, co-founder of Brainjar, pre...
Implementing error-proof, business-critical Machine Learning, presentation by Deevid De Meyer, co-founder of Brainjar, pre...
Implementing error-proof, business-critical Machine Learning, presentation by Deevid De Meyer, co-founder of Brainjar, pre...
Próxima SlideShare
Cargando en…5
×

Implementing error-proof, business-critical Machine Learning, presentation by Deevid De Meyer, co-founder of Brainjar, presented at the Trustworthy and Ethical AI Conference on Feb 13th in Brussels

69 visualizaciones

Publicado el

This presentation by Deevid De Meyer outlines how Brainjar uses human-centric design and explainability to create machine learning systems that work together with humans to improve efficiency while reducing error rate.

Publicado en: Tecnología
  • Sé el primero en comentar

  • Sé el primero en recomendar esto

Implementing error-proof, business-critical Machine Learning, presentation by Deevid De Meyer, co-founder of Brainjar, presented at the Trustworthy and Ethical AI Conference on Feb 13th in Brussels

  1. 1. Error-proof Machine Learning with explainability and human-centric design
  2. 2. WE ARE AN END-TO-END ARTIFICIAL INTELLIGENCE SOFTWARE FACTORY Human-in-the-loop AI systems inevitably lose accuracy over time. Brainjar builds systems that are able to continually improve by working together with their human colleagues. Technology agnostic Because of our broad technology expertise, we always choose the best solution for problem. This can be AI API, pre-trained model or custom model and is always cloud agnostic. End-to-end Brainjar works end-to-end with the customer to build and integrate the AI solution. We start with a data analysis, followed by labeling, model training, front-end building and deployment.
  3. 3. WE ARE PART OF THE RACCOONS GROUP Software prototyping AI end-to-end Chatbot framework Blockchain services Full stack Javascript Quantum computing Edge AI Digital production house AI Business Strategy
  4. 4. AI is...
  5. 5. NOT new (but has changed) Early AI (1950’s – 2000’s) • Deterministic, human-made algorithms • Centered around planning or abstract reasoning • Scaled horribly • Required maintenance of structured “knowledge systems” Modern AI (2000’s – Now) • “Learning” algorithms -> Machine Learning • Centered around human senses and prediction • Can scale with data • Can process unstructured data Source: Stanford news
  6. 6. Probabilistic (Modern) AI makes mistakes! • ~95-99% accuracy is typical (depending on usecase) • Requires strategy for 5% (human review) • Algorithms can provide “confidence level”, allowing humans to only review edge cases Source: wikipedia
  7. 7. LESSON 1 Think about mistakes
  8. 8. NOT Self-Learning! Machine Learning learns from VERIFIED data • Many tasks require human verification for the system to learn & improve • Initial training can be done using historic data • Human feedback still required during production Source: Medium
  9. 9. NOT Self-Learning! support question central support domain experts support question ML algorithm domain experts support question ML algorithm domain experts Example: Mail Forwarding (1) (2) (3)
  10. 10. LESSON 2 Learn from your users
  11. 11. Human-Centered Machine Learning Cooperation benefits all parties • AI eliminates tedium • Humans focus on edge cases • Two key principle builds trust with management • High quality data is produced • AI improves/adapts continuously Source: wikipedia
  12. 12. LESSON 3 Learn your users
  13. 13. Our work
  14. 14. CASE 1 ORDER ADMINSTRATION AT TVH TVH Group is a Belgian company active in the market of forklift trucks, aerial work platforms, agricultural tractors and industrial vehicles. The company has an order department that handles orders from customers via email. TVH has a large order administration department. O R D E R A D M I N I S T R AT I O N Customers can place orders by sending an email with their request to TVH. E M A I L O R D E R S The rental administrators handle these requests by reading the mails and registering the order in the ERP system. R E P E T I T I V E W O R K
  15. 15. A R T I F I C I A L I N T E L L I G E N C E Determine if the email is a rental order or not & extracts all entities in the email Present an intuitive verification step for users with automatic retraining Structured order is placed in the ERP system C O N N E C T T O M A I L B O X SOLUTION Connects with existing mailbox & gets triggered when there is an incoming mail V E R I F I C A T I O N P L A C E O R D E R
  16. 16. Customers of TVH send orders via email to the order administration. A typical example is shown below. CONNECT TO MAILBOX
  17. 17. V E R I F I C A T I O N P L A C E O R D E RC O N N E C T T O M A I L B O X A R T I F I C I A L I N T E L L I G E N C E SOLUTION Determine if the email is a rental order or not & extracts all entities in the email Present an intuitive verification step for users with automatic retraining Structured order is placed in the ERP system Connects with existing mailbox & gets triggered when there is an incoming mail
  18. 18. Automatically determines if the email is a rental order & extracts all entities in the email. ARTIFICIAL INTELLIGENCE
  19. 19. P L A C E O R D E R V E R I F I C AT I O N SOLUTION Determine if the email is a rental order or not & extracts all entities in the email Present an intuitive verification step for users with automatic retraining Structured order is placed in the ERP system Connects with existing mailbox & gets triggered when there is an incoming mail C O N N E C T T O M A I L B O X A R T I F I C I A L I N T E L L I G E N C E
  20. 20. P r e s e n t a n i n t u i t i v e v e r i f i c a t i o n s t e p f o r u s e r s w i t h a u t o m a t i c r e t r a i n i n g . VERIFICATION
  21. 21. P L A C E O R D E R S t r u c t u r e d o r d e r i s p l a c e d i n t h e E R P s y s t e m P L A C E O R D E R SOLUTION Determine if the email is a rental order or not & extracts all entities in the email Present an intuitive verification step for users with automatic retraining Structured order is placed in the ERP system Connects with existing mailbox & gets triggered when there is an incoming mail C O N N E C T T O M A I L B O X A R T I F I C I A L I N T E L L I G E N C E V E R I F I C A T I O N
  22. 22. BUSINESS CASE The current order administration in each branch of TVH has 7 full time equivalents handling over 400 incoming orders via email per day. 5 MIN T I M E P E R E M A I L 1 MIN BEFORE: AFTER: 80% T I M E R E D U C T I O N
  23. 23. THE KLASSIF.AI SOLUTION K l a s s i f . a i h a s i d e n t i f i e d f o u r k e y c o m p o n e n t s t h a t a l l o w o r g a n i s a t i o n s t o l e v e r a g e m a c h i n e l e a r n i n g t o i m p r o v e t h e i r d o c u m e n t p r o c e s s i n g w o r k f l o w s . C o m b i n i n g t h e s e c o m p o n e n t s a l l o w s K l a s s i f . a i t o p r o c e s s d o c u m e n t s a t c o m p u t e r s p e e d s w i t h h u m a n r e l i a b i l i t y . For your experts, it’s business as usual. Meanwhile, Klassif.ai is gathering training data to learn the ins and outs of your documents. The result is a training set that represents the real world. L A B E L I N G Just like any mature software project needs versioning and logging, so does a machine learning model! Klassif.ai provides a model history, along with statistics that show you the model’s performance on real, unseen data. G O V E R N A N C E Klassif.ai integrates seamlessly into your current document processing workflow, enriching your unstructured documents and connecting them to your structured systems. I N T E G R AT I O N Every document type requires its own approach. That’s why Klassif.ai classifies each document before extracting the required information. After all, an invoice needs to be processed differently to an employee contract. C L A S S I F I C AT I O N & E X T R A C T I O N
  24. 24. KLASSIF.AI Platform features OCR USER MANAGEMENT AUTOMATIC RETRAINING SECURITY NOTIFICATIONS ADMIN PANEL MULTILINGUAL SUPPORT SCALABILITY MODULAR AI MODELS ON-PREMISE OR IN THE CLOUD
  25. 25. PROJECT APPROACH K L A S S I F . A I We map and visualise your business processes 1 . I N TA K E Your domain experts generate data while doing their work as usual 4 . L A B E L We customise our platform based on your way of working 3 . C U S T O M I S E We integrate our services with your systems 2 . I N T E G R AT E We train a machine learning algorithm that meets your needs 5 . T R A I N M O D E L We ready the platform for your experts 6 . D E P L O Y P O C L E A D T I M E : 1 - 3 M O N T H S P R O D U C T I O N R E A D Y: 2 - 5 M O N T H S
  26. 26. CASE 2 HEALTHCARE DATA EXTRACTION AT RIZIV RIZIV is the Belgian federal institute for health and disability insurance. The institute is, among others, responsible for handling files on medical accidents. Hospitals scan all case documents and email them to RIZIV. An administrator at RIZIV makes sure the case is complete. C A S E D O C U M E N T S A doctor from RIZIV goes over all case documents and decides on refund of medical costs. R E V I E W B Y D O C T O R An intern prints all case documents and sorts these on date before scanning them again. M A N U A L W O R K
  27. 27. O C R A R T I F I C I A L I N T E L L I G E N C E V E R I F I C A T I O N B Y D O C T O R Convert scanned document to computer readable text Extract all entities such as, doctor name, procedure date and procedure type from document Present an intuitive verification step for doctors with automatic retraining U P L O A D D O C U M E N T S SOLUTION Doctors can upload all case documents into application
  28. 28. U P L O A D D O C U M E N T S O C R SOLUTION Convert scanned document to computer readable text Extract all entities such as, doctor name, procedure date and procedure type from document Present an intuitive verification step for doctors with automatic retraining Doctors can upload all case documents into application A R T I F I C I A L I N T E L L I G E N C E V E R I F I C A T I O N B Y D O C T O R
  29. 29. A R T I F I C I A L I N T E L L I G E N C EU P L O A D D O C U M E N T S O C R SOLUTION Convert scanned document to computer readable text Extract all entities such as, doctor name, procedure date and procedure type from document Present an intuitive verification step for doctors with automatic retraining Doctors can upload all case documents into application V E R I F I C A T I O N B Y D O C T O R
  30. 30. P L A C E O R D E R S t r u c t u r e d o r d e r i s p l a c e d i n t h e E R P s y s t e m V E R I F I C AT I O N B Y D O C T O RU P L O A D D O C U M E N T S O C R SOLUTION Convert scanned document to computer readable text Extract all entities such as, doctor name, procedure date and procedure type from document Present an intuitive verification step for doctors with automatic retraining Doctors can upload all case documents into application A R T I F I C I A L I N T E L L I G E N C E
  31. 31. Present an intuitive verification step for doctors with automatic retraining VERIFICATION BY DOCTOR
  32. 32. CASE 3 DIGITALISING UNEMPLOYMENT BENEFITS AT HVW HVW, the Belgian relief fund for unemployment benefits, is a public institution for social security which pays unemployed citizens. Each month HVW pays more than 120.000 benefit recipients their replacement income. HVW has a large administration department who processes all unemployment benefits B E N E F I T A D M I N I S T R AT I O N The majority of unemployment forms are still submitted on paper PA P E R F O R M S Due to human mistakes, often error are introduced during the input step requiring extra steps to fix mistakes E R R O R P R O N E
  33. 33. H A N D W R I T I N G R E C O G N I T I O N A R T I F I C I A L I N T E L L I G E N C E V E R I F I C A T I O N C o n v e r t h a n d w r i t t e n c o n t a c t i n f o r m a t i o n t o c o m p u t e r r e a d a b l e t e x t E x t r a c t c a l e n d a r i n f o r m a t i o n f r o m p h o t o P r e s e n t a n i n t u i t i v e v e r i f i c a t i o n s t e p f o r u s e r s w i t h a u t o m a t i c r e t r a i n i n g U P L O A D P H O T O U s e r s c a n u p l o a d a p h o t o o f t h e i r u n e m p l o y m e n t c a r d SOLUTION
  34. 34. U s e r s c a n u p l o a d a p h o t o o f t h e i r u n e m p l o y m e n t c a r d UPLOAD PHOTO
  35. 35. U P L O A D P H O T O U s e r s c a n u p l o a d a p h o t o o f t h e i r u n e m p l o y m e n t c a r d H A N D W R I T I N G R E C O G N I T I O N C o n v e r t h a n d w r i t t e n c o n t a c t i n f o r m a t i o n t o c o m p u t e r r e a d a b l e t e x t A R T I F I C I A L I N T E L L I G E N C E E x t r a c t c a l e n d a r i n f o r m a t i o n f r o m p h o t o V E R I F I C A T I O N P r e s e n t a n i n t u i t i v e v e r i f i c a t i o n s t e p f o r u s e r s w i t h a u t o m a t i c r e t r a i n i n g SOLUTION
  36. 36. A R T I F I C I A L I N T E L L I G E N C E E x t r a c t c a l e n d a r i n f o r m a t i o n f r o m p h o t o U P L O A D P H O T O U s e r s c a n u p l o a d a p h o t o o f t h e i r u n e m p l o y m e n t c a r d H A N D W R I T I N G R E C O G N I T I O N C o n v e r t h a n d w r i t t e n c o n t a c t i n f o r m a t i o n t o c o m p u t e r r e a d a b l e t e x t V E R I F I C A T I O N P r e s e n t a n i n t u i t i v e v e r i f i c a t i o n s t e p f o r u s e r s w i t h a u t o m a t i c r e t r a i n i n g SOLUTION
  37. 37. P L A C E O R D E R S t r u c t u r e d o r d e r i s p l a c e d i n t h e E R P s y s t e m V E R I F I C AT I O N P r e s e n t a n i n t u i t i v e v e r i f i c a t i o n s t e p f o r u s e r s w i t h a u t o m a t i c r e t r a i n i n g U P L O A D P H O T O U s e r s c a n u p l o a d a p h o t o o f t h e i r u n e m p l o y m e n t c a r d H A N D W R I T I N G R E C O G N I T I O N C o n v e r t h a n d w r i t t e n c o n t a c t i n f o r m a t i o n t o c o m p u t e r r e a d a b l e t e x t A R T I F I C I A L I N T E L L I G E N C E E x t r a c t c a l e n d a r i n f o r m a t i o n f r o m p h o t o SOLUTION
  38. 38. P r e s e n t a n i n t u i t i v e v e r i f i c a t i o n s t e p f o r u s e r s w i t h a u t o m a t i c r e t r a i n i n g VERIFICATION
  39. 39. CASE 4 AI PRESENTATION ANALIST In an international organisation a lot of effort is expended in keeping the personnel up-to-date using training videos. T R A I N I N G To help train their trainers, we built an AI system that provides feedback on their presentation style. A U T O M AT E F E E D B A C K Often trainers have extensive knowledge on their subject but are not natural born presenters. T R A I N T H E T R A I N E R
  40. 40. A R C H I T E C T U R E A R T I F I C I A L I N T E L L I G E N C E U S E R I N T E R F A C E Our clever architecture only enables the expensive cloud components on demand. Audio and video are separated for further processing. Several AI components are combined to analyse the audio and video of the presentation. Present an intuitive user interface for presenters to view the feedback. SOLUTION U P L O A D V I D E O Users can upload their videos to our application for processing
  41. 41. U s e r s c a n v i e w t h e f e e d b a c k a b o u t t h e i r p r e s e n t a t i o n USER INTERFACE
  42. 42. U s e r s c a n v i e w t h e f e e d b a c k a b o u t t h e i r p r e s e n t a t i o n USER INTERFACE
  43. 43. U s e r s c a n v i e w t h e f e e d b a c k a b o u t t h e i r p r e s e n t a t i o n USER INTERFACE
  44. 44. And what about explainability? Explainability is mainly relevant for decision making AI (which we haven’t been asked to build yet) • Still early days • Be careful about feedback to external users! ➢ Micro loan approval experiment Source: wikipedia
  45. 45. The choice among algorithm categories can partially be made based on the user persona's ability to intervene at different parts of a machine learning pipeline. If the user is allowed to modify the training data, then pre-processing can be used. If the user is allowed to change the learning algorithm, then in-processing can be used. If the user can only treat the learned model as a black box without any ability to modify the training data or learning algorithm, then only post-processing can be used. AIF360 recommends the earliest mediation category in the pipeline that the user has permission to apply because it gives the most flexibility and opportunity to correct bias as much as possible. If possible, all algorithms from all permissible categories should be tested because the ultimate performance depends on dataset characteristics: there is no one best algorithm independent of dataset
  46. 46. NEED FOR A HUMAN IN THE LINK? LET’S GET STARTED! CONTACT US STAY IN TOUCH info@brainjar.ai +32 (0) 473 36 15 55 Gaston Geenslaan 11, B4 3001 Leuven https://brainjar.ai https://twitter.com/brainjarai https://nl.linkedin.com/company/brainjarbe

×