Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models

Marlon Dumas
Marlon DumasProfessor at University of Tartu | Co-Founder at Apromore en University of Tartu and Apromore Pty Ltd
••
Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models David Chapela-Campa1, Ismail Benchekroun2, Opher Baron2, Marlon Dumas1, Dmitry Krass2, and Arik Senderovich3 21st International Conference on Business Process Management (BPM 2023) 1 University of Tartu, Estonia 2 University of Toronto, Canada 3 York University, Canada
Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models 2 Introduction
Business Process Simulation (BPS) 3 BPS allows users to address “what-if” analysis questions. What would be the cycle time of the process if the rate of arrival of new cases doubles? Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
Business Process Simulation (BPS) 4 BPS models can be manually created by modeling experts. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
Business Process Simulation (BPS) 5 BPS models can be manually created by modeling experts. Use of process mining techniques to automatically discover BPS models from business process event logs. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
Business Process Simulation (BPS) 6 How to assess the quality of a BPS model? Automatic assessment. Useful to detect the sources of deviations. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models 7 Proposed Framework
Quality of a BPS model 8 How to assess the quality of a BPS model? Comparing an event log with a BPS model. Variety of different BPS models formats. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models Process event log
Quality of a BPS model 9 How to assess the quality of a BPS model? Generate K simulated event logs. Compare individually and report the average and confidence interval. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models 9 Process event log K simulated event logs
Quality of a BPS model 10 A BPS model can be very accurate in one aspect (e.g., control-flow), yet very different in another (e.g., processing times). Three main dimensions: control-flow, temporal, congestion. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models Process event log Simulated event log
Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models 11 Proposed Framework Control-flow measures
Control-flow: Control-Flow Log Distance 12 Control-Flow Log Distance (CFLD): given two event logs L1 and L2, (minimum) average distance to transform each case in L1 into another case in L2, such that each case in L1 is paired to a different case in L2. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models Camargo, M., Dumas, M., Rojas, O.G.: Discovering generative models from event logs: data-driven simulation vs deep learning. PeerJ Comput. Sci. 7, e577 (2021) Process event log Simulated event log
Control-flow: Control-Flow Log Distance 13 Control-Flow Log Distance (CFLD): given two event logs L1 and L2, (minimum) average distance to transform each case in L1 into another case in L2, such that each case in L1 is paired to a different case in L2. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models Process event log Simulated event log A B C D A B C D A C B D A E F G H A E F G I A B C D A C B D A E F G A E F G H A E F G H
Control-flow: Control-Flow Log Distance 14 Control-Flow Log Distance (CFLD): given two event logs L1 and L2, (minimum) average distance to transform each case in L1 into another case in L2, such that each case in L1 is paired to a different case in L2. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models Process event log Simulated event log A B C D A B C D A C B D A E F G H A E F G I A B C D A C B D A E F G A E F G H A E F G H 0
Control-flow: Control-Flow Log Distance 15 Control-Flow Log Distance (CFLD): given two event logs L1 and L2, (minimum) average distance to transform each case in L1 into another case in L2, such that each case in L1 is paired to a different case in L2. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models Process event log Simulated event log A B C D A B C D A C B D A E F G H A E F G I A B C D A C B D A E F G A E F G H A E F G H 0 0 0.2
Control-flow: Control-Flow Log Distance 16 Control-Flow Log Distance (CFLD): given two event logs L1 and L2, (minimum) average distance to transform each case in L1 into another case in L2, such that each case in L1 is paired to a different case in L2. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models CFLD = 0+0+0.75+0+0.2 5 = 0.19
Control-flow: N-Gram Distance 17 N-Gram Distance (NGD): given two event logs L1 and L2, and a positive integer 𝑛, difference in the frequencies of the 𝑛-grams observed in both L1 and L2. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models Leemans, S.J.J., Syring, A.F., van der Aalst, W.M.P.: Earth movers’ stochastic conformance checking. In: BPM Forum 2019. LNBIP, vol. 360, pp. 127–143. Springer (2019) Process event log Simulated event log
Control-flow: N-Gram Distance 18 N-Gram Distance (NGD): given two event logs L1 and L2, and a positive integer 𝑛, difference in the frequencies of the 𝑛-grams observed in both L1 and L2. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models Process event log Simulated event log A B C D A B C D A C B D A E F G H A E F G I A B C D A C B D A E F G A E F G H A E F G H N = 3
Control-flow: N-Gram Distance 19 N-Gram Distance (NGD): given two event logs L1 and L2, and a positive integer 𝑛, difference in the frequencies of the 𝑛-grams observed in both L1 and L2. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models Process event log Simulated event log A B C D A B C D A C B D A E F G H A E F G I A B C D A C B D A E F G A E F G H A E F G H N = 3
Control-flow: N-Gram Distance 20 N-Gram Distance (NGD): given two event logs L1 and L2, and a positive integer 𝑛, difference in the frequencies of the 𝑛-grams observed in both L1 and L2. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models Process event log Simulated event log A B C D A B C D A C B D A E F G H A E F G I A B C D A C B D A E F G A E F G H A E F G H N = 3
Control-flow: N-Gram Distance 21 N-Gram Distance (NGD): given two event logs L1 and L2, and a positive integer 𝑛, difference in the frequencies of the 𝑛-grams observed in both L1 and L2. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models Process event log Simulated event log A B C D A B C D A C B D A E F G H A E F G I A B C D A C B D A E F G A E F G H A E F G H N = 3
Control-flow: N-Gram Distance 22 N-Gram Distance (NGD): given two event logs L1 and L2, and a positive integer 𝑛, difference in the frequencies of the 𝑛-grams observed in both L1 and L2. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models 0 1 2 3 4 5 6 _ _ A _ A B _ A C _ A E A B C A C B A E F B C D C B D E F G F G H F G I C D _ Process event log Simulated event log
Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models 23 Proposed Framework Temporal measures
Process event log Simulated event log Temporal: Absolute Event Distribution 24 Absolute Event Distribution (AED): given two event logs L1 and L2, distance between the time series of the events in L1 and L2. How different they are distributed through the event log. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
Process event log Simulated event log Temporal: Absolute Event Distribution 25 Absolute Event Distribution (AED): given two event logs L1 and L2, distance between the time series of the events in L1 and L2. How different they are distributed through the event log. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
Temporal: Absolute Event Distribution 26 Absolute Event Distribution (AED): given two event logs L1 and L2, distance between the time series of the events in L1 and L2. How different they are distributed through the event log. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models 06-10-2022 10am – 11am 07-10-2022 11am – 12pm
Temporal: Absolute Event Distribution 27 Absolute Event Distribution (AED): given two event logs L1 and L2, distance between the time series of the events in L1 and L2. How different they are distributed through the event log. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models Earth mover's distance
Temporal: Circadian Event Distribution 28 Circadian Event Distribution (CED): given two event logs L1 and L2, distance between the time series of the events in L1 and L2, for each day of the week. How different they are distributed through each day of the week. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models Monday Tuesday Wednesday Thursday 10am – 11am Friday 11am – 12pm
Temporal: Circadian Event Distribution 29 Circadian Event Distribution (CED): given two event logs L1 and L2, distance between the time series of the events in L1 and L2, for each day of the week. How different they are distributed through each day of the week. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models EMD Monday Monday
Temporal: Relative Event Distribution 30 Relative Event Distribution (RED): given two event logs L1 and L2, distance between the time series of the events in L1 and L2, with respect to the start of their case. How different they are distributed within each process case. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models 00:00:00 01:01:47
Temporal: Relative Event Distribution Relative Event Distribution (RED): given two event logs L1 and L2, distance between the time series of the events in L1 and L2, with respect to the start of their case. How different they are distributed within each process case. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models EMD
Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models 32 Proposed Framework Congestion measures
Congestion: Case Arrival Rate 33 Case Arrival Rate (CAR): given two event logs L1 and L2, distance between how the case arrivals are distributed in L1 and L2. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models 06-10-2022 10am – 11am
Congestion: Case Arrival Rate 34 Case Arrival Rate (CAR): given two event logs L1 and L2, distance between how the case arrivals are distributed in L1 and L2. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models EMD
Congestion: Cycle Time Distribution 35 Cycle Time Distribution (CTD): given two event logs L1 and L2, distance between the distribution of cycle times in L1 and L2. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models 01:07:02
Congestion: Cycle Time Distribution 36 Cycle Time Distribution (CTD): given two event logs L1 and L2, distance between the distribution of cycle times in L1 and L2. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models EMD
Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models 37 Evaluation
Evaluation 38 Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models EQ1: Are the proposed measures able to discern the impact of different known modifications to a BPS model? EQ2: Is the N-Gram Distance’s performance significantly different from the CFLD’s performance? No modifications Control-flow Gateway probabilities Case arrival rate Activity durations Resource contention Working calendars Extraneous delays
Evaluation 39 Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
Evaluation 40 Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models EQ1: Are the proposed measures able to discern the impact of different known modifications to a BPS model?
Evaluation 41 Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models EQ2: Is the N-Gram Distance’s performance significantly different from the CFLD’s performance? Kendall rank correlation coefficient 1.0
Evaluation 42 Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models EQ3: Given two BPS models discovered by existing automated BPS model discovery techniques in real-life scenarios, are the proposed measures able to identify the strengths and weaknesses of each technique?
Evaluation 43 Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models EQ3: Given two BPS models discovered by existing automated BPS model discovery techniques in real-life scenarios, are the proposed measures able to identify the strengths and weaknesses of each technique? 4 real-life processes: each split into disjoint training and test.
Evaluation 44 Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models EQ3: Given two BPS models discovered by existing automated BPS model discovery techniques in real-life scenarios, are the proposed measures able to identify the strengths and weaknesses of each technique? Automatically discover BPS model with SIMOD and Service Miner.
Evaluation 45 Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models EQ3: Given two BPS models discovered by existing automated BPS model discovery techniques in real-life scenarios, are the proposed measures able to identify the strengths and weaknesses of each technique?
Evaluation 46 Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models EQ3: Given two BPS models discovered by existing automated BPS model discovery techniques in real-life scenarios, are the proposed measures able to identify the strengths and weaknesses of each technique?
Evaluation 47 Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models EQ3: Given two BPS models discovered by existing automated BPS model discovery techniques in real-life scenarios, are the proposed measures able to identify the strengths and weaknesses of each technique?
Evaluation 48 Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models EQ3: Given two BPS models discovered by existing automated BPS model discovery techniques in real-life scenarios, are the proposed measures able to identify the strengths and weaknesses of each technique?
Evaluation 49 Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models EQ4: Does the 1-WD report the same insights in real-life scenarios as the EMD?
Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models 50 Conclusion
Conclusion 51 Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models Proposed a framework to measure the quality of a BPS model: decomposing into three perspectives (control-flow, temporal, and congestion), and defined measures for each of these perspectives. The measures proved their ability to detect the alterations in their corresponding perspectives. Beyond capturing the quality of BPS model and identifying the sources of discrepancies, the measures can also assist in eliciting areas for improvement in these techniques. The presented computationally efficient alternatives led to similar conclusions.
Future Work 52 Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models Explore the applicability of the proposed measures to other process mining problems, e.g., concept drift detection and variant analysis. Studying how to assess the quality of BPS models in the context of object-centric event logs. Study other quality measures for BPS models adapted from the field of generative machine learning, for example, by using a discriminative model that attempts to distinguish between data generated by the BPS model and real data.
1 de 52

Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models

••
Descargar para leer sin conexión
Datos y análisis

Business Process Simulation (BPS) is an approach to analyze the performance of business processes under different scenarios. For example, BPS allows us to estimate what would be the cycle time of a process if one or more resources became unavailable. The starting point of BPS is a process model annotated with simulation parameters (a BPS model). BPS models may be manually designed, based on information collected from stakeholders and empirical observations, or automatically discovered from execution data. Regardless of its origin, a key question when using a BPS model is how to assess its quality. In this paper, we propose a collection of measures to evaluate the quality of a BPS model w.r.t. its ability to replicate the observed behavior of the process. We advocate an approach whereby different measures tackle different process perspectives. We evaluate the ability of the proposed measures to discern the impact of modifications to a BPS model, and their ability to uncover the relative strengths and weaknesses of two approaches for automated discovery of BPS models. The evaluation shows that the measures not only capture how close a BPS model is to the observed behavior, but they also help us to identify sources of discrepancies. Presentation delivered by David Chapela-Campa at the BPM'2023 conference, Utrecht, September 2023.

Marlon Dumas
Marlon DumasProfessor at University of Tartu | Co-Founder at Apromore en University of Tartu and Apromore Pty Ltd

Recomendados

Walking the Way from Process Mining to AI-Driven Process Optimization por
Walking the Way from Process Mining to AI-Driven Process OptimizationWalking the Way from Process Mining to AI-Driven Process Optimization
Walking the Way from Process Mining to AI-Driven Process OptimizationMarlon Dumas
672 vistas•27 diapositivas
Business Process Optimization: Status and Perspectives por
Business Process Optimization: Status and PerspectivesBusiness Process Optimization: Status and Perspectives
Business Process Optimization: Status and PerspectivesMarlon Dumas
544 vistas•18 diapositivas
Fundamentals of business process management and BPMN por
Fundamentals of business process management and BPMNFundamentals of business process management and BPMN
Fundamentals of business process management and BPMNGregor Polančič
2K vistas•55 diapositivas
Business Process Modeling por
Business Process ModelingBusiness Process Modeling
Business Process ModelingAng Chen
4.5K vistas•47 diapositivas
BPMN 2.0 overview por
BPMN 2.0 overviewBPMN 2.0 overview
BPMN 2.0 overviewsolomd
8K vistas•19 diapositivas
Introduction to BPM, Business Process Management, BPM por
Introduction to BPM, Business Process Management, BPMIntroduction to BPM, Business Process Management, BPM
Introduction to BPM, Business Process Management, BPMSarbashrestha Panda
17.2K vistas•28 diapositivas

Más contenido relacionado

La actualidad más candente

Business Process Management - What is it, and why all the buzz? por
Business Process Management - What is it, and why all the buzz?Business Process Management - What is it, and why all the buzz?
Business Process Management - What is it, and why all the buzz?Bonitasoft
2.3K vistas•30 diapositivas
Architecture + Process: BPM for Enterprise Architects por
Architecture + Process: BPM for Enterprise ArchitectsArchitecture + Process: BPM for Enterprise Architects
Architecture + Process: BPM for Enterprise ArchitectsMichael zur Muehlen
6.4K vistas•34 diapositivas
Introduction to BPM por
Introduction to BPMIntroduction to BPM
Introduction to BPMSandy Kemsley
17K vistas•81 diapositivas
Process Mining and Predictive Process Monitoring in Apromore por
Process Mining and Predictive Process Monitoring in ApromoreProcess Mining and Predictive Process Monitoring in Apromore
Process Mining and Predictive Process Monitoring in ApromoreMarlon Dumas
713 vistas•34 diapositivas
BPM (Business Process Management) Introduction por
BPM (Business Process Management) IntroductionBPM (Business Process Management) Introduction
BPM (Business Process Management) IntroductionIntegrify
27.8K vistas•24 diapositivas
Advanced Biofuel Technologies por
Advanced Biofuel TechnologiesAdvanced Biofuel Technologies
Advanced Biofuel TechnologiesGerard B. Hawkins
3.7K vistas•100 diapositivas

La actualidad más candente(20)

Business Process Management - What is it, and why all the buzz? por Bonitasoft
Business Process Management - What is it, and why all the buzz?Business Process Management - What is it, and why all the buzz?
Business Process Management - What is it, and why all the buzz?
Bonitasoft•2.3K vistas
Architecture + Process: BPM for Enterprise Architects por Michael zur Muehlen
Architecture + Process: BPM for Enterprise ArchitectsArchitecture + Process: BPM for Enterprise Architects
Architecture + Process: BPM for Enterprise Architects
Michael zur Muehlen•6.4K vistas
Introduction to BPM por Sandy Kemsley
Introduction to BPMIntroduction to BPM
Introduction to BPM
Sandy Kemsley•17K vistas
Process Mining and Predictive Process Monitoring in Apromore por Marlon Dumas
Process Mining and Predictive Process Monitoring in ApromoreProcess Mining and Predictive Process Monitoring in Apromore
Process Mining and Predictive Process Monitoring in Apromore
Marlon Dumas•713 vistas
BPM (Business Process Management) Introduction por Integrify
BPM (Business Process Management) IntroductionBPM (Business Process Management) Introduction
BPM (Business Process Management) Introduction
Integrify•27.8K vistas
Advanced Biofuel Technologies por Gerard B. Hawkins
Advanced Biofuel TechnologiesAdvanced Biofuel Technologies
Advanced Biofuel Technologies
Gerard B. Hawkins•3.7K vistas
Software prototyping por Herry Prihandoko
Software prototypingSoftware prototyping
Software prototyping
Herry Prihandoko•148 vistas
Business Process Modeling with BPMN 2.0 - Second edition por Gregor Polančič
Business Process Modeling with BPMN 2.0 - Second editionBusiness Process Modeling with BPMN 2.0 - Second edition
Business Process Modeling with BPMN 2.0 - Second edition
Gregor Polančič•73K vistas
Business Process Modeling por Sandy Kemsley
Business Process ModelingBusiness Process Modeling
Business Process Modeling
Sandy Kemsley•21.9K vistas
Programming in UML: An Introduction to fUML and Alf por Ed Seidewitz
Programming in UML: An Introduction to fUML and AlfProgramming in UML: An Introduction to fUML and Alf
Programming in UML: An Introduction to fUML and Alf
Ed Seidewitz•28K vistas
What is BPM? por BOC Group
What is BPM?What is BPM?
What is BPM?
BOC Group•3.6K vistas
Process mining in business process management por Ramez Al-Fayez
Process mining in business process managementProcess mining in business process management
Process mining in business process management
Ramez Al-Fayez•3.4K vistas
Using Innoslate for Model-Based Systems Engineering por Elizabeth Steiner
Using Innoslate for Model-Based Systems EngineeringUsing Innoslate for Model-Based Systems Engineering
Using Innoslate for Model-Based Systems Engineering
Elizabeth Steiner•417 vistas
Process Mining - Chapter 7 - Conformance Checking por Wil van der Aalst
Process Mining - Chapter 7 - Conformance CheckingProcess Mining - Chapter 7 - Conformance Checking
Process Mining - Chapter 7 - Conformance Checking
Wil van der Aalst•3K vistas
Process architecture - Part II por Marcello La Rosa
Process architecture - Part IIProcess architecture - Part II
Process architecture - Part II
Marcello La Rosa•12.7K vistas
Waste to Energy Complex Briefing por GlobalReEnergy
Waste to Energy Complex Briefing Waste to Energy Complex Briefing
Waste to Energy Complex Briefing
GlobalReEnergy•1.3K vistas
Automated Program Repair Keynote talk por Abhik Roychoudhury
Automated Program Repair Keynote talkAutomated Program Repair Keynote talk
Automated Program Repair Keynote talk
Abhik Roychoudhury•5.7K vistas
Requirements Gathering And Management por Alan McSweeney
Requirements Gathering And ManagementRequirements Gathering And Management
Requirements Gathering And Management
Alan McSweeney•6.8K vistas
Camunda BPM 7.2: Connectors, Data, Scripting (English) por camunda services GmbH
Camunda BPM 7.2: Connectors, Data, Scripting (English)Camunda BPM 7.2: Connectors, Data, Scripting (English)
Camunda BPM 7.2: Connectors, Data, Scripting (English)
camunda services GmbH•5.1K vistas
Business modeling with UML Eriksson-Penker Notation por Massimo Talia
Business modeling with UML Eriksson-Penker NotationBusiness modeling with UML Eriksson-Penker Notation
Business modeling with UML Eriksson-Penker Notation
Massimo Talia•246 vistas

Similar a Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models

Process Capability por
Process CapabilityProcess Capability
Process CapabilityThomas Abraham
7K vistas•41 diapositivas
Process Mining and Predictive Process Monitoring por
Process Mining and Predictive Process MonitoringProcess Mining and Predictive Process Monitoring
Process Mining and Predictive Process MonitoringMarlon Dumas
3.3K vistas•40 diapositivas
Statistical Process Control - SPC por
Statistical Process Control - SPCStatistical Process Control - SPC
Statistical Process Control - SPCPrasenjit Puri
153 vistas•25 diapositivas
Six Sigma Dfss Application In Data Accarucy por
Six Sigma Dfss Application In Data AccarucySix Sigma Dfss Application In Data Accarucy
Six Sigma Dfss Application In Data Accarucyxyhfun
529 vistas•28 diapositivas
A process to improve the accuracy of mk ii fp to cosmic charles symons por
A process to improve the accuracy of mk ii fp to cosmic charles symonsA process to improve the accuracy of mk ii fp to cosmic charles symons
A process to improve the accuracy of mk ii fp to cosmic charles symonsIWSM Mensura
349 vistas•30 diapositivas
Multi-Perspective Comparison of Business Processes Variants Based on Event Logs por
Multi-Perspective Comparison of Business Processes Variants Based on Event LogsMulti-Perspective Comparison of Business Processes Variants Based on Event Logs
Multi-Perspective Comparison of Business Processes Variants Based on Event LogsMarlon Dumas
557 vistas•14 diapositivas

Similar a Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models(20)

Process Capability por Thomas Abraham
Process CapabilityProcess Capability
Process Capability
Thomas Abraham•7K vistas
Process Mining and Predictive Process Monitoring por Marlon Dumas
Process Mining and Predictive Process MonitoringProcess Mining and Predictive Process Monitoring
Process Mining and Predictive Process Monitoring
Marlon Dumas•3.3K vistas
Statistical Process Control - SPC por Prasenjit Puri
Statistical Process Control - SPCStatistical Process Control - SPC
Statistical Process Control - SPC
Prasenjit Puri•153 vistas
Six Sigma Dfss Application In Data Accarucy por xyhfun
Six Sigma Dfss Application In Data AccarucySix Sigma Dfss Application In Data Accarucy
Six Sigma Dfss Application In Data Accarucy
xyhfun•529 vistas
A process to improve the accuracy of mk ii fp to cosmic charles symons por IWSM Mensura
A process to improve the accuracy of mk ii fp to cosmic charles symonsA process to improve the accuracy of mk ii fp to cosmic charles symons
A process to improve the accuracy of mk ii fp to cosmic charles symons
IWSM Mensura•349 vistas
Multi-Perspective Comparison of Business Processes Variants Based on Event Logs por Marlon Dumas
Multi-Perspective Comparison of Business Processes Variants Based on Event LogsMulti-Perspective Comparison of Business Processes Variants Based on Event Logs
Multi-Perspective Comparison of Business Processes Variants Based on Event Logs
Marlon Dumas•557 vistas
Advantages Of Linear Programming Models por Brenda Torres
Advantages Of Linear Programming ModelsAdvantages Of Linear Programming Models
Advantages Of Linear Programming Models
Brenda Torres•3 vistas
Robust PID Controller Design for Non-Minimum Phase Systems using Magnitude Op... por IRJET Journal
Robust PID Controller Design for Non-Minimum Phase Systems using Magnitude Op...Robust PID Controller Design for Non-Minimum Phase Systems using Magnitude Op...
Robust PID Controller Design for Non-Minimum Phase Systems using Magnitude Op...
IRJET Journal•41 vistas
Comprehensive acquisition management methodology rev6 por clifpt
Comprehensive acquisition management methodology rev6Comprehensive acquisition management methodology rev6
Comprehensive acquisition management methodology rev6
clifpt•44 vistas
3D-DRESD AC por Marco Santambrogio
3D-DRESD AC3D-DRESD AC
3D-DRESD AC
Marco Santambrogio•408 vistas
Se notes por Noor Ul Hudda Memon
Se notesSe notes
Se notes
Noor Ul Hudda Memon•1.7K vistas
Gestión de la calidad sem 2 por youffre
Gestión de la calidad sem 2Gestión de la calidad sem 2
Gestión de la calidad sem 2
youffre•118 vistas
Lpg detector final ppt por Agara Mudhalvan
Lpg detector final pptLpg detector final ppt
Lpg detector final ppt
Agara Mudhalvan•147 vistas
Practical Guides on Programming with Big Number Library in Scientific Researches por theijes
Practical Guides on Programming with Big Number Library in Scientific ResearchesPractical Guides on Programming with Big Number Library in Scientific Researches
Practical Guides on Programming with Big Number Library in Scientific Researches
theijes•31 vistas
Unbiased, Fine-Grained Description of Processes Performance from Event Data por Vadim Denisov
Unbiased, Fine-Grained Description of Processes Performance from Event DataUnbiased, Fine-Grained Description of Processes Performance from Event Data
Unbiased, Fine-Grained Description of Processes Performance from Event Data
Vadim Denisov•34 vistas
BUSINESS PROCESS MODELED WITH BPMN AND CTL MODEL CHECKING por IJCI JOURNAL
BUSINESS PROCESS MODELED WITH BPMN AND CTL MODEL CHECKINGBUSINESS PROCESS MODELED WITH BPMN AND CTL MODEL CHECKING
BUSINESS PROCESS MODELED WITH BPMN AND CTL MODEL CHECKING
IJCI JOURNAL•2 vistas
DECISIONS IN A SUPPLY CHAIN MODELING FOR COMPARATIVE EVALUATION STRATEGIES IN... por IAEME Publication
DECISIONS IN A SUPPLY CHAIN MODELING FOR COMPARATIVE EVALUATION STRATEGIES IN...DECISIONS IN A SUPPLY CHAIN MODELING FOR COMPARATIVE EVALUATION STRATEGIES IN...
DECISIONS IN A SUPPLY CHAIN MODELING FOR COMPARATIVE EVALUATION STRATEGIES IN...
IAEME Publication•267 vistas
Decisions in a supply chain modeling for comparative evaluation strategies in... por IAEME Publication
Decisions in a supply chain modeling for comparative evaluation strategies in...Decisions in a supply chain modeling for comparative evaluation strategies in...
Decisions in a supply chain modeling for comparative evaluation strategies in...
IAEME Publication•294 vistas
Software Architecture in the age of Cloud Computing por Jaroslav Gergic
Software Architecture in the age of Cloud ComputingSoftware Architecture in the age of Cloud Computing
Software Architecture in the age of Cloud Computing
Jaroslav Gergic•525 vistas
Learning Accurate Business Process Simulation Models from Event Logs via Auto... por Marlon Dumas
Learning Accurate Business Process Simulation Models from Event Logs via Auto...Learning Accurate Business Process Simulation Models from Event Logs via Auto...
Learning Accurate Business Process Simulation Models from Event Logs via Auto...
Marlon Dumas•421 vistas

Más de Marlon Dumas

Discovery and Simulation of Business Processes with Probabilistic Resource Av... por
Discovery and Simulation of Business Processes with Probabilistic Resource Av...Discovery and Simulation of Business Processes with Probabilistic Resource Av...
Discovery and Simulation of Business Processes with Probabilistic Resource Av...Marlon Dumas
63 vistas•15 diapositivas
Learning When to Treat Business Processes: Prescriptive Process Monitoring wi... por
Learning When to Treat Business Processes: Prescriptive Process Monitoring wi...Learning When to Treat Business Processes: Prescriptive Process Monitoring wi...
Learning When to Treat Business Processes: Prescriptive Process Monitoring wi...Marlon Dumas
35 vistas•19 diapositivas
Augmented Business Process Management por
Augmented Business Process ManagementAugmented Business Process Management
Augmented Business Process ManagementMarlon Dumas
1.2K vistas•23 diapositivas
Process Mining and Data-Driven Process Simulation por
Process Mining and Data-Driven Process SimulationProcess Mining and Data-Driven Process Simulation
Process Mining and Data-Driven Process SimulationMarlon Dumas
299 vistas•29 diapositivas
Modeling Extraneous Activity Delays in Business Process Simulation por
Modeling Extraneous Activity Delays in Business Process SimulationModeling Extraneous Activity Delays in Business Process Simulation
Modeling Extraneous Activity Delays in Business Process SimulationMarlon Dumas
148 vistas•42 diapositivas
Prescriptive Process Monitoring Under Uncertainty and Resource Constraints por
Prescriptive Process Monitoring Under Uncertainty and Resource ConstraintsPrescriptive Process Monitoring Under Uncertainty and Resource Constraints
Prescriptive Process Monitoring Under Uncertainty and Resource ConstraintsMarlon Dumas
92 vistas•23 diapositivas

Más de Marlon Dumas(20)

Discovery and Simulation of Business Processes with Probabilistic Resource Av... por Marlon Dumas
Discovery and Simulation of Business Processes with Probabilistic Resource Av...Discovery and Simulation of Business Processes with Probabilistic Resource Av...
Discovery and Simulation of Business Processes with Probabilistic Resource Av...
Marlon Dumas•63 vistas
Learning When to Treat Business Processes: Prescriptive Process Monitoring wi... por Marlon Dumas
Learning When to Treat Business Processes: Prescriptive Process Monitoring wi...Learning When to Treat Business Processes: Prescriptive Process Monitoring wi...
Learning When to Treat Business Processes: Prescriptive Process Monitoring wi...
Marlon Dumas•35 vistas
Augmented Business Process Management por Marlon Dumas
Augmented Business Process ManagementAugmented Business Process Management
Augmented Business Process Management
Marlon Dumas•1.2K vistas
Process Mining and Data-Driven Process Simulation por Marlon Dumas
Process Mining and Data-Driven Process SimulationProcess Mining and Data-Driven Process Simulation
Process Mining and Data-Driven Process Simulation
Marlon Dumas•299 vistas
Modeling Extraneous Activity Delays in Business Process Simulation por Marlon Dumas
Modeling Extraneous Activity Delays in Business Process SimulationModeling Extraneous Activity Delays in Business Process Simulation
Modeling Extraneous Activity Delays in Business Process Simulation
Marlon Dumas•148 vistas
Prescriptive Process Monitoring Under Uncertainty and Resource Constraints por Marlon Dumas
Prescriptive Process Monitoring Under Uncertainty and Resource ConstraintsPrescriptive Process Monitoring Under Uncertainty and Resource Constraints
Prescriptive Process Monitoring Under Uncertainty and Resource Constraints
Marlon Dumas•92 vistas
Robotic Process Mining por Marlon Dumas
Robotic Process MiningRobotic Process Mining
Robotic Process Mining
Marlon Dumas•333 vistas
Accurate and Reliable What-If Analysis of Business Processes: Is it Achievable? por Marlon Dumas
Accurate and Reliable What-If Analysis of Business Processes: Is it Achievable?Accurate and Reliable What-If Analysis of Business Processes: Is it Achievable?
Accurate and Reliable What-If Analysis of Business Processes: Is it Achievable?
Marlon Dumas•51 vistas
Process Mining: A Guide for Practitioners por Marlon Dumas
Process Mining: A Guide for PractitionersProcess Mining: A Guide for Practitioners
Process Mining: A Guide for Practitioners
Marlon Dumas•141 vistas
Optimización de procesos basada en datos por Marlon Dumas
Optimización de procesos basada en datosOptimización de procesos basada en datos
Optimización de procesos basada en datos
Marlon Dumas•386 vistas
Process Mining and AI for Continuous Process Improvement por Marlon Dumas
Process Mining and AI for Continuous Process ImprovementProcess Mining and AI for Continuous Process Improvement
Process Mining and AI for Continuous Process Improvement
Marlon Dumas•207 vistas
Prescriptive Process Monitoring for Cost-Aware Cycle Time Reduction por Marlon Dumas
Prescriptive Process Monitoring for Cost-Aware Cycle Time ReductionPrescriptive Process Monitoring for Cost-Aware Cycle Time Reduction
Prescriptive Process Monitoring for Cost-Aware Cycle Time Reduction
Marlon Dumas•362 vistas
Mine Your Simulation Model: Automated Discovery of Business Process Simulatio... por Marlon Dumas
Mine Your Simulation Model: Automated Discovery of Business Process Simulatio...Mine Your Simulation Model: Automated Discovery of Business Process Simulatio...
Mine Your Simulation Model: Automated Discovery of Business Process Simulatio...
Marlon Dumas•654 vistas
On the Road to AI-Infused Process Execution por Marlon Dumas
On the Road to AI-Infused Process ExecutionOn the Road to AI-Infused Process Execution
On the Road to AI-Infused Process Execution
Marlon Dumas•270 vistas
Process Mining 2.0: From Insights to Actions por Marlon Dumas
Process Mining 2.0: From Insights to ActionsProcess Mining 2.0: From Insights to Actions
Process Mining 2.0: From Insights to Actions
Marlon Dumas•742 vistas
Process Mining in Action: Self-service data science for business teams por Marlon Dumas
Process Mining in Action: Self-service data science for business teamsProcess Mining in Action: Self-service data science for business teams
Process Mining in Action: Self-service data science for business teams
Marlon Dumas•407 vistas
Enterprise Systems por Marlon Dumas
Enterprise SystemsEnterprise Systems
Enterprise Systems
Marlon Dumas•796 vistas
Process Mining Meets Causal Machine Learning: Discovering Causal Rules From E... por Marlon Dumas
Process Mining Meets Causal Machine Learning: Discovering Causal Rules From E...Process Mining Meets Causal Machine Learning: Discovering Causal Rules From E...
Process Mining Meets Causal Machine Learning: Discovering Causal Rules From E...
Marlon Dumas•437 vistas
Identifying Candidate Routines for Robotic Process Automation From Unsegmente... por Marlon Dumas
Identifying Candidate Routines for Robotic Process Automation From Unsegmente...Identifying Candidate Routines for Robotic Process Automation From Unsegmente...
Identifying Candidate Routines for Robotic Process Automation From Unsegmente...
Marlon Dumas•444 vistas
Analitica de datos e inteligencia artificial para procesos de negocios por Marlon Dumas
Analitica de datos e inteligencia artificial para procesos de negociosAnalitica de datos e inteligencia artificial para procesos de negocios
Analitica de datos e inteligencia artificial para procesos de negocios
Marlon Dumas•356 vistas

Último

Inawisdom Quick Sight por
Inawisdom Quick SightInawisdom Quick Sight
Inawisdom Quick SightPhilipBasford
7 vistas•27 diapositivas
VoxelNet por
VoxelNetVoxelNet
VoxelNettaeseon ryu
17 vistas•21 diapositivas
Amy slides.pdf por
Amy slides.pdfAmy slides.pdf
Amy slides.pdfStatsCommunications
5 vistas•13 diapositivas
[DSC Europe 23][AI:CSI] Dragan Pleskonjic - AI Impact on Cybersecurity and P... por
[DSC Europe 23][AI:CSI] Dragan Pleskonjic - AI Impact on Cybersecurity and P...[DSC Europe 23][AI:CSI] Dragan Pleskonjic - AI Impact on Cybersecurity and P...
[DSC Europe 23][AI:CSI] Dragan Pleskonjic - AI Impact on Cybersecurity and P...DataScienceConferenc1
8 vistas•36 diapositivas
apple.pptx por
apple.pptxapple.pptx
apple.pptxhoneybeeqwe
6 vistas•15 diapositivas
Listed Instruments Survey 2022.pptx por
Listed Instruments Survey 2022.pptxListed Instruments Survey 2022.pptx
Listed Instruments Survey 2022.pptxsecretariat4
93 vistas•12 diapositivas

Último(20)

Inawisdom Quick Sight por PhilipBasford
Inawisdom Quick SightInawisdom Quick Sight
Inawisdom Quick Sight
PhilipBasford•7 vistas
VoxelNet por taeseon ryu
VoxelNetVoxelNet
VoxelNet
taeseon ryu•17 vistas
Amy slides.pdf por StatsCommunications
Amy slides.pdfAmy slides.pdf
Amy slides.pdf
StatsCommunications•5 vistas
[DSC Europe 23][AI:CSI] Dragan Pleskonjic - AI Impact on Cybersecurity and P... por DataScienceConferenc1
[DSC Europe 23][AI:CSI] Dragan Pleskonjic - AI Impact on Cybersecurity and P...[DSC Europe 23][AI:CSI] Dragan Pleskonjic - AI Impact on Cybersecurity and P...
[DSC Europe 23][AI:CSI] Dragan Pleskonjic - AI Impact on Cybersecurity and P...
DataScienceConferenc1•8 vistas
apple.pptx por honeybeeqwe
apple.pptxapple.pptx
apple.pptx
honeybeeqwe•6 vistas
Listed Instruments Survey 2022.pptx por secretariat4
Listed Instruments Survey 2022.pptxListed Instruments Survey 2022.pptx
Listed Instruments Survey 2022.pptx
secretariat4•93 vistas
Inawsidom - Data Journey por PhilipBasford
Inawsidom - Data JourneyInawsidom - Data Journey
Inawsidom - Data Journey
PhilipBasford•8 vistas
[DSC Europe 23][Cryptica] Martin_Summer_Digital_central_bank_money_Ideas_init... por DataScienceConferenc1
[DSC Europe 23][Cryptica] Martin_Summer_Digital_central_bank_money_Ideas_init...[DSC Europe 23][Cryptica] Martin_Summer_Digital_central_bank_money_Ideas_init...
[DSC Europe 23][Cryptica] Martin_Summer_Digital_central_bank_money_Ideas_init...
DataScienceConferenc1•5 vistas
Construction Accidents & Injuries por Bisnar Chase Personal Injury Attorneys
Construction Accidents & InjuriesConstruction Accidents & Injuries
Construction Accidents & Injuries
Bisnar Chase Personal Injury Attorneys•6 vistas
Lack of communication among family.pptx por ahmed164023
Lack of communication among family.pptxLack of communication among family.pptx
Lack of communication among family.pptx
ahmed164023•14 vistas
Data about the sector workshop por info828217
Data about the sector workshopData about the sector workshop
Data about the sector workshop
info828217•29 vistas
Shreyas hospital statistics.pdf por samithavinal
Shreyas hospital statistics.pdfShreyas hospital statistics.pdf
Shreyas hospital statistics.pdf
samithavinal•5 vistas
CRM stick or twist workshop por info828217
CRM stick or twist workshopCRM stick or twist workshop
CRM stick or twist workshop
info828217•14 vistas
[DSC Europe 23] Stefan Mrsic_Goran Savic - Evolving Technology Excellence.pptx por DataScienceConferenc1
[DSC Europe 23] Stefan Mrsic_Goran Savic - Evolving Technology Excellence.pptx[DSC Europe 23] Stefan Mrsic_Goran Savic - Evolving Technology Excellence.pptx
[DSC Europe 23] Stefan Mrsic_Goran Savic - Evolving Technology Excellence.pptx
DataScienceConferenc1•11 vistas
6498-Butun_Beyinli_Cocuq-Daniel_J.Siegel-Tina_Payne_Bryson-2011-259s.pdf por 10urkyr34
6498-Butun_Beyinli_Cocuq-Daniel_J.Siegel-Tina_Payne_Bryson-2011-259s.pdf6498-Butun_Beyinli_Cocuq-Daniel_J.Siegel-Tina_Payne_Bryson-2011-259s.pdf
6498-Butun_Beyinli_Cocuq-Daniel_J.Siegel-Tina_Payne_Bryson-2011-259s.pdf
10urkyr34•7 vistas
LIVE OAK MEMORIAL PARK.pptx por ms2332always
LIVE OAK MEMORIAL PARK.pptxLIVE OAK MEMORIAL PARK.pptx
LIVE OAK MEMORIAL PARK.pptx
ms2332always•7 vistas
META.pptx por vasanthan19012003
META.pptxMETA.pptx
META.pptx
vasanthan19012003•6 vistas
Data Journeys Hard Talk workshop final.pptx por info828217
Data Journeys Hard Talk workshop final.pptxData Journeys Hard Talk workshop final.pptx
Data Journeys Hard Talk workshop final.pptx
info828217•11 vistas
Product Research sample.pdf por AllenSingson
Product Research sample.pdfProduct Research sample.pdf
Product Research sample.pdf
AllenSingson•33 vistas
Employees attrition por MaryAlejandraDiaz
Employees attritionEmployees attrition
Employees attrition
MaryAlejandraDiaz•5 vistas

Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models

  • 1. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models David Chapela-Campa1, Ismail Benchekroun2, Opher Baron2, Marlon Dumas1, Dmitry Krass2, and Arik Senderovich3 21st International Conference on Business Process Management (BPM 2023) 1 University of Tartu, Estonia 2 University of Toronto, Canada 3 York University, Canada
  • 2. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models 2 Introduction
  • 3. Business Process Simulation (BPS) 3 BPS allows users to address “what-if” analysis questions. What would be the cycle time of the process if the rate of arrival of new cases doubles? Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
  • 4. Business Process Simulation (BPS) 4 BPS models can be manually created by modeling experts. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
  • 5. Business Process Simulation (BPS) 5 BPS models can be manually created by modeling experts. Use of process mining techniques to automatically discover BPS models from business process event logs. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
  • 6. Business Process Simulation (BPS) 6 How to assess the quality of a BPS model? Automatic assessment. Useful to detect the sources of deviations. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
  • 7. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models 7 Proposed Framework
  • 8. Quality of a BPS model 8 How to assess the quality of a BPS model? Comparing an event log with a BPS model. Variety of different BPS models formats. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models Process event log
  • 9. Quality of a BPS model 9 How to assess the quality of a BPS model? Generate K simulated event logs. Compare individually and report the average and confidence interval. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models 9 Process event log K simulated event logs
  • 10. Quality of a BPS model 10 A BPS model can be very accurate in one aspect (e.g., control-flow), yet very different in another (e.g., processing times). Three main dimensions: control-flow, temporal, congestion. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models Process event log Simulated event log
  • 11. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models 11 Proposed Framework Control-flow measures
  • 12. Control-flow: Control-Flow Log Distance 12 Control-Flow Log Distance (CFLD): given two event logs L1 and L2, (minimum) average distance to transform each case in L1 into another case in L2, such that each case in L1 is paired to a different case in L2. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models Camargo, M., Dumas, M., Rojas, O.G.: Discovering generative models from event logs: data-driven simulation vs deep learning. PeerJ Comput. Sci. 7, e577 (2021) Process event log Simulated event log
  • 13. Control-flow: Control-Flow Log Distance 13 Control-Flow Log Distance (CFLD): given two event logs L1 and L2, (minimum) average distance to transform each case in L1 into another case in L2, such that each case in L1 is paired to a different case in L2. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models Process event log Simulated event log A B C D A B C D A C B D A E F G H A E F G I A B C D A C B D A E F G A E F G H A E F G H
  • 14. Control-flow: Control-Flow Log Distance 14 Control-Flow Log Distance (CFLD): given two event logs L1 and L2, (minimum) average distance to transform each case in L1 into another case in L2, such that each case in L1 is paired to a different case in L2. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models Process event log Simulated event log A B C D A B C D A C B D A E F G H A E F G I A B C D A C B D A E F G A E F G H A E F G H 0
  • 15. Control-flow: Control-Flow Log Distance 15 Control-Flow Log Distance (CFLD): given two event logs L1 and L2, (minimum) average distance to transform each case in L1 into another case in L2, such that each case in L1 is paired to a different case in L2. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models Process event log Simulated event log A B C D A B C D A C B D A E F G H A E F G I A B C D A C B D A E F G A E F G H A E F G H 0 0 0.2
  • 16. Control-flow: Control-Flow Log Distance 16 Control-Flow Log Distance (CFLD): given two event logs L1 and L2, (minimum) average distance to transform each case in L1 into another case in L2, such that each case in L1 is paired to a different case in L2. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models CFLD = 0+0+0.75+0+0.2 5 = 0.19
  • 17. Control-flow: N-Gram Distance 17 N-Gram Distance (NGD): given two event logs L1 and L2, and a positive integer 𝑛, difference in the frequencies of the 𝑛-grams observed in both L1 and L2. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models Leemans, S.J.J., Syring, A.F., van der Aalst, W.M.P.: Earth movers’ stochastic conformance checking. In: BPM Forum 2019. LNBIP, vol. 360, pp. 127–143. Springer (2019) Process event log Simulated event log
  • 18. Control-flow: N-Gram Distance 18 N-Gram Distance (NGD): given two event logs L1 and L2, and a positive integer 𝑛, difference in the frequencies of the 𝑛-grams observed in both L1 and L2. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models Process event log Simulated event log A B C D A B C D A C B D A E F G H A E F G I A B C D A C B D A E F G A E F G H A E F G H N = 3
  • 19. Control-flow: N-Gram Distance 19 N-Gram Distance (NGD): given two event logs L1 and L2, and a positive integer 𝑛, difference in the frequencies of the 𝑛-grams observed in both L1 and L2. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models Process event log Simulated event log A B C D A B C D A C B D A E F G H A E F G I A B C D A C B D A E F G A E F G H A E F G H N = 3
  • 20. Control-flow: N-Gram Distance 20 N-Gram Distance (NGD): given two event logs L1 and L2, and a positive integer 𝑛, difference in the frequencies of the 𝑛-grams observed in both L1 and L2. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models Process event log Simulated event log A B C D A B C D A C B D A E F G H A E F G I A B C D A C B D A E F G A E F G H A E F G H N = 3
  • 21. Control-flow: N-Gram Distance 21 N-Gram Distance (NGD): given two event logs L1 and L2, and a positive integer 𝑛, difference in the frequencies of the 𝑛-grams observed in both L1 and L2. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models Process event log Simulated event log A B C D A B C D A C B D A E F G H A E F G I A B C D A C B D A E F G A E F G H A E F G H N = 3
  • 22. Control-flow: N-Gram Distance 22 N-Gram Distance (NGD): given two event logs L1 and L2, and a positive integer 𝑛, difference in the frequencies of the 𝑛-grams observed in both L1 and L2. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models 0 1 2 3 4 5 6 _ _ A _ A B _ A C _ A E A B C A C B A E F B C D C B D E F G F G H F G I C D _ Process event log Simulated event log
  • 23. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models 23 Proposed Framework Temporal measures
  • 24. Process event log Simulated event log Temporal: Absolute Event Distribution 24 Absolute Event Distribution (AED): given two event logs L1 and L2, distance between the time series of the events in L1 and L2. How different they are distributed through the event log. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
  • 25. Process event log Simulated event log Temporal: Absolute Event Distribution 25 Absolute Event Distribution (AED): given two event logs L1 and L2, distance between the time series of the events in L1 and L2. How different they are distributed through the event log. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
  • 26. Temporal: Absolute Event Distribution 26 Absolute Event Distribution (AED): given two event logs L1 and L2, distance between the time series of the events in L1 and L2. How different they are distributed through the event log. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models 06-10-2022 10am – 11am 07-10-2022 11am – 12pm
  • 27. Temporal: Absolute Event Distribution 27 Absolute Event Distribution (AED): given two event logs L1 and L2, distance between the time series of the events in L1 and L2. How different they are distributed through the event log. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models Earth mover's distance
  • 28. Temporal: Circadian Event Distribution 28 Circadian Event Distribution (CED): given two event logs L1 and L2, distance between the time series of the events in L1 and L2, for each day of the week. How different they are distributed through each day of the week. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models Monday Tuesday Wednesday Thursday 10am – 11am Friday 11am – 12pm
  • 29. Temporal: Circadian Event Distribution 29 Circadian Event Distribution (CED): given two event logs L1 and L2, distance between the time series of the events in L1 and L2, for each day of the week. How different they are distributed through each day of the week. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models EMD Monday Monday
  • 30. Temporal: Relative Event Distribution 30 Relative Event Distribution (RED): given two event logs L1 and L2, distance between the time series of the events in L1 and L2, with respect to the start of their case. How different they are distributed within each process case. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models 00:00:00 01:01:47
  • 31. Temporal: Relative Event Distribution Relative Event Distribution (RED): given two event logs L1 and L2, distance between the time series of the events in L1 and L2, with respect to the start of their case. How different they are distributed within each process case. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models EMD
  • 32. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models 32 Proposed Framework Congestion measures
  • 33. Congestion: Case Arrival Rate 33 Case Arrival Rate (CAR): given two event logs L1 and L2, distance between how the case arrivals are distributed in L1 and L2. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models 06-10-2022 10am – 11am
  • 34. Congestion: Case Arrival Rate 34 Case Arrival Rate (CAR): given two event logs L1 and L2, distance between how the case arrivals are distributed in L1 and L2. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models EMD
  • 35. Congestion: Cycle Time Distribution 35 Cycle Time Distribution (CTD): given two event logs L1 and L2, distance between the distribution of cycle times in L1 and L2. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models 01:07:02
  • 36. Congestion: Cycle Time Distribution 36 Cycle Time Distribution (CTD): given two event logs L1 and L2, distance between the distribution of cycle times in L1 and L2. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models EMD
  • 37. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models 37 Evaluation
  • 38. Evaluation 38 Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models EQ1: Are the proposed measures able to discern the impact of different known modifications to a BPS model? EQ2: Is the N-Gram Distance’s performance significantly different from the CFLD’s performance? No modifications Control-flow Gateway probabilities Case arrival rate Activity durations Resource contention Working calendars Extraneous delays
  • 39. Evaluation 39 Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
  • 40. Evaluation 40 Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models EQ1: Are the proposed measures able to discern the impact of different known modifications to a BPS model?
  • 41. Evaluation 41 Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models EQ2: Is the N-Gram Distance’s performance significantly different from the CFLD’s performance? Kendall rank correlation coefficient 1.0
  • 42. Evaluation 42 Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models EQ3: Given two BPS models discovered by existing automated BPS model discovery techniques in real-life scenarios, are the proposed measures able to identify the strengths and weaknesses of each technique?
  • 43. Evaluation 43 Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models EQ3: Given two BPS models discovered by existing automated BPS model discovery techniques in real-life scenarios, are the proposed measures able to identify the strengths and weaknesses of each technique? 4 real-life processes: each split into disjoint training and test.
  • 44. Evaluation 44 Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models EQ3: Given two BPS models discovered by existing automated BPS model discovery techniques in real-life scenarios, are the proposed measures able to identify the strengths and weaknesses of each technique? Automatically discover BPS model with SIMOD and Service Miner.
  • 45. Evaluation 45 Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models EQ3: Given two BPS models discovered by existing automated BPS model discovery techniques in real-life scenarios, are the proposed measures able to identify the strengths and weaknesses of each technique?
  • 46. Evaluation 46 Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models EQ3: Given two BPS models discovered by existing automated BPS model discovery techniques in real-life scenarios, are the proposed measures able to identify the strengths and weaknesses of each technique?
  • 47. Evaluation 47 Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models EQ3: Given two BPS models discovered by existing automated BPS model discovery techniques in real-life scenarios, are the proposed measures able to identify the strengths and weaknesses of each technique?
  • 48. Evaluation 48 Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models EQ3: Given two BPS models discovered by existing automated BPS model discovery techniques in real-life scenarios, are the proposed measures able to identify the strengths and weaknesses of each technique?
  • 49. Evaluation 49 Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models EQ4: Does the 1-WD report the same insights in real-life scenarios as the EMD?
  • 50. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models 50 Conclusion
  • 51. Conclusion 51 Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models Proposed a framework to measure the quality of a BPS model: decomposing into three perspectives (control-flow, temporal, and congestion), and defined measures for each of these perspectives. The measures proved their ability to detect the alterations in their corresponding perspectives. Beyond capturing the quality of BPS model and identifying the sources of discrepancies, the measures can also assist in eliciting areas for improvement in these techniques. The presented computationally efficient alternatives led to similar conclusions.
  • 52. Future Work 52 Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models Explore the applicability of the proposed measures to other process mining problems, e.g., concept drift detection and variant analysis. Studying how to assess the quality of BPS models in the context of object-centric event logs. Study other quality measures for BPS models adapted from the field of generative machine learning, for example, by using a discriminative model that attempts to distinguish between data generated by the BPS model and real data.

Notas del editor

  1. So, the first thing we need to know is, what is business process simulation?
  2. BPS aims to replicate the execution of a process, to mimic the behavior of the process, in a certain scenario (set of resources, etc.) analyzing its performance (KPIs) This allows users… The starting point is a BPS model… A process model annotated with a set of simulation parameters that define the scenario (resources, calendars, activity durations…). [NEXT]
  3. BPS models may be manually created based on information collected via interviews or empirical observations.. Or [NEXT…]
  4. they may be automatically discovered from execution data recorded in process-aware information systems (event logs) Regardless of the origin, a key question when using a BPS model is… [NEXT]
  5. how to assess its quality? Several approaches have been proposed to address this problem. However, these approaches are either manual and qualitative or they produce a single number that does not allow one to identify the source(s) of deviations between the BPS model and the observed reality
  6. First we need to decide what to compare when assessing the quality of a BPS model. What we are comparing is a BPS model, with a PROCESS What we usually have is…
  7. …an event log! Now, the first thing we asked ourselves was: should we compare a BPS model against a event log? But it is true that BPS models do not follow a standard structure… They can be formed by queue systems, but less of more models (resources, more complex waiting times), and they will change during time with new research. Thus, what we can do is simulate an event log out of the BPS model
  8. and compare log to log. K runs and compute the avg and conf int
  9. Abstract event logs into time-series or histograms and compare them
  10. We have two event logs, we are focusing on the control-flow, so the first step is to… [NEXT]
  11. obtain the activity sequences of each event log. Then, we compute the Damerau-Levenshtein (string edit distance) distance between each pair of cases… [NEXT]
  12. For exampe, [comment examples], we repeat this for each case Once we have all the pairings computed, we compute the matching between cases of one log to another (such as each case in one log is matched to one case in the other event log, with no repetitions… [NEXT]
  13. While minimizing the sum of distances using the Hungarian algorithm for optimal alignment. Finally, the CFLD measure is the average of these distances… [NEXT]
  14. The computational complexity of computing the DL-distance for all possible pairings is O(N2 ×MTL3) where N is the number of traces in the logs (assuming both logs have an equal number of cases, which holds in our setting) and MTL is the maximum trace length. Since all pairings are put into a matrix to compute the optimal alignment of cases (the one that minimizes the total sum of distances), CFLD’s memory complexity is quadratic on the number of cases. The optimal alignment of traces using the Hungarian algorithm has a cubic complexity on the number of cases.
  15. In the same way than for the CFLD, we are focusing on the control-flow, so the first step is to obtain the activity sequences of each event log. Leemans et al. measure the quality of a stochastic process model by mapping the model and a log to their Directly-Follows Graph (DFG), viewing each DFG as a histogram, and measuring the distance between these histograms. We note that the histogram of 2-grams of a log is equal to the histogram of its DFG. Given this observation, we generalize the approach of to n-grams, noting that the histogram of n-grams of a log is equal to the (n-1)th-Markovian abstraction of the log. Then… [NEXT]
  16. Let’s assume a size of N=3, so the N-grams are 3-grams (sequences of three activites). We compute all 3-grams observed in both logs, considering two dummy activities in the start and end of each trace. Then… [NEXT]
  17. We measure the frequency of each N-gram in each log…
  18. We measure the frequency of each N-gram in each log…
  19. And compute the sum of absolute differences between them, normalized by the sum of frequencies of all n-grams (value between 0-1). NGD is considerably more efficient than CFLD, as the construction of the histogram of n-grams is linear on the number of events in the log, and the same goes for computing the differences between the n-gram histograms.
  20. For the temporal measures, we first do the opposite of the control-flow, we abstract from the control-flow information… [NEXT]
  21. retaining only the events (in this case start and end)… [NEXT]
  22. Then we discretize these events into bins of 1h in the following way. Obtaining a time-series with the number of events happening in each hour of the process timeline.
  23. Once we have the temporal distribution (not a probabilistic distribution, but just the events occurring in the timeline), we compare both time-series with the EMD to measure the distance. We measure the trend.
  24. The same process is followed for the next measure, but in this case discretized to weekdays. In this way, we measure the seasonality of the events happening in the process.
  25. The temporal distribution of events of each day of the week is compared and then we compute the average distance of the 7 days.
  26. Finally, for the third one, we focus on how the events are distributed within their corresponding trace. For this, we compute the time from the case arrival to the event happening and bin it in hours.
  27. Finally, for the third one, we focus on how the events are distributed within their corresponding trace. For this, we compute the time from the case arrival to the event happening and bin it in hours.
  28. For the case arrival rate we want to measure how different the arrival of cases is. So, we retain only the events denoting the arrival of each case (start of first activity instance). Then we build the distribution in the same way than the previous metrics… [NEXT]
  29. and compare with EMD.