SlideShare una empresa de Scribd logo
1 de 35
Descargar para leer sin conexión
@hughleo01 Liberty Information Technology
Liberty IT, Belfast and Dublin
Test trend analysis:Towards robust,
reliable and timely tests
Hugh McCamphill
@hughleo01 Liberty Information Technology
2Liberty IT @hughleo01
Indianapolis Dover Belfast GS LIU DCs Europe RDC Russia DRDC
Redmond DC
Global DRDC
& Utilities
Kansas City DC
Latin America RDC
GS LIU DRDC
Portsmouth DC
GS LIU DC
Turkey DCs India DCs Asia RDC China DCs
Wausau
Fairfield
Boston
Seattle Blanchardstown
11 major IT offices
7 country data centers (DCs)
DRDC = Disaster Recovery DC
3 US domestic data centers (DCs)
4 international regional data centers
LIU = Liberty International Underwriters
More than 5,000+ IT
employees around the globe
Australia
About Us
3Liberty IT
Hugh McCamphill
Principal Software Engineer in Test
Writing Automation since 2004
@hughleo01
@hughleo01 Liberty Information Technology
4Liberty IT @hughleo01
Which tests are slow
Which steps are slow
By Martin Lewison from Forest Hills, NY, U.S.A. - Cedar Point and Oberlin's Commencement Uploaded by
Astros4477, CC BY-SA 2.0, https://commons.wikimedia.org/w/index.php?curid=27339553
@hughleo01 Liberty Information Technology
5Liberty IT @hughleo01
Tumbling Dice : Decorated Anti-tank Blocks near Harkess Rocks, Bamburgh, Northumberland
Image Copyright Richard West. This work is licensed under the Creative Commons Attribution-Share Alike
2.0 Generic Licence
Which tests are intermittently passing
What are common failures across tests
@hughleo01 Liberty Information Technology
6Liberty IT @hughleo01
“Create commons cute robot mural, Toronto"
by https://www.flickr.com/photos/margonaut/ is licensed under CC BY 2.0
Where are the issues?
@hughleo01 Liberty Information Technology
7Liberty IT @hughleo01
@hughleo01 Liberty Information Technology
8Liberty IT @hughleo01
@hughleo01 Liberty Information Technology
9Liberty IT @hughleo01
~ Henry David Thoreau
@hughleo01 Liberty Information Technology
10Liberty IT @hughleo01
@hughleo01 Liberty Information Technology
11Liberty IT @hughleo01
Example dashboard
@hughleo01 Liberty Information Technology
12Liberty IT @hughleo01
RunTests Results
Post
Results
A Simple Process Flow
@hughleo01 Liberty Information Technology
13Liberty IT @hughleo01
A time series database is a
software system that is optimized
for handling time series data,
arrays of numbers indexed
by time
Elasticsearch is a tool for storing, searching, and
analyzing structured and unstructured data
@hughleo01 Liberty Information Technology
14Liberty IT @hughleo01
Setting up a mapping
{"mappings":{"result":{
"properties":{
"project":{"type":"string"},
"environment":{"type":"string"},
"duration":{"type":"double"},
"name":{"type":"string","index":"not_analyzed"},
"classname":{"type":"string","index":"not_analyzed"},
"message":{"type":"string","index":"not_analyzed"},
"status":{"type":"string"},
"created":{"type":"date","format":"yyyy-MM-dd HH:mm:ss"}}}}
@hughleo01 Liberty Information Technology
15Liberty IT @hughleo01
Setting up a mapping
{"mappings":{"result":{
"properties":{
"project":{"type":"string"},
"environment":{"type":"string"},
"duration":{"type":"double"},
"name":{"type":"string","index":"not_analyzed"},
"classname":{"type":"string","index":"not_analyzed"},
"message":{"type":"string","index":"not_analyzed"},
"status":{"type":"string"},
"created":{"type":"date","format":"yyyy-MM-dd HH:mm:ss"}}}}
@hughleo01 Liberty Information Technology
16Liberty IT @hughleo01
Setting up a mapping
{"mappings":{"result":{
"properties":{
"project":{"type":"string"},
"environment":{"type":"string"},
"duration":{"type":"double"},
"name":{"type":"string","index":"not_analyzed"},
"classname":{"type":"string","index":"not_analyzed"},
"message":{"type":"string","index":"not_analyzed"},
"status":{"type":"string"},
"created":{"type":"date","format":"yyyy-MM-dd HH:mm:ss"}}}}
@hughleo01 Liberty Information Technology
17Liberty IT @hughleo01
Setting up a mapping
{"mappings":{"result":{
"properties":{
"project":{"type":"string"},
"environment":{"type":"string"},
"duration":{"type":"double"},
"name":{"type":"string","index":"not_analyzed"},
"classname":{"type":"string","index":"not_analyzed"},
"message":{"type":"string","index":"not_analyzed"},
"status":{"type":"string"},
"created":{"type":"date","format":"yyyy-MM-dd HH:mm:ss"}}}}
curl -XPUT “http://localhost:9200/test-result” –
d @mapping_schema_framework.json
@hughleo01 Liberty Information Technology
18Liberty IT @hughleo01
{"index":{"_index":"test-result","_type":"functional" }}
{"project":“TestProject",
"environment":"QA",
"classname":“TestProject.Foo.Bar",
"name":"ShouldBeAbleToAddItem",
"duration":"47.12",
"created":"2016-09-14 21:35:53",
"status":"Passed"}
curl -XPOST “http://localhost:9200/test-
result/_bulk?pretty” --data-binary @testResults.json
Posting of results
@hughleo01 Liberty Information Technology
19Liberty IT @hughleo01
A Simple Process Flow
RunTests Results
Post
Results Visualize
@hughleo01 Liberty Information Technology
20Liberty IT @hughleo01
Kibana is an open source data visualization plugin for ElasticSearch
@hughleo01 Liberty Information Technology
21Liberty IT @hughleo01
@hughleo01 Liberty Information Technology
22Liberty IT @hughleo01
New visualization results in a single bucket
@hughleo01 Liberty Information Technology
23Liberty IT @hughleo01
Splitting error messages into buckets of top occurring error messages
@hughleo01 Liberty Information Technology
24Liberty IT @hughleo01
Showing top ten most occurring error messages
@hughleo01 Liberty Information Technology
25Liberty IT @hughleo01
Splitting error messages into buckets of tests affected by that error
@hughleo01 Liberty Information Technology
26Liberty IT @hughleo01
Error Messages
Individual Tests
Splitting error messages into buckets of tests affected by that error
@hughleo01 Liberty Information Technology
27Liberty IT @hughleo01
Top Failure Reasons By Test
@hughleo01 Liberty Information Technology
28Liberty IT @hughleo01
Top Slowest Tests
@hughleo01 Liberty Information Technology
29Liberty IT @hughleo01
Top Failing Tests
@hughleo01 Liberty Information Technology
30Liberty IT @hughleo01
Step Time Versus Number Of Times Executed
@hughleo01 Liberty Information Technology
31Liberty IT @hughleo01
Step Time Versus Number Of Times Executed
@hughleo01 Liberty Information Technology
32Liberty IT @hughleo01
Step Time Versus Number Of Times Executed
@hughleo01 Liberty Information Technology
33Liberty IT @hughleo01
Example Dashboard With Time Filter
@hughleo01 Liberty Information Technology
34Liberty IT @hughleo01
References
http://martinfowler.com/articles/nonDeterminism.htm
https://watirmelon.blog/2015/11/11/your-tests-arent-flaky/
@hughleo01 Liberty Information Technology
35Liberty IT @hughleo01
Thank you
Hugh McCamphill
@hughleo01
Run Tests Results
Post
Results Visualize
In Summary

Más contenido relacionado

La actualidad más candente

Towards Knowledge Graph based Representation, Augmentation and Exploration of...
Towards Knowledge Graph based Representation, Augmentation and Exploration of...Towards Knowledge Graph based Representation, Augmentation and Exploration of...
Towards Knowledge Graph based Representation, Augmentation and Exploration of...
Sören Auer
 

La actualidad más candente (20)

Knowledge Graph Introduction
Knowledge Graph IntroductionKnowledge Graph Introduction
Knowledge Graph Introduction
 
The Bounties of Semantic Data Integration for the Enterprise
The Bounties of Semantic Data Integration for the Enterprise The Bounties of Semantic Data Integration for the Enterprise
The Bounties of Semantic Data Integration for the Enterprise
 
Applying large scale text analytics with graph databases
Applying large scale text analytics with graph databasesApplying large scale text analytics with graph databases
Applying large scale text analytics with graph databases
 
Enterprise knowledge graphs
Enterprise knowledge graphsEnterprise knowledge graphs
Enterprise knowledge graphs
 
Transforming Your Data with GraphDB: GraphDB Fundamentals, Jan 2018
Transforming Your Data with GraphDB: GraphDB Fundamentals, Jan 2018Transforming Your Data with GraphDB: GraphDB Fundamentals, Jan 2018
Transforming Your Data with GraphDB: GraphDB Fundamentals, Jan 2018
 
Tools and techniques for data science
Tools and techniques for data scienceTools and techniques for data science
Tools and techniques for data science
 
Pandas, Data Wrangling & Data Science
Pandas, Data Wrangling & Data SciencePandas, Data Wrangling & Data Science
Pandas, Data Wrangling & Data Science
 
Integration of data ninja services with oracle spatial and graph
Integration of data ninja services with oracle spatial and graphIntegration of data ninja services with oracle spatial and graph
Integration of data ninja services with oracle spatial and graph
 
Do it on your own - From 3 to 5 Star Linked Open Data with RMLio
Do it on your own - From 3 to 5 Star Linked Open Data with RMLioDo it on your own - From 3 to 5 Star Linked Open Data with RMLio
Do it on your own - From 3 to 5 Star Linked Open Data with RMLio
 
Towards Knowledge Graph based Representation, Augmentation and Exploration of...
Towards Knowledge Graph based Representation, Augmentation and Exploration of...Towards Knowledge Graph based Representation, Augmentation and Exploration of...
Towards Knowledge Graph based Representation, Augmentation and Exploration of...
 
Linking Open, Big Data Using Semantic Web Technologies - An Introduction
Linking Open, Big Data Using Semantic Web Technologies - An IntroductionLinking Open, Big Data Using Semantic Web Technologies - An Introduction
Linking Open, Big Data Using Semantic Web Technologies - An Introduction
 
[Conference] Cognitive Graph Analytics on Company Data and News
[Conference] Cognitive Graph Analytics on Company Data and News[Conference] Cognitive Graph Analytics on Company Data and News
[Conference] Cognitive Graph Analytics on Company Data and News
 
Knowledge graphs on the Web
Knowledge graphs on the WebKnowledge graphs on the Web
Knowledge graphs on the Web
 
Powerful Information Discovery with Big Knowledge Graphs –The Offshore Leaks ...
Powerful Information Discovery with Big Knowledge Graphs –The Offshore Leaks ...Powerful Information Discovery with Big Knowledge Graphs –The Offshore Leaks ...
Powerful Information Discovery with Big Knowledge Graphs –The Offshore Leaks ...
 
A New Year in Data Science: ML Unpaused
A New Year in Data Science: ML UnpausedA New Year in Data Science: ML Unpaused
A New Year in Data Science: ML Unpaused
 
iRODS UGM 2018 Fair data management and DISQOVERability
iRODS UGM 2018 Fair data management and DISQOVERabilityiRODS UGM 2018 Fair data management and DISQOVERability
iRODS UGM 2018 Fair data management and DISQOVERability
 
Towards an Open Research Knowledge Graph
Towards an Open Research Knowledge GraphTowards an Open Research Knowledge Graph
Towards an Open Research Knowledge Graph
 
Python for Data Science
Python for Data SciencePython for Data Science
Python for Data Science
 
First Steps in Semantic Data Modelling and Search & Analytics in the Cloud
First Steps in Semantic Data Modelling and Search & Analytics in the CloudFirst Steps in Semantic Data Modelling and Search & Analytics in the Cloud
First Steps in Semantic Data Modelling and Search & Analytics in the Cloud
 
Property graph vs. RDF Triplestore comparison in 2020
Property graph vs. RDF Triplestore comparison in 2020Property graph vs. RDF Triplestore comparison in 2020
Property graph vs. RDF Triplestore comparison in 2020
 

Destacado

Excuse me, sir, do you have a moment to talk about tests in Kotlin
Excuse me, sir, do you have a moment to talk about tests in KotlinExcuse me, sir, do you have a moment to talk about tests in Kotlin
Excuse me, sir, do you have a moment to talk about tests in Kotlin
leonsabr
 

Destacado (20)

Colorful world-of-visual-automation-testing-latest
Colorful world-of-visual-automation-testing-latestColorful world-of-visual-automation-testing-latest
Colorful world-of-visual-automation-testing-latest
 
End-to-End Test Automation for Both Horizontal and Vertical Scale
End-to-End Test Automation for Both Horizontal and Vertical ScaleEnd-to-End Test Automation for Both Horizontal and Vertical Scale
End-to-End Test Automation for Both Horizontal and Vertical Scale
 
Protractor framework – how to make stable e2e tests for Angular applications
Protractor framework – how to make stable e2e tests for Angular applicationsProtractor framework – how to make stable e2e tests for Angular applications
Protractor framework – how to make stable e2e tests for Angular applications
 
Better Bullshit Driven Development [SeleniumCamp 2017]
Better Bullshit Driven Development [SeleniumCamp 2017]Better Bullshit Driven Development [SeleniumCamp 2017]
Better Bullshit Driven Development [SeleniumCamp 2017]
 
Testing Metrics - Making your tests visible
Testing Metrics - Making your tests visibleTesting Metrics - Making your tests visible
Testing Metrics - Making your tests visible
 
Grading the Quality of Selenium Tests
Grading the Quality of Selenium TestsGrading the Quality of Selenium Tests
Grading the Quality of Selenium Tests
 
Design patterns in test automation
Design patterns in test automationDesign patterns in test automation
Design patterns in test automation
 
Excuse me, sir, do you have a moment to talk about tests in Kotlin
Excuse me, sir, do you have a moment to talk about tests in KotlinExcuse me, sir, do you have a moment to talk about tests in Kotlin
Excuse me, sir, do you have a moment to talk about tests in Kotlin
 
SoapUI one key to all doors
SoapUI one key to all doorsSoapUI one key to all doors
SoapUI one key to all doors
 
How does Java 8 exert hidden power on Test Automation?
How does Java 8 exert hidden power on Test Automation?How does Java 8 exert hidden power on Test Automation?
How does Java 8 exert hidden power on Test Automation?
 
WixAutomation - Test State Pattern - Selenium Camp 2017
WixAutomation - Test State Pattern - Selenium Camp 2017WixAutomation - Test State Pattern - Selenium Camp 2017
WixAutomation - Test State Pattern - Selenium Camp 2017
 
Test Automation Architecture in Microservices
Test Automation Architecture in MicroservicesTest Automation Architecture in Microservices
Test Automation Architecture in Microservices
 
An easy way to automate complex UI
An easy way to automate complex UIAn easy way to automate complex UI
An easy way to automate complex UI
 
Fabulous Tests on Spock and Groovy
Fabulous Tests on Spock and GroovyFabulous Tests on Spock and Groovy
Fabulous Tests on Spock and Groovy
 
Page Objects Done Right - selenium conference 2014
Page Objects Done Right - selenium conference 2014Page Objects Done Right - selenium conference 2014
Page Objects Done Right - selenium conference 2014
 
Unit Testing TypeScript
Unit Testing TypeScriptUnit Testing TypeScript
Unit Testing TypeScript
 
Combining Heritrix and PhantomJS for Better Crawling of Pages with Javascript
Combining Heritrix and PhantomJS for Better Crawling of Pages with JavascriptCombining Heritrix and PhantomJS for Better Crawling of Pages with Javascript
Combining Heritrix and PhantomJS for Better Crawling of Pages with Javascript
 
Testing NodeJS, REST APIs and MongoDB with UFT
Testing NodeJS, REST APIs and MongoDB with UFTTesting NodeJS, REST APIs and MongoDB with UFT
Testing NodeJS, REST APIs and MongoDB with UFT
 
Angular js automation using protractor
Angular js automation using protractorAngular js automation using protractor
Angular js automation using protractor
 
API Testing with Frisby and Mocha
API Testing with Frisby and MochaAPI Testing with Frisby and Mocha
API Testing with Frisby and Mocha
 

Similar a Test trend analysis: Towards robust reliable and timely tests

The Semantic Knowledge Graph
The Semantic Knowledge GraphThe Semantic Knowledge Graph
The Semantic Knowledge Graph
Trey Grainger
 
Building a Real-time Data Pipeline: Apache Kafka at LinkedIn
Building a Real-time Data Pipeline: Apache Kafka at LinkedInBuilding a Real-time Data Pipeline: Apache Kafka at LinkedIn
Building a Real-time Data Pipeline: Apache Kafka at LinkedIn
DataWorks Summit
 

Similar a Test trend analysis: Towards robust reliable and timely tests (20)

Searching for Meaning
Searching for MeaningSearching for Meaning
Searching for Meaning
 
The Relevance of the Apache Solr Semantic Knowledge Graph
The Relevance of the Apache Solr Semantic Knowledge GraphThe Relevance of the Apache Solr Semantic Knowledge Graph
The Relevance of the Apache Solr Semantic Knowledge Graph
 
How to Build a Semantic Search System
How to Build a Semantic Search SystemHow to Build a Semantic Search System
How to Build a Semantic Search System
 
The Semantic Knowledge Graph
The Semantic Knowledge GraphThe Semantic Knowledge Graph
The Semantic Knowledge Graph
 
Docker Summit MongoDB - Data Democratization
Docker Summit MongoDB - Data Democratization Docker Summit MongoDB - Data Democratization
Docker Summit MongoDB - Data Democratization
 
Building a Real-Time Data Pipeline: Apache Kafka at LinkedIn
Building a Real-Time Data Pipeline: Apache Kafka at LinkedInBuilding a Real-Time Data Pipeline: Apache Kafka at LinkedIn
Building a Real-Time Data Pipeline: Apache Kafka at LinkedIn
 
Koshy june27 140pm_room210_c_v4
Koshy june27 140pm_room210_c_v4Koshy june27 140pm_room210_c_v4
Koshy june27 140pm_room210_c_v4
 
Building a Real-time Data Pipeline: Apache Kafka at LinkedIn
Building a Real-time Data Pipeline: Apache Kafka at LinkedInBuilding a Real-time Data Pipeline: Apache Kafka at LinkedIn
Building a Real-time Data Pipeline: Apache Kafka at LinkedIn
 
From keyword-based search to language-agnostic semantic search
From keyword-based search to language-agnostic semantic searchFrom keyword-based search to language-agnostic semantic search
From keyword-based search to language-agnostic semantic search
 
President Election of Korea in 2017
President Election of Korea in 2017President Election of Korea in 2017
President Election of Korea in 2017
 
Thinkful DC - Intro to Data Science
Thinkful DC - Intro to Data Science Thinkful DC - Intro to Data Science
Thinkful DC - Intro to Data Science
 
Application Development & Database Choices: Postgres Support for non Relation...
Application Development & Database Choices: Postgres Support for non Relation...Application Development & Database Choices: Postgres Support for non Relation...
Application Development & Database Choices: Postgres Support for non Relation...
 
Graphs & Big Data - Philip Rathle and Andreas Kollegger @ Big Data Science Me...
Graphs & Big Data - Philip Rathle and Andreas Kollegger @ Big Data Science Me...Graphs & Big Data - Philip Rathle and Andreas Kollegger @ Big Data Science Me...
Graphs & Big Data - Philip Rathle and Andreas Kollegger @ Big Data Science Me...
 
Getting to Know Your Data with R
Getting to Know Your Data with RGetting to Know Your Data with R
Getting to Know Your Data with R
 
Agile Data Science 2.0
Agile Data Science 2.0Agile Data Science 2.0
Agile Data Science 2.0
 
Bigdatacooltools
BigdatacooltoolsBigdatacooltools
Bigdatacooltools
 
Paper presentation
Paper presentationPaper presentation
Paper presentation
 
Agile Data Science 2.0
Agile Data Science 2.0Agile Data Science 2.0
Agile Data Science 2.0
 
MongoDB Schema Design: Practical Applications and Implications
MongoDB Schema Design: Practical Applications and ImplicationsMongoDB Schema Design: Practical Applications and Implications
MongoDB Schema Design: Practical Applications and Implications
 
Introduction to PySpark
Introduction to PySparkIntroduction to PySpark
Introduction to PySpark
 

Último

CHEAP Call Girls in Pushp Vihar (-DELHI )🔝 9953056974🔝(=)/CALL GIRLS SERVICE
CHEAP Call Girls in Pushp Vihar (-DELHI )🔝 9953056974🔝(=)/CALL GIRLS SERVICECHEAP Call Girls in Pushp Vihar (-DELHI )🔝 9953056974🔝(=)/CALL GIRLS SERVICE
CHEAP Call Girls in Pushp Vihar (-DELHI )🔝 9953056974🔝(=)/CALL GIRLS SERVICE
9953056974 Low Rate Call Girls In Saket, Delhi NCR
 
AI Mastery 201: Elevating Your Workflow with Advanced LLM Techniques
AI Mastery 201: Elevating Your Workflow with Advanced LLM TechniquesAI Mastery 201: Elevating Your Workflow with Advanced LLM Techniques
AI Mastery 201: Elevating Your Workflow with Advanced LLM Techniques
VictorSzoltysek
 
TECUNIQUE: Success Stories: IT Service provider
TECUNIQUE: Success Stories: IT Service providerTECUNIQUE: Success Stories: IT Service provider
TECUNIQUE: Success Stories: IT Service provider
mohitmore19
 

Último (20)

Optimizing AI for immediate response in Smart CCTV
Optimizing AI for immediate response in Smart CCTVOptimizing AI for immediate response in Smart CCTV
Optimizing AI for immediate response in Smart CCTV
 
call girls in Vaishali (Ghaziabad) 🔝 >༒8448380779 🔝 genuine Escort Service 🔝✔️✔️
call girls in Vaishali (Ghaziabad) 🔝 >༒8448380779 🔝 genuine Escort Service 🔝✔️✔️call girls in Vaishali (Ghaziabad) 🔝 >༒8448380779 🔝 genuine Escort Service 🔝✔️✔️
call girls in Vaishali (Ghaziabad) 🔝 >༒8448380779 🔝 genuine Escort Service 🔝✔️✔️
 
Pharm-D Biostatistics and Research methodology
Pharm-D Biostatistics and Research methodologyPharm-D Biostatistics and Research methodology
Pharm-D Biostatistics and Research methodology
 
%in tembisa+277-882-255-28 abortion pills for sale in tembisa
%in tembisa+277-882-255-28 abortion pills for sale in tembisa%in tembisa+277-882-255-28 abortion pills for sale in tembisa
%in tembisa+277-882-255-28 abortion pills for sale in tembisa
 
How To Troubleshoot Collaboration Apps for the Modern Connected Worker
How To Troubleshoot Collaboration Apps for the Modern Connected WorkerHow To Troubleshoot Collaboration Apps for the Modern Connected Worker
How To Troubleshoot Collaboration Apps for the Modern Connected Worker
 
VTU technical seminar 8Th Sem on Scikit-learn
VTU technical seminar 8Th Sem on Scikit-learnVTU technical seminar 8Th Sem on Scikit-learn
VTU technical seminar 8Th Sem on Scikit-learn
 
Right Money Management App For Your Financial Goals
Right Money Management App For Your Financial GoalsRight Money Management App For Your Financial Goals
Right Money Management App For Your Financial Goals
 
call girls in Vaishali (Ghaziabad) 🔝 >༒8448380779 🔝 genuine Escort Service 🔝✔️✔️
call girls in Vaishali (Ghaziabad) 🔝 >༒8448380779 🔝 genuine Escort Service 🔝✔️✔️call girls in Vaishali (Ghaziabad) 🔝 >༒8448380779 🔝 genuine Escort Service 🔝✔️✔️
call girls in Vaishali (Ghaziabad) 🔝 >༒8448380779 🔝 genuine Escort Service 🔝✔️✔️
 
CHEAP Call Girls in Pushp Vihar (-DELHI )🔝 9953056974🔝(=)/CALL GIRLS SERVICE
CHEAP Call Girls in Pushp Vihar (-DELHI )🔝 9953056974🔝(=)/CALL GIRLS SERVICECHEAP Call Girls in Pushp Vihar (-DELHI )🔝 9953056974🔝(=)/CALL GIRLS SERVICE
CHEAP Call Girls in Pushp Vihar (-DELHI )🔝 9953056974🔝(=)/CALL GIRLS SERVICE
 
Introducing Microsoft’s new Enterprise Work Management (EWM) Solution
Introducing Microsoft’s new Enterprise Work Management (EWM) SolutionIntroducing Microsoft’s new Enterprise Work Management (EWM) Solution
Introducing Microsoft’s new Enterprise Work Management (EWM) Solution
 
MarTech Trend 2024 Book : Marketing Technology Trends (2024 Edition) How Data...
MarTech Trend 2024 Book : Marketing Technology Trends (2024 Edition) How Data...MarTech Trend 2024 Book : Marketing Technology Trends (2024 Edition) How Data...
MarTech Trend 2024 Book : Marketing Technology Trends (2024 Edition) How Data...
 
AI Mastery 201: Elevating Your Workflow with Advanced LLM Techniques
AI Mastery 201: Elevating Your Workflow with Advanced LLM TechniquesAI Mastery 201: Elevating Your Workflow with Advanced LLM Techniques
AI Mastery 201: Elevating Your Workflow with Advanced LLM Techniques
 
TECUNIQUE: Success Stories: IT Service provider
TECUNIQUE: Success Stories: IT Service providerTECUNIQUE: Success Stories: IT Service provider
TECUNIQUE: Success Stories: IT Service provider
 
The Top App Development Trends Shaping the Industry in 2024-25 .pdf
The Top App Development Trends Shaping the Industry in 2024-25 .pdfThe Top App Development Trends Shaping the Industry in 2024-25 .pdf
The Top App Development Trends Shaping the Industry in 2024-25 .pdf
 
Direct Style Effect Systems - The Print[A] Example - A Comprehension Aid
Direct Style Effect Systems -The Print[A] Example- A Comprehension AidDirect Style Effect Systems -The Print[A] Example- A Comprehension Aid
Direct Style Effect Systems - The Print[A] Example - A Comprehension Aid
 
%in Stilfontein+277-882-255-28 abortion pills for sale in Stilfontein
%in Stilfontein+277-882-255-28 abortion pills for sale in Stilfontein%in Stilfontein+277-882-255-28 abortion pills for sale in Stilfontein
%in Stilfontein+277-882-255-28 abortion pills for sale in Stilfontein
 
Unveiling the Tech Salsa of LAMs with Janus in Real-Time Applications
Unveiling the Tech Salsa of LAMs with Janus in Real-Time ApplicationsUnveiling the Tech Salsa of LAMs with Janus in Real-Time Applications
Unveiling the Tech Salsa of LAMs with Janus in Real-Time Applications
 
Microsoft AI Transformation Partner Playbook.pdf
Microsoft AI Transformation Partner Playbook.pdfMicrosoft AI Transformation Partner Playbook.pdf
Microsoft AI Transformation Partner Playbook.pdf
 
%in Bahrain+277-882-255-28 abortion pills for sale in Bahrain
%in Bahrain+277-882-255-28 abortion pills for sale in Bahrain%in Bahrain+277-882-255-28 abortion pills for sale in Bahrain
%in Bahrain+277-882-255-28 abortion pills for sale in Bahrain
 
HR Software Buyers Guide in 2024 - HRSoftware.com
HR Software Buyers Guide in 2024 - HRSoftware.comHR Software Buyers Guide in 2024 - HRSoftware.com
HR Software Buyers Guide in 2024 - HRSoftware.com
 

Test trend analysis: Towards robust reliable and timely tests

  • 1. @hughleo01 Liberty Information Technology Liberty IT, Belfast and Dublin Test trend analysis:Towards robust, reliable and timely tests Hugh McCamphill
  • 2. @hughleo01 Liberty Information Technology 2Liberty IT @hughleo01 Indianapolis Dover Belfast GS LIU DCs Europe RDC Russia DRDC Redmond DC Global DRDC & Utilities Kansas City DC Latin America RDC GS LIU DRDC Portsmouth DC GS LIU DC Turkey DCs India DCs Asia RDC China DCs Wausau Fairfield Boston Seattle Blanchardstown 11 major IT offices 7 country data centers (DCs) DRDC = Disaster Recovery DC 3 US domestic data centers (DCs) 4 international regional data centers LIU = Liberty International Underwriters More than 5,000+ IT employees around the globe Australia About Us
  • 3. 3Liberty IT Hugh McCamphill Principal Software Engineer in Test Writing Automation since 2004 @hughleo01
  • 4. @hughleo01 Liberty Information Technology 4Liberty IT @hughleo01 Which tests are slow Which steps are slow By Martin Lewison from Forest Hills, NY, U.S.A. - Cedar Point and Oberlin's Commencement Uploaded by Astros4477, CC BY-SA 2.0, https://commons.wikimedia.org/w/index.php?curid=27339553
  • 5. @hughleo01 Liberty Information Technology 5Liberty IT @hughleo01 Tumbling Dice : Decorated Anti-tank Blocks near Harkess Rocks, Bamburgh, Northumberland Image Copyright Richard West. This work is licensed under the Creative Commons Attribution-Share Alike 2.0 Generic Licence Which tests are intermittently passing What are common failures across tests
  • 6. @hughleo01 Liberty Information Technology 6Liberty IT @hughleo01 “Create commons cute robot mural, Toronto" by https://www.flickr.com/photos/margonaut/ is licensed under CC BY 2.0 Where are the issues?
  • 7. @hughleo01 Liberty Information Technology 7Liberty IT @hughleo01
  • 8. @hughleo01 Liberty Information Technology 8Liberty IT @hughleo01
  • 9. @hughleo01 Liberty Information Technology 9Liberty IT @hughleo01 ~ Henry David Thoreau
  • 10. @hughleo01 Liberty Information Technology 10Liberty IT @hughleo01
  • 11. @hughleo01 Liberty Information Technology 11Liberty IT @hughleo01 Example dashboard
  • 12. @hughleo01 Liberty Information Technology 12Liberty IT @hughleo01 RunTests Results Post Results A Simple Process Flow
  • 13. @hughleo01 Liberty Information Technology 13Liberty IT @hughleo01 A time series database is a software system that is optimized for handling time series data, arrays of numbers indexed by time Elasticsearch is a tool for storing, searching, and analyzing structured and unstructured data
  • 14. @hughleo01 Liberty Information Technology 14Liberty IT @hughleo01 Setting up a mapping {"mappings":{"result":{ "properties":{ "project":{"type":"string"}, "environment":{"type":"string"}, "duration":{"type":"double"}, "name":{"type":"string","index":"not_analyzed"}, "classname":{"type":"string","index":"not_analyzed"}, "message":{"type":"string","index":"not_analyzed"}, "status":{"type":"string"}, "created":{"type":"date","format":"yyyy-MM-dd HH:mm:ss"}}}}
  • 15. @hughleo01 Liberty Information Technology 15Liberty IT @hughleo01 Setting up a mapping {"mappings":{"result":{ "properties":{ "project":{"type":"string"}, "environment":{"type":"string"}, "duration":{"type":"double"}, "name":{"type":"string","index":"not_analyzed"}, "classname":{"type":"string","index":"not_analyzed"}, "message":{"type":"string","index":"not_analyzed"}, "status":{"type":"string"}, "created":{"type":"date","format":"yyyy-MM-dd HH:mm:ss"}}}}
  • 16. @hughleo01 Liberty Information Technology 16Liberty IT @hughleo01 Setting up a mapping {"mappings":{"result":{ "properties":{ "project":{"type":"string"}, "environment":{"type":"string"}, "duration":{"type":"double"}, "name":{"type":"string","index":"not_analyzed"}, "classname":{"type":"string","index":"not_analyzed"}, "message":{"type":"string","index":"not_analyzed"}, "status":{"type":"string"}, "created":{"type":"date","format":"yyyy-MM-dd HH:mm:ss"}}}}
  • 17. @hughleo01 Liberty Information Technology 17Liberty IT @hughleo01 Setting up a mapping {"mappings":{"result":{ "properties":{ "project":{"type":"string"}, "environment":{"type":"string"}, "duration":{"type":"double"}, "name":{"type":"string","index":"not_analyzed"}, "classname":{"type":"string","index":"not_analyzed"}, "message":{"type":"string","index":"not_analyzed"}, "status":{"type":"string"}, "created":{"type":"date","format":"yyyy-MM-dd HH:mm:ss"}}}} curl -XPUT “http://localhost:9200/test-result” – d @mapping_schema_framework.json
  • 18. @hughleo01 Liberty Information Technology 18Liberty IT @hughleo01 {"index":{"_index":"test-result","_type":"functional" }} {"project":“TestProject", "environment":"QA", "classname":“TestProject.Foo.Bar", "name":"ShouldBeAbleToAddItem", "duration":"47.12", "created":"2016-09-14 21:35:53", "status":"Passed"} curl -XPOST “http://localhost:9200/test- result/_bulk?pretty” --data-binary @testResults.json Posting of results
  • 19. @hughleo01 Liberty Information Technology 19Liberty IT @hughleo01 A Simple Process Flow RunTests Results Post Results Visualize
  • 20. @hughleo01 Liberty Information Technology 20Liberty IT @hughleo01 Kibana is an open source data visualization plugin for ElasticSearch
  • 21. @hughleo01 Liberty Information Technology 21Liberty IT @hughleo01
  • 22. @hughleo01 Liberty Information Technology 22Liberty IT @hughleo01 New visualization results in a single bucket
  • 23. @hughleo01 Liberty Information Technology 23Liberty IT @hughleo01 Splitting error messages into buckets of top occurring error messages
  • 24. @hughleo01 Liberty Information Technology 24Liberty IT @hughleo01 Showing top ten most occurring error messages
  • 25. @hughleo01 Liberty Information Technology 25Liberty IT @hughleo01 Splitting error messages into buckets of tests affected by that error
  • 26. @hughleo01 Liberty Information Technology 26Liberty IT @hughleo01 Error Messages Individual Tests Splitting error messages into buckets of tests affected by that error
  • 27. @hughleo01 Liberty Information Technology 27Liberty IT @hughleo01 Top Failure Reasons By Test
  • 28. @hughleo01 Liberty Information Technology 28Liberty IT @hughleo01 Top Slowest Tests
  • 29. @hughleo01 Liberty Information Technology 29Liberty IT @hughleo01 Top Failing Tests
  • 30. @hughleo01 Liberty Information Technology 30Liberty IT @hughleo01 Step Time Versus Number Of Times Executed
  • 31. @hughleo01 Liberty Information Technology 31Liberty IT @hughleo01 Step Time Versus Number Of Times Executed
  • 32. @hughleo01 Liberty Information Technology 32Liberty IT @hughleo01 Step Time Versus Number Of Times Executed
  • 33. @hughleo01 Liberty Information Technology 33Liberty IT @hughleo01 Example Dashboard With Time Filter
  • 34. @hughleo01 Liberty Information Technology 34Liberty IT @hughleo01 References http://martinfowler.com/articles/nonDeterminism.htm https://watirmelon.blog/2015/11/11/your-tests-arent-flaky/
  • 35. @hughleo01 Liberty Information Technology 35Liberty IT @hughleo01 Thank you Hugh McCamphill @hughleo01 Run Tests Results Post Results Visualize In Summary