Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood
Automated Reporting of GUI Design Violations for Mobile Apps
1. Kevin Moran,
Boyang Li,
Carlos Bernal-Cárdenas,
Dan Jelf, and
Denys Poshyvanyk
Automated Reporting of GUI
DesignViolations for Mobile Apps
40th International Conference
on Software Engineering (ICSE’18)
Wednesday, May 30th, 2018
29. EXISTING WORK
• Utilizes Perceptual Image Differencing (PID) for detecting Differences
• Performs clustering of difference areas and detects potential faulty html elements
• Predicts Root Causes by Probabilistic modeling
Detecting Presentation Failures in Web Applications*
*Detection and Localization of HTML Presentation Failures Using Computer Vision-Based Techniques. Sonal Mahajan, William G. J. Halfond. In
Proceedings of the 8th IEEE International Conference on Software Testing, Verification and Validation (ICST). April 2015.
*Using Visual Symptoms for Debugging Presentation Failures in Web Applications. Sonal Mahajan, Bailan Li, Pooyan Behnamghader, William G.J.
Halfond. In Proceedings of the 9th IEEE International Conference on Software Testing, Verification, and Validation (ICST). April 2016.
32. EXISTING LIMITATIONS
• Tailored for web pages and is not directly applicable to native mobile
app GUIs
• Does not generate expressive reports to facilitate communication
between teams
33. EXISTING LIMITATIONS
• Tailored for web pages and is not directly applicable to native mobile
app GUIs
• Does not generate expressive reports to facilitate communication
between teams
• Predictive modeling of visual symptoms may not be precise enough
62. List of Potential
DesignViolations
Layout DV Resolver
• <x,y>
• <height,width>
Text DV Resolver
• <image>
• <text> CHA + CQ
Resource DV Resolver
• <image>
• Missing/
Extra CHA + CQ
STAGE 3: DESIGN VIOLATION DETECTION
79. STUDY 1: EFFECTIVENESS AND PERFORMANCE
XML and
Image Manipulation
Synthetic DV
Injection
Runtime
Screenshots & XML
80. STUDY 1: EFFECTIVENESS AND PERFORMANCE
LayoutViolations
TextViolations
ResourceViolations
100 Screens
200 DVs
XML and
Image Manipulation
Synthetic DV
Injection
Runtime
Screenshots & XML
82. STUDY 2: GVT UTILITY
LayoutViolations
TextViolations
ResourceViolations
45 Screens
40 DesignViolations
10 Academic and
Industrial Android Developers
25%
CP, DP, R
Usability and
User Preference
Questions
83. STUDY 2: RESULTS
Easy to understand?
Reports were Useful?
Would Ike to use often?
Hard to Read?
Too Complex?
0 1.25 2.5 3.75 5
Developer UX/UP
Responses
Developer CP, DP and R
AgreeDisagree
84. STUDY 3: INDUSTRIAL APPLICABILITY
20 Industrial Android
Developers & Designers
Usability and
User Preference
Questions
GVTTool
Semi-Structured
Interviews with
Managers
85. STUDY 3: RESULTS
0 1 2 3 4 5
Industrial Developer/Designer Responses
AgreeDisagree
Using GVT tool helped to improve the quality of
mobile applications produced by Huawei
The GVT tool helped my team communicate with
other teams regarding GUI design violations
The GVT tool is able to accurately report existing
design violations in production applications
The GVT has helped you to reduce the time
required for verifying design violations
The GVT tool allowed for better transfer of
the design to the implementation of the app
88. STUDY 3: RESULTS
“When the page is more complicated, the tool is more helpful.”
“[DVs] can be detected pretty well... [but the tool is] not very
flexible. For example, a switch component in the design is
open, but the switch is closed in the implementation"
89. STUDY 3: RESULTS
“When the page is more complicated, the tool is more helpful.”
“[DVs] can be detected pretty well... [but the tool is] not very
flexible. For example, a switch component in the design is
open, but the switch is closed in the implementation"
“Currently, most errors are layout errors, so [the] tool is
accurate. Static components are basically detected, [but]
maybe the next extension should focus on dynamic
components.”
90. STUDY 3: RESULTS
“When the page is more complicated, the tool is more helpful.”
“[DVs] can be detected pretty well... [but the tool is] not very
flexible. For example, a switch component in the design is
open, but the switch is closed in the implementation"
“Currently, most errors are layout errors, so [the] tool is
accurate. Static components are basically detected, [but]
maybe the next extension should focus on dynamic
components.”
“If you consider the time savings on
discussion and arguments between the two
teams, this tool saves us a lot of time”
94. • Add support for detecting design violations for dynamic content
FUTURE WORK
95. • Add support for detecting design violations for dynamic content
• Improve the user interface and usability of the tool
FUTURE WORK
96. • Add support for detecting design violations for dynamic content
• Improve the user interface and usability of the tool
• Gather further feedback from developers and designers
FUTURE WORK
110. LIMITATIONS
• Impose restrictions on the app design process
• Cannot enforce the implementation of design specifications
• Can be expensive or prohibitive to operate
111. PROBLEM FORMALIZATION
Screens and GUI-Components
DesignViolations
S = {GC1{GC2{GCi}, GC3}}<latexit sha1_base64="fUnIc/N9nW4lcCs00VSw9yHs4w0=">AAACCnicdVDLSgMxFM3UV62vqks3oUVwIUNmqn0shGIXuqxotdApJZOmbWjmQZIRyjB7N/6KGxcqbv0Cd/6N6bSCih7IzeGce0nucUPOpELow8gsLC4tr2RXc2vrG5tb+e2daxlEgtAWCXgg2i6WlDOfthRTnLZDQbHncnrjjhtT/+aWCskC/0pNQtr18NBnA0aw0lIvX7iEJ9CJzxo9K612WpmTHEJ9l5zESXr5IjLtqnVctiAySwhVENLErpWtqg0tE6Uogjmavfy70w9I5FFfEY6l7FgoVN0YC8UIp0nOiSQNMRnjIe1o6mOPym6c7pLAfa304SAQ+vgKpur3iRh7Uk48V3d6WI3kb28q/uV1IjWodmPmh5GiPpk9NIg4VAGcBgP7TFCi+EQTTATTf4VkhAUmSseX0yF8bQr/Jy3brJno4qhYP52nkQV7oAAOgAUqoA7OQRO0AAF34AE8gWfj3ng0XozXWWvGmM/sgh8w3j4BN/aY3g==</latexit><latexit sha1_base64="fUnIc/N9nW4lcCs00VSw9yHs4w0=">AAACCnicdVDLSgMxFM3UV62vqks3oUVwIUNmqn0shGIXuqxotdApJZOmbWjmQZIRyjB7N/6KGxcqbv0Cd/6N6bSCih7IzeGce0nucUPOpELow8gsLC4tr2RXc2vrG5tb+e2daxlEgtAWCXgg2i6WlDOfthRTnLZDQbHncnrjjhtT/+aWCskC/0pNQtr18NBnA0aw0lIvX7iEJ9CJzxo9K612WpmTHEJ9l5zESXr5IjLtqnVctiAySwhVENLErpWtqg0tE6Uogjmavfy70w9I5FFfEY6l7FgoVN0YC8UIp0nOiSQNMRnjIe1o6mOPym6c7pLAfa304SAQ+vgKpur3iRh7Uk48V3d6WI3kb28q/uV1IjWodmPmh5GiPpk9NIg4VAGcBgP7TFCi+EQTTATTf4VkhAUmSseX0yF8bQr/Jy3brJno4qhYP52nkQV7oAAOgAUqoA7OQRO0AAF34AE8gWfj3ng0XozXWWvGmM/sgh8w3j4BN/aY3g==</latexit><latexit sha1_base64="fUnIc/N9nW4lcCs00VSw9yHs4w0=">AAACCnicdVDLSgMxFM3UV62vqks3oUVwIUNmqn0shGIXuqxotdApJZOmbWjmQZIRyjB7N/6KGxcqbv0Cd/6N6bSCih7IzeGce0nucUPOpELow8gsLC4tr2RXc2vrG5tb+e2daxlEgtAWCXgg2i6WlDOfthRTnLZDQbHncnrjjhtT/+aWCskC/0pNQtr18NBnA0aw0lIvX7iEJ9CJzxo9K612WpmTHEJ9l5zESXr5IjLtqnVctiAySwhVENLErpWtqg0tE6Uogjmavfy70w9I5FFfEY6l7FgoVN0YC8UIp0nOiSQNMRnjIe1o6mOPym6c7pLAfa304SAQ+vgKpur3iRh7Uk48V3d6WI3kb28q/uV1IjWodmPmh5GiPpk9NIg4VAGcBgP7TFCi+EQTTATTf4VkhAUmSseX0yF8bQr/Jy3brJno4qhYP52nkQV7oAAOgAUqoA7OQRO0AAF34AE8gWfj3ng0XozXWWvGmM/sgh8w3j4BN/aY3g==</latexit>
(GCm
i ⇡ GCr
j ) ^ (GCm
i 6= GCr
j )
=) DV 2 {GCm
i , GCr
j }<latexit sha1_base64="xgaNEjkEl6iZoQFEvQTxuPwBpk4=">AAACTHicdVFNbxMxFPQGaEv4aKBHLk9ESEVCK2/oR3qrWiR6LBJpK8Vp5PW+pKa2d7G90Gi1f7AXJG78Cy4cKELCSRNREIxkaTQzT34ep4WSzlP6JWrcun1naXnlbvPe/QcPV1uPHh+5vLQCeyJXuT1JuUMlDfa89ApPCotcpwqP0/P9qX/8Aa2TuXnrJwUONB8bOZKC+yANWxmsv94fVrI+1cB4Udj8AqbCu/rUPgf2EbMx3owYfP/bZwyY1GFNdPDqKHADrFpkXyxirB622jTudJPNrQRo/JLSbUoD6exsJd0OJDGdoU3mOBy2PrMsF6VG44XizvUTWvhBxa2XQmHdZKXDgotzPsZ+oIZrdINq1kYNz4KSwSi34RgPM/XmRMW1cxOdhqTm/sz97U3Ff3n90o+6g0qaovRoxPVFo1KBz2FaLWTSovBqEggXVoZdQZxxy4UPH9AMJSxeCv8nvU68E9M3G+3dvXkbK+QJeUrWSUK2yS45IIekRwS5JF/Jd3IVfYq+RT+in9fRRjSfWSN/oLH0C+ISsuQ=</latexit><latexit sha1_base64="xgaNEjkEl6iZoQFEvQTxuPwBpk4=">AAACTHicdVFNbxMxFPQGaEv4aKBHLk9ESEVCK2/oR3qrWiR6LBJpK8Vp5PW+pKa2d7G90Gi1f7AXJG78Cy4cKELCSRNREIxkaTQzT34ep4WSzlP6JWrcun1naXnlbvPe/QcPV1uPHh+5vLQCeyJXuT1JuUMlDfa89ApPCotcpwqP0/P9qX/8Aa2TuXnrJwUONB8bOZKC+yANWxmsv94fVrI+1cB4Udj8AqbCu/rUPgf2EbMx3owYfP/bZwyY1GFNdPDqKHADrFpkXyxirB622jTudJPNrQRo/JLSbUoD6exsJd0OJDGdoU3mOBy2PrMsF6VG44XizvUTWvhBxa2XQmHdZKXDgotzPsZ+oIZrdINq1kYNz4KSwSi34RgPM/XmRMW1cxOdhqTm/sz97U3Ff3n90o+6g0qaovRoxPVFo1KBz2FaLWTSovBqEggXVoZdQZxxy4UPH9AMJSxeCv8nvU68E9M3G+3dvXkbK+QJeUrWSUK2yS45IIekRwS5JF/Jd3IVfYq+RT+in9fRRjSfWSN/oLH0C+ISsuQ=</latexit><latexit sha1_base64="xgaNEjkEl6iZoQFEvQTxuPwBpk4=">AAACTHicdVFNbxMxFPQGaEv4aKBHLk9ESEVCK2/oR3qrWiR6LBJpK8Vp5PW+pKa2d7G90Gi1f7AXJG78Cy4cKELCSRNREIxkaTQzT34ep4WSzlP6JWrcun1naXnlbvPe/QcPV1uPHh+5vLQCeyJXuT1JuUMlDfa89ApPCotcpwqP0/P9qX/8Aa2TuXnrJwUONB8bOZKC+yANWxmsv94fVrI+1cB4Udj8AqbCu/rUPgf2EbMx3owYfP/bZwyY1GFNdPDqKHADrFpkXyxirB622jTudJPNrQRo/JLSbUoD6exsJd0OJDGdoU3mOBy2PrMsF6VG44XizvUTWvhBxa2XQmHdZKXDgotzPsZ+oIZrdINq1kYNz4KSwSi34RgPM/XmRMW1cxOdhqTm/sz97U3Ff3n90o+6g0qaovRoxPVFo1KBz2FaLWTSovBqEggXVoZdQZxxy4UPH9AMJSxeCv8nvU68E9M3G+3dvXkbK+QJeUrWSUK2yS45IIekRwS5JF/Jd3IVfYq+RT+in9fRRjSfWSN/oLH0C+ISsuQ=</latexit>
(<x,y><height,width><text><image>)
Presentation Failures
{DV1, DV2, ...DVi} 2 {GCm
i , GCr
j }
PF 2 {Sm
, Sr
}<latexit sha1_base64="yXwZIsk0lnWWH3xnETRAGLTCsiI=">AAACVXicdVFdb9MwFHWyMUb5WIHHvVirkHioIifA1r1NDI09dhrdJtVd5LjOamY7wb5BVFH+JLzAT+FlmtO10kBwJVtH55x7bR9npZIOCPkVhGvrDzYebj7qPH7y9NlW9/mLM1dUlosRL1RhLzLmhJJGjECCEhelFUxnSpxn14etfv5VWCcL8wnmpZhodmVkLjkDT6VdTUF8A6trmTeYfqnYFNP6w1ka97Hfk34URS2Q1KvSeO3jYVrL5lL3W/C5ubS0obSzmgIzYVZzhkfLllPvxqetM+32SJQM4ne7MSbRG0L2CPEg2d+NBwmOI7KoHlrWMO1+p9OCV1oY4Io5N45JCZOaWZBciaZDKydKxq/ZlRh7aJgWblIvYmnwK89McV5YvwzgBXu/o2baubnOvFMzmLm/tZb8lzauIB9MamnKCoThdwfllcJQ4DZjPJVWcFBzDxi30t8V8xmzjIP/iY4PYfVS/H8wSqL9iJy87R28X6axibbRDnqNYrSHDtAxGqIR4ugH+h2EwVrwM7gJ18ONO2sYLHteoj8q3LoFJMWyDg==</latexit><latexit sha1_base64="yXwZIsk0lnWWH3xnETRAGLTCsiI=">AAACVXicdVFdb9MwFHWyMUb5WIHHvVirkHioIifA1r1NDI09dhrdJtVd5LjOamY7wb5BVFH+JLzAT+FlmtO10kBwJVtH55x7bR9npZIOCPkVhGvrDzYebj7qPH7y9NlW9/mLM1dUlosRL1RhLzLmhJJGjECCEhelFUxnSpxn14etfv5VWCcL8wnmpZhodmVkLjkDT6VdTUF8A6trmTeYfqnYFNP6w1ka97Hfk34URS2Q1KvSeO3jYVrL5lL3W/C5ubS0obSzmgIzYVZzhkfLllPvxqetM+32SJQM4ne7MSbRG0L2CPEg2d+NBwmOI7KoHlrWMO1+p9OCV1oY4Io5N45JCZOaWZBciaZDKydKxq/ZlRh7aJgWblIvYmnwK89McV5YvwzgBXu/o2baubnOvFMzmLm/tZb8lzauIB9MamnKCoThdwfllcJQ4DZjPJVWcFBzDxi30t8V8xmzjIP/iY4PYfVS/H8wSqL9iJy87R28X6axibbRDnqNYrSHDtAxGqIR4ugH+h2EwVrwM7gJ18ONO2sYLHteoj8q3LoFJMWyDg==</latexit><latexit sha1_base64="yXwZIsk0lnWWH3xnETRAGLTCsiI=">AAACVXicdVFdb9MwFHWyMUb5WIHHvVirkHioIifA1r1NDI09dhrdJtVd5LjOamY7wb5BVFH+JLzAT+FlmtO10kBwJVtH55x7bR9npZIOCPkVhGvrDzYebj7qPH7y9NlW9/mLM1dUlosRL1RhLzLmhJJGjECCEhelFUxnSpxn14etfv5VWCcL8wnmpZhodmVkLjkDT6VdTUF8A6trmTeYfqnYFNP6w1ka97Hfk34URS2Q1KvSeO3jYVrL5lL3W/C5ubS0obSzmgIzYVZzhkfLllPvxqetM+32SJQM4ne7MSbRG0L2CPEg2d+NBwmOI7KoHlrWMO1+p9OCV1oY4Io5N45JCZOaWZBciaZDKydKxq/ZlRh7aJgWblIvYmnwK89McV5YvwzgBXu/o2baubnOvFMzmLm/tZb8lzauIB9MamnKCoThdwfllcJQ4DZjPJVWcFBzDxi30t8V8xmzjIP/iY4PYfVS/H8wSqL9iJy87R28X6axibbRDnqNYrSHDtAxGqIR4ugH+h2EwVrwM7gJ18ONO2sYLHteoj8q3LoFJMWyDg==</latexit>
118. HUAWEI COLLABORATION METHODOLOGY
• Bi-weekly meetings with designers and developers
• More than 12 releases of the tool, starting with a CLI
119. HUAWEI COLLABORATION METHODOLOGY
• Bi-weekly meetings with designers and developers
• More than 12 releases of the tool, starting with a CLI
• Acceptance tests designed by UI/UX experts
120. HUAWEI COLLABORATION METHODOLOGY
• Bi-weekly meetings with designers and developers
• More than 12 releases of the tool, starting with a CLI
• Acceptance tests designed by UI/UX experts
• Deployed to a controlled user base for feedback