Niruphan & Karthikeyan delivered a session on "Adopt Non-Functional Testing to realise the full potential of Connected Devices" at ATAGTR2020 ATAGTR2020 was the 5th Edition of Global Testing Retreat. Niruphan is a Senior Non-Functional Testing Consultant with over 10 years of experience. Currently he is pursuing his IT career in London, United Kingdom Karthikeyan is a Non-Functional Lead with 8+ years of experience. Has excellent technical skills with a focus on continuous improvement and client centricity The video recording of the session is now available on the following link: https://youtu.be/5yenwStw4cw To know more about #ATAGTR2020, please visit: https://gtr.agiletestingalliance.org/

1. Adopt Non-Functional Testing to realise the
full potential of Connected Devices
Niruphan Rajendran
Cognizant Worldwide Limited
Karthikeyan
Lakshminarayanan
Cognizant Worldwide Limited

2. Introduction
IoT Ecosystem and Challenges
A View & A Deep-Dive on IoT NFT
Solution – An Overview
Solution – Non Functional Requirements
Benefits
AGENDA

3. Introduction
How companies can offer services that ensure
efficiency, safety of devices, smooth operations
and intuitive interface design?
How people can operate safely at home or office
without much impact caused by smart IoT
devices?
IoT
‘Hot topic for decades, but radically
improved to solve societal challenges’
NFT
‘Drive quality at speed with Customer
User Experience’
According to Gartner, the number of IoT devices is set to reach 25 billion by 2025
and worth of $1.1 trillion by 2026

4. IoT Ecosystem and Challenges
Connecting IoT devices [Device to Device] and testing its performance & security in near real time scenario is a key challenge. There is a need for
handling device messages every milli second in a real-time scenario is crucial aspect of Non-Functional testing.
KEY
CHALLENGES
1
Non-functional validation of end to end IoT
Ecosystem
2
Firmware validation and verification across IoT
Devices
3
Setup the continuous performance test
suite on IoT Devices
4
Industry Benchmarks and KPIs to test &
validate IoT Devices
Device / Client Layer
UI/Network Responsiveness
iOS
Android
End User
Mobile App
IoT Devices
Validation between Hub to DevicesPlatform to HubPlatform Verification & Validation
Cloud Platform
APIs
API GatewayDynamoDBLambda
Functions
Cosmos DB API
Management
APIs ZigbeeIoT Hub
SMART Devices
NFT Pain Point

5. A View & A Deep-Dive on IoT NFT
Hotspot:
▪ Test for peak hour, validate scalability to
peak volume
▪ App Unresponsiveness, Fragmentation
▪ Battery Consumption & Network provider
bandwidth
NFT Focus:
▪ Better page response and loading time
▪ Scalable & Reliable
▪ Availability & Maintainability
▪ Application Stability & Durability
Focus on End User first
Phase1
Device / Client Layer
App Validation and Data Consumption
iOS
Android
End User
Mobile App
Storage & Persistence
Cloud Platform
APIs
API GatewayDynamoDBLambda
Functions
Cosmos DB API
Management
IoT Devices
IoT Device and Data Ingestion/Simulation
APIs ZigbeeIoT Hub
SMART Devices
Tune backend systems next
Phase2
Hotspot:
▪ Server Capacity on each component
▪ Custom code for data processing
▪ API & Query performance
▪ Data Handling, Resource Consumption
▪ Processing Millions of messages
NFT Focus:
▪ Data consumption per second
▪ High Availability
▪ Better processing and staying powerful
▪ Data Management
Deep-dive into IoT devices
Phase3
Hotspot:
▪ Data Ingestion & Simulation using different
protocols
▪ Protocol Transmission
▪ Data processing & secure validation within
Device to Device
NFT Focus:
▪ Scalability, Stability, Security &
Sustainability
Is it possible to validate at IoT device
level?
• Turn on/off bulbs
• Change colour of the bulb
• Turn thermostat/heater on/off
• Turn on/off smart plug etc
Use
Cases

6. Solution – An Overview
Debug Adapter
Sniffing
Automated
transactions
Simulate
IoT Devices
Sniffer helped to get a view on below
▪ Custom code had built in, to pair & configure the device
▪ Simulate the messages (ex: Device ON or OFF) under different network conditions
▪ Sniff the calls using custom chip between two IoT devices to ensure the minimum edge nodes are connected to
give better response time
▪ Using customized monitoring tool check the routing mechanism to understand if request process meets the NFRs
▪ Monitor the encryption techniques to see if it meets the security standards
▪ Overall breakdown analysis on Device to Device communication
Performance
Phase3
▪ Security testing approach that uses binary code analysis to
ensure the security of device firmware
▪ Reversing the firmware binaries using open source tools &
customized code to extract and to perform automated scan to
identify the common threats
▪ Implementation of continuous source-code quality & security
assurance in IoT ecosystem
Security

7. Solution – Non Functional Requirements
NFT Category NFT Test Type Non-Functional Requirements
Performance
Load Simulation
Response time SLA b/w Centralized Device & IoT Device
Network loss between D2D
Server Attributes Frame Control in Zigbee Network
Poll Period Transmission
Fall Poll mode
Device Stability
Different & Maximum number of devices to be connected
Network Transmission &
Connectivity
Validate the Frame Control
Ensure Network provider bandwidth
Caching Compression
Node & Routing Mechanism
Node processing
Call transmission between client and server
Security
Encryption
Encryption methodology
Network Stability
Firmware Analysis
IoT Device Security
Static Device Firmware Analysis
Firmware Assurance
Prevention of exploitation of user space processes
This table illustrates different set of Non-Functional requirements at Device level which will evaluate performance and security at higher standards

8. Benefits
Aided 100% Reusability due to Automated Regression
suite set up on IoT Ecosystem to support frequent changes
Outline 25+ Non-Functional Requirements
at Device level to meet the standards
Security related issues and bad
practices implemented in firmware
files (both source & binaries)
Best-in Class framework to assure
different sets of Non-functional tests
This diverse solution helps across several industries
like Home Automation, Healthcare Industries etc.,
Authorize Resource Planning &
Optimization across landscape
Features can help IoT companies to track,
manage assets and create contactless
user experience

9. Q&A
THANK YOU