The document discusses upgrading servers for customers remotely. It proposes: 1. Customers subscribe to an MQTT broker to receive upgrade notifications. 2. When a new firmware is released, customers are notified via email. 3. Customers can select devices to upgrade from their dashboard and initiate the upgrade by sending an MQTT message to the upgrade appliance.

1. lundi 13 mars 2023
VMAF DEPLOYMENT & UPGRADE
Thierry GAYET

2.  Because the VMAF server will need to de deployed to bytel, the
question on how NN6 will deliver the firmware is now an actual
question.
 In order to not be too specific to TESTTREE, a thinking have been done
in order to see how server may be install or upgraded by using generic
and common way to proceed.
GOAL / INTRODUCTION
2

3. INTRODUCTION

4. VMAF – ARCH.
4
STREAMPROBE
Rabbit MQ
VMAF
Pool of vmaf ip address
(scalling) HTTPS
REST
API
+
OAUTH2
PROTOCOL : HTTPS
IPV4(s)
PORTS
OAUTH2 TOKEN
IPV4
PORT
USERNAME
PASSWORD
QUEUE_NAME

5.  The streamprobe gray interface manage :
 Database (mariadb start/stop, reset db
 Network (ip interfaces, DNS, NTP
 Security (iptables, ldaps, fail2ban, rsyslog)
 Storage (fstab, create partition, format, LUKS encryption)
 Upgrade
 Boot
 Actions (reboot,shutdown, hardware inventory
STREAMPROBE’S BUILDROOT
5

6.  The VMAF appliance should not be specific to testtree bu must be
usable by any project
 That’s why the vmaf appliance has been designed with standard
interfaces :
 A standard REST API for commands/ input request
 A common message broker (rabbitMQ) for responses
GOAL
6

7. DEVSECOPS CYCLE

8. DEVSECOPS CYCLE
8

9. DEVSECOPS CYCLE
9

10. DEVSECOPS CYCLE
10

11. NEW APPROACH FOR UPGRADING

12. 12
INTERNET
CUSTOMER
NOSQL
DB
BACKEND
FRONTEND /
GUI
DOCKER REGISTRY
PACKAGE REGISTRY
LICENCE REGISTRY
PRODUCTION
INTEGRATION
DEVELOPMENT
NN6 NETWORK
MQTT
MQTT CLIENT
GITLAB-CI/CD

13.  A customer :
 View all of his devices with their version and licence
 View all release in relation of the devices
 Can synchronise upgrade items with the cutomer upgrade appliance
 Can download upgrade items within a pgp encrypted archive
 Can launch an upgrade process from the main ui to the customer’s site
 NN6 team :
 View all device with their current release
 Locate all customer on a worldmap
 Investigate on equipment using encrypted vpn (over a ssl tummel)
REQUIREMENTS
13

14. 14
SECURED
NETWORK
DMZ
INTERNET
NN6 NETWORK
DOCKER REGISTRY
(CUSTOMER)
UPGRADE BACKEND
• HELM / CHART
• PROVISIONNING (ANSIBLE,
SALT STACK, … )
• …
….
manual
deploy
DOCKER REGISTRY
(PROD)
MQTT broker
MQTT
CLIENT
registry
monitoring
Manual/auto
docker sync
Docker
push
(subscribe)
(publish)
https://mqtt.enensys.com
dockerregistry.enensys.com
CUSTOMER NETWORK
BACKEND
FRONTEND / GUI
NOSQL DB
https://dashboard.enensys.com
MQTT CLIENT
PACKAGE REGISTRY
(PROD)
DOCKER REGISTRY
(CUSTOMER)
Package
push
packerregistry.enensys.com
LOCAL
GUI
LICENCE REGISTRY
lm.enensys.com LICENCE
CUSTOMER)
UPGRADE
APPLIANCE

15. INSTALLING A SERVER

16.  Streamprobe
 Baremetal
 Native BIOS : boot from usb key generated from the img
 Idrack (DELL) : using the img key
 Ipmi (SUPERMICRO) : boot from usb key generated from the img
 VMWARE ESXI
 Ova image
 Mediacast
 Baremetal (supermicro)
 VMWARE ESXI
 Boot from an ISO image then launch an installer that register RPMS
Inventory of the way to install (1/2)
16

17.  Smartgate vt / vt2
 Baremetal :
 Boot from ISO image then install docker (docker swarm orchestrator)
 VMWARE ESXI
 Boot from ISO image then install packages
 eBox (medicast Mobile + lte gcsas server)
 Fedora image install manually then install qemu images (gcow2)
Inventory of the way to install (2/2)
17

18.  Which ALMA release V8 or v9 ?
 Using an ISO image burn on an USB key, or directly the ISO image
(http://mirror.almalinux.ikoula.com/9.2/isos/x86_64/)
 Compatible with:
 VMWARE ESXI
 BAREMETAL
 IDRACK (DELL)
 IPMI (SUPERMICRO)
ALMA
18

19.  May use ALMA linux but we require ALPINE as possible
 Does not need any base install
 HELM/CHARTs is the best deployment method to push docker
container and/or services
K8S
19

20. UPGRADING A SERVER

21.  A customer may :
 Have several location with UPGRADE APPLIANCE
 Inventory all local appliance (by type/release)
 Thus, upgrade may control several point through MQTT connexion
Requrements
21

22. 22
SECURED
NETWORK
DMZ
INTERNET
NN6 NETWORK
DOCKER REGISTRY
(CUSTOMER)
UPGRADE BACKEND
• HELM / CHART
• PROVISIONNING (ANSIBLE,
SALT STACK, … )
• …
….
manual
deploy
DOCKER REGISTRY
(PROD)
MQTT broker
MQTT
CLIENT
registry
monitoring
Manual/auto
docker sync
Docker
push
(subscribe)
(publish)
https://mqtt.enensys.com
dockerregistry.enensys.com
CUSTOMER NETWORK
BACKEND
NOSQL DB
https://dashboard.enensys.com
MQTT CLIENT
PACKAGE REGISTRY
(PROD)
DOCKER REGISTRY
(CUSTOMER)
Package
push
packerregistry.enensys.com
LOCAL
GUI
LICENCE REGISTRY
lm.enensys.com LICENCE
CUSTOMER)
FRONTEND / GUI
UPGRADE
APPLIANCE
Licence
push
Architecture based on an ALMA OS
Architecture based on an ALMA OS
ALMA OS

23. 23
SECURED
NETWORK
DMZ
INTERNET
NN6 NETWORK
DOCKER REGISTRY
(CUSTOMER)
UPGRADE BACKEND
• HELM / CHART
• PROVISIONNING (ANSIBLE,
SALT STACK, … )
• …
….
manual
deploy
DOCKER REGISTRY
(PROD)
MQTT broker
MQTT
CLIENT
registry
monitoring
Manual/auto
docker sync
Docker
push
(subscribe)
(publish)
https://mqtt.enensys.com
dockerregistry.enensys.com
CUSTOMER NETWORK
BACKEND
NOSQL DB
https://dashboard.enensys.com
MQTT CLIENT
PACKAGE REGISTRY
(PROD)
DOCKER REGISTRY
(CUSTOMER)
Package
push
packerregistry.enensys.com
LOCAL
GUI
LICENCE REGISTRY
lm.enensys.com LICENCE
CUSTOMER)
FRONTEND / GUI
UPGRADE
APPLIANCE
Licence
push
Architecture based on Kubernetes (K8S)
K8S

24. Upgrade for customer network connected
to Internet (ONLINE)

25. 25
SECURED
NETWORK
DMZ
INTERNET
NN6 NETWORK
DOCKER REGISTRY
(CUSTOMER)
UPGRADE BACKEND
• HELM / CHART
• PROVISIONNING (ANSIBLE,
SALT STACK, … )
• …
….
manual
deploy
DOCKER REGISTRY
(PROD)
MQTT broker
MQTT
CLIENT
registry
monitoring
Manual/auto
docker sync
Docker
push
(subscribe)
(publish)
https://mqtt.enensys.com
dockerregistry.enensys.com
CUSTOMER NETWORK
BACKEND
NOSQL DB
https://dashboard.enensys.com
MQTT CLIENT
PACKAGE REGISTRY
(PROD)
DOCKER REGISTRY
(CUSTOMER)
Package
push
packerregistry.enensys.com
LOCAL
GUI
LICENCE REGISTRY
lm.enensys.com LICENCE
CUSTOMER)
0. At the very beginning the customer network (DMZ) subscribe to the MQTT
FRONTEND / GUI
UPGRADE
APPLIANCE
Licence
push

26. 26
SECURED
NETWORK
DMZ
INTERNET
NN6 NETWORK
DOCKER REGISTRY
(CUSTOMER)
UPGRADE BACKEND
• HELM / CHART
• PROVISIONNING (ANSIBLE,
SALT STACK, … )
• …
….
manual
deploy
DOCKER REGISTRY
(PROD)
MQTT broker
MQTT
CLIENT
registry
monitoring
Manual/auto
docker sync
Docker
push
(subscribe)
(publish)
https://mqtt.enensys.com
dockerregistry.enensys.com
CUSTOMER NETWORK
BACKEND
NOSQL DB
https://dashboard.enensys.com
MQTT CLIENT
PACKAGE REGISTRY
(PROD)
DOCKER REGISTRY
(CUSTOMER)
Package
push
packerregistry.enensys.com
LOCAL
GUI
LICENCE REGISTRY
lm.enensys.com LICENCE
CUSTOMER)
1. A new firwmare is released officially, an email may be sent to the customer
by the backend
FRONTEND / GUI
UPGRADE
APPLIANCE
Licence
push

27. 27
SECURED
NETWORK
DMZ
INTERNET
NN6 NETWORK
DOCKER REGISTRY
(CUSTOMER)
UPGRADE BACKEND
• HELM / CHART
• PROVISIONNING (ANSIBLE,
SALT STACK, … )
• …
….
manual
deploy
DOCKER REGISTRY
(PROD)
MQTT broker
MQTT
CLIENT
registry
monitoring
Manual/auto
docker sync
Docker
push
(subscribe)
(publish)
https://mqtt.enensys.com
dockerregistry.enensys.com
CUSTOMER NETWORK
BACKEND
FRONTEND / GUI
NOSQL DB
https://dashboard.enensys.com
MQTT CLIENT
PACKAGE REGISTRY
(PROD)
DOCKER REGISTRY
(CUSTOMER)
Package
push
packerregistry.enensys.com
LOCAL
GUI
LICENCE REGISTRY
lm.enensys.com LICENCE
CUSTOMER)
2. The backend will detect a new release for a customer devices and send an email
may be sent to the customer by the backend
UPGRADE
APPLIANCE
Licence
push

28. 28
SECURED
NETWORK
DMZ
INTERNET
NN6 NETWORK
DOCKER REGISTRY
(CUSTOMER)
UPGRADE BACKEND
• HELM / CHART
• PROVISIONNING (ANSIBLE,
SALT STACK, … )
• …
….
manual
deploy
DOCKER REGISTRY
(PROD)
MQTT broker
MQTT
CLIENT
registry
monitoring
Manual/auto
docker sync
Docker
push
(subscribe)
(publish)
https://mqtt.enensys.com
dockerregistry.enensys.com
CUSTOMER NETWORK
BACKEND
FRONTEND / GUI
NOSQL DB
https://dashboard.enensys.com
MQTT CLIENT
PACKAGE REGISTRY
(PROD)
DOCKER REGISTRY
(CUSTOMER)
Package
push
packerregistry.enensys.com
LOCAL
GUI
LICENCE REGISTRY
lm.enensys.com LICENCE
CUSTOMER)
3. Now, the customer check his dashboard of all its devices and the release
note and can decide which on to upgrade (just on, a subset or all)
UPGRADE
APPLIANCE
Licence
push

29. 29
SECURED
NETWORK
DMZ
INTERNET
NN6 NETWORK
DOCKER REGISTRY
(CUSTOMER)
UPGRADE BACKEND
• HELM / CHART
• PROVISIONNING (ANSIBLE,
SALT STACK, … )
• …
….
manual
deploy
DOCKER REGISTRY
(PROD)
MQTT broker
MQTT
CLIENT
registry
monitoring
Manual/auto
docker sync
Docker
push
(subscribe)
(publish)
https://mqtt.enensys.com
dockerregistry.enensys.com
CUSTOMER NETWORK
BACKEND
NOSQL DB
https://dashboard.enensys.com
MQTT CLIENT
PACKAGE REGISTRY
(PROD)
DOCKER REGISTRY
(CUSTOMER)
Package
push
packerregistry.enensys.com
LOCAL
GUI
LICENCE REGISTRY
lm.enensys.com LICENCE
CUSTOMER)
5. A message is sent to the upgrade appliance by sending a message
throught MQTT.
FRONTEND / GUI
UPGRADE
APPLIANCE
Licence
push

30. 30
SECURED
NETWORK
DMZ
INTERNET
NN6 NETWORK
DOCKER REGISTRY
(CUSTOMER)
UPGRADE BACKEND
• HELM / CHART
• PROVISIONNING (ANSIBLE,
SALT STACK, … )
• …
….
manual
deploy
DOCKER REGISTRY
(PROD)
MQTT broker
MQTT
CLIENT
registry
monitoring
Manual/auto
docker sync
Docker
push
(subscribe)
(publish)
https://mqtt.enensys.com
dockerregistry.enensys.com
CUSTOMER NETWORK
BACKEND
NOSQL DB
https://dashboard.enensys.com
MQTT CLIENT
PACKAGE REGISTRY
(PROD)
DOCKER REGISTRY
(CUSTOMER)
Package
push
packerregistry.enensys.com
LOCAL
GUI
LICENCE REGISTRY
lm.enensys.com LICENCE
CUSTOMER)
6. A synchronisation is done to get the new firmware (docker, package,
licences, … )
FRONTEND / GUI
UPGRADE
APPLIANCE
Licence
push

31. 31
SECURED
NETWORK
DMZ
INTERNET
NN6 NETWORK
DOCKER REGISTRY
(CUSTOMER)
UPGRADE BACKEND
• HELM / CHART
• PROVISIONNING (ANSIBLE,
SALT STACK, … )
• …
….
manual
deploy
DOCKER REGISTRY
(PROD)
MQTT broker
MQTT
CLIENT
registry
monitoring
Manual/auto
docker sync
Docker
push
(subscribe)
(publish)
https://mqtt.enensys.com
dockerregistry.enensys.com
CUSTOMER NETWORK
BACKEND
NOSQL DB
https://dashboard.enensys.com
MQTT CLIENT
PACKAGE REGISTRY
(PROD)
DOCKER REGISTRY
(CUSTOMER)
Package
push
packerregistry.enensys.com
LOCAL
GUI
LICENCE REGISTRY
lm.enensys.com LICENCE
CUSTOMER)
7. Finaly the upgrade appliance upgrade all devices (by set, not all in the
same time), then send a feedback over MQTT to NN6 (for the support
team).
FRONTEND / GUI
UPGRADE
APPLIANCE
Licence
push

32. Upgrade for customer network not
connected to Internet (OFFLINE)

33. 33
SECURED
NETWORK
DMZ
INTERNET
NN6 NETWORK
DOCKER REGISTRY
(CUSTOMER)
UPGRADE BACKEND
• HELM / CHART
• PROVISIONNING (ANSIBLE,
SALT STACK, … )
• …
….
manual
deploy
DOCKER REGISTRY
(PROD)
MQTT broker
MQTT
CLIENT
registry
monitoring
Docker
push
https://mqtt.enensys.com
dockerregistry.enensys.com
CUSTOMER NETWORK
BACKEND
NOSQL DB
https://dashboard.enensys.com
MQTT CLIENT
PACKAGE REGISTRY
(PROD)
DOCKER REGISTRY
(CUSTOMER)
Package
push
packerregistry.enensys.com
LOCAL
GUI
LICENCE REGISTRY
lm.enensys.com LICENCE
CUSTOMER)
1. A new firwmare is released officially, an email may be sent to the customer
by the backend
FRONTEND / GUI
UPGRADE
APPLIANCE

34. 34
SECURED
NETWORK
DMZ
INTERNET
NN6 NETWORK
DOCKER REGISTRY
(CUSTOMER)
UPGRADE BACKEND
• HELM / CHART
• PROVISIONNING (ANSIBLE,
SALT STACK, … )
• …
….
manual
deploy
DOCKER REGISTRY
(PROD)
MQTT broker
MQTT
CLIENT
registry
monitoring
Docker
push
https://mqtt.enensys.com
dockerregistry.enensys.com
CUSTOMER NETWORK
BACKEND
NOSQL DB
https://dashboard.enensys.com
MQTT CLIENT
PACKAGE REGISTRY
(PROD)
DOCKER REGISTRY
(CUSTOMER)
Package
push
packerregistry.enensys.com
LOCAL
GUI
LICENCE REGISTRY
lm.enensys.com LICENCE
CUSTOMER)
2. The backend will detect a new release for a customer devices and send an email
may be sent to the customer by the backend
FRONTEND / GUI
UPGRADE
APPLIANCE
Licence
push

35. 35
SECURED
NETWORK
DMZ
INTERNET
NN6 NETWORK
DOCKER REGISTRY
(CUSTOMER)
UPGRADE BACKEND
• HELM / CHART
• PROVISIONNING (ANSIBLE,
SALT STACK, … )
• …
….
manual
deploy
DOCKER REGISTRY
(PROD)
MQTT broker
MQTT
CLIENT
registry
monitoring
Docker
push
https://mqtt.enensys.com
dockerregistry.enensys.com
CUSTOMER NETWORK
BACKEND
FRONTEND / GUI
NOSQL DB
https://dashboard.enensys.com
MQTT CLIENT
PACKAGE REGISTRY
(PROD)
DOCKER REGISTRY
(CUSTOMER)
Package
push
packerregistry.enensys.com
LOCAL
GUI
LICENCE REGISTRY
lm.enensys.com LICENCE
CUSTOMER)
3. Now, the customer check his dashboard of all its devices and the release
note and can decide which on to upgrade (just on, a subset or all)
UPGRADE
APPLIANCE
Licence
push

36. 36
SECURED
NETWORK
DMZ
INTERNET
NN6 NETWORK
DOCKER REGISTRY
(CUSTOMER)
UPGRADE BACKEND
• HELM / CHART
• PROVISIONNING (ANSIBLE,
SALT STACK, … )
• …
….
manual
deploy
DOCKER REGISTRY
(PROD)
MQTT broker
MQTT
CLIENT
registry
monitoring
Docker
push
https://mqtt.enensys.com
dockerregistry.enensys.com
CUSTOMER NETWORK
BACKEND
NOSQL DB
https://dashboard.enensys.com
MQTT CLIENT
PACKAGE REGISTRY
(PROD)
DOCKER REGISTRY
(CUSTOMER)
Package
push
packerregistry.enensys.com
LOCAL
GUI
LICENCE REGISTRY
lm.enensys.com LICENCE
CUSTOMER)
4. An encrypted (pgp) tarball is now downloaded
PGP
TARBALL
download
ENCRYPTED
TARBALL
(static update)
Generate
FRONTEND / GUI
UPGRADE
APPLIANCE
Licence
push

37. 37
SECURED
NETWORK
DMZ
INTERNET
NN6 NETWORK
DOCKER REGISTRY
(CUSTOMER)
UPGRADE BACKEND
• HELM / CHART
• PROVISIONNING (ANSIBLE,
SALT STACK, … )
• …
….
manual
deploy
DOCKER REGISTRY
(PROD)
MQTT broker
MQTT
CLIENT
registry
monitoring
Docker
push
https://mqtt.enensys.com
dockerregistry.enensys.com
CUSTOMER NETWORK
BACKEND
NOSQL DB
https://dashboard.enensys.com
MQTT CLIENT
PACKAGE REGISTRY
(PROD)
DOCKER REGISTRY
(CUSTOMER)
Package
push
packerregistry.enensys.com
LOCAL
GUI
LICENCE REGISTRY
lm.enensys.com LICENCE
CUSTOMER)
ENCRYPTED
TARBALL
(static update)
FRONTEND / GUI
UPGRADE
APPLIANCE
Licence
push
5. Now the cutomer can upload the encrypted (pgp) tarball that contains all update
for the customer’s devices.
Upload

38. 38
SECURED
NETWORK
DMZ
INTERNET
NN6 NETWORK
DOCKER REGISTRY
(CUSTOMER)
UPGRADE BACKEND
• HELM / CHART
• PROVISIONNING (ANSIBLE,
SALT STACK, … )
• …
….
manual
deploy
DOCKER REGISTRY
(PROD)
MQTT broker
MQTT
CLIENT
registry
monitoring
Manual/auto
docker sync
Docker
push
(subscribe)
(publish)
https://mqtt.enensys.com
dockerregistry.enensys.com
CUSTOMER NETWORK
BACKEND
NOSQL DB
https://dashboard.enensys.com
MQTT CLIENT
PACKAGE REGISTRY
(PROD)
DOCKER REGISTRY
(CUSTOMER)
Package
push
packerregistry.enensys.com
LOCAL
GUI
LICENCE REGISTRY
lm.enensys.com LICENCE
CUSTOMER)
6. Finaly the upgrade appliance upgrade all devices (by set, not all in the
same time) ; it may be interesting to got a feedback on the serveur
upgraded !
FRONTEND / GUI
UPGRADE
APPLIANCE
Licence
push

39. ENENSYS
4A rue des Buttes
CS 37734
35 577 Cesson-Sévigné – France
Phone (+33) 1 70 72 51 70
Email contact@test-tree.com
www.enensys.com
39