1. 1
TOPIC
Running Kafka and Spark on Raspberry PI
with Azure and some .NET magic
Marco Parenzan

2. 2
Thanks to our partners

3. Senior Solution Architect in beanTech
Microsoft Azure MVP
Community Lead 1nn0va // Pordenone
1nn0va After Hour (ogni martedì sera alle 21)
https://www.youtube.com/c/Community1nn0va
Linkedin: https://www.linkedin.com/in/marcoparenzan/
Marco Parenzan

4. Marco Parenzan
marco_parenzan
marcoparenzan
marcoparenzan
Marco Parenzan

5. @cloudgen_verona
#CloudGen2021

6. 6
.NET for data(science)
• .NET Spark
• .NET Interactive (notebooks)
• Azure Synapse Analytics
• ML.NET…
All good for IoT scenarios!

7. 7
.NET for ARM/edge
.NET (Core)
Jeff Geerling (You Tube)

8. 8
More ARM

9. ““Questo matrimonio
non s’ha da fare…”

10. “ Sì….può….fareeeee!

11. 11
Edge Scenario
IoT Edge…quale è lo scenario equivalente OpenSource?
Non voglio usare i container…voglio sfruttare una rete di rpi

12. 12
Apache Kafka and Apache Spark

13. 13
VNET
The complete vision
Edge00
PI3B+
Device(GPIO)
Edge01
PI400/4
Kafka
Edge02
PI4/4
Spark
Edge99
IoTHub
Nano00
ESP32WROOM
Device//GPIO
TP-LINK
MR6400 4G
Edge03
PI3B+
(MQTT Broker)
VPN
Private Network?
Azure
Edge
192.168.2.100
192.168.2.101 192.168.2.102 192.168.2.103

14. 14

15. 15

16. 16

17. 17
Enabled SSH and VNC
DHCP with MAC
reservation
Issues:
- networking: PI does not
broadcast its name out
of the box(?!?!?!)
- Samba/NFS/…
- Reaching (issues with
Windows 10)

18. 18
Install .NET 6.0 on Raspberry PI
Download
https://dotnet.microsoft.com/download/dotnet/thank-
you/sdk-6.0.100-rc.1-linux-arm32-binaries
sudo mkdir -p $HOME/.dotnet
sudo tar zxf dotnet-sdk-3.1.100-linux-arm.tar.gz -C
$HOME/.dotnet
echo 'export DOTNET_ROOT=$HOME/.dotnet' >> ~/.bashrc
echo 'export PATH=$PATH:$HOME/.dotnet' >> ~/.bashrc
source ~/.bashrc
• On edge00 and edge03
(edge02 has 3.1 for .net
spark)
https://docs.microsoft.com/en-us/dotnet/iot/deployment

19. 19
Visual Studio Code
It works!
• And install VSCode
(https://code.visualstudio.com/docs/setup/raspberry-pi)
• sudo apt update
• sudo apt install code
• But...

20. 20
Minimal Visual Studio 2019 for NanoFramework in the 2022 era

21. 21
No Containers?
• Not using containers
• More simple manipulation of config files
• Images not ready for ARM

22. 22
The scenario
WordCount as a Sample (WCaaS )
Wordsignal, command, event (vote?)
19 can become «event1»..»event9»
In device terms: a button on digital input for each «word»

23. 23
Edge00: PI3B+ device gw
Produce events
Send to Kafka
Issues:
• Kafka written in Java
• But std .NET package uses native libkafka (C++) from
Confluent that is available only for x86
• Require rebuilding (not found a ready rebuild)
• Used a native implementation (not maintained)
• Not tested on ARM
• Not clean working
• MQTT listener on kafka does not exist (only plugins
controlled by kafka)

24. 24
Edge01: PI400 Kafka
Collect events events with Kafka
Install OpenJDK11
Issues
• Consumes lot of memory (required PI4)
• Issues with configuration (listeners, also on WSL)
• Complicated configuration (SASL-SCRAM, certificate
store)Industrial scenario (tampering)

25. 25
Edge02: PI4 Spark
Works on PI3 (even if too slow!)
Aggregating events
Install OpenJDK8 for 2.4.8 (not 11)
Issues
• Issues(?) with 3.x (not really tested)
• Alignment between .net spark with spark
• Problems with watermarking that is fundamental on
streaming

26. 26
Edge03:
• Handle processed data and route to the cloud
• Planned installing an MQTT broker (Mosquitto?)
• Installed .NET 6
• Monitor files on a shared folder on edge02
• sudo mkdir /mnt/data
• sudo apt-get install sshfs
• sudo sshfs -o allow_other,default_permissions
pi@edge02.local:/home/pi/data /mnt/data
• Issues
• <empty>?
• Broker not tested

28. 28

29. 29
Minus/Issues
• PI3 is slowOnly PI4+
• PI4 is slow for desktop
• Kafka is difficult to configure
• Spark has a complicated processing/execution model
• In general documentation is sparse and not homogeneus
(PI/Kafka/Spark)

30. 30
Plus
• .NET 6 works fine
• Also Java works fine Kafka and Spark
• PI4 is powerful
• It can be a potential platform for Kafka and Spark on edge

31. 31
Some links
https://github.com/dotnet/spark
https://docs.microsoft.com/en-
us/dotnet/spark/tutorials/get-started?tabs=linux
https://code.visualstudio.com/docs/setup/raspberry-pi
https://dotnet.microsoft.com/download/dotnet/thank-
you/sdk-3.1.413-linux-arm32-binaries
https://github.com/dotnet/spark/releases
https://docs.microsoft.com/en-us/dotnet/iot/deployment
https://mvnrepository.com/artifact/org.apache.spark/spark-
streaming_2.12/3.0.1

32. Thank you
Any questions?
marcoparenzan marco_parenzan marcoparenzan
https://github.com/marcoparenzan/kafka-spark-on-pi-with-dotnet