1. PRECISION AGRICULTURE SUPPORT
USING SCALA/SPARK
Project Report
SRIRAM RV
SPRING SEMESTER
ADVISOR: PROFESSOR BRAD RUBIN

2. 2
Table of Contents
1.0 PURPOSE OF PROJECT.......................................................................................................................4
2.0 PROJECT DESCRIPTION.....................................................................................................................4
2.0 Why Agriculture Data.........................................................................................................................5
3.0 DATASET ..............................................................................................................................................5
3.1 Data Source:........................................................................................................................................5
3.2 Details about Dataset: .........................................................................................................................5
3.3 Sample Data........................................................................................................................................5
Weather data .........................................................................................................................................5
Moisture Data........................................................................................................................................5
Image Data............................................................................................................................................6
3.4 Schema................................................................................................................................................6
Weather data .........................................................................................................................................6
Moisture Data........................................................................................................................................6
3.5 Data Description: ................................................................................................................................7
Weather Data: .......................................................................................................................................7
Moisture Data........................................................................................................................................7
4.0 PROJECT IMPLEMENTATION...........................................................................................................8
4.1 Data Ingestion using Kafka.................................................................................................................8
4.2 Kafka producer....................................................................................................................................8
4.4 Kafka Broker.......................................................................................................................................9
4.5 Kafka Consumer ...............................................................................................................................10
5.0 ADDITIONAL TOOLS........................................................................................................................10
5.1 Maven ...............................................................................................................................................10
5.2 Scala Build tool.................................................................................................................................11
5.3 Git .....................................................................................................................................................11
6.0 OUTPUT INTERPRETATION............................................................................................................12
7.0 IMPROVING THE KAFKA ARCHITECTURE.................................................................................12
7.1. Making kafka architecture more robust ...........................................................................................12
7.2. Having dedicated Kafka Broker to improve performance ...............................................................13

3. 3
8.0 FUTURE RESEARCH.........................................................................................................................13
9.0. CONCLUSION....................................................................................................................................13
BIBLIOGRAPHY.......................................................................................................................................14

4. 4
1.0 PURPOSE OF PROJECT
Big data tools over last few years has been focused on both structured and unstructured data.
However, image processing is one area where it needs more of attention and it has been my
area of interest too. With the help of this project, I will get an opportunity to experiment with
streaming images and weather data captured in the UST greenhouse, and also get a feel for
image processing with Scala/Spark on Hadoop more generally.
I will gain experience in technologies such as Scala, Spark, Spark streaming, and image
processing in the domain of food technology that will give me skills that I cannot otherwise
obtain in the GPS curriculum.
2.0 PROJECT DESCRIPTION
The purpose of the project is to stream real-time weather data captured by both direct sensors
and RGB images captured by the drones to perform image processing and weather data
analytics leveraging the Scala/Spark ecosystem on a Hadoop computing cluster. Since image
processing and streaming with Spark are knew technologies to GPS, part of the project will
focus on experimenting with different tools and find out more reliable way of storing images
and streamed data in HDFS.
The UST greenhouse will be growing plants for the Precision Agriculture project run by the UST
School of Engineering. The greenhouse has a local weather station that will be broadcasting
weather data such as temperature, humidity, light intensity, barometric pressure, position
(latitude/longitude), wind speed and direction and rainfall. The broadcast will be continuous at
10 second intervals (in CSV format) .The equipment in the greenhouse is a prototype for field
use which is useful for both analysis of plant health and creating a model for each of the six
plant species that will be grown. In addition, high resolution images will be taken of the plants
in the visible and near IR regions of the light spectrum. The periodicity of these images will be
every couple of days.

5. 5
2.0 Why Agriculture Data
With the help of agricultural data, I will get an opportunity to experiment withstreaming images
and weather data captured in the UST greenhouse. Data captured in greenhouse is so much detailed
and gives me experience on working with data from food technology.
3.0 DATASET
3.1 Data Source:
The data source used for this project is the live streaming of weather and moisture data captured using
sensors through Arduino chip and Streamed using Kafka producer.
3.2 Details about Dataset:
 The sensor data were captured for every second.
 Total number of days of weather data stored in HDFS is 90 days.
 Total number of days of moisture data stored in HDFS is 85days.
 Total number of days of image data stored is 90 days.
3.3 Sample Data
Weather data
Fig 1: Sensor Weather data from Arduino
Moisture Data

6. 6
Fig 2: Sensor data from Arduino
Image Data
Image data was captured every alternative day over a period of 90 days .
Fig 3: Images from the greenhouse
3.4 Schema
Weather data
Date Time Wind
direction
Wind
Speed
Humidity Temperature Rain Pressure Battery Light
Level
Table 1 : Weather Data Schema
Moisture Data
Date Time Moist
2
Moist
6
Moist
8
Moist
11
Moist
10
Moist
1
Moist
9
Moist
7
Moist
5
Temp Par
Table 2 : Weather Data Schema

7. 7
3.5 Data Description:
Weather Data:
Date & time : Timestamp of the recording
Wind Direction: Direction of wind
Wind Speed: Speed of wind
Wind Gust: Gust of wind
Humidity: Percentage of water in air
Temperature: Temperature
Rain: Rain percentage
Pressure: Air pressure
Battery: Battery of Arduino
Light: Light exposure
Moisture Data
Moist 2: Moisture of plot 2
Moist 6: Moisture of plot 6
Moist 8: Moisture of plot 8
Moist 11: Moisture of plot 5
Moist 10: Moisture of plot 10
Moist 1: Moisture of plot 1
Moist 9: Moisture of plot 9
Moist 7: Moisture of plot 7
Moist 5: Moisture of plot 5
Temp: Soil temperature
PAR: Moisture metrics

8. 8
4.0 PROJECT IMPLEMENTATION
4.1 Data Ingestion using Kafka
Kafka is the distributed messaging system which allows to transmit moisture and weather data from
Arduino chip to the HDFS. Kafka Architecture depends mainly on three components producer, broker and
consumer. Zookeeper is used to monitor the frequency of data following in and out of the Kafka broker.
The Below diagram is the architectural diagram of precision agriculture project. Kafka producer streams
the data that is produced in the greenhouse and sends it to the kafka broker. Kafka producer gets the
addresses of the broker thought zookeeper.
Fig 4: Kafka Architectural Diagram
4.2 Kafka producer
Kafka Producer is sender side of the Kafka distributed messaging system. Producer splits the messages to
their respective topics and sends to brokers based on topics. Producer also gets the address of the Kafka
brokers which is attached to the header of packet while sending the data.
The weather data, moisture data and image data differentiated using different topics such as “weather-data”,
”moisture-data” and “image-data”.
Below is the snippet to set up the Kafka producer with key and value set as string serialization. Bootstrap
server is the broker ID list of the Kafka broker.

9. 9
Fig 5 : Configuring the kafka producer
Below is the snippet that is used to create message object which contains topic and messages to be sent to
the Kafka broker. Send function of Kafka producer binds the Kafka configuration instance with messages,
sends it to the broker.
Fig 6: Sending the message to kafka broker
4.4 Kafka Broker
Kafka Broker is the server side of the kafka distributed messaging system which is capable of handling
hundreds and hundreds of read and write operation per second. It can elastically expand without downtime.
Data Streams are partitioned and spread over a cluster of machines to allow data streams larger than
capability of single machine. The Kafka broker can be monitored using Zookeeper using port number
2181.By default Kafka broker comes with retention period of 168 hours.

10. 10
Fig 7 : Monitoring the messages using Zookeeper
4.5 Kafka Consumer
Kafka Consumer is receiver side of the kafka distributed messaging system that fetches the data topic wise
from the brokers. Consumer runs in cluster and also stores the data in the HDFS for further processing.
Below is the sample consumer code which connects to the PA cluster. Topic set contains list of topics that
we are interested to fetch from the broker.
Fig 8 : Configuring Kafka Consumer
5.0 ADDITIONAL TOOLS
5.1 Maven
Maven was used as the dependency management to bring in all the jar from the server to the local repository.
This dependency injection help to develop the code from the windows environment .Maven helped to
specify the version of spark and kafka that was used and all the jar files related that version of spark was
stored in the local repository.

11. 11
Fig 9 : Dependency Injection
5.2 Scala Build tool
Scala Build tool (SBT) was used to create the package and jar files which was transferred to cluster and vm
using winscp.
Fig 10 : SBT build
5.3 Git
Git is online code repository for storing all the code related to project. It offers all of the distributed
revision control and source code management (SCM). Git was used for precision agriculture project
repository to store the code online and share with team.
Below is the git link for the precision agriculture.
https://github.com/sri303030/Data-Ingestion-using-Kafka

12. 12
6.0 OUTPUT INTERPRETATION
The Streamed data with the help of consumer is send to the HDFS and stored as two different folder to
distinguish between weather data and moisture data.
Below is the output from the weather data folder
Fig 11: weather data folder
Below is output from the moisture data folder
Fig 12: Moisture Data Folder
7.0 IMPROVING THE KAFKA ARCHITECTURE
Kafka Architecture can be improved in two ways:
1. Making kafka architecture more robust.
2. Having dedicated Kafka Broker to improve performance
7.1. Making kafka architecture more robust
In precision agriculture project, both broker and consumer were running on the same system as the
requirement of data ingestion was to store data in HDFS. In order make the architecture more robust,
consumer system must be a remote system or cluster which have the access to kafka broker this way the
architecture will be more robust and in case of failure in kafka broker the data can be retrived from
consumer.

13. 13
7.2. Having dedicated Kafka Broker to improve performance
Kafka Broker runs as part of the cluster in precision agriculture project . In order to avoid noise in the
cluster broker must be a dedicated system or set of systems. It also helps to eradicate the overhead that
kafka broker has got over hadoop environment and speeds up all the processes.
8.0 FUTURE RESEARCH
1. Implement the bridging between HDFS and SparkSQL and store table as persistent data in hive.
2. Implement real time machine learning using Spark Mllib
3. Connect the live data to the reporting tool and analyze live data and create useful reports.
9.0. CONCLUSION
Kafka is rapidly growing distributed messaging system having various application in the field of
engineering. Thus with the help precision agriculture project, agricultural data from greenhouse was
captured and streamed to hadoop environment using kafka and spark. This project also gave me exposure
to handle different big data problems in real time situation and helped me understand kafka architecture.

14. 14
BIBLIOGRAPHY
http://kafka.apache.org/
Rahul Jain (2014) Real time Analytics with Apache Kafka and Apache Spark
Wang, H., Can, D., Kazemzadeh, A., Bar, F., & Narayanan, S. (2012). A System for Real- time Twitter
Sentiment Analysis of 2012 U.S. Presidential Election Cycle. Paper presented at the Proceedings of
the 50th Annual Meeting of the Association for Computational Linguistics, Jeju, Republic of
Korea. http://www.aclweb.org/anthology/P12-3020