SlideShare una empresa de Scribd logo
1 de 23
Descargar para leer sin conexión
Process Automation for
Hydrological Data Mapping
over GIS Software


By Rohan Jain (08AG1016)
Introduction
● Weather data is available from various
  organisations like IMD, CWC through their
  stations spanning all over the country,
  periodically.
● The data available from these places can be
  used for further processing.
● Processing is done via various GIS Software
  available.
● ArcGIS is one such popular software. It is
  used for this project
Introduction: Problem
● Data is not available in format ArcGIS support
● So it cannot be directly imported
● Manually importing 10s of thousands of data
  is not possible.
● Hence data needs to be automatically
  converted into an ArcGIS format.
● But again data from all the sources is not in a
  standardised format.
● So each data source needs special attention
Objectives
● Automatic conversion of existing
  hydrological data of Mahanadi river basin
  into a universal time-series format
● Mapping of the data into ArcHydro model of
  the ArcGIS software
Study Area: Description
● Mahandi river basin, located between
  longitudes 800 30' and 870 E, and latitudes
  190 21' and 230 35' N
● 4.3% of the total geographical area of India
● Mahanadi was notorious for its devastating
  floods.
● Hirakud Dam, one of the longest dams
  improved the situation greatly.
Mahanadi river
basin
Study Area: Data Available
● Data from India Meteorological Department
  and Central Water Commission (CWC)
● Rainfall data
● Escape Discharge data
● Water Level Data
● Data from remote sensing
Methodology: Requirements
● ArcGIS (Version 9.3)
● ArcHydro tools (Version 1.4) and ArcHydro
  data model
● Python Programming Language (Version >
  2.6)
● External Python Libraries
  ○ xlrd (for reading spreadsheets)
  ○ dbfpy (for writing dBase files)
Methodology: Study Material
● Book: ArcHydro - GIS for Water Resources
  by David R. Maidment[7]
● Book: Arc Hydro Tools - Tutorials
● GIS Course Content - University of Texas
● Web Resources, Lectures made available by
  ESRI[8] (ArcGIS Developer organisation)
Methodology
● For interfacing with ArcGIS dBase (*.dbf)
  database file format used
● dBase is a popular database and ArcGIS
  relies on it itself for storing data, so a good
  choice for using it for our task
● Python libraries available (dbfpy)
● For data model to store the time series, used
  the TimeSeries model from ArcHydro data
  models.
Methodology: Data Model
● FeatureID: ID of the feature for which this
  time series data exists. IMD Stations, CWC
  Gauges etc.
● TSTypeID: ID of the time series type. We
  have Precipitaion, Discharge, Water Level
  etc defined
● TSDateTime: The date and time of individual
  data
● TSValue: Individual data value
Methodology: Automation
1. The data obtained from various organisations
   is converted into a format which follows
   python data structures.
2. Separate (dBase) files contain information
   about HydroIDs (which will help find
   FeatureID). The information is extracted and
   used to find FeatureIDs for station names
3. Time Series is generated and then further
   published as dBase files for use with ArcGIS
   software.
The Data Conversion
Process
Methodology: Code Written
● Modules
  ○ These are for generic tasks which are applicable to
      all data sources
   ○ timeseries.py
     ■ Takes care of timeseries related internal tasks
      ■ Also generates the dBase files
   ○ stations.py:
     ■ Process the HydroIDs (FeatureIDs in Time
         Series database)
      ■ Fetches ID - Name info about the stations
Methodology: Code Written
● Individual Data Source Scripts
  ○ Since each data source provides information in a
    different format, they all need a separate script.
  ○ These scripts process the raw data to pythonic
    format and then generate time series database
● Written in Python Programming Language
● Total roughly 450 lines of python code
● A C/Java equivalent will easily measure 2-3
  times
Results
● Set up an initial project with correct directory
  hierarchy and install python + the required
  libraries
● Then, on execution of the scripts the time
  series files are generated automatically
● The time series files can then be imported
  into ArcGIS table
Results: Loading Data




Loading data
into a Time
Series table in
ArcCatalog
Result: Loading Data




ArcCatalog data loading dialogs
Result: Loading Data
                       Displaying data
                       after being
                       imported.
Result: Processing Data
                   Processing the
                   data in
                   ArcMap using
                   ArcHydro tools
Result: Processing Data




ArcMap Processing the Discharge Time Series
Future Work
● Rewrite the modules using Object Oriented
  Approach to improve the code quality and
  future additions of code easier
● Apart from this Rainfall, Discharge, Water
  Level series more data can be obtained and
  added
Thank You

Más contenido relacionado

La actualidad más candente

Solar ponds and its applications
Solar ponds and its applicationsSolar ponds and its applications
Solar ponds and its applicationsKunal Mahajan
 
Introducing Cloudera DataFlow (CDF) 2.13.19
Introducing Cloudera DataFlow (CDF) 2.13.19Introducing Cloudera DataFlow (CDF) 2.13.19
Introducing Cloudera DataFlow (CDF) 2.13.19Cloudera, Inc.
 
Thermal energy storage system
Thermal energy storage systemThermal energy storage system
Thermal energy storage systemAbhinav Bhaskar
 
A Spatial Decision Support System (SDSS) for Understanding and Reducing Long-...
A Spatial Decision Support System (SDSS) for Understanding and Reducing Long-...A Spatial Decision Support System (SDSS) for Understanding and Reducing Long-...
A Spatial Decision Support System (SDSS) for Understanding and Reducing Long-...Global Risk Forum GRFDavos
 
Dmm302 - Sap Hana Data Warehousing: Models for Sap Bw and SQL DW on SAP HANA
Dmm302 - Sap Hana Data Warehousing: Models for Sap Bw and SQL DW on SAP HANA Dmm302 - Sap Hana Data Warehousing: Models for Sap Bw and SQL DW on SAP HANA
Dmm302 - Sap Hana Data Warehousing: Models for Sap Bw and SQL DW on SAP HANA Luc Vanrobays
 
SAP Business Objects Universe Designer & Introduction to Business Intelligenc...
SAP Business Objects Universe Designer & Introduction to Business Intelligenc...SAP Business Objects Universe Designer & Introduction to Business Intelligenc...
SAP Business Objects Universe Designer & Introduction to Business Intelligenc...BigClasses Com
 
Review of Biomass Energy Sources
Review of Biomass Energy SourcesReview of Biomass Energy Sources
Review of Biomass Energy SourcesHimanshu Paghdal
 
Powering Custom Apps at Facebook using Spark Script Transformation
Powering Custom Apps at Facebook using Spark Script TransformationPowering Custom Apps at Facebook using Spark Script Transformation
Powering Custom Apps at Facebook using Spark Script TransformationDatabricks
 
Snowflake: Your Data. No Limits (Session sponsored by Snowflake) - AWS Summit...
Snowflake: Your Data. No Limits (Session sponsored by Snowflake) - AWS Summit...Snowflake: Your Data. No Limits (Session sponsored by Snowflake) - AWS Summit...
Snowflake: Your Data. No Limits (Session sponsored by Snowflake) - AWS Summit...Amazon Web Services
 
Solar Radiation Geometry, Solar Thermal Conversion and Applications
Solar Radiation Geometry, Solar Thermal Conversion and ApplicationsSolar Radiation Geometry, Solar Thermal Conversion and Applications
Solar Radiation Geometry, Solar Thermal Conversion and ApplicationsDr Ramesh B T
 
Database management system
Database management system   Database management system
Database management system 9535814851
 
Future energy scenario of world
Future energy scenario of worldFuture energy scenario of world
Future energy scenario of worldAsma1122
 
Phase Change Material (PCM) - Seminar.pptx
Phase Change Material (PCM) - Seminar.pptxPhase Change Material (PCM) - Seminar.pptx
Phase Change Material (PCM) - Seminar.pptxShahidTavar
 
The path to success with graph database and graph data science_ Neo4j GraphSu...
The path to success with graph database and graph data science_ Neo4j GraphSu...The path to success with graph database and graph data science_ Neo4j GraphSu...
The path to success with graph database and graph data science_ Neo4j GraphSu...Neo4j
 
Remote Sensing: Meaning, Concept and Components | Geography
Remote Sensing: Meaning, Concept and Components | GeographyRemote Sensing: Meaning, Concept and Components | Geography
Remote Sensing: Meaning, Concept and Components | GeographySrimantaKarak
 

La actualidad más candente (20)

Solar ponds and its applications
Solar ponds and its applicationsSolar ponds and its applications
Solar ponds and its applications
 
Introducing Cloudera DataFlow (CDF) 2.13.19
Introducing Cloudera DataFlow (CDF) 2.13.19Introducing Cloudera DataFlow (CDF) 2.13.19
Introducing Cloudera DataFlow (CDF) 2.13.19
 
Snowflake Overview
Snowflake OverviewSnowflake Overview
Snowflake Overview
 
Thermal energy storage system
Thermal energy storage systemThermal energy storage system
Thermal energy storage system
 
A Spatial Decision Support System (SDSS) for Understanding and Reducing Long-...
A Spatial Decision Support System (SDSS) for Understanding and Reducing Long-...A Spatial Decision Support System (SDSS) for Understanding and Reducing Long-...
A Spatial Decision Support System (SDSS) for Understanding and Reducing Long-...
 
Introduction to Microsoft Azure Cloud
Introduction to Microsoft Azure CloudIntroduction to Microsoft Azure Cloud
Introduction to Microsoft Azure Cloud
 
Dmm302 - Sap Hana Data Warehousing: Models for Sap Bw and SQL DW on SAP HANA
Dmm302 - Sap Hana Data Warehousing: Models for Sap Bw and SQL DW on SAP HANA Dmm302 - Sap Hana Data Warehousing: Models for Sap Bw and SQL DW on SAP HANA
Dmm302 - Sap Hana Data Warehousing: Models for Sap Bw and SQL DW on SAP HANA
 
SAP Business Objects Universe Designer & Introduction to Business Intelligenc...
SAP Business Objects Universe Designer & Introduction to Business Intelligenc...SAP Business Objects Universe Designer & Introduction to Business Intelligenc...
SAP Business Objects Universe Designer & Introduction to Business Intelligenc...
 
Review of Biomass Energy Sources
Review of Biomass Energy SourcesReview of Biomass Energy Sources
Review of Biomass Energy Sources
 
Powering Custom Apps at Facebook using Spark Script Transformation
Powering Custom Apps at Facebook using Spark Script TransformationPowering Custom Apps at Facebook using Spark Script Transformation
Powering Custom Apps at Facebook using Spark Script Transformation
 
Snowflake: Your Data. No Limits (Session sponsored by Snowflake) - AWS Summit...
Snowflake: Your Data. No Limits (Session sponsored by Snowflake) - AWS Summit...Snowflake: Your Data. No Limits (Session sponsored by Snowflake) - AWS Summit...
Snowflake: Your Data. No Limits (Session sponsored by Snowflake) - AWS Summit...
 
Unit 01 dbms
Unit 01 dbmsUnit 01 dbms
Unit 01 dbms
 
Solar Radiation Geometry, Solar Thermal Conversion and Applications
Solar Radiation Geometry, Solar Thermal Conversion and ApplicationsSolar Radiation Geometry, Solar Thermal Conversion and Applications
Solar Radiation Geometry, Solar Thermal Conversion and Applications
 
Database management system
Database management system   Database management system
Database management system
 
Overview of thermodynamics
Overview of thermodynamicsOverview of thermodynamics
Overview of thermodynamics
 
PowerBI Training
PowerBI Training PowerBI Training
PowerBI Training
 
Future energy scenario of world
Future energy scenario of worldFuture energy scenario of world
Future energy scenario of world
 
Phase Change Material (PCM) - Seminar.pptx
Phase Change Material (PCM) - Seminar.pptxPhase Change Material (PCM) - Seminar.pptx
Phase Change Material (PCM) - Seminar.pptx
 
The path to success with graph database and graph data science_ Neo4j GraphSu...
The path to success with graph database and graph data science_ Neo4j GraphSu...The path to success with graph database and graph data science_ Neo4j GraphSu...
The path to success with graph database and graph data science_ Neo4j GraphSu...
 
Remote Sensing: Meaning, Concept and Components | Geography
Remote Sensing: Meaning, Concept and Components | GeographyRemote Sensing: Meaning, Concept and Components | Geography
Remote Sensing: Meaning, Concept and Components | Geography
 

Destacado

JHydro - an implementation of the digital watershed
JHydro - an implementation of the digital watershedJHydro - an implementation of the digital watershed
JHydro - an implementation of the digital watershedsilli
 
Impacts of landuse change on sediment transport in the yali reservoir catchment
Impacts of landuse change on sediment transport in the yali reservoir catchmentImpacts of landuse change on sediment transport in the yali reservoir catchment
Impacts of landuse change on sediment transport in the yali reservoir catchmentCPWF Mekong
 
Climate Smart Landscape-Based Integrated Watershed Management: Experiences fr...
Climate Smart Landscape-Based Integrated Watershed Management: Experiences fr...Climate Smart Landscape-Based Integrated Watershed Management: Experiences fr...
Climate Smart Landscape-Based Integrated Watershed Management: Experiences fr...CIFOR-ICRAF
 
Thesis Presentation
Thesis PresentationThesis Presentation
Thesis Presentationgawump
 
Application of RS and GIS in Groundwater Prospects Zonation
Application of RS and GIS in Groundwater Prospects ZonationApplication of RS and GIS in Groundwater Prospects Zonation
Application of RS and GIS in Groundwater Prospects ZonationVishwanath Awati
 
Presentation on Aerosols, cloud properties
Presentation on Aerosols, cloud properties Presentation on Aerosols, cloud properties
Presentation on Aerosols, cloud properties Esayas Meresa
 
Analysis of runoff for vishwamitri river watershed using scs cn method and ge...
Analysis of runoff for vishwamitri river watershed using scs cn method and ge...Analysis of runoff for vishwamitri river watershed using scs cn method and ge...
Analysis of runoff for vishwamitri river watershed using scs cn method and ge...vishvam Pancholi
 
Soil Erosion for Vishwamitri River watershed using RS and GIS
Soil Erosion for Vishwamitri River watershed using RS and GISSoil Erosion for Vishwamitri River watershed using RS and GIS
Soil Erosion for Vishwamitri River watershed using RS and GISvishvam Pancholi
 
Presentation on remote sensing & gis and watershed copy
Presentation on remote sensing & gis and watershed   copyPresentation on remote sensing & gis and watershed   copy
Presentation on remote sensing & gis and watershed copydivya sahgal
 
REMOTE SENSING & GIS APPLICATIONS IN WATERSHED MANAGEMENT
REMOTE SENSING & GIS APPLICATIONS IN WATERSHED MANAGEMENT REMOTE SENSING & GIS APPLICATIONS IN WATERSHED MANAGEMENT
REMOTE SENSING & GIS APPLICATIONS IN WATERSHED MANAGEMENT Sumant Diwakar
 
Iirs overview -Remote sensing and GIS application in Water Resources Management
Iirs overview -Remote sensing and GIS application in Water Resources ManagementIirs overview -Remote sensing and GIS application in Water Resources Management
Iirs overview -Remote sensing and GIS application in Water Resources ManagementTushar Dholakia
 
APPLICATIONS OF REMOTE SENSING AND GIS IN WATERSHED MANAGEMENT
APPLICATIONS OF REMOTE SENSING AND GIS IN WATERSHED MANAGEMENTAPPLICATIONS OF REMOTE SENSING AND GIS IN WATERSHED MANAGEMENT
APPLICATIONS OF REMOTE SENSING AND GIS IN WATERSHED MANAGEMENTSriram Chakravarthy
 
Application of gis and remote sensing in agriculture
Application of gis and remote sensing in agricultureApplication of gis and remote sensing in agriculture
Application of gis and remote sensing in agricultureRehana Qureshi
 
Application of Remote Sensing in Agriculture
Application of Remote Sensing in AgricultureApplication of Remote Sensing in Agriculture
Application of Remote Sensing in AgricultureUTTAM KUMAR
 
Introduction to remote sensing and gis
Introduction to remote sensing and gisIntroduction to remote sensing and gis
Introduction to remote sensing and gisMohsin Siddique
 
Integration of the MODFLOW Lak7 package in the FREEWAT GIS modelling environment
Integration of the MODFLOW Lak7 package in the FREEWAT GIS modelling environmentIntegration of the MODFLOW Lak7 package in the FREEWAT GIS modelling environment
Integration of the MODFLOW Lak7 package in the FREEWAT GIS modelling environmentMassimiliano Cannata
 

Destacado (20)

JHydro - an implementation of the digital watershed
JHydro - an implementation of the digital watershedJHydro - an implementation of the digital watershed
JHydro - an implementation of the digital watershed
 
Jaysukh C Songara
Jaysukh C SongaraJaysukh C Songara
Jaysukh C Songara
 
Spatial Earth Profile2
Spatial Earth Profile2Spatial Earth Profile2
Spatial Earth Profile2
 
Impacts of landuse change on sediment transport in the yali reservoir catchment
Impacts of landuse change on sediment transport in the yali reservoir catchmentImpacts of landuse change on sediment transport in the yali reservoir catchment
Impacts of landuse change on sediment transport in the yali reservoir catchment
 
Climate Smart Landscape-Based Integrated Watershed Management: Experiences fr...
Climate Smart Landscape-Based Integrated Watershed Management: Experiences fr...Climate Smart Landscape-Based Integrated Watershed Management: Experiences fr...
Climate Smart Landscape-Based Integrated Watershed Management: Experiences fr...
 
Thesis Presentation
Thesis PresentationThesis Presentation
Thesis Presentation
 
Gnd water
Gnd waterGnd water
Gnd water
 
Application of RS and GIS in Groundwater Prospects Zonation
Application of RS and GIS in Groundwater Prospects ZonationApplication of RS and GIS in Groundwater Prospects Zonation
Application of RS and GIS in Groundwater Prospects Zonation
 
Intergrated Water Resourcses system
Intergrated Water Resourcses systemIntergrated Water Resourcses system
Intergrated Water Resourcses system
 
Presentation on Aerosols, cloud properties
Presentation on Aerosols, cloud properties Presentation on Aerosols, cloud properties
Presentation on Aerosols, cloud properties
 
Analysis of runoff for vishwamitri river watershed using scs cn method and ge...
Analysis of runoff for vishwamitri river watershed using scs cn method and ge...Analysis of runoff for vishwamitri river watershed using scs cn method and ge...
Analysis of runoff for vishwamitri river watershed using scs cn method and ge...
 
Soil Erosion for Vishwamitri River watershed using RS and GIS
Soil Erosion for Vishwamitri River watershed using RS and GISSoil Erosion for Vishwamitri River watershed using RS and GIS
Soil Erosion for Vishwamitri River watershed using RS and GIS
 
Presentation on remote sensing & gis and watershed copy
Presentation on remote sensing & gis and watershed   copyPresentation on remote sensing & gis and watershed   copy
Presentation on remote sensing & gis and watershed copy
 
REMOTE SENSING & GIS APPLICATIONS IN WATERSHED MANAGEMENT
REMOTE SENSING & GIS APPLICATIONS IN WATERSHED MANAGEMENT REMOTE SENSING & GIS APPLICATIONS IN WATERSHED MANAGEMENT
REMOTE SENSING & GIS APPLICATIONS IN WATERSHED MANAGEMENT
 
Iirs overview -Remote sensing and GIS application in Water Resources Management
Iirs overview -Remote sensing and GIS application in Water Resources ManagementIirs overview -Remote sensing and GIS application in Water Resources Management
Iirs overview -Remote sensing and GIS application in Water Resources Management
 
APPLICATIONS OF REMOTE SENSING AND GIS IN WATERSHED MANAGEMENT
APPLICATIONS OF REMOTE SENSING AND GIS IN WATERSHED MANAGEMENTAPPLICATIONS OF REMOTE SENSING AND GIS IN WATERSHED MANAGEMENT
APPLICATIONS OF REMOTE SENSING AND GIS IN WATERSHED MANAGEMENT
 
Application of gis and remote sensing in agriculture
Application of gis and remote sensing in agricultureApplication of gis and remote sensing in agriculture
Application of gis and remote sensing in agriculture
 
Application of Remote Sensing in Agriculture
Application of Remote Sensing in AgricultureApplication of Remote Sensing in Agriculture
Application of Remote Sensing in Agriculture
 
Introduction to remote sensing and gis
Introduction to remote sensing and gisIntroduction to remote sensing and gis
Introduction to remote sensing and gis
 
Integration of the MODFLOW Lak7 package in the FREEWAT GIS modelling environment
Integration of the MODFLOW Lak7 package in the FREEWAT GIS modelling environmentIntegration of the MODFLOW Lak7 package in the FREEWAT GIS modelling environment
Integration of the MODFLOW Lak7 package in the FREEWAT GIS modelling environment
 

Similar a Btp presentation

Understanding Hadoop
Understanding HadoopUnderstanding Hadoop
Understanding HadoopAhmed Ossama
 
SF Big Analytics 20191112: How to performance-tune Spark applications in larg...
SF Big Analytics 20191112: How to performance-tune Spark applications in larg...SF Big Analytics 20191112: How to performance-tune Spark applications in larg...
SF Big Analytics 20191112: How to performance-tune Spark applications in larg...Chester Chen
 
11. From Hadoop to Spark 1:2
11. From Hadoop to Spark 1:211. From Hadoop to Spark 1:2
11. From Hadoop to Spark 1:2Fabio Fumarola
 
Analyzing Data at Scale with Apache Spark
Analyzing Data at Scale with Apache SparkAnalyzing Data at Scale with Apache Spark
Analyzing Data at Scale with Apache SparkNicola Ferraro
 
Hoodie: How (And Why) We built an analytical datastore on Spark
Hoodie: How (And Why) We built an analytical datastore on SparkHoodie: How (And Why) We built an analytical datastore on Spark
Hoodie: How (And Why) We built an analytical datastore on SparkVinoth Chandar
 
Incremental Processing on Large Analytical Datasets with Prasanna Rajaperumal...
Incremental Processing on Large Analytical Datasets with Prasanna Rajaperumal...Incremental Processing on Large Analytical Datasets with Prasanna Rajaperumal...
Incremental Processing on Large Analytical Datasets with Prasanna Rajaperumal...Databricks
 
Spark Meetup at Uber
Spark Meetup at UberSpark Meetup at Uber
Spark Meetup at UberDatabricks
 
Data Analytics and Machine Learning: From Node to Cluster on ARM64
Data Analytics and Machine Learning: From Node to Cluster on ARM64Data Analytics and Machine Learning: From Node to Cluster on ARM64
Data Analytics and Machine Learning: From Node to Cluster on ARM64Ganesh Raju
 
BKK16-404B Data Analytics and Machine Learning- from Node to Cluster
BKK16-404B Data Analytics and Machine Learning- from Node to ClusterBKK16-404B Data Analytics and Machine Learning- from Node to Cluster
BKK16-404B Data Analytics and Machine Learning- from Node to ClusterLinaro
 
BKK16-408B Data Analytics and Machine Learning From Node to Cluster
BKK16-408B Data Analytics and Machine Learning From Node to ClusterBKK16-408B Data Analytics and Machine Learning From Node to Cluster
BKK16-408B Data Analytics and Machine Learning From Node to ClusterLinaro
 
Streamsets and spark in Retail
Streamsets and spark in RetailStreamsets and spark in Retail
Streamsets and spark in RetailHari Shreedharan
 
Analytic Insights in Retail Using Apache Spark with Hari Shreedharan
Analytic Insights in Retail Using Apache Spark with Hari ShreedharanAnalytic Insights in Retail Using Apache Spark with Hari Shreedharan
Analytic Insights in Retail Using Apache Spark with Hari ShreedharanDatabricks
 
Spark Driven Big Data Analytics
Spark Driven Big Data AnalyticsSpark Driven Big Data Analytics
Spark Driven Big Data Analyticsinoshg
 
Dache: A Data Aware Caching for Big-Data using Map Reduce framework
Dache: A Data Aware Caching for Big-Data using Map Reduce frameworkDache: A Data Aware Caching for Big-Data using Map Reduce framework
Dache: A Data Aware Caching for Big-Data using Map Reduce frameworkSafir Shah
 

Similar a Btp presentation (20)

Understanding Hadoop
Understanding HadoopUnderstanding Hadoop
Understanding Hadoop
 
Geospatial Data Abstraction Library (GDAL) Enhancement for ESDIS (GEE)
Geospatial Data Abstraction Library (GDAL) Enhancement for ESDIS (GEE)Geospatial Data Abstraction Library (GDAL) Enhancement for ESDIS (GEE)
Geospatial Data Abstraction Library (GDAL) Enhancement for ESDIS (GEE)
 
HDF and netCDF Data Support in ArcGIS
HDF and netCDF Data Support in ArcGISHDF and netCDF Data Support in ArcGIS
HDF and netCDF Data Support in ArcGIS
 
SF Big Analytics 20191112: How to performance-tune Spark applications in larg...
SF Big Analytics 20191112: How to performance-tune Spark applications in larg...SF Big Analytics 20191112: How to performance-tune Spark applications in larg...
SF Big Analytics 20191112: How to performance-tune Spark applications in larg...
 
HDF-EOS Data Product Developer's Guide
HDF-EOS Data Product Developer's GuideHDF-EOS Data Product Developer's Guide
HDF-EOS Data Product Developer's Guide
 
11. From Hadoop to Spark 1:2
11. From Hadoop to Spark 1:211. From Hadoop to Spark 1:2
11. From Hadoop to Spark 1:2
 
Analyzing Data at Scale with Apache Spark
Analyzing Data at Scale with Apache SparkAnalyzing Data at Scale with Apache Spark
Analyzing Data at Scale with Apache Spark
 
Hoodie: How (And Why) We built an analytical datastore on Spark
Hoodie: How (And Why) We built an analytical datastore on SparkHoodie: How (And Why) We built an analytical datastore on Spark
Hoodie: How (And Why) We built an analytical datastore on Spark
 
Incremental Processing on Large Analytical Datasets with Prasanna Rajaperumal...
Incremental Processing on Large Analytical Datasets with Prasanna Rajaperumal...Incremental Processing on Large Analytical Datasets with Prasanna Rajaperumal...
Incremental Processing on Large Analytical Datasets with Prasanna Rajaperumal...
 
Spark Meetup at Uber
Spark Meetup at UberSpark Meetup at Uber
Spark Meetup at Uber
 
design_doc
design_docdesign_doc
design_doc
 
Data Analytics and Machine Learning: From Node to Cluster on ARM64
Data Analytics and Machine Learning: From Node to Cluster on ARM64Data Analytics and Machine Learning: From Node to Cluster on ARM64
Data Analytics and Machine Learning: From Node to Cluster on ARM64
 
BKK16-404B Data Analytics and Machine Learning- from Node to Cluster
BKK16-404B Data Analytics and Machine Learning- from Node to ClusterBKK16-404B Data Analytics and Machine Learning- from Node to Cluster
BKK16-404B Data Analytics and Machine Learning- from Node to Cluster
 
BKK16-408B Data Analytics and Machine Learning From Node to Cluster
BKK16-408B Data Analytics and Machine Learning From Node to ClusterBKK16-408B Data Analytics and Machine Learning From Node to Cluster
BKK16-408B Data Analytics and Machine Learning From Node to Cluster
 
Streamsets and spark in Retail
Streamsets and spark in RetailStreamsets and spark in Retail
Streamsets and spark in Retail
 
Analytic Insights in Retail Using Apache Spark with Hari Shreedharan
Analytic Insights in Retail Using Apache Spark with Hari ShreedharanAnalytic Insights in Retail Using Apache Spark with Hari Shreedharan
Analytic Insights in Retail Using Apache Spark with Hari Shreedharan
 
Big Data Processing
Big Data ProcessingBig Data Processing
Big Data Processing
 
Spark Driven Big Data Analytics
Spark Driven Big Data AnalyticsSpark Driven Big Data Analytics
Spark Driven Big Data Analytics
 
Working with HDF and netCDF Data in ArcGIS: Tools and Case Studies
Working with HDF and netCDF Data in ArcGIS: Tools and Case StudiesWorking with HDF and netCDF Data in ArcGIS: Tools and Case Studies
Working with HDF and netCDF Data in ArcGIS: Tools and Case Studies
 
Dache: A Data Aware Caching for Big-Data using Map Reduce framework
Dache: A Data Aware Caching for Big-Data using Map Reduce frameworkDache: A Data Aware Caching for Big-Data using Map Reduce framework
Dache: A Data Aware Caching for Big-Data using Map Reduce framework
 

Último

Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024
Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024
Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024Victor Rentea
 
Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...
Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...
Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...apidays
 
"I see eyes in my soup": How Delivery Hero implemented the safety system for ...
"I see eyes in my soup": How Delivery Hero implemented the safety system for ..."I see eyes in my soup": How Delivery Hero implemented the safety system for ...
"I see eyes in my soup": How Delivery Hero implemented the safety system for ...Zilliz
 
Apidays New York 2024 - The value of a flexible API Management solution for O...
Apidays New York 2024 - The value of a flexible API Management solution for O...Apidays New York 2024 - The value of a flexible API Management solution for O...
Apidays New York 2024 - The value of a flexible API Management solution for O...apidays
 
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers:  A Deep Dive into Serverless Spatial Data and FMECloud Frontiers:  A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FMESafe Software
 
AXA XL - Insurer Innovation Award Americas 2024
AXA XL - Insurer Innovation Award Americas 2024AXA XL - Insurer Innovation Award Americas 2024
AXA XL - Insurer Innovation Award Americas 2024The Digital Insurer
 
MINDCTI Revenue Release Quarter One 2024
MINDCTI Revenue Release Quarter One 2024MINDCTI Revenue Release Quarter One 2024
MINDCTI Revenue Release Quarter One 2024MIND CTI
 
[BuildWithAI] Introduction to Gemini.pdf
[BuildWithAI] Introduction to Gemini.pdf[BuildWithAI] Introduction to Gemini.pdf
[BuildWithAI] Introduction to Gemini.pdfSandro Moreira
 
Rising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdf
Rising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdfRising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdf
Rising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdfOrbitshub
 
CNIC Information System with Pakdata Cf In Pakistan
CNIC Information System with Pakdata Cf In PakistanCNIC Information System with Pakdata Cf In Pakistan
CNIC Information System with Pakdata Cf In Pakistandanishmna97
 
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...Jeffrey Haguewood
 
FWD Group - Insurer Innovation Award 2024
FWD Group - Insurer Innovation Award 2024FWD Group - Insurer Innovation Award 2024
FWD Group - Insurer Innovation Award 2024The Digital Insurer
 
Corporate and higher education May webinar.pptx
Corporate and higher education May webinar.pptxCorporate and higher education May webinar.pptx
Corporate and higher education May webinar.pptxRustici Software
 
Architecting Cloud Native Applications
Architecting Cloud Native ApplicationsArchitecting Cloud Native Applications
Architecting Cloud Native ApplicationsWSO2
 
Manulife - Insurer Transformation Award 2024
Manulife - Insurer Transformation Award 2024Manulife - Insurer Transformation Award 2024
Manulife - Insurer Transformation Award 2024The Digital Insurer
 
Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...
Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...
Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...apidays
 
Emergent Methods: Multi-lingual narrative tracking in the news - real-time ex...
Emergent Methods: Multi-lingual narrative tracking in the news - real-time ex...Emergent Methods: Multi-lingual narrative tracking in the news - real-time ex...
Emergent Methods: Multi-lingual narrative tracking in the news - real-time ex...Zilliz
 
DBX First Quarter 2024 Investor Presentation
DBX First Quarter 2024 Investor PresentationDBX First Quarter 2024 Investor Presentation
DBX First Quarter 2024 Investor PresentationDropbox
 
Why Teams call analytics are critical to your entire business
Why Teams call analytics are critical to your entire businessWhy Teams call analytics are critical to your entire business
Why Teams call analytics are critical to your entire businesspanagenda
 

Último (20)

Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024
Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024
Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024
 
Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...
Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...
Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...
 
"I see eyes in my soup": How Delivery Hero implemented the safety system for ...
"I see eyes in my soup": How Delivery Hero implemented the safety system for ..."I see eyes in my soup": How Delivery Hero implemented the safety system for ...
"I see eyes in my soup": How Delivery Hero implemented the safety system for ...
 
Apidays New York 2024 - The value of a flexible API Management solution for O...
Apidays New York 2024 - The value of a flexible API Management solution for O...Apidays New York 2024 - The value of a flexible API Management solution for O...
Apidays New York 2024 - The value of a flexible API Management solution for O...
 
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers:  A Deep Dive into Serverless Spatial Data and FMECloud Frontiers:  A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
 
AXA XL - Insurer Innovation Award Americas 2024
AXA XL - Insurer Innovation Award Americas 2024AXA XL - Insurer Innovation Award Americas 2024
AXA XL - Insurer Innovation Award Americas 2024
 
MINDCTI Revenue Release Quarter One 2024
MINDCTI Revenue Release Quarter One 2024MINDCTI Revenue Release Quarter One 2024
MINDCTI Revenue Release Quarter One 2024
 
[BuildWithAI] Introduction to Gemini.pdf
[BuildWithAI] Introduction to Gemini.pdf[BuildWithAI] Introduction to Gemini.pdf
[BuildWithAI] Introduction to Gemini.pdf
 
Rising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdf
Rising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdfRising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdf
Rising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdf
 
CNIC Information System with Pakdata Cf In Pakistan
CNIC Information System with Pakdata Cf In PakistanCNIC Information System with Pakdata Cf In Pakistan
CNIC Information System with Pakdata Cf In Pakistan
 
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...
 
FWD Group - Insurer Innovation Award 2024
FWD Group - Insurer Innovation Award 2024FWD Group - Insurer Innovation Award 2024
FWD Group - Insurer Innovation Award 2024
 
Corporate and higher education May webinar.pptx
Corporate and higher education May webinar.pptxCorporate and higher education May webinar.pptx
Corporate and higher education May webinar.pptx
 
Architecting Cloud Native Applications
Architecting Cloud Native ApplicationsArchitecting Cloud Native Applications
Architecting Cloud Native Applications
 
Manulife - Insurer Transformation Award 2024
Manulife - Insurer Transformation Award 2024Manulife - Insurer Transformation Award 2024
Manulife - Insurer Transformation Award 2024
 
Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...
Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...
Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...
 
Emergent Methods: Multi-lingual narrative tracking in the news - real-time ex...
Emergent Methods: Multi-lingual narrative tracking in the news - real-time ex...Emergent Methods: Multi-lingual narrative tracking in the news - real-time ex...
Emergent Methods: Multi-lingual narrative tracking in the news - real-time ex...
 
DBX First Quarter 2024 Investor Presentation
DBX First Quarter 2024 Investor PresentationDBX First Quarter 2024 Investor Presentation
DBX First Quarter 2024 Investor Presentation
 
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
 
Why Teams call analytics are critical to your entire business
Why Teams call analytics are critical to your entire businessWhy Teams call analytics are critical to your entire business
Why Teams call analytics are critical to your entire business
 

Btp presentation

  • 1. Process Automation for Hydrological Data Mapping over GIS Software By Rohan Jain (08AG1016)
  • 2. Introduction ● Weather data is available from various organisations like IMD, CWC through their stations spanning all over the country, periodically. ● The data available from these places can be used for further processing. ● Processing is done via various GIS Software available. ● ArcGIS is one such popular software. It is used for this project
  • 3. Introduction: Problem ● Data is not available in format ArcGIS support ● So it cannot be directly imported ● Manually importing 10s of thousands of data is not possible. ● Hence data needs to be automatically converted into an ArcGIS format. ● But again data from all the sources is not in a standardised format. ● So each data source needs special attention
  • 4. Objectives ● Automatic conversion of existing hydrological data of Mahanadi river basin into a universal time-series format ● Mapping of the data into ArcHydro model of the ArcGIS software
  • 5. Study Area: Description ● Mahandi river basin, located between longitudes 800 30' and 870 E, and latitudes 190 21' and 230 35' N ● 4.3% of the total geographical area of India ● Mahanadi was notorious for its devastating floods. ● Hirakud Dam, one of the longest dams improved the situation greatly.
  • 7. Study Area: Data Available ● Data from India Meteorological Department and Central Water Commission (CWC) ● Rainfall data ● Escape Discharge data ● Water Level Data ● Data from remote sensing
  • 8. Methodology: Requirements ● ArcGIS (Version 9.3) ● ArcHydro tools (Version 1.4) and ArcHydro data model ● Python Programming Language (Version > 2.6) ● External Python Libraries ○ xlrd (for reading spreadsheets) ○ dbfpy (for writing dBase files)
  • 9. Methodology: Study Material ● Book: ArcHydro - GIS for Water Resources by David R. Maidment[7] ● Book: Arc Hydro Tools - Tutorials ● GIS Course Content - University of Texas ● Web Resources, Lectures made available by ESRI[8] (ArcGIS Developer organisation)
  • 10. Methodology ● For interfacing with ArcGIS dBase (*.dbf) database file format used ● dBase is a popular database and ArcGIS relies on it itself for storing data, so a good choice for using it for our task ● Python libraries available (dbfpy) ● For data model to store the time series, used the TimeSeries model from ArcHydro data models.
  • 11. Methodology: Data Model ● FeatureID: ID of the feature for which this time series data exists. IMD Stations, CWC Gauges etc. ● TSTypeID: ID of the time series type. We have Precipitaion, Discharge, Water Level etc defined ● TSDateTime: The date and time of individual data ● TSValue: Individual data value
  • 12. Methodology: Automation 1. The data obtained from various organisations is converted into a format which follows python data structures. 2. Separate (dBase) files contain information about HydroIDs (which will help find FeatureID). The information is extracted and used to find FeatureIDs for station names 3. Time Series is generated and then further published as dBase files for use with ArcGIS software.
  • 14. Methodology: Code Written ● Modules ○ These are for generic tasks which are applicable to all data sources ○ timeseries.py ■ Takes care of timeseries related internal tasks ■ Also generates the dBase files ○ stations.py: ■ Process the HydroIDs (FeatureIDs in Time Series database) ■ Fetches ID - Name info about the stations
  • 15. Methodology: Code Written ● Individual Data Source Scripts ○ Since each data source provides information in a different format, they all need a separate script. ○ These scripts process the raw data to pythonic format and then generate time series database ● Written in Python Programming Language ● Total roughly 450 lines of python code ● A C/Java equivalent will easily measure 2-3 times
  • 16. Results ● Set up an initial project with correct directory hierarchy and install python + the required libraries ● Then, on execution of the scripts the time series files are generated automatically ● The time series files can then be imported into ArcGIS table
  • 17. Results: Loading Data Loading data into a Time Series table in ArcCatalog
  • 18. Result: Loading Data ArcCatalog data loading dialogs
  • 19. Result: Loading Data Displaying data after being imported.
  • 20. Result: Processing Data Processing the data in ArcMap using ArcHydro tools
  • 21. Result: Processing Data ArcMap Processing the Discharge Time Series
  • 22. Future Work ● Rewrite the modules using Object Oriented Approach to improve the code quality and future additions of code easier ● Apart from this Rainfall, Discharge, Water Level series more data can be obtained and added