This document discusses shipping and weather data from 1662-1855 recorded in captains' logs. It contains over 290,000 data points across 120 weather parameters. The data has been added to GIS databases and published online to allow exploration and comparison of shipping routes and weather patterns over time to support climate research. Tools like time sliders and filtering by nationality facilitate investigation of this large dataset. Lessons include joining data losslessly, sharing openly to enable reuse and collaboration, and using the data to further climate studies.

1. Andrew Zolnai
Cambridge UK
www.zolnai.ca
Shipping and weather data
CLIWOC ships captains logs 1662-1855
Data points for climate research
Image: Royal Museum Greenwich

2. Climatological data
• Ships captains logs data from 1662 to 1855
• Ships and routing details and location
• Weather recording of over 120 parameters
• Release 1.x (2004) and Release 2.x (2007)

3. Shipping data
• First added to Esri file geodatabse
– The complete (weather and location) dataset has
almost 290,000 points, and is almost 250Mb database
– The locations were extracted into a geodatabase (70
Mb), then into a compressed file geodatabase (10 Mb)
• Posted for ArcGIS Explorer upload on original
website 1750 – 1850 data from Release 1.5
– George Dailey further developed this for education
– and posted it on ESRI's Education Blog
• And map package shares Release 2.0
– layer files showing them in 25-year time-slices
– and for Captains Cook and de la Perouse

5. Time-enable shipping
• Daniel Schober reposted this as map package
– Here layers are time-enabled through the field "Time"
which is the UTC time in the format YYYYMMDD
• Re-reposted here and as a YouTube video
– data are no longer cast into arbitrarily time-sliced
feature-classes
– Time Slider allows to step through all 252,917 points
from 1662 to 1855
– it beautifully depicts the various colonial activities
driven by, for example, spice trade in the East and
West Indies for the British, Dutch, French and Spanish
– in other words Time Slider greatly facilitates the
investigation of this large data set

6. Static vs. dynamic web maps
• Amazon Web Service ‘small' elastic service
• Mapcentia Geocloud2 stack
• Leaflet frontend w Stamen toner, postGIS backend
• unlimited data, no performance hits
• CLIWOC all nations by year (1750-1775)
• Use time stamps to aggregate layers
• Posted as ArcGIS Online map service
• But stock server has performance issues
• So Filter data by nationality to limit fetches

8. Dynamic maps II
• Workflow
– of real interest was to compare nations shipping over
time, rather then animation per se
– thus create a pair of layers to step through
• Context
– add Rumsey 1812 map for backdrop from ArcGIS Online
– add current wind data from NOAA METAR with continents
mask to show offshore wind patterns
• Posted as CLIWOC Compare
– move slider to 1802-1812 to view with Rumsey backdrop
– turn on Continent Mask + NOAA METAR for current wind

10. Restore climate data
• CLIWOC posted all shipboard data it had
– from British, Dutch, French and Spanish sources
– with look-up tables for each different source
• Merged wind force and direction data
– table joins with lookup tables gives direction and speed
data consistent numeric values, but…
– wind (class, Beaufort and speed) keeps 277,581 points
– wind direction however boosts count to 471,214 points
– and joining wind direction and force to graduate and
orient the symbols drops the count slightly to 431,651

12. Lessons learned
• Join and aggregate data in a lossless manner
• Use File Geodatabases to allow mega datasets
• Filter data by parameters of interest:
– nationalities in pairs to compare
– step-thru animation to see evolution
• Share on-line to foster re-use & collaboration:
– feature template CLIWOCwinds
– web map CLIWOC Wind Force+Direction
– and layer package WindDutch1812

14. Take-aways
• Very rich dataset from
160 - 350 years ago
• Lots of data strung out
along ship tracks
• Navies had standards
• But issues w data
recording and capture
• Compare Old Weather
NOAA ICOADS, &c.
• Many visualisations
• Few share &/or re-use
• Foster online collaboration
• Further climate research