2013/03/11 Group Meeting, Tsinghua University, Beijing, China.

1. Threp
A Lightweight Remap Framework
for the Earth System Model
吴竑 (Xunzhang) xunzhangthu@gmail.com
2013/03/09

2. Background
Data management between components in the Earth System Model(coupler)
Data exchange between grids in each Models(atmosphere, land, ocean, ice, ...)
Grids have become diverse(triple pole, hexagon, unstructured)
Interpolation algorithms have become complex(mixed/coupled, special handling)

3. Motivation
Current popular universal remapping libraries:
ESMF-Regridding

4. Motivation
Current popular universal remapping libraries:
Inflexible data structure(latitude bin, local traverse) 3D Cartesian coordinates
Tooold(SCRIP1.4 released at 2001) Unstructured grid supported
Serial Parallel
ESMF-Regridding

5. Motivation
Current popular universal remapping libraries:
Inflexible data structure(latitude bin, local traverse) Appeared as a app, no interface supplied
3D Coords
Tooold(SCRIP1.4 released at 2001) Complicated hierarchy, really hard to improve
Unstructured grid supported
Serial Thoughtless readability
Parallel
ESMF-Regridding

6. Motivation
More versatility, flexible data structure
Remapping programming model in the Earth System Model(throw details to Threp)
Readable, lightweight, efficient remapping framework

7. Goal of Threp
Universal(Earth System Model)
Users develop just like building blocks
More grids(regular, rectilinear, curvilinear, unstructured) supported
Mask supported(source, destination)
Extrapolation supported(e.g., border of land and sea)
Parallel and Scalable

8. Encapsulation
Abstract…
Abstract…
Abstract…

9. Outline
Remapping on a sphere
Design of Threp
Implementation
Bilinear in action
Demo
Future work

10. Two Stage
Stage 1: Weights Generation Stage 2: Regridding Data
𝑘
𝑤11 𝑤12 … 𝑤1𝑘
𝑤21 𝑤22 … 𝑤2𝑘 𝑑𝑎𝑡𝑎 𝐴 𝑗 = 𝑤 𝑖 ∙ 𝑑𝑎𝑡𝑎(𝑃𝑖 )
A= … 𝑖=1
𝑤 𝑗1 𝑤 𝑗2 … 𝑤 𝑗𝑘
…
𝑑𝑎𝑡𝑎 𝑑𝑠𝑡_𝑖𝑛𝑑𝑥 = 𝐴 ∙ 𝑑𝑎𝑡𝑎[𝑠𝑟𝑐_𝑖𝑛𝑑𝑥]
𝐴 𝑗 (on dst grid)
𝑃1 , 𝑃2 … 𝑃 𝑘 (on src grid)
𝑤1, 𝑤2 … 𝑤 𝑘 (wgts)

11. Search Neighbors
IDW: nearest k points
Bilinear: nearest quadrangle
Bicubic: nearest quadrangle or nearest 16 points find nearest k + select
Patch: nearest patch
Least Square Method: nearest k points

12. Matrix-Vector Multiply
𝑤11 𝑤12 … 𝑤1𝑘
𝑤21 𝑤22 … 𝑤2𝑘
𝑘 × 𝑛𝑢𝑚 𝑑𝑠𝑡
Sparse matrix A = … , sparsity = .
𝑛𝑢𝑚 𝑠𝑟𝑐 ×𝑛𝑢𝑚 𝑑𝑠𝑡
𝑤𝑗1 𝑤𝑗2 … 𝑤𝑗𝑘
… 𝑛𝑢𝑚 𝑠𝑟𝑐 ×𝑛𝑢𝑚 𝑑𝑠𝑡
𝑇
dst_𝑖𝑛𝑑𝑥 = 1 1 … 1 2 2 … 2 … … 𝑛𝑢𝑚 𝑑𝑠𝑡 𝑛𝑢𝑚 𝑑𝑠𝑡 … 𝑛𝑢𝑚 𝑑𝑠𝑡

13. Outline
Remap model in the Earth System Model
Design of Threp
Implementation
Bilinear in action
Demo
Future work

14. 2d Tree Construction

15. Nearest neighbor search in a 2d tree demo
Average: 𝑙𝑜𝑔𝑁
Worse: 𝑁

16. Remap on a sphere
Periodic Boundary
Construct left ghost boundary:
(lon, 0.5) -> (lon, -0.5)
Construct right ghost boundary:
(lon, 359.5) -> (lon, 360.5)
Index recovery map/table:
index of (lon, ghost_lat) ->index of (lon, lat)
Left ghost right ghost
boundary boundary

17. Remap on a sphere
Pole Singularities
if highest lat(dst grid)> highest lat(src grid):
use north pole region interp pnts
continue
if lowest lat(dst grid)< lowest lat(src grid):
use south pole region interp pnts
continue
normal case…

18. A Map-Reduce application
Map: Stage1
Reduce: Stage 2

19. Outline
Remap model in the Earth System Model
Design of Threp
Implementation
Bilinear in action
Demo
Future work

20. Architecture
bilinear idw patch conserve
interp
search solver prediction geometry
io
NetCDF
MPI/TCP-IP Pydoop
clusters

21. Embrace Python
User friendly(geoscience user) Recursive efficiency:
Easy to code, easy to read loop version of kd-tree
More integrated libraries(no more repeated wheels) Auto type checking:

22. Embrace Python
< 4000 LOC! (so far)
12th day, available!

23. Grid File
SCRIP Grid Format
2D lat/lon coords(degrees/radians units supported)
Atmosphere grid with mask or no mask

24. Nearest k + Select
Heuristic

25. Solver
-solve weights for each dst grid pnt
Class +__init__()
+solve_quadratics()
+select_legal_root()
Idw_Solver Bilinear_Solver Bilinear_Solver
-solve idw weights -solve bilinear weights -solve bilinear weights
+__init__()
+solve()
+__init__()
+solve()
+__init__()
+solve()
Solver
Bilinear
Interp -Bilinear remapping
-Interpolation management class class
+__init__()
+__init__() +local_functions()
+interp() +interp()
+remap() +remap()
+check_wgt()
+check_wgtsum()
+check_all_masks()
+dst_distribute()
Interp
+dst_merge() Idw
+indx_recovery() -Idw remapping class
+compact_remap_matrix() +__init__()
+learn() +local_functions()
+deliver() +interp()
+remap()
Predictor Bilinear_Predictor
-Preditor class -Bilinear predictor class Predictor
+__init__() +__init__()
+predict() +predict()

26. MPI
Python MPI library: mpi4py
Linear speedup(stage1)
4min->2min->1min->30sec->…(not complete)
Very fast for serial stage2, <0.2sec.
necessary to parallelize?
callback of MPI_Reduce interface?

27. MapReduce
Not finished!

28. Outline
Remap model in the Earth System Model
Design of Threp
Implementation
Bilinear in action
Demo
Future work

29. Bilinear Algorithm
𝑓 𝑥, 𝑦 = 𝑎 + 𝑏𝑥 + 𝑐𝑦 + 𝑑𝑥𝑦
𝑓 𝑥 𝑘 , 𝑦 𝑘 = 𝑓 0,0 1 − 𝑥 𝑘 (1 − 𝑦 𝑘 ) + 𝑓 1,0 𝑥 𝑘 (1 − 𝑦 𝑘 ) + 𝑓 0,1 𝑦 𝑘 1 − 𝑥 𝑘 + 𝑓 1,1 𝑥 𝑘 𝑦 𝑘
𝑓 𝑥 𝑘 , 𝑦 𝑘 = 𝑓 0,0 𝑤1 + 𝑓 1,0 𝑤2 + 𝑓 0,1 𝑤3 + 𝑓 1,1 𝑤4
For irregular bilinear box, a similar way to calculate𝑤1, 𝑤2, 𝑤3, 𝑤4 (2011/8, wuhong)

30. Extrapolation Case
Source grid masked
atm2ocn & ocn2atm
Destination grid masked
ignore
Coupled bilinear
Can extrapolation as accurate as interpolation?
Yes, a property that…

31. Still Two Stage
𝑓 𝑥1, 𝑦1 = 𝑎 + 𝑏𝑥1 + 𝑐𝑦1 + 𝑑𝑥1 𝑦1 𝑓 = 𝑓 𝑥1, 𝑦1 𝑓 𝑥1, 𝑦1 𝑓 𝑥1, 𝑦1 𝑓 𝑥1, 𝑦1 𝑇
𝑓 𝑥2, 𝑦2 = 𝑎 + 𝑏𝑥2 + 𝑐𝑦2 + 𝑑𝑥2 𝑦2
𝑓 = 𝐶𝑦 1 𝑥1 𝑦1 𝑥1 𝑦1
𝑓 𝑥3, 𝑦3 = 𝑎 + 𝑏𝑥3 + 𝑐𝑦3 + 𝑑𝑥3 𝑦3
1 𝑥2 𝑦2 𝑥2 𝑦2 𝑇
𝑓 𝑥4, 𝑦4 = 𝑎 + 𝑏𝑥4 + 𝑐𝑦4 + 𝑑𝑥4 𝑦4 𝐶= 𝑦= 𝑎 𝑏 𝑐 𝑑
1 𝑥3 𝑦3 𝑥3 𝑦3
1 𝑥4 𝑦4 𝑥4 𝑦4 4×4
Solving inversion of a 4×4
Matrix C
𝑓 𝑥 𝑘 , 𝑦 𝑘 = 𝑎 + 𝑏𝑥 𝑘 + 𝑐𝑦 𝑘 + 𝑑𝑥 𝑘 𝑦 𝑘 𝑓 𝑘 = 𝑧𝑦
𝑓 𝑘 = 𝑧𝐶 −1 𝑓 𝑧 = [1 𝑥 𝑘 𝑦 𝑘 𝑥 𝑘 𝑦 𝑘 ]
𝑓 𝑘 = 𝑤𝑓 𝑤 = 𝑧𝐶−1

32. Piece of Results
T42 -> POP43:
are: 8.74e-5
mre: 0.00596
Better than idw!
POP43->T42: Better than idw mixed bilinear!
are: 0.000238
mre: 0.0829

33. Outline
Remap model in the Earth System Model
Design of Threp
Implementation
Bilinear in action
Demo
Future work

34. Discussion
Unstructured grid source
Periodic boundary problem of unstructured grid
Specificity conservative remapping
High dimensional remapping
Restart Threp

35. Outline
Remap model in the Earth System Model
Design of Threp
Implementation
Bilinear in action
Demo
Future work

36. Roadmap
Performance: More sub modules in Threp:
PyPy interpreter derivation
C extension of kernel code integral
mapreduce version of Threp More algorithms:
More grids: patch
triple poles bicubic
unstructured conservative(first & second)
Graduate!!

37. Conclusion
Threp abstract remapping model in the Earth System Model
Threp is a parallel remapping framework, supports masked/unstructured grid
Threp is lightweight, easy to start
A extrapolation coupled bilinear algorithm implemented in Threp, better accuracy
High performance scalability
Embrace Threp!!

38. Open Source
https://github.com/xunzhang/Threp

39. Acknowledge
Google.com
Stackoverflow.com
Lanning Wang< wangln@bnu.edu.cn >
Shunqiang Song< pinewall@gmail.com >
Li Liu< liuli03@mails.tsinghua.edu.cn >
Yushu Chen<>

40. Thank you! Questions?

41. Singularity(backup)

42. Ill-condition(backup)