1. Middleware Frameworks for Adaptive Executions and Visualizations of Climate and Weather Applications on Grids SathishVadhiyar Grid Applications Research Lab Supercomputer Education and Research Centre Indian Institute of Science Bangalore February 16, 2011 Yahoo! Hadoop India Summit, Indian Institute of Science

2. Outline Parallel Simulation and Visualization Resource Constraints Impact on Climate Simulations Adaptive Integrated Framework Framework Contradictory Objectives Decision Algorithm Steering the Visualizations Results Progress of Simulation and Visualization Adaptation of Parameters Potential for Cloud Computing February 16, 2011 Yahoo! Hadoop India Summit, Indian Institute of Science

3. Parallel Simulation and Visualization Critical climate applications like cyclone tracking require High-fidelity high-resolution simulation High-performance computations Massive amount of output On-the-fly remote visualization Real-time guidance to policy and decision makers Joint analysis by geographically distributed climate scientists High-performance simulations Parallel I/O Remote visualization DISK Network Figure: Simultaneous simulation and remote visualization using stable storage February 16, 2011 Yahoo! Hadoop India Summit, Indian Institute of Science

5. High I/O bandwidth

6. Limited network bandwidth

7. Limited storage spaceSIM VIS Simulation Process Visualization Process Stable Storage Network Figure: Illustration of resource constraints on simulation February 16, 2011 Yahoo! Hadoop India Summit, Indian Institute of Science

8. Impact on climate simulations Rapid accumulation of data in the stable storage Eventual unavailability of storage Stalling of simulation Low temporal resolution Loss of visualization February 16, 2011 Yahoo! Hadoop India Summit, Indian Institute of Science

11. Determine near-optimal parameters

12. Schedules climate simulation application

15. Decision Algorithm Objectives Maximize rate of simulation Maximize temporal resolution Enable continuous visualization Ensure availability of storage Contradictory Objectives February 16, 2011 Yahoo! Hadoop India Summit, Indian Institute of Science

16. Decision Algorithm Input Simulation resolution Network bandwidth Remaining disk space Output Number of processors for simulation Output frequency Optimization Based Algorithm February 16, 2011 Yahoo! Hadoop India Summit, Indian Institute of Science

17. Optimization-based Approach Causes of faster consumption of storage space Faster execution time Limited network bandwidth High frequency of output Objectives Optimal processor allocation Best possible output frequency Judicious use of storage Maximize simulation ratewithin the constraints related to continuous visualization, acceptable output frequency, I/O bandwidth, disk space and network bandwidth February 16, 2011 Yahoo! Hadoop India Summit, Indian Institute of Science

18. Problem Formulation Objective function: minimize t Table: Decision Variables Time Constraint: Time to solve + Time to output ≤ Time to transfer (1) February 16, 2011 Yahoo! Hadoop India Summit, Indian Institute of Science

19. Constraints Disk Constraint: Net input to the disk ≤ Remaining disk space (2) (3) Bound Constraints: Bounds for t and z (4) (5) February 16, 2011 Yahoo! Hadoop India Summit, Indian Institute of Science

20. Experiments Simulation: Weather Research and Forecasting Model v3.0.1 Visualization: VisIt v1.12.0 Climate Application: Tracking Cyclone Aila Modeled area: 32x106 sq. km. from 60ºE - 120ºE and 10ºS - 40ºN Formed: 23th May 2009, Dissipated: 26th May 2009 Figure: Visualization of Perturbation Pressure showing the track of Aila Table: Resolutions for different Pressure Values February 16, 2011 Yahoo! Hadoop India Summit, Indian Institute of Science

21. Experiments Table: Simulation and Visualization Configurations February 16, 2011 Yahoo! Hadoop India Summit, Indian Institute of Science

22. Faster rate of simulation Simulation stalls in Greedy-Threshold approach Simulation Progress Figure: For cross-continent configuration February 16, 2011 Yahoo! Hadoop India Summit, Indian Institute of Science

23. Visualization Progress Faster rate of visualization Lags behind in attempt to visualize every time step initially INCREASING LAG Figure: For intra-country configuration February 16, 2011 Yahoo! Hadoop India Summit, Indian Institute of Science

24. Less than 50% disk space used Higher rate of disk space consumption Disk Space Utilization Figure: For intra-country configuration February 16, 2011 Yahoo! Hadoop India Summit, Indian Institute of Science

25. Adaptivity Figure: For inter-departmentconfiguration February 16, 2011 Yahoo! Hadoop India Summit, Indian Institute of Science

26. February 16, 2011 Yahoo! Hadoop India Summit, Indian Institute of Science Steering the Visualization

27. February 16, 2011 Yahoo! Hadoop India Summit, Indian Institute of Science Steering Across the Ocean! Auto-changing number of procs to maintain QoS Changing Resolution of Simulation Changing Visualization Frequency Changing number of procs from 96 to 80

28. Ship the simulations to a cloud Use resource management services of clouds to find a “nearby” large storage This will eliminate the storage problem/constraint But new research challenges: Storage can spill over; Need to maintain metadata of storage repositories Simulation->Storage->Visualization will now involve multiple hops Hence added benefits due to large storage-as-service in cloud will have to balanced against loss in performance February 16, 2011 Yahoo! Hadoop India Summit, Indian Institute of Science Potential for Clouds

29. The infrastructure has to be expanded to include multiple simultaneous multi-user visualizations of multiple independent simulations Such independent simulations are natural for executions on clouds. February 16, 2011 Yahoo! Hadoop India Summit, Indian Institute of Science Potential for Clouds

30. To minimize lag between simulation and visualization site – choosing representative frames Multiple visualization-simulation framework Applying for other applications February 16, 2011 Yahoo! Hadoop India Summit, Indian Institute of Science Future Work

31. PreetiMalakar (Phd student) Dr. Vijay Natarajan(Co-researcher) February 16, 2011 Yahoo! Hadoop India Summit, Indian Institute of Science Acknowledgements

32. February 16, 2011 Yahoo! Hadoop India Summit, Indian Institute of Science Thank You!