This document introduces distributed computing and compares the architectures of high-performance computing (HPC) and big data systems. It discusses that distributed computing works by taking large jobs and breaking them into smaller parallel tasks, and provides examples of HPC systems that use head and compute nodes and big data systems that use name and data nodes. The document also notes some lessons learned, such as that hybrid HPC systems can be useless and bugs found late in processing large datasets can be difficult to diagnose.

3. Introduce

4. Workloads
Distributed compute in a nutshell (where many nuts > few nuts)

5. Workloads

6. Distributed Compute Developer
Big Compute Big Penguin Big Data

7. BIG ANYTHING
Hello All Worlds

8. Big Co$t?
That’d be like Microsoft, right?

9. Microsoft Research Genomics

10. Inspect
Distributed compute in a nutshell (with many little nuts)

11. Inspect

12. HPC
Head Node
Broker Nodes
Compute Nodes
Allows on-premises
And hybrid option
Compare Architectures
Big Data
Name Node
Data Nodes
Allows cloud or on-premises
no hybrid option

13. Hadoop
HPC

14. All distributed compute
works on the basis of
taking a large JOB and
breaking it to many
smaller TASKS which
are then parallelised

15. Develop

16. Deploy

17. Examples
How do you take your compute?

18. OUCH
Lessons learned from getting too close to the coalface

20. A broken cluster is no
place to be diagnosing

22. Scalability < Elasticity

24. Hybrid HPC is next to
useless

26. 95% through a
petabyte is a bad place
to find a bug

27. Thank you!