This document provides information about a high performance computing course, including its objectives, learning aspects, textbooks, and course contents. The goal of the course is to provide a survey of high performance computing and discuss topics related to accessing HPC resources, developing applications for those resources, and executing developed applications. The course will help students design, analyze, and implement high performance computational science and engineering applications and focus on advanced computer architectures, parallel algorithms, languages, and performance-oriented computing.
2. Course Objective
The goal of this course is to provide a survey of high
performance computing.
The course will discuss topics related to accessing high
performance computing resources, developing applications
for those resources, and executing developed applications.
Wednesday, September 15,
2021
2
3. Learning Aspects..
The students will be able to design, analysis, and
implementation, of high performance computational
science and engineering applications.
The course focuses on advanced computer architectures,
parallel algorithms, parallel languages, and performance-
oriented computing
Wednesday, September 15,
2021
3
4. Books
TEXT BOOKS:
”Introduction to Parallel Processing”, Latest Edition,
By: M Sasikumar, Dinesh Shikhare, P.Ravi Prakash
“Parallel Programming in C with MPI and OpenMP”
Latest Edition, By: Michael J. Quinn
“Introduction to Parallel Computing”, Latest Edition,
By: Annanth Grama, Anshul Gupta, George karypis,
Vipin Kumar
Wednesday, September 15,
2021
4
5. Books
REFERENCE BOOKS:
“Grid Computing: A practical Guide to Technology
and Applications”, Latest Edition,
By: Ahmer Abbas
Wednesday, September 15,
2021
5
6. Course Contents
INTRODUCTION TO HPC:
Importance of HPC in Science and Engineering, Introduction to
Parallel Computers and Computational Overheads
ARCHITECTURES AND BUILDING BLOCKS:
Supercomputer Architecture, Vector Machines, Parallel
Processors, Data Parallel Processors, Single-Instruction-Multiple-
Data, Multiple-Instruction-Multiple-Data, Pipelining, Vectorization,
Parallelization, Comparison of Serial, Parallel and Vector
Architectures
Wednesday, September 15,
2021
6
7. Course Contents
PERFORMANCE MEASURES AND MODELS:
Speed-up limitations, Theoretical Timings, Efficiencies, Amdahl's
Law and Extensions, Scaled Speed-up, Pipeline Speed-up, Data
Dependency Reduction. Data flow, Loop reordering, Loop
optimizations, Implementation Principal of Locality, Caches and
Buffers.
PARALLEL ALGORITHMS:
Purely Parallel Algorithms, Parallelization of Algorithms, Parallel
linear algebra routines, Cyclic Reduction; Iterative Algorithms
(Jacobi, Gauss-Seidel and Red-Black Orderings); Divide-and-Conquer
Algorithms, Adaptive Quadrature, Correct Termination, Block
Decomposition Methods, Massively Data Parallel Algorithms, Array
notation.
Wednesday, September 15,
2021
7
8. Course Contents
PARALLEL PROGRAMMING PACKAGES:
Parallel Languages and Compilers, Fortran90 and HPC Fortran,
Parallel and Vector C Code, Massage passing interface (MPI),
Parallel Virtual Machine (PVM), Layout, Align, Replicate, Masking,
Shifting, Spreading, Broadcasting, Forall Loops, Levels of
Abstraction, Models of Computation and Parallel Overheads, Levels
of Abstraction, Multiple Program Counters in Hardware; Multi-
Thread Models, with Primary Sources of Overhead;; Task-Parallel
versus Data-Parallel Programming Models; Further Sources of
Overhead; Loop Transformations; Data Transformations;
Dependence Analysis; Compiler Strategies.
Wednesday, September 15,
2021
8
9. Course Contents
RESULTS AND BENCHMARKING:
Memory Architecture and Memory Access Times and Associated
Sources of Overhead; Multi-Process Execution Model; Performance
Tuning via Overhead Reduction; Task Scheduling; Data Partitioning
and its Effect on Performance.
Restructuring for Parallel Performance
Wednesday, September 15,
2021
9
11. Introduction to High-Performance Computing
What is High Performance Computing?
High‐speed computing. Originally implemented only in
supercomputers for scientific research
Used for scientific research and computational science
Main area of discipline is developing parallel processing
algorithms and software so that programs can be divided into
small independent parts and can be executed simultaneously by
separate processors
HPC systems have shifted from supercomputer to computing
clusters
Wednesday, September 15,
2021
11
13. What is Cluster
Cluster is a group of machines interconnected
in a way that they work together as a single
system
Terminology
Node – individual machine in a cluster
Head/Master node – connected to both
the private network of the cluster and a
public network and are used to access a
given cluster. Responsible for providing
user an environment to work and
distributing task among other nodes
Compute nodes – connected to only the
private network of the cluster and are
generally used for running jobs assigned
to them by the head node(s)
Wednesday, September 15,
2021
13
14. Types of Cluster
Storage
Storage clusters provide a consistent file system image
Allowing simultaneous read and write to a single shared file system
High‐availability (HA)
ƒ
Provide continuous availability of services by eliminating single
points of failure
Load‐balancing
ƒ
Sends network service requests to multiple cluster nodes to balance
the requested load among the cluster nodes
High‐performance
ƒ
Use cluster nodes to perform concurrent calculations
Allows applications to work in parallel to enhance the performance
of the applications
Also referred to as computational clusters or grid computing
Wednesday, September 15,
2021
14
15. Benefits of Cluster
Reduced Cost
The price of off‐the‐shelf consumer desktops has fallen in recent years,
and this drop in price has corresponded with a vast increase in their
processing power and performance. The average desktop PC today is
many times more powerful than the first mainframe computers.
Processing Power
The parallel processing power of a high‐performance cluster can, in
many cases, prove more cost effective than a mainframe with similar
power. This reduced price‐per‐unit of power enables enterprises to get
a greater ROI(Return On Investment).
Scalability
Perhaps the greatest advantage of computer clusters is the scalability
they offer. While mainframe computers have a fixed processing
capacity, computer clusters can be easily expanded as requirements
change by adding additional nodes to the network
Wednesday, September 15,
2021
15
16. Benefits of Cluster
Improved Network Technology
In clusters, computers are typically connected via a single virtual
local area network (VLAN), and the network treats each
computer as a separate node. Information can be passed
throughout these networks with very little lag, ensuring that
data doesn’t bottleneck between nodes.
Availability
When a mainframe computer fails, the entire system fails.
However, if a node in a computer cluster fails, its operations can
be simply transferred to another node within the cluster,
ensuring that there is no interruption in service
Wednesday, September 15,
2021
16
17. When We Need High Performance Computing?
Case -1: Complete a time-consuming operation in less time
I am an automotive engineer
I need to design a new car that consumes less gasoline
I’d rather have the design completed in 6 months than in 2 years
I want to test my design using computer simulations rather than
building very expensive prototypes and crashing them
Case 2: Complete an operation under a tight deadline
I work for a weather prediction agency
I am getting input from weather stations/sensors
I’d like to predict tomorrow’s forecast today
Wednesday, September 15,
2021
17
18. When We Need High Performance Computing?
Case 3: Perform a high number of operations per seconds
I am an engineer at Amazon.com
My Web server gets 1,000 hits per seconds
I’d like my web server and databases to handle 1,000 transactions
per seconds so that customers do not experience bad delays
Wednesday, September 15,
2021
18
19. Where are we using HPC?
Used to solve complex modeling problems in a spectrum of disciplines
Topics include:
o Artificial intelligence o Physical oceanography
o Climate modeling o Plasma physics
o Automotive engineering o Quantum physics
o Cryptographic analysis o Quantum chemistry
o Geophysics o Nuclear physics
o Molecular biology o Solid state physics
o Molecular dynamics o Structural dynamics
HPC is currently applied to business uses as well
data warehouses
transaction processing
Wednesday, September 15,
2021
19
20. From concept to engineering, from design to test and
manufacturing, from weather prediction to medical discoveries, our
day to day life depends more and more on HPC simulations
Safer products, accurate predictions, research, etc.
High-performance compute clusters provide the most efficient,
flexible, cost effective HPC environment for any HPC simulation
Wednesday, September 15,
2021
20
21. Changing times
From 1986 – 2002, microprocessors were speeding like a rocket,
increasing in performance an average of 50% per year
Since then, it’s dropped to about 20% increase per year..
Wednesday, September 15,
2021
21
22. The Problem
Up to now, performance increases have been
attributed to increasing density of transistors
But there are inherent problems
A little Physics lesson:
Smaller transistors = faster processors
Faster processors = increased power consumption
Increased power consumption = increased heat
Increased heat = unreliable processors
Wednesday, September 15,
2021
22
23. An intelligent solution
Move away from single‐core systems to multicore processors
“core” = processing unit
Introduction of parallelism..!!
But …
Adding more processors doesn’t help much if programmers
aren’t aware of them..!!
… or don’t know how to use them
Serial programs don’t benefit from this approach (in most
cases)
Wednesday, September 15,
2021
23
24. Parallel Computing
Form of computation in which many calculations are carried out
simultaneously, operating on the principle that large problems can
often be divided into smaller ones, which are then solved
concurrently i.e. "in parallel“
So, we need to rewrite serial programs so that they’re parallel.
Write translation programs that automatically convert serial progra
ms into parallel programs.
This is very difficult to do.
Success has been limited
Wednesday, September 15,
2021
24
25. Parallel Computing
Example
Compute n values and add them together.
Serial solution:
Wednesday, September 15,
2021
25
26. Parallel Computing
Example
We have p cores, p much smaller than n.
Each core performs a partial sum of approximately n/p values.
Wednesday, September 15,
2021
26
27. Parallel Computing
Example
After each core completes execution of the code, is a private
variable my_sum contains the sum of the values computed by
its calls to Compute_next_value.
Ex., n = 200, then
• Serial – will take 200 addition
• Parallel (for 8 cores)
– Each core will perform n/p = 25 addition
– And master will perform 8 more addition + 8 receive
operation
– Total 41 operation.
Wednesday, September 15,
2021
27
28. Parallel Computing
Some coding constructs can be recognized by an automatic program
generator, and converted to a parallel construct.
However, it’s likely that the result will be a very inefficient program.
Sometimes the best parallel solution is to step back and devise an en
tirely new algorithm.
Parallel computer programs are more difficult to write than
sequential programs
Potential problems
Race condition (output depending on sequence or timing of
other events)
Communication and synchronization between the different
subtasks
Wednesday, September 15,
2021
28