1. Course Policies and outcomes

2. Chapter 1Chapter 1
Introduction to ParallelIntroduction to Parallel
ComputingComputing
Dr. Muhammad Hanif Durad
Department of Computer and Information Sciences
Pakistan Institute Engineering and Applied Sciences
hanif@pieas.edu.pk

3. Dr. Hanif Durad 3
Lecture Outline
 Motivating Parallelism
 Scope of Parallel Computing Applications
 Organization and Contents of the Course
 Texts and References

4. Parallel computing
 Parallel Processing is a term used to denote a large
class of techniques that are used to provide
simultaneous data processing tasks for the purpose
of
 Save time and/or money
 Solve larger problems
 Parallel computing is the simultaneous use of
multiple compute resources to solve a
computational problem 4
parallelcomputing-120214042541-phpapp02-130408023815-phpapp01.ppt

5. The Universe is Parallel
5
 Galaxy formation
 Planetary movement
 Weather and ocean patterns
 Tectonic plate drift
 Rush hour traffic
 Automobile assembly line
 Building a jet
 Ordering a hamburger
at the drive through.

6. Areas of Parallel Computing
 Physics – applied, nuclear, particle, condensed matter,
high pressure, fusion, photonics
 Bioscience, Biotechnology, Genetics
 Chemistry, Molecular Sciences
 Geology, Seismology
 Mechanical Engineering - from prosthetics to spacecraft
 Electrical Engineering, Circuit Design, Microelectronics
 Computer Science, Mathematics
Dr. Hanif Durad 6

7. Why Use Parallel Computing?
 Save time and/or money: In theory, throwing more
resources at a task will shorten its time to completion,
with potential cost savings. Parallel computers can be
built from cheap, commodity components.
 Solve larger problems: Many problems are so large
and/or complex that it is impractical or impossible to
solve them on a single computer, especially given limited
computer memory.
 Better response times: As the computing tasks are
engaged by a group of processors, the tasks are
completed in a smaller amount of time 7

8. Dr. Hanif Durad 8
Hardware Models
02Distributed_Computing.pptx

9. Ways to Classify Parallel
Computers
 One of the more widely used classifications, in use
since 1966, is called Flynn's Taxonomy
The 4 possible classifications according to Flynn’s
are :
 Single Instruction, Single Data (SISD)
 Single Instruction, Multiple Data (SIMD)
 Multiple Instruction, Single Data (MISD)
 Multiple Instruction, Multiple Data (MIMD)
Dr. Hanif Durad 9

10. Dr. Hanif Durad 10
Recent Trends in Computing

11. Technology Trends: Moore’s
Law
Dr. Hanif Durad 11
Slide from Maurice Herlihy
Clock speed
flattening
sharply
Transistor
count still
rising
6963_L1.ppt

12. Techology Trends: Power Issues
Dr. Hanif Durad 12
From www.electronics-cooling.com/.../jan00_a2f2.jpg6963_L1.ppt

13. Power Perspective
Dr. Hanif Durad 13
GigaFlop/s
MegaWatts
0.001
0.01
0.1
1
10
100
1000
10000
100000
1000000
10000000
100000000
1000000000
1960 1970 1980 1990 2000 2010 2020
Performance (Gflops)
Power
Slide source: Bob Lucas
6963_L1.ppt

14. Increase speed processor
 Greater no. of transistors
 Operation can be done in fewer clock cycles
 Increased clock speed
 More operations per unit time
 Example
 8088/8086 : 5 Mhz, 29000 transistors
 E6700 Core 2 Duo: 2.66 GHz, 291 million
transistor
lecture1-100131132656-phpapp02.pptx

15. The Multi-Core Paradigm Shift
 Key ideas:
 Movement away from increasingly complex
processor design and faster clocks
 Replicated functionality (i.e., parallel) is simpler to
design
 Resources more efficiently utilized
 Huge power management advantages
Dr. Hanif Durad 15
6963_L1.ppt

16. Multicore
 A multi-core processor is one processor that contains two
or more complete functional units. Such chips are now
the focus of Intel and AMD. A multi-core chip is a form
of SMP
lecture1-100131132656-phpapp02.pptx
parallelprogramming-130823023925-phpapp01.pptx

17. Concept of GPGPU (General-
Purpose Computing on GPUs)
 Idea:
 Potential for very high performance at low cost
 Architecture well suited for certain kinds of parallel applications (data
parallel)
 Demonstrations of 30-100X speedup over CPU
 Early challenges:
 Architectures very customized to graphics problems (e.g., vertex and
fragment processors)
 Programmed using graphics-specific programming models or libraries
 Recent trends:
 Some convergence between commodity and GPUs and their associated
parallel programming models
Dr. Hanif Durad 17
6963_L1.ppt

18. Parallel Programming Model
 Parallel programming models in common use:
 Shared Memory (without threads)
 Threads
 Distributed Memory / Message Passing
 Data Parallel
 Hybrid
 Single Program Multiple Data (SPMD)
 Multiple Program Multiple Data (MPMD)
 These models are NOT specific to a particular type of machine
or memory architecture
 Any of these models can be implemented on any underlying hardware
18
parallelprogramming-130823023925-phpapp01.pptx, p-60/114

19. Classes of Parallel Computers
 Cluster computing
 A cluster is a group of loosely coupled computers that
work together closely, so that in some respects they
can be regarded as a single computer
 Massive parallel processing
 A massively parallel processor (MPP) is a single
computer with many networked processors.
 MPPs have many of the same characteristics as
clusters, but MPPs have specialized interconnect
networks Dr. Hanif Durad 19
parallelprogramming-130823023925-phpapp01.pptx, p-99/114

20. Classes of Parallel Computers
 Grid computing
 Compared to clusters, grids tend to be more loosely
coupled, heterogeneous, and geographically dispersed
Dr. Hanif Durad 20

21. Cloud Computing
 A cloud provider has 100s of thousands of nodes (aka
servers).
 Cloud computing is massively-parallel computing with
multi-processors (i.e. many multi-core processors)
 In principle, your application may run on one, two, …
thousands of servers (i.e. processors)
 For your application to run on one, two, … thousands of
servers, your application code or data must be
parallelized.
 i.e. Split up into independent or relatively independent parts.
Dr. Hanif Durad 21
02Distributed_Computing.pptx

22. Dr. Hanif Durad 22
Organization and Contents of this
Course
1. Fundamentals: This part of the class covers basic parallel
platforms, principles of algorithm design, group
communication primitives, and analytical modeling
techniques.
2. Parallel Programming: This part of the class deals with
programming using message passing libraries and threads.
3. Parallel Algorithms: This part of the class covers basic
algorithms for matrix computations, graphs, sorting,
discrete optimization, and dynamic programming.
1+2=CIS-546 PC
3=CIS-645 PA

23. Summary
 Introduction to Parallel Computing
 Technology Trends
 Multicore
 GPGPU (General-Purpose Computing on GPUs)
 Parallel Programming Model
 Classes of Parallel Computers
 Cluster computing
 Massive parallel processing (MPP)
 Grid computing
 Cloud Computing
Dr. Hanif Durad 23

24. Dr. Hanif Durad 24
References
 Grama A., Gupta A., Karypis G., and Kumar V., Introduction
to Parallel Computing, 2nd ed., Addison-Wesley, 2003. (Main
Text Book)
 Thomas Rauber, Gudula Rünger, Parallel Programming: For
Multicore and Cluster Systems, Springer, 2010.
 Peter Pacheco, An Introduction to Parallel Programming,
Elsevier, 2011.
 Michael J. Quinn , Parallel Programming in C with MPI and
OpenMP, McGraw-Hill, 2004.

25. References
 Mostafa Abd-El-Barr and Hesham El-Rewini, Advanced
Computer Architecture and Parallel Processing, John Wiley
and Sons, 2005.
 Culler D. E., Singh J. P., and Gupta A., Parallel Computer
Architecture: A Hardware/Software Approach, Morgan
Kaufmann Publisher, 1999 .
 Foster I., Designing and Building Parallel Programs,
Addision-Wesley, 1995.
Dr. Hanif Durad 25