Lectures of CS-721 (Network Performance Evaluation) taught for the Virtual University by Junaid Qadir.
To access other resources, visit http://sites.google.com/site/netperfeval
2. Today’s agenda:
1 What is Network Performance Evaluation
(NPE)?
2 About this course: contents and resources
3. 1
What is NPE?
ˉ Importance of Computer Networks
ˉ Importance of NPE
ˉ What does performance mean?
4. NPE (design, analysis and
evaluation)
1 For a given packet arrival pattern and desired
performance (e.g., low packet drops), what
should be the size of router buffers? (design
question)
2 What is the quantitative improvement in
performance of the router after the latest
software (or hardware) upgrade? (analysis/
evaluation question)
5. Goals of NPE
1 Comparison of design alternatives
2 System dimensioning
3 Relationship (causal/ correlational) modeling
4 Performance description in appropriate metrics.
10. The Science and Art of NPE
Part A
12 lectures
12 lectures
Lecture 2-13
Statistical
The Science of background
The Art of NPE
NPE (grammar of
science)
11. The Science and Art of NPE
Part A
12 lectures
The Science of NPE
"The fundamental principle of science, the
definition almost, is this: the sole test of the
validity of any idea is experiment." -
Richard Feynman.
'In questions of science the authority of a
thousand is not worth the humble
reasoning of a single individual.' Galileo
Galilei.
12. The Science and Art of NPE
Part A
12 lectures
Statistical background
It is better to be approximately right than be exactly wrong – John
Tukey
13. The Science and Art of NPE
Part A
12 lectures
The Art of NPE
“All models are wrong; some are
useful." – George Box.
"Excellence in statistical graphics
consists of complex ideas
communicated with clarity, precision,
and efficiency." – Edward Tufte.
15. Empirical/ Experimental NPE
Empirical/
Part B
experimental NPE
13 lectures
Lecture 14-26
Workload Design of Internet
Modeling Experiments Measurement
16. Empirical/ Experimental NPE
Part B
13 lectures
Workload Modeling
"The essence of modeling, as opposed to just
observing and recording, is one of abstraction.
This means two things: generalization and
simplification." - Dror Feitelson.
17. Empirical/ Experimental NPE
Part B
13 lectures
Design of Experiments
Does the „magic
tablet‟ work as
advertised or is it
just a gimmick?
Hypothesis testing is at the very core of the scientific method.
18. Empirical/ Experimental NPE
Part B
13 lectures
Internet Measurement
"Measure what is measurable, and make
measurable what is not so." - Galileo
Galilei
19. Module 3/3
Simulation/ Modeling based
NPE
Part C
18 lectures
Lecture 2-13 Lecture 14-26 Lecture 27-44
20. Simulation/ Modeling based NPE
Part C
18 lectures
Lecture 27-44
Probability and
Stochastic Analytical Simulation
Processes Modeling Modeling
Preliminaries
21. Simulation/ Modeling based NPE
Part C
18 lectures
Stochastic Background
PROBABILITY
RANDOM VARIABLE’S VALUE
22. Simulation/ Modeling based NPE
Part C
18 lectures
Analytical Modeling
SERVER
PACKET
QUEUE
ROUTER
"The essence of modeling is one of abstraction.
This means two things: generalization and
simplification."
23. Simulation/ Modeling based NPE
Part C
18 lectures
Simulation Modeling
Computer based simulation modeling can be performed when
experimenting with real system is not practical or efficient and when
simpler analytical methods do not exist
24. Course conclusion
Summary of this course
(Lecture 45)
Lecture 2-13 Lecture 14-26 Lecture 27-44
29. Reference for today’s lecture
Chapter 1 [Raj Jain]
Find more lecture resources at the course‟s companion site:
http://sites.google.com/site/netperfeval
30. Credits/ Acknowledgments
Books
Raj Jain, “The Art of Computer Systems Performance Analysis”
Figures and Images:
Car Crash: http://www.pclaunches.com/other_stuff/audi_uses_supercomputer_for_crash_simulations.php
Galileo‟s Portrait: http://en.wikipedia.org/wiki/File:Justus_Sustermans_-_Portrait_of_Galileo_Galilei,_1636.jpg
George Box‟s Picture: http://gallery.socionix.com/d/49618-2/1990_George_Box.jpg
Edward Tufte‟s Picture: http://www.freebase.com/view/en/envisioning_information
VU LMS: http://vulms.vu.edu.pk
Clipart supplied with Microsoft Office
These resources have been used in these lecture slides for educational purpose under the fair use doctrine.
The ownership of these resources, if copyrighted, is retained by their respective copyright owners.
Welcome to the first lecture of this new course offered by Virtual University titled: 'Network Performance and Evaluation’ being offered by Virtual University. I’m the instructor for this course and my name is JunaidQadir.Firstly, a few things about myself.I’mJunaidQadir, currently an Assistant Professor at SEECS, NUST. I received my PhD in Computer Engineering in May 2008 from the University of New South Wales (UNSW), Sydney, Australia where I worked on the research topic of "Improving broadcast performance in multi-radio multi-channel multi-rate wireless mesh networks"'.Currently, I am the director of the Cognet lab which operates under the PTCL, Cisco, NUST IP technology center of excellence situated at SEECS, NUST.Before my PhD studies, I have worked in the networking industry for around 4 years where I gained hands-on experience working on the largest data networks in the country. Amongst other places, I have worked at PTCL’s Pakistan Internet Exchange (PIE) and at the national data networks of National Telecommunication Corporation (NTC). I completed my bachelor’s degree in electrical engineering from University of Engineering and Technology (UET), Lahore, Pakistan in 2000.
Today, I want to talk about two things. Firstly, I want to introduce you to the notion of NPE and show you how fundamental it is to the efficient operation of computer and communication networks.Secondly, I want to talk about this particular course: about myself, my pedagogical approach to teaching this course, the outline of this course, and the relevant resources such as the course textbook, the course guidebook, and the course websites.
So we begin with what is NPE:Importance of NPE and its basic tasks:With computer networks playing such a central role in modern life, it is imperative that: 1) we design high-performance reliable networks and 2) we have the ability to analyze and evaluation the performance of computer networks.NPE, like probability, can often contradict intuition and therefore it would be foolhardy to only rely on common-sense and gut feeling for evaluating network performance. As an example of NPE results contradicting common-sense consider the Braess' Paradox1 which states that adding extra capacity to a network when the moving entities selfishly choose their route, can in some cases reduce overall performance. This contradicts the conventional wisdom which implies that the solution to all network problems is to increase capacity but this paradox is indicating that it may not always be productive to do so. Since we cannot rely on intuition and gut feeling only for NPE, this motivates us to develop formal tools to facilitate analysis and evaluation. NPE is also workload and metric specific, changing which can lead to dramatically different results. 1 http://en.wikipedia.org/wiki/Braess%27s_paradox
There are two broad categories of NPE questions: design questions, and analysis or evaluation questions.
NPE is about quantifying the service delivered by a computer or communication system. The goal of a performance evaluation may either be:1) a comparison of design alternatives, i.e. quantify the improvement brought by a design optionComparison of designs requires a well-defined load model: e.g., we might be interested in: comparing the power consumption of several server farm configurations; however, the exact value of its intensity does not have to be identified. 2) system dimensioning, i.e. determine the size of all system components for a given planned utilization. With a performance goal in mind (e.g., a service level agreement, SLA, of 99.999% uptime), what infrastructure is needed? System dimensioning requires a detailed estimation of the load intensity. Like any prediction exercise, this is very hazardous and liable to going very wrong.3) determining causal relationship model between design parameter and performance metric of choice4) describing the performance of a system in terms on performance metrics.
In the first module of this course, we are going to learn about the philosophy of NPE and discover that it is both a work of art and a work of science.
Scientific method is a very important method for gaining knowledge in which we focus on building knowledge through repeatable and controlled (empirical) observations. It is a useful method for NPE since it minimizes the effect of biases and forms conclusion only on the basis of experimental evidence.
Statistics is an integral tool that we use to deal with uncertainty. It helps us to quantify uncertainty and to make informed decisions in the face of uncertain circumstances. It is the grammar of science and is used by all the physical and social sciences since all measurements are beset by errors and it’s necessary for any meaningful conclusions from such data that we are able to quantify the amount of error (not in the sense of a blunder but in the sense of natural unavoidable uncertainty) To read more about the example of the man with two watches, refer to ‘Statistical Modeling: A Fresh Approach’ by Daniel Kaplan.
The model is the most basic element of the scientific method. Everything done in science is done with models. A model is any simplification, substitute or stand-in for what you are actually studying or trying to predict. Models are used because they are convenient substitutes that facilitates experimentation, analysis or observation. We've all heard about hypotheses and theories, especially in physics and chemistry. Theories usually comprise some idea that scientists have about how nature works, but that they aren't totally sure. Hypotheses and theories are merely particular kinds of models that we will refer to below as abstract models. The scientific method is a procedure for the construction and verification of models. We note here that science progresses by continuously trading current models for better models whenever they may become available and there's no right answer at which we would stop our search for better models.It must be remembered always that all models are only approximations of reality. They are used since they are useful for learning about the system and how it will behave if the system parameters are changed. The role of a NPE model can be compared to a city map. A city map has two key properties. It is an abstract representation of the real situation in that the distances between any two points on the map are not in geographical proportion, and it is simple because it is uncluttered by unimportant real-world physical details. The natural urge is to create a NPE "map” adorned with an abundance of physical detail because that would seem to constitute a more faithful representation of the computer system being analyzed. In spite of this urge, you should strive instead to make your NPE models as simple and abstract as a city map. Adding complexity does not guarantee accuracy. (Adapted from Neil Gunther’s book: ‘Analyzing Computer System Performance with Perl::PDQ’.)Unfortunately, there is no simple recipe for constructing a city map (or a NPE model). Einstein reputedly said that things should be as simple as possible, but no simpler. That should certainly be the goal for applying NPE, but like drawing any good map there are aspects that remain more in the realm of art than science.
In the second module of this course, we are going to learn about the use of empirical methods (experiment based) to perform NPE.
To perform NPE of any system, it is important that we know what input load would be applied to such a system. All performance evaluation is contingent on what workload is applied to the system, and it is possible (and is usually the case) that performance conclusions be reversed if we make different set of assumptions about what the workload would be like. We will see in the subpart 1 of the module on empirical methods (Part B of this course) that workload modeling is essentially an abstraction (or simplification) process in which we generalize the actual workload by its parameterized model so that we can perform evaluation on the system by tweaking the workload model and observing how the performance varies in response. "The essence of modeling, as opposed to just observing and recording, is one of abstraction. This means two things: generalization and simplification." - DrorFeitelson.The following is an excerpt from DrorFeitelson’s book1 on Workload Modeling:An important property of good models is simplicity. A good model doesn’t just define new useful quantities—it also leaves out many useless ones. The act of modeling distills the cumulative experience gained from performing experimental measurements, and sets then in a format that can be used as the basis for further progress [14]. In fact, this is also the basis for natural science, where experimental observations are summarized in simple laws that are actually a model of hownature operates.It should be stressed that finding new models is not easy. The largest obstacle is actually noticing that a new model is needed. It is very tempting to interpret experimental results in the light of prevailing preconceptions. This runs the risk of fitting the measurements to the theory, rather than the theory to the measurements.NPE analysts use models for various tasks: describing a variable (lets say the input to the system under consideration which is not known apriori exactly but known statistically) or to describe a process or system under study (a model may be more amenable to analysis or experimental manipulation). Both approximation facilitates (and in some cases makes possible) efficient analysis and evaluation. 1http://www.cs.huji.ac.il/~feit/wlmod/
Design of experiments uses hypothesis testing to ascertain the validity of a claim (or a hypothesis) which explains some observable phenomena. We seek to differentiate real effects from coincidental or chance effects which may also produce the same result (due to chance only).
An important part of the empirical method is measuring and learning how to measure.
In the thirdmodule of this course, we are going to learn about the use of modeling and simulation to perform NPE.
We will focus on the necessary stochastic and probability background that we shall be using for the remainder of this course.
"One of life's more disagreeable activities, namely, waiting in line, is the delightful subject of this book. One might reasonably ask, "What does it profit study such unpleasant phenomena?" from preface to Queueing Systems, Vol. 1, Leonard Kleinrock.In analytical NPE, we model the system under consideration using mathematical symbols that are mathematically related e.g, the system may be represented as a set of equations or inequalities. a sequence on sourcing statistics. Analytical models are typically mathematical models for which closed-form solutions exist and are one of the quickest methods of NPE. However, they are applicable to only a subclass of NPE systems. In this course, we will learn the utility of mathematical modeling. Mathematical models govern our economy and help forecast our weather. They predict who will win the election and decide whether your bank loan application should be granted. However, the man on the street knows little about what mathematical models are and how they work. In this course, we will use NPE to explain how mathematical models are designed, built, and validated.The most important framework that we shall study in this context would be queueing theory. It is amazing how much we can model using the plain concept of a queue of customers waiting for service from a server. We can model a router or a switch as a queue in which packets line up in a queue to obtain switching service: using queueing theory, we can obtain average case statistics like how much time does a packet wait on average in the system, at any given time, how many packets are queued in the router buffers waiting for service?
In many cases, real life systems are too complex and may not be even approximated closely by analytical models. In such cases, we usually recourse to simulation methods in which we build computational models (or computer program models) of the real system that we would like to model. Thereafter, we can analyze the system by executing the computation model (or program) and make conclusions on the basis of such executions. Simulation modeling is a very general technique and is used much in NPE studies especially for evaluation of algorithms, and systems of computer networks.
In this course, our aim is to present performance evaluation in an accessible way which can be understood and implemented by all computer engineers and scientists. The foundations of performance evaluation are in statistics and queuing theory, therefore, this course traditionally involves a lot of mathematics. However, I will aim to present ideas intuitively and in the context of concrete applications. I will also post links to resources that you can use to further develop your intuition. We will see in this course that we can often use computer simulations to develop intuition about intractable problems that can be solved by analytical models. Similarly, I would encourage you to explore active demo material including applets which will be linked to in the companion site of this course. You would learn that you can gain a lot of intuition by practicing through these demos and applets.