Simulation and Performance Evaluation - aa 2014-15

Past academic year courses: bottom page

Welcome to the 2014-15 web site of Simulation and Performance Evaluation. This is a dynamic, one-way communication channel, you can interact (with some parsimony, please!) sending me e-mails. I know more modern tools can be adopted, but I'm lazy, and I don't want to adopt a moodle platform or open a facebook prfile for the course.

The billboard is used to convey messages and announcements also to those that cannot be in class, so please check it regularly.

Slides and other teaching material (including assignements) will be posted here in due time.


Tuesday 9:00--11:00 and Thursday 14:00--16:00, Room A203

Bill board

  • Course Opening: The course starts Thursday Feb. 19, from 14-16 PM room A203.
  • Wednesday March 11: I have activated a forum for the Homeworks on DoL-ESSE3. Not sure if it works, so we might end up using something else, but give it a try.
  • The lesson of Thursday March 19 is suspended for the ICT Days. Enjoy them!
  • Moodle Platform. The moodle platform for our course has been activated through your ESSE3 account.
    Please, register immediately. It will be used to assign and collect the assignments. Is you do not do the assignements you cannot pass the exam.
  • Lessons are suspended for Easter from April 2 to April 12
  • Tuestay April 29 Luca and Michele will dedicate the 2 hours of the lesson to discuss and check the first assignment with you. Thursday 30 the lesson is suspended, so you can recover from the assignemtn effort ... or finish it!
  • Extended deadline for early delivery of Assignments 2 and 3: now July 19.

Program and Methodology

Please, refer to ESSE3 and the syllabus relative to Simulation and Performance Evaluation for the official course content.

This said, let's try to understand a bit better the course goals and the course content, which, up to some point, will also be twisted and tweked based on how we came to know each other during the first few lessons. As mentioned in the general page of the course site, Performance Evaluation (PE for short) is central to all systems design, and ICT systems are no exception. Event driven simulations of just Simulations to be brief, are one of the means to evaluate performance, but since very often the term "simulation" is used with different meanings, and since it is unfortunately widespread to think about simulations in bery different terms, we explicit it in the title that Simulation techniques and methodology are part of this course.

The main objective of the course is defining what PE means, what are the applications areas, and why it is (or should be) a centerpiece of any system design ... including computers, software, the Internet, and applications in general. Very often performance is just seen as speed ("how fast is your system"?), but it is not a great intellectual feat to understand that speed alone is not sufficient: even a racing car can be the fastest of the lot, but if it breaks down it does not win the race! So reliability is part of performance, availability too, and dependability even more, specially for mission-critical systems.

One of the major issues we will have to clarify is that performance is rarely (if ever) a simple number, but it has to be expressed in stochastic terms, and often as complex vectors of time-dependent variables, and this explain why the starting point of our course is some reasonable knowledge of Probability and Statistics (which in turn imply some reasonable knowledge of Analysis). And we give for granted that, as you all are graduate students of Computer Science, you are all much better than me in programming techniques and software engineering, so that the implementation of simple algorthms, drawting some graphs and manipulating matrices and programs outputs is not a problem.

We will try to make a logical path that starts froms systems, simple or complex as they are, understand, given the system, what are the relevant metrics that define their perfromance, and what are the dependencies between different metrics and different subsystems. Once we have understood what are the metrics of interest and their relationships, we have to take a tough decision to evaluate performance: shall we measure them on the real system, simulate the system, or build an analytical model?

We will start from measures, or better, from the interpretation of measure points and from the design of a correct experimental setup do do the measures. In some cases we might use a simulator as if it were a real system, because this gives us more flexibility ... and makes experiments much, much cheaper!

Next we will introduce simulations, understand what they really are from a mathematical point of view, and learn the basics to design a simulator.

Finally we will make some examples of systems that are prone to an analytical modeling and how these models can be solved to yield results that are otherwise impossible to obtain.


We do not have an official textbook, because the course covers the enormously vast topic of Perfroamcne Evaluation with a "breadth first" approach, so that we would end up using chapters of many different books, and it is not worth buying them. However, I report here some selected books that for one reason or another are relevant for the course ... and try to explain why.

Book No. 1:
K.S. Trivedi,
"Probability and Statistics with Reliability, Queuing, and Computer Science Applications",
Wiley-Interscience; 2001
ISBN: 0-471-33341-7
can be bought on-line AT Amazon or directly from the Wiley Web Site .
2-3 copies are available in the University and Faculties Libraries;
The first five Chapters plus Chapter 10 of this book contain all the background in probability and statistics we need for the course. Indeed, I think there is much more than what you already know, but I do not have control on the courses you took at the Bachelor, thus I cannot be more precise. In conclusion: don't be too scared if some of the material looks new or if some details are beyond your knowledge, we'll rehearse these more advanced parts. If instead you have problems with Bayes rule, feel uncomfortable with percentiles and find difficult to compute the probability that an exponential RV with mean m is larger than 2m, then it is probably better to devote some time to study these Chapters.

Book No. 2:
R. Jain,
"The Art of Computer Systems Performance Analysis: Techniques for Experimental Design, Measurement, Simulation, and Modeling,"
Wiley- Interscience, 1991,
This book is very old, so I do not advise to buy it, but it is still the only one I'm aware of that tackles the entire range of Perfromance Analysis techniques in the context of Computer Science, thus it is a good reference for the subject as a whole, while taking examples are "real cases" is today a bit outdated.

Book No. 3:
G. Bolch, S. Greiner, H. de Meer, K.S. Trivedi,
"Queueing Networks and Markov Chains : Modeling and Performance Evaluation With Computer Science Applications",
Wiley-Interscience; 1998
ISBN: 0-471-19366-6
(can be bought on-line AT Amazon or directly from the Wiley Web Site)
The following is a very good and advanced book on the use of Markov Chains and Queueing Networks for performance evaluation, but it is definitely not needed for our course, where analytical modeling will be at most 1/3 of the course.


The exam is composed of the assignments we give you during the course through the moodle platform via ESSE3, plus and oral discussion and colloquium. All assignments are individual, meaning that each of you have to deliver a report (often limited in length, to avoid that you write too much). There will be 3 assignments during the course, each of which should not take more than 8-10 hours to complete. Further details will be given later on.

The oral discussion can include exercises, clarifications on the assignements and reports and theoretical, open questions as well. The outcome is a blend of the quality of your reports and the oral discussion, with "more weight" on the assignments if the oral confirms the result; however, you cannot come with assignments perfectly done to the oral, and then be unable to even discuss what is their meaning, in this case the oral dominates, and you may even fail the exam.

All reports must be written in English and also the clarity of the language is part of the evaluation, while the oral can be taken both in English or in Italian as you prefer.

Michele Segata and Luca Baldesi will help me with assignments and their evaluation.

Forthcoming Dates

  • June - July 2015

Refer to ESSE3 for the remaining burocracy.

Please remember to register for the exam before delivering your material and reserve for the oral discussion!

Teaching Material

The slides are provided to help taking notes. "Reading" the slides is definitely not enough to study the course. Additinal material, if present, normally refers to somthing we discussed in class, but may have different notation or cover more than we actually did.

  • Course presentation, rules, exams, and the program at large.
  • The systems we consider for performance evaluation, thge need for a model, an example of possible systems with many facets, common concetual mistakes in PE, the three methodologies for PE: Measures, Simulations, Analysis
    • PDF Slides 24/2/2015, Version 2.1. This should be the final version.
  • Rehearsal on probability theory
    • PDF Slides 10/3/2015, Version 5.1. Final version with typo corrections and comments based on in-class discussion.
  • Stochastic Processes
    • PDF Slides 23/3/2015, Version 2.1. Stationarity, independence, simple memory structure, Marov Processes, Autocorrelation and wide-sense stationarity. Final version.
  • Manipulation and representation of measured data
    • PDF Slides 27/3/2015, Version 2.5. Good and bad measures, simple plots, estimates, mean, variance, etc., and identification of dataset characteristics, memory and correlation, GPS errors.
    • Datasets, R scripts and plots of the examples we used in class
  • A Primer on R
  • The Art of Modeling
    • PDF Slides 21-04-2015, Version 3, Transmitting bits & Moving cars, Protocols, DTMCs, CTMCs, ...
  • Continuous time queueing systems
    • PDF Slides 04-05-2015, M/M/1, M/M/m, M/M/infty, M/G/1, plots, ...
  • Queueing networks
  • Discrete Event Simulation
  • More on Confidence Intervals and MLE
  • Examples of Modeling in ICT


The assignments are parte of the exam, indeed a large part of it. We have three different assignemnts, each requiring 10-20 hours of work ... provided you have already studied the relative theory! There are three assignemnts, one on measures, one on simulation, and one on modeling. Each one assignes up to 11 points, so that overall you can "collect" 33 points, that then you have to confirm with the oral discussion.

  • A template in LaTex explaining how to write a decent (if not good) report. You are free to use any word processor you like, but don't complain or blame it on us if the result is bad and/or the format requested difficult to achieve.
  • Assignment 1: Measures estimation and analysis.
    • Assignment text
    • Data folder - Here you find the assignemnt again and all the personalized data-sets, labled with your surname.
    • If you have opted for R, here are some hints and instructions.
      To install R, you can do the following depending on your OS:
      • Linux: Just open your terminal and type
        sudo apt-get install r-base
      • Mac OS X: If you have MacPorts installed, open your terminal and type
        sudo port install R
        If you don't have MacPorts, you can download R binaries from the official R website
      • Windows: Download the binaries from the official R website
      For a more efficient and nice coding, you can download some free tools online:
      • RStudio: This is an IDE for R which includes the text editor, the R console, the plot window, and some more useful things in a single window. You can download it here
      • Vim R plugin: If you hate IDEs and you are used to vim there is a plugin that splits the terminal between vim for source editing and the R console. You can then highlight lines of code and send them to the R console (and many more stuff). Careful: if you are not already very familiar with vim, forget about this! It will take you too much time to set it up and running. If you know vim and you plan to use R a lot, then have a look at this quick and dirty guide for installing it and at the official documentation
      To get started with R, here you can find tons of examples about basic operations, data loading, computing statistics, plotting results, etc.
  • Assignment 2: Simulation of a queuing network.
  • Assignment 3: Approximation and interpretation of a queuing network.

Meeting me

The most obvious place to query, ask for clarifications, rise doubts and so on is the class, so that everybody can intervene and be aware of clarifications. Further minor queries can be done at the end (or before) the lesson.

If additional time is needed for complex queries or long problems send me an e-mail for an appointment. Try to specify the reason in the e-mail, this helps me preparing the answer and helps you formalizing it. I do not set up a formal receiving hour: you can come at any time, just take an appointment.

Please, avoid dropping-by the office. Normally I cannot receive you on the fly and the only result is a waste of time. If you have small quick doubts I can also try to solve them via e-mail, so that you don't need to come to my office for a 30 second answer!

Academic Years