Universitatea Babeş-Bolyai Cluj-Napoca
Facultatea de Matematică şi Informatică
Ciclul de studii: Licenţă
Domeniul: Calculatoare şi tehnologia informaţiei
Programul de studii: Ingineria informatiei - in limba engleza
Limba de predare: Engleză

SYLLABUS

I. General data
Code	Subject	
MMP0006	Information Theory	

Semester	Hours: C+S+L+P	Category	Status	
5	2+2+0+0	speciality	compulsory	

II. Full status faculty members
Name and	Scientific	Didactic	Chair	Type of activity	
surname	title	title		C	S	L	
LISEI Hannelore-Inge	Ph.D.	Assoc.Prof.	Applied Mathematics	*	*		

Associated faculty members
Name and	Scientific	Institution	Type of	Type of activity	
surname	title		position	C	S	L	

III. Course objectives
We will give the fundamental concepts of information theory, such as entropy, mutual information,
properties and examples of information sources, results about data compression and optimal codes,
channel capacity, transmission rates, Kolmogorov complexity etc.	

IV. Course contents
► Review of basic notions of probability theory 
► Entropy, informational divergence, joint entropy, conditional entropy, mutual information, , 
     conditional information  
► Fano’s inequality and data processing lemma
► Information sources, entropy rate, Markov chains 
► Data compression, optimal codes, Huffman codes
► Channel capacity, the Channel Coding Theorem
► Maximum entropy distributions, universal source coding
► Kolmogorov complexity, minimum description length principle	

V. Bibliography
1. Blaga, P., Calculul probabilităţilor şi statistică matematică. Vol. II. Curs şi culegere de probleme, UBB, Cluj-Napoca, 1994
2. Cover T.M, Thomas J.A, 	Elements of Information Theory, 2nd edition, Wiley-Interscience, 2006
3. MacKay D., Information Theory, Inference and Learning Algorithms, Cambridge University Press, 2003
4. Murgan A. T., Principiile teoriei Informaţiei în ingineria informaţiei şi a comunicaţiilor, Editura Academiei Române, Bucureşti, 1998
5. W. Yeung, R.W., A First Course in Information Theory, Kluwer Academic Publishers, 2005	

VI. Thematic of didactic activities per weeks
Course 1:
Introduction and brief history of information theory. Review of basic notions of probability theory
(discrete and continuous random variables, probability mass function, cumulative distribution function,
density function, expectation and variance of a random variable, conditional probabilities)
Seminar 1:
Problems from classic probability theory, applications of conditional probabilities
Course 2:
Review of the properties of the logarithmic function, entropy, joint entropy, conditional entropy, chain rule for entropy
Seminar 2:
Examples and problems concerning the computation of entropy, joint entropy, conditional entropy
Course 3:
Mutual information, informational divergence, conditional information  (definitions, examples and chain rules)
Seminar 3:
Examples and problems concerning the computation of mutual information, informational divergence, conditional information
Course 4:
Basic inequalities of information theory, convex functions, Jensen’s inequality, Gibbs inequality and their consequences
Seminar 4:
Examples and problems concerning the applications of the basic inequalities from information theory
Course 5:
Data processing theorem and its consequences, Fano’s inequality, information sources
Seminar 5:
Generating uniformly distributed random numbers, the inversion method and applications
Course 6:
Entropy rate, Markov chains (definitions and properties)
Seminar 6:
Examples of stationary Markov chains, directed graphs, stationary distributions
Course 7:
Source code, data compression, Kraft inequality, optimal codes 
Seminar 7:
Examples of optimal codes
Course 8:
Optimal codes, Huffman codes and their properties, Shannon coding
Seminar 8:
Examples of Huffman codes and Shannon codes
Course 9:
Channel capacity, the Channel Coding Theorem, zero-error codes
Seminar 9:
Examples for calculating the capacity of channels
Course 10:
Fano’s inequality and the converse to the Coding Theorem
Seminar 10:
Examples for calculating the capacity of channels
Course 11:
Gaussian channels, application of information theory in statistics
Seminar 11:
Examples and applications in statistics
Course 12:
Maximum entropy distributions, entropy rates for Gaussian processes, Burg’s Maximum Entropy
Theorem
Seminar 12:
Examples of maximum entropy distributions, entropy rates for Gaussian processes
Course 13 :
Universal codes and channel capacity, universal source coding for binary sequences examples
Seminar 13:
Examples of universal coding
Course 14: 
Kolmogorov complexity, minimum description length principle
Seminar 14: 
Applications of Kolmogorov complexity	

VII. Didactic methods used
Lecture, proofs, explanation, discussion, examples, synthesis, team work, individual study, homework,
modeling at the computer 	

VIII. Assessment
The final note will be given by:
evaluation at the seminar: 30%
evaluation at the exam:70%	

IX. Additional references
1.	Applebaum, D., Probability and Information, Cambridge University Press, 2008
2.	Gallager, R.G., Information Theory and Reliable Communication, Wiley, 1968
3.	Gray R.M., Entropy and Information Theory, Springer, 1999	

	Dean,							Chair head,
         Prof.  ŢÂMBULEA Leon, PhD                               Prof.   AGRATINI Octavian, PhD

	Date,							Course responsible,
        27.09.2010        					    Assoc.Prof. LISEI Hannelore-Inge, Ph.D.