syllabus
1. Information regarding the programme
1.1 Higher education institution 	Babeş Bolyai University	
1.2 Faculty	Faculty of Mathematics and Computer Science	
1.3 Department	Mathematics	
1.4 Field of study	Information Engineering	
1.5 Study cycle	Bachelor	
1.6 Study programme / Qualification 		

2. Information regarding the discipline 
2.1 Name of the discipline	Information Theory	
2.2 Course coordinator 	Assoc. Professor Ph.D. Radu Trîmbiţaş	
2.3 Seminar coordinator	TA . Ph.D. Radu Zapotinschi	
2.4. Year of study	3	2.5 Semester	5	2.6. Type of evaluation	E	2.7 Type of discipline	Compulsory	

3. Total estimated time (hours/semester of didactic activities) 
3.1 Hours per week 	4	Of which: 3.2 course	2	3.3 seminar/laboratory	2	
3.4 Total hours in the curriculum 	56	Of which: 3.5 course	28	3.6 seminar/laboratory	28	
Time allotment:	hours	
Learning using manual, course support, bibliography, course notes	20	
Additional documentation (in libraries, on electronic platforms, field documentation) 	10	
Preparation for seminars/labs, homework, papers, portfolios and essays	20	
Tutorship	14	
Evaluations	4	
Other activities: test+exam preparatives	1	
3.7 Total individual study hours 	50	
3.8 Total hours per semester	125	
3.9 Number of ECTS credits	5	

4. Prerequisites (if necessary)
4.1. curriculum	•	Calculus, probabilities	
4.2. competencies	•	Programming in MATLAB	

5. Conditions (if necessary)
5.1. for the course	•	videoprojector (if available)	
5.2.  for the seminar /lab activities	•	computer network	

6. Specific competencies acquired 
Professional competencies	•	Knowledge, understanding and use of basic concepts of theoretical Computer Science
•	Knowledge, understanding and use of the fundamental methods, processes and tools of Software Engineering
•	Understanding of basic concepts of Mathematics and use them to problem-solving activities.
•	Ability to understand and approach problems of modeling nature from other sciences	
Transversal competencies	•	Good programming skills in high-level languages 
•	Abilities to develop and maintain software systems
•	Deep understanding of a real-world organization (structure and processes carrying information)
•	Ability to analyze, synthesize and model fenomena and processes from various fields (economy, science, research, education) using adequate mathematical, statistical, computational and software engineering methods.	

7. Objectives of the discipline (outcome of the acquired competencies)
7.1 General objective of the discipline
	•	Introduction to basic concepts of Information Theory
•	Practical Applications	
7.2 Specific objective of the discipline
	•	fundamental concepts of information theory, such as entropy, mutual information,properties and examples of information sources
•	 results about data compression and optimal codes, channel capacity, transmission rates, Kolmogorov complexity 	

8. Content
8.1 Course	Teaching methods	Remarks	
1.	Introduction and brief history of information theory. Review of basic notions of probability theory (discrete and continuous random variables, probability mass function, cumulative distribution function, density function, expectation and variance of a random variable, conditional probabilities)	Lecture, proofs, explanation, discussion, examples, synthesis, team work, individual study, homework, modeling at the computer		
2.	Review of the properties of the logarithmic function, entropy, joint entropy, conditional entropy, chain rule for entropy	Lecture, proofs, explanation, discussion, examples, synthesis, team work, individual study, homework, modeling at the computer		
3.	Mutual information, informational divergence, conditional information  (definitions, examples and chain rules)	Lecture, proofs, explanation, discussion, examples, synthesis, team work, individual study, homework, modeling at the computer		
4.	Basic inequalities of information theory, convex functions, Jensen’s inequality, Gibbs inequality and their consequences
	Lecture, proofs, explanation, discussion, examples, synthesis, team work, individual study, homework, modeling at the computer		
5.	Data processing theorem and its consequences, Fano’s inequality, information sources
6.	Entropy rate, Markov chains (definitions and properties)	Lecture, proofs, explanation, discussion, examples, synthesis, team work, individual study, homework, modeling at the computer		
7.	Source code, data compression, Kraft inequality, optimal codes 	Lecture, proofs, explanation, discussion, examples, synthesis, team work, individual study, homework, modeling at the computer		
8.	Optimal codes, Huffman codes and their properties	Lecture, proofs, explanation, discussion, examples, synthesis, team work, individual study, homework, modeling at the computer		
9.	Shannon-Fano-Elias coding, properties	Lecture, proofs, explanation, discussion, examples, synthesis, team work, individual study, homework, modeling at the computer		
10.	Shannon codes, competitive optimality of Shannon codes	Lecture, proofs, explanation, discussion, examples, synthesis, team work, individual study, homework, modeling at the computer		
11.	Channel capacity, the Channel Coding Theorem, zero-error codes	Lecture, proofs, explanation, discussion, examples, synthesis, team work, individual study, homework, modeling at the computer		
12.	Gaussian channels, application of information theory in statistics	Lecture, proofs, explanation, discussion, examples, synthesis, team work, individual study, homework, modeling at the computer		
13.	Maximum entropy distributions, entropy rates for Gaussian processes, Burg’s Maximum Entropy Theorem	Lecture, proofs, explanation, discussion, examples, synthesis, team work, individual study, homework, modeling at the computer		
14.	Universal codes and channel capacity, universal source coding for binary sequences examples	Lecture, proofs, explanation, discussion, examples, synthesis, team work, individual study, homework, modeling at the computer		
			
Bibliography
1. Blaga, P., Calculul probabilitatilor si statistica matematica. Vol. II. Curs si culegere de probleme, UBB, Cluj-Napoca, 1994
2. Cover T.M, Thomas J.A, 	Elements of Information Theory, 2nd edition, Wiley-Interscience, 2006
3. MacKay D., Information Theory, Inference and Learning Algorithms, Cambridge University Press, 2003
4. Murgan A. T., Principiile teoriei Informatiei în ingineria informatiei si a comunicatiilor, Editura Academiei Române, Bucuresti, 1998
5. W. Yeung, R.W., A First Course in Information Theory, Kluwer Academic Publishers, 2005 

6. Applebaum, D., Probability and Information, Cambridge University Press, 2008

7. Gallager, R.G., Information Theory and Reliable Communication, Wiley, 1968

8. Gray R.M., Entropy and Information Theory, Springer, 1999

	
8.2 Seminar / laboratory	Teaching methods	Remarks	
1.	Problems from classic probability theory, applications of conditional probabilities	Discussion, examples,  individual study, homework, modeling at the computer		
2.	Examples and problems concerning the computation of entropy, joint entropy, conditional entropy	Discussion, examples,  individual study, homework, modeling at the computer		
3.	Examples and problems concerning the computation of mutual information, informational divergence, conditional information	Discussion, examples,  individual study, homework, modeling at the computer		
4.	Examples and problems concerning the applications of the basic inequalities from information theory	Discussion, examples,  individual study, homework, modeling at the computer		
5.	Generating uniformly distributed random numbers, the inversion method and applications	Discussion, examples,  individual study, homework, modeling at the computer		
6.	Examples of stationary Markov chains, directed graphs, stationary distributions	Discussion, examples,  individual study, homework, modeling at the computer		
7.	Examples of optimal codes	Discussion, examples,  individual study, homework, modeling at the computer		
8.	Examples of Huffman codes	Discussion, examples,  individual study, homework, modeling at the computer		
9.	Examples of  Shannon-Fano-Elias  codes	Discussion, examples,  individual study, homework, modeling at the computer		
10.	Examples of  Shannon codes	Discussion, examples,  individual study, homework, modeling at the computer		
11.	Examples for calculating the capacity of channels	Discussion, examples,  individual study, homework, modeling at the computer		
12.	Examples and applications in statistics	Discussion, examples,  individual study, homework, modeling at the computer		
13.	Examples of maximum entropy distributions, entropy rates for Gaussian processes	Discussion, examples,  individual study, homework, modeling at the computer		
14.	Examples of universal coding	Discussion, examples,  individual study, homework, modeling at the computer		
Bibliography
1. Cover T.M, Thomas J.A, 	Elements of Information Theory, 2nd edition, Wiley-Interscience, 2006
2. MacKay D., Information Theory, Inference and Learning Algorithms, Cambridge University Press, 2003

	

9. Corroborating the content of the discipline with the expectations of the epistemic community, professional associations and representative employers within the field of the program

•	The course exists in the studying program of all major universities in Romania and abroad;
•	The content of the course is considered as important for average programming skills by the software companies interested in information coding, transmission and processing

	

10. Evaluation
Type of activity	10.1 Evaluation criteria	10.2 Evaluation methods	10.3 Share in the grade (%)	
10.4 Course	Knowing basic principles of Information Theory		60%	
	Applications to information transmission and coding			
10.5 Seminar/lab activities	Ability to implement concepts and algorithms		40%	
	Ability to implement application involving  Information Theory concepts			
10.6 Minimum performance standards	
•	at least 5 for exam and at least 5 for seminar activity	

Date			Signature of course coordinator 	    Signature of seminar coordinator
  26 September 2012	               Assoc. Prof. PH.D. Radu Trîmbiţaş  Radu Zapotinschi, Ph.D 

Date of approval  					  Signature of the head of department	
...........................................					...........…............................