Checking date: 24/04/2023

Course: 2023/2024

Information Theory
Master in Advanced Communications Technologies (Plan: 436 - Estudio: 278)

Coordinating teacher: KOCH , TOBIAS MIRCO

Department assigned to the subject: Signal and Communications Theory Department

Type: Electives
ECTS Credits: 6.0 ECTS


Requirements (Subjects that are assumed to be known)
Students should have a solid basis in probability and calculus, as well as pleasure with mathematics. Having taken a course on Digital Communications / Communication Theory is also helpful.
This course teaches the fundamentals of Information Theory. Students will acquire a profound understanding of: - Information-theoretic quantities, such as entropy, Kullback-Leibler divergence, and mutual information. - Mathematical tools commonly used in Information Theory, such as Jensen's inequality. - The concepts and fundamental theorems of data compression. - The application of Information Theory in Machine Learning.
Skills and learning outcomes
Description of contents: programme
This course teaches the fundamentals of Information Theory. The topics covered in this course are as follows: 1) Fundamental quantities and concepts in Information Theory: entropy, Kullback-Leibler divergence, mutual information and Jensen's inequality. 2) Lossless data compression: uniquely decodable and instantaneous source codes, Kraft's inequality, analysis of the optimal codeword length, Huffman codes, and universal compression. 3) Information theory and machine learning: Generalization error, empirical risk minimization, classical statistical learning generalization guarantees, information theoretic generalization bounds.
Learning activities and methodology
Lectures: The basic concepts will be mainly taught at the blackboard. We will follow closely the book "Elements of Information Theory" by Cover & Thomas (see Basic Bibliography). Exercises: In order to deepen the understanding of the taught material, every two weeks students have to hand in the solutions to a set of problems. These solutions will be graded from 1 to 10, the average grade over the whole semester will constitute part of the grade of the continuous assessment.
Assessment System
  • % end-of-term-examination 0
  • % of continuous assessment (assigments, laboratory, practicals...) 100
Calendar of Continuous assessment
Basic Bibliography
  • Thomas M. Cover and Joy A. Thomas. Elements of Information Theory. Second Edition. 2006
Additional Bibliography
  • Abbas El Gamal and Young-Han Kim. Network Information Theory. First Edition. 2011
  • Imre Csiszár and János Körner. Information Theory: Coding Theorems for Discrete Memoryless Systems. Second Edition. 2011
  • Robert G. Gallager. Information Theory and Reliable Communication. First Edition. 1968

The course syllabus may change due academic events or other reasons.