Course: 2023/2024

Information Theory

(18066)

Requirements (Subjects that are assumed to be known)

Students should have a solid basis in probability and calculus, as well as pleasure with mathematics.

This course teaches the fundamentals of Information Theory. Students will acquire a profound understanding of:
- Information-theoretic quantities, such as entropy, Kullback-Leibler divergence, and mutual information.
- Mathematical tools commonly used in Information Theory, such as Jensen's inequality.
- The concepts and fundamental theorems of data compression.
- The application of Information Theory in Machine Learning.

Skills and learning outcomes

Description of contents: programme

This course teaches the fundamentals of Information Theory. The topics covered in this course are as follows:
1) Fundamental quantities and concepts in Information Theory: entropy, Kullback-Leibler divergence, mutual information and Jensen's inequality.
2) Lossless data compression: uniquely decodable and instantaneous source codes, Kraft's inequality, analysis of the optimal codeword length, Huffman codes, and universal compression.
3) Information theory and machine learning: Generalization error, empirical risk minimization, classical statistical learning generalization guarantees, information theoretic generalization bounds.

Learning activities and methodology

Lectures:
The basic concepts will be mainly taught at the blackboard. We will follow closely the book "Elements of Information Theory" by Cover & Thomas (see Basic Bibliography).
Exercises:
In order to deepen the understanding of the taught material, every two weeks students have to hand in the solutions to a set of problems. These solutions will be graded from 1 to 10, the average grade over the whole semester will constitute part of the grade of the continuous assessment.

Assessment System

- % end-of-term-examination 0
- % of continuous assessment (assigments, laboratory, practicals...) 100

Basic Bibliography

- Thomas M. Cover and Joy A. Thomas. Elements of Information Theory. Second Edition. 2006

Additional Bibliography

- Abbas El Gamal and Young-Han Kim. Network Information Theory. First Edition. 2011
- Imre Csiszár and János Körner. Information Theory: Coding Theorems for Discrete Memoryless Systems. Second Edition. 2011
- Robert G. Gallager. Information Theory and Reliable Communication. First Edition. 1968

The course syllabus may change due academic events or other reasons.