Checking date: 24/04/2023

Course: 2023/2024

Information Theory
(18066)
Master in Information Health Engineering (Plan: 427 - Estudio: 359)
EPI

Coordinating teacher: KOCH , TOBIAS MIRCO

Department assigned to the subject: Signal and Communications Theory Department

Type: Electives
ECTS Credits: 6.0 ECTS

Course:
Semester:

Requirements (Subjects that are assumed to be known)
Students should have a solid basis in probability and calculus, as well as pleasure with mathematics.
Objectives
This course teaches the fundamentals of Information Theory. Students will acquire a profound understanding of: - Information-theoretic quantities, such as entropy, Kullback-Leibler divergence, and mutual information. - Mathematical tools commonly used in Information Theory, such as Jensen's inequality. - The concepts and fundamental theorems of data compression. - The application of Information Theory in Machine Learning.
Skills and learning outcomes
Description of contents: programme
This course teaches the fundamentals of Information Theory. The topics covered in this course are as follows: 1) Fundamental quantities and concepts in Information Theory: entropy, Kullback-Leibler divergence, mutual information and Jensen's inequality. 2) Lossless data compression: uniquely decodable and instantaneous source codes, Kraft's inequality, analysis of the optimal codeword length, Huffman codes, and universal compression. 3) Information theory and machine learning: Generalization error, empirical risk minimization, classical statistical learning generalization guarantees, information theoretic generalization bounds.
Learning activities and methodology
Lectures: The basic concepts will be mainly taught at the blackboard. We will follow closely the book "Elements of Information Theory" by Cover & Thomas (see Basic Bibliography). Exercises: In order to deepen the understanding of the taught material, every two weeks students have to hand in the solutions to a set of problems. These solutions will be graded from 1 to 10, the average grade over the whole semester will constitute part of the grade of the continuous assessment.
Assessment System
• % end-of-term-examination 0
• % of continuous assessment (assigments, laboratory, practicals...) 100
Calendar of Continuous assessment
Basic Bibliography
• Thomas M. Cover and Joy A. Thomas. Elements of Information Theory. Second Edition. 2006