Checking date: 11/05/2022

Course: 2022/2023

Information Theory
Study: Master in Information Health Engineering (359)

Coordinating teacher: KOCH , TOBIAS MIRCO

Department assigned to the subject: Signal and Communications Theory Department

Type: Electives
ECTS Credits: 6.0 ECTS


Requirements (Subjects that are assumed to be known)
Students should have a solid basis in probability and calculus, as well as pleasure with mathematics.
Basic competences CB6 Having and understanding the knowledge that provides a basis or opportunity to be original in the development and/or application of ideas, often in a research context CB7 Students know how to apply their acquired knowledge and problem-solving skills in new or unfamiliar settings within broader (or multidisciplinary) contexts related to their field of study. CB8 Students are able to integrate knowledge and to face the complexity of making judgments based on information that, being incomplete or limited, includes reflections on the social and ethical responsibilities linked to the application of their knowledge and judgments. CB9 Students know how to communicate their conclusions and the knowledge and ultimate reasons behind them to specialised and non-specialised audiences in a clear and unambiguous way. CB10 Students have the learning skills that will enable them to continue studying in a way that will be largely self-directed or autonomous. General competences CG2 Ability to apply the knowledge of skills and research methods related to engineering. CG3 Ability to apply the knowledge of research skills and methods related to Life Sciences. CG4 Ability to contribute to the widening of the frontiers of knowledge through an original research, part of which merits publication referenced at an international level. CG5 Ability to perform a critical analysis and an evaluation and synthesis of new and complex ideas. CG6 Ability to communicate with the academic and scientific community and with society in general about their fields of knowledge in the modes and languages commonly used in their international scientific community. Specific competences CE8 Ability to easily handle with the mathematical concepts and foundations necessary for the analysis, design and implementation of automatic learning algorithms for their operation under given specifications. CE9 Ability in the handling of advanced automatic learning techniques for their application in the field of biomedicine.
Skills and learning outcomes
Description of contents: programme
This course teaches the fundamentals of Information Theory, which concerns data compression and transmission in digital communication systems. The topics covered in this course are as follows: 1) Fundamental quantities and concepts in Information Theory: entropy, Kullback-Leibler divergence, mutual information, and Jensen's inequality. 2) Lossless data compression: uniquely decodable and instantaneous source codes, Kraft's inequality, analysis of the optimal codeword length, Huffman codes, universal compression, and arithmetic coding. 3) Lossy data compression: the rate-distortion theorem and properties of the rate-distortion function.
Learning activities and methodology
AF3 Theoretical and practical lessons - 33.5 hours AF4 Lab sessions - 10.5 hours AF5 Office hours - 6 hours AF6 Group work - 30 hours AF7 Individual student work - 62 hours AF8 Continuous and final assessments - 4 hours Lectures (AF3): The basic concepts will be mainly taught at the blackboard. We will follow closely the book "Elements of Information Theory" by Cover & Thomas (see Basic Bibliography). Exercises (AF6/AF7): In order to deepen the understanding of the taught material, every two weeks students have to hand in the solutions to a set of problems. These solutions will be graded from 1 to 10, the average grade over the whole semester will constitute the grade of the continuous assessment. Laboratory Classes (AF4): There will be laboratory classes where students have the opportunity to deepen the concepts learned in class by means of computer exercises. Laboratory classes will also be used to discuss the homework exercises.
Assessment System
  • % end-of-term-examination 60
  • % of continuous assessment (assigments, laboratory, practicals...) 40
Calendar of Continuous assessment
Basic Bibliography
  • Thomas M. Cover and Joy A. Thomas. Elements of Information Theory. Second Edition. 2006
Additional Bibliography
  • Abbas El Gamal and Young-Han Kim. Network Information Theory. First Edition. 2011
  • Imre Csiszár and János Körner. Information Theory: Coding Theorems for Discrete Memoryless Systems. Second Edition. 2011
  • Robert G. Gallager. Information Theory and Reliable Communication. First Edition. 1968

The course syllabus may change due academic events or other reasons.