Information Theory (ELEN E6717)

General Information

Lecturer	Prof. Vittorio Castelli Office Hours: Tuesday 4:50 - 6:40, Mudd 1312 (knock on the back door to the EE department office) , phone number 854-3131 Please feel free to send me questions at my e-mail address: [email protected]
Day & Time Class	Tue, 6:50-9:30
Location:	Room 222 Pupin Bldg
Teaching Assistant	Anmo Kim. e-mail: [email protected] Office Hours: Thursday, 2-4 PM, 1247 Mudd.
Credits for Course:	3
Class Type:	Lecture
Prerequisites:	Prerequisite: Stat-IEOR W3658 (Probability) or equivalent introduction to Statistics OR a course in stochastic processes. Suggested: Calculus (integration, differentiation, summation of series). Corequisite: EE E6711
Description:	Relationship of Information Theory and other disciplines. Fundamental quantities: entropy, mutual information, relative entropy, and basic theorems. Lossless Data Compression: the source coding theorem, Kraft Inequality, optimal codes, Huffman, Shannon and Arithmetic codes. Transmitting information: Channel Capacity for discrete memoryless channels, Gaussian Channels. Lossy compression: rate distortion theory. Information theory and Statistics: method of types, Sanov's theorem, Chernoff bound. Dictionary-based source coding.
Required Text:	Cover and Thomas, Elements of Information Theory John Wiley & Sons.
Reference Text(s):	Information Theory, Inference, and Learning Algorithms D. MacKay. Cambridge University Press, 2003. Coding and information theory R. W. Hamming. Prentice-Hall, c1980. Rate distortion theory a mathematical basis for data compression. Toby Berger, Prentice-Hall 1971. Principles and practice of information theory Richard E. Blahut. Addison-Wesley c1987. Coding and information theory Steven Roman. Springer-Verlag 1992 Basic concepts in information theory and coding : the adventures of secret agent 00111 Solomon W. Golomb, Robert E. Peile, Robert A. Scholtz. Plenum Publishing Co., 1994. Claude Elwood Shannon : collected papers edited by N.J.A. Sloane, Aaron D. Wyner. IEEE Press c1993.
Homework(s):	Weekly Assignments
Project(s):	None
Paper(s):	None
Midterm Exam(s):	120 minute midterm, closed book, "open notes", where "open notes" will be defined in class.
Final Exam:	120 minutes in class exam, closed book, "open notes", where "open notes" will be defined in class.
Grading	Midterm: 35%, Final 45%, Homework 25% (total = 105%, 5% bonus points !)
Homework Submissions:	TBA