3 Concepts: Information

Course Material

Lectures

» Lecture 1: What is Information Theory?
» Lecture 2: Mathematical Preliminaries
» Lecture 3: Source Coding: Theory
    • Cover & Thomas, Chapter 2.
» Lecture 4: Source Coding: Practice
    • Cover & Thomas, Chapter 5.
    • Witten, Neal, and Cleary, "Arithmetic coding for data compression". See link below.
» Lecture 5: MDL Principle
» Lecture 6: MDL Principle (contd.)
    • Cover & Thomas, Chapter 7.

Highly Recommended:

• Thomas Cover & Joy Thomas,
  Elements of Information Theory.
  John Wiley & Sons, New York, 1991. (3rd edition, 2006).
  
  You do not have to get this book. Necessary material will be distributed in the class (and made available in the course folder).

Additional Reading:

• Bell T. C., Cleary J. G., Witten I. H.,
  Text Compression.
  Prentice Hall, Englewood Cliffs, New Jersey, 1990.
  
  
• Blahut R. E.,
  Principles and Practice of Information Theory.
  Addison-Wesley, New York, 1987.
  
  
• Csiszar, Imre; Shields, Paul C.,
  Information Theory And Statistics: A Tutorial
  Lightning Source Inc., 2004. Available on-line in LaTeX (!).
  
  
• Deco G., Obradovic D.,
  An Information-Theoretic Approach to Neural Computing.
  Springer-Verlag, New York, 1996.
  
  
• Frey B. J.,
  Graphical Models for Machine Learning and Digital Communication.
  MIT Press, Cambridge, Massachusetts.
  
  
• Gray R. M.,
  Entropy and Information Theory.
  Springer, 1991. Available on-line.
  
  
• Peter Grünwald
  The Minimum Description Length Principle
  MIT Press, 2007. Sample chapters.
  
  
• Debra A. Lelewer and Daniel S. Hirschberg,
  Data compression.
  ACM Computing Surveys, 19:3, 1987, pp. 261–296.
  
  
• David MacKay,
  Information Theory, Inference and Learning Algorithms.
  Cambridge University Press, 2003.
  
  
• McEliece R. J.,
  The Theory of Information and Coding: A Mathematical Framework for Communication.
  Addison-Wesley, 1977. Reprinted 1984 by C.U.P. in Encyclopedia of Mathematics.
  
  
• James P. Sethna,
  Statistical Mechanics: Entropy, Order Parameters, and Complexity.
  Oxford University Press, 2006.
  
  
• Claude E. Shannon,
  A Mathematical Theory of Communication.
  Bell System Technical Journal, vol. 27, pp. 379-423 and 623-656, July and October, 1948.
  
  
• David Solomon,
  Data Compression: The Complete Reference.
  3rd edition, Springer, 2004.
  
  
• Jorma Rissanen,
  Information and Complexity in Statistical Modeling.
  Springer, 2007. (The book is based on lecture notes, available on web: PDF.)
  
  
• Williams R. N.,
  Adaptive Data Compression.
  Kluwer Academic Publishers, USA, 1991.
  
  
• Witten I. H., Neal R. M., Cleary J. G.,
  Arithmetic coding for data compression.
  Communications of the ACM, 30:6, 1987, pp. 520–540. (PDF).
  
  

Links

» mdl-research.org
» DataCompression.info
» Compression Pointers
» MIT Course on Information and Entropy
» Stanford Information Theory Class
» Entropy on the World Wide Web
» G.J. Chaitin Home Page
» David Dowe's Minimum Encoding Length Inference Page
» Bayesians Worldwide
» Quantum Information
» Information about gzip

3 Concepts: Information	2007