COMP2610/COMP6261 — Information Theory Lecture 2: First Steps and Basic Probability Mark Reid and Aditya Menon ResearchSchoolofComputerScience TheAustralianNationalUniversity July 23rd, 2014 MarkReidandAdityaMenon (ANU) COMP2610/COMP6261—InformationTheory Semester2 1/37 Last time Administrivia Course overview and outline Informal introduction to information and uncertainty MarkReidandAdityaMenon (ANU) COMP2610/COMP6261—InformationTheory Semester2 2/37 This time First steps towards principles of communication (cid:73) A general system for communication (cid:73) The noisy channel model (cid:73) What is a notion of “information”? Introduction to concepts in probability (cid:73) Joint, marginal, and conditional distributions (cid:73) Worked examples of probability manipulation MarkReidandAdityaMenon (ANU) COMP2610/COMP6261—InformationTheory Semester2 3/37 Outline 1 A General Communication System 2 The Role of Uncertainty 3 Basic Concepts In Probability 4 Relating Joint, Marginal and Conditional Distributions 5 Wrapping Up MarkReidandAdityaMenon (ANU) COMP2610/COMP6261—InformationTheory Semester2 4/37 1 A General Communication System 2 The Role of Uncertainty 3 Basic Concepts In Probability 4 Relating Joint, Marginal and Conditional Distributions 5 Wrapping Up MarkReidandAdityaMenon (ANU) COMP2610/COMP6261—InformationTheory Semester2 5/37 A General Communication System Source : The information source that generates the message to be communicated Encoder : Operates on the message to produce a signal suitable for transmission Channel : The medium used to transmit the signal Decoder : Reconstructs the message from the signal Destination : The entity that receives the message MarkReidandAdityaMenon (ANU) COMP2610/COMP6261—InformationTheory Semester2 6/37 Communication over Noisy Channels A channel is some medium for transmitting messages A noisy channel is a channel that potentially introduces errors in the sender’s message The Problem of Communication “The fundamental problem of communication is that of reproducing at one point either exactly or approximately a message selected at another point.” (Claude Shannon, 1948) MarkReidandAdityaMenon (ANU) COMP2610/COMP6261—InformationTheory Semester2 7/37 Example: Telephone Network “Hi”% “Hi”% Source : Aditya Encoder : Telephone handset Channel : Analogue telephone line Decoder : Telephone handset Destination : Mark MarkReidandAdityaMenon (ANU) COMP2610/COMP6261—InformationTheory Semester2 8/37 Examples of Noisy Communication Channels Other examples of noisy channels: A radio communication link Reproducing cells A magnetic hard disk drive (cid:73) Information does not need to involve physical movement What would the other components be for each of these channels? MarkReidandAdityaMenon (ANU) COMP2610/COMP6261—InformationTheory Semester2 9/37 1 A General Communication System 2 The Role of Uncertainty 3 Basic Concepts In Probability 4 Relating Joint, Marginal and Conditional Distributions 5 Wrapping Up MarkReidandAdityaMenon (ANU) COMP2610/COMP6261—InformationTheory Semester2 10/37