ECE 515
Information Theory


Assignments

Assignment 1 - Due September 28, 2018
  1. Consider two binary random variables X and Y with joint probability distribution p(x,y) given by
    p(0,0) = 1/2, p(0,1) = 1/4, p(1,0) = 0, and p(1,1) = 1/4.
    Find the values of H(X), H(Y), H(X|Y), H(Y|X), H(XY), and I(X;Y).
  2. A discrete memoryless source has symbols x1, ..., x6 with p(x1) = 1/2 and p(x2) = 1/4. Determine the upper and lower bounds on the entropy of this source.
  3. A jar contains 5 black balls and 10 white balls. Experiment X involves randomly drawing a ball out of the jar. Experiment Y involves randomly drawing a ball with the ball drawn in experiment X not replaced in the jar. One is interested in the colour of the drawn ball.
    (a) How much uncertainty does experiment X contain?
    (b) What is the uncertainty in experiment Y given that the first ball is black?
    (c) What is the uncertainty in experiment Y given that the first ball is white?
    (d) How much uncertainty does experiment Y contain?
  4. Let X be a random variable whose entropy H(X) is 8 bits. Suppose that Y(X) is a deterministic function that takes on a different value for each value of X.
    (a) What is H(Y)?
    (b) What is H(Y|X)?
    (c) What is H(X|Y)?
    (d) What is I(X;Y)?
    (e) Suppose now that the deterministic function Y(X) is not invertible, so that different values of X may correspond to the same value of Y(X). In this case, what can be said about H(Y), and also about H(X|Y)?
  5. A Division Series is a five game baseball series held in October that ends as soon as a team wins 3 games. Define a random variable X that represents the outcome of the series between two teams A and B. Three possible values of X are BBB, ABAA, and BBAAA (many more outcomes can occur). Let Y be the number of games played which range from 3 to 5. Assuming teams A and B are equally matched and that the games are independent, calculate H(X), H(Y), H(Y|X) and H(X|Y).
  6. In a country, 25% of the people are blond and 75% of all blond people have blue eyes. In addition, 50% of the people have blue eyes. How much information is received in each of the following cases:
    (a) if we know that a person is blond and we are told the colour (blue/not blue) of their eyes,
    (b) if we know that a person has blue eyes and we are told the colour (blond/not blond) of their hair,
    (c) if we are told both the colour of their hair and that of their eyes.

Assignment 2 - Due 2018
Assignment 3 - Due 2018
Assignment 4 - Due 2018
Assignment 5 - Due 2018