ECE 515
Information Theory

Assignment Submission

Assignments

1. Consider two binary random variables X and Y with joint probability distribution p(x,y) given by
   p(0,0) = 1/2, p(0,1) = 1/4, p(1,0) = 0, and p(1,1) = 1/4.
   Find the value of
   (a) H(X)
   (b) H(Y)
   (c) H(X|Y)
   (d) H(Y|X)
   (e) H(XY)
   (f) I(X;Y)
2. Consider a discrete random variable X with 2ⁿ+1 symbols x_i, i = 1, 2, …, 2ⁿ+1. Determine the upper and lower bounds on the entropy and the corresponding symbol probabilities when
   (a) p(x₁)=0
   (b) p(x₁)=1/2
3. A jar contains 5 black balls and 10 white balls. Experiment X involves randomly drawing a ball out of the jar. Next, experiment Y involves randomly drawing a ball with the ball drawn in experiment X not replaced in the jar. One is interested in the colour of the drawn ball.
   (a) How much uncertainty does experiment X contain?
   (b) What is the uncertainty in experiment Y given that the first ball is black?
   (c) What is the uncertainty in experiment Y given that the first ball is white?
   (d) How much uncertainty does experiment Y contain?
4. Let X be a random variable whose entropy H(X) is 8 bits. Suppose that Y(X) is a deterministic function that takes on a different value for each value of X.
   (a) What is H(Y)?
   (b) What is H(Y|X)?
   (c) What is H(X|Y)?
   (d) What is I(X;Y)?
   (e) Suppose now that the deterministic function Y(X) is not invertible, so that different values of X may correspond to the same value of Y(X). In this case, what can be said about H(Y), and also about H(X|Y)?
5. The Stanley Cup Final is a seven-game hockey series that ends as soon as either team wins 4 games. Let X be the random variable that represents the outcome of the Stanley Cup Final between teams A and B. For example, some possible values of X are AAAA, BABABAB, and BBBAAAA. Let Y be random variable for the number of games played, which ranges from 4 to 7. Assuming A and B are equally matched and that the games are independent, determine
   (a) H(X)
   (b) H(Y)
   (c) H(Y|X)
   (d) H(X|Y)
6. In a country, 25% of the people are blond and 75% of all blond people have blue eyes. In addition, 50% of the people have blue eyes. How much information is received in each of the following cases
   (a) if we know that a person is blond and we are told the colour (blue/not blue) of their eyes
   (b) if we know that a person has blue eyes and we are told the colour (blond/not blond) of their hair
   (c) if we are told both the colour of their hair and that of their eyes.
7. A roulette wheel is subdivided into 38 numbered compartments of various colours. The distribution of the compartments according to colour is 2 green, 18 red, and 18 black.
   
   The experiment consists of throwing a small ball onto the rotating roulette wheel. The event that the ball comes to rest in one of the 38 compartments is equally probable for each compartment.
   (a) How much information is received if the colour is revealed?
   (b) How much information is received if both the colour and number are revealed?
   (c) If the color is known, what is the remaining uncertainty about the number?

Aaron Gulliver
2025-09-16