Select Page

Show transcribed image text Suppose that we have a probability distribution on the set {1, …, n}, i.e. a sequence p = (p_1, …, p_n) of probabilities in the set P_n vector where P bar_n:= {p element of [0, 1]^n: Sigma^n_i=1 p_i = 1}, P_n ={p element of (0, 1)^n: Sigma^n_i = 1 p_i =1}. A fundamental quantity for a probability distribution p is its entropy S(p):= sigma^n_i=1 P_i log p_i. (We extend the function x log x to 0 by continuity, so that 0 log 0:= 0.) The entropy of p measures the disorder or lack of information in p. Using Lagrange multipliers, find the critical point q of S on the set P_n. Compute the value of S at q. Prove that S reaches its maximum on P bar_n at q.

Suppose that we have a probability distribution on the set {1, …, n}, i.e. a sequence p = (p_1, …, p_n) of probabilities in the set P_n vector where P bar_n:= {p element of [0, 1]^n: Sigma^n_i=1 p_i = 1}, P_n ={p element of (0, 1)^n: Sigma^n_i = 1 p_i =1}. A fundamental quantity for a probability distribution p is its entropy S(p):= sigma^n_i=1 P_i log p_i. (We extend the function x log x to 0 by continuity, so that 0 log 0:= 0.) The entropy of p measures the disorder or lack of information in p. Using Lagrange multipliers, find the critical point q of S on the set P_n. Compute the value of S at q. Prove that S reaches its maximum on P bar_n at q.