Solved–Assignment 05– Solution

Description

1. (10 points) Use the Matlab function randn() to generate a data sample of N points drawn from a Gaussian distribution with mean _true = 10 and standard deviation _true = 4. Consider the problem of using the data to get an estimate b of this Gaussian mean, assuming it is unknown, when the standard deviation _true is known.

Consider using one of the two prior prior distributions on the mean: (i) a Gaussian prior with mean

_prior = 10:5 and standard deviation _prior = 1 and (ii) a uniform prior over (9:5; 11:5).

Consider various sample sizes N = 5, 10, 20, 40, 60, 80, 100, 500, 10³; 10⁴. For each sample size N, repeat the following experiment M 100 times: generate the data, get the maximum likelihood estimate b^ML, get the maximum-a-posteriori estimates b^MAP1 and b^MAP2, and measure the relative errors jb _truej= _true for all three estimates.

Plot a single graph that shows the relative errors for each value of N as a box plot (use the Matlab boxplot() function), for each of the three estimates.

Interpret what you see in the graph. (i) What happens to the error as N increases ? (ii) Which of the three estimates will you prefer and why ?

2. (10 points) Use the Matlab function rand() to generate a data sample of N points from the uniform distribution on (0; 1). Transform the resulting data x to generate a transformed data sample where each datum y := ( 1= ) log(x) with = 5. The transformed data y will have some distribution with parameter ; what is its analytical form ? Use a Gamma prior on the parameter , where the Gamma distribution has parameters = 5:5 and = 1.

Consider various sample sizes N = 5, 10, 20, 40, 60, 80, 100, 500, 10³; 10⁴. For each sample size N, repeat the following experiment M 100 times: generate the data, get the maximum like-lihood estimate b^ML, get the Bayesian estimate as the posterior mean b^{PosteriorMean}, and measure the relative errors jb _truej= _true for both the estimates.

Derive a formula for the posterior mean.

Plot a single graph that shows the relative errors for each value of N as a box plot (use the Matlab boxplot() function), for both the estimates.

Interpret what you see in the graph. (i) What happens to the error as N increases ? (ii) Which of the two estimates will you prefer and why ?

3. (5 points) Consider a 2-dimensional data sample (assuming an extremely large sample size N) such that the data are drawn from a uniform distribution on a circle (i.e., a ring; the boundary of a disc, but not its interior) with center as the origin and radius r. Suppose you decide to fit a multivariate (2-dimensional) Gaussian distribution to this data by maximizing the likelihood

p

function. The multivariate Gaussian is P (x; ; C) := 1= (2 )^djCj) exp( 0:5(x )^>C ¹(x )), where, for our case, dimension d = 2, x and are vectors of size d 1, C is a matrix of size d d, and jCj is the determinant of C.

Derive the mathematical formula, in terms of r, for the estimated mean and the estimated covariance matrix. You may use http://www.ee.ic.ac.uk/hp/staff/dmb/matrix/calculus. html to compute derivatives of the likelihood function with respect to the vector parameter and matrix parameter C.

Where is the mode of this Gaussian situated within R² ? Do you think this Gaussian fits the data well ? Is it a good model ? Why or why not ?

Generate a large sample that is uniformly distributed on a circle with center origin and radius r. Compute the maximum likelihood estimates for the mean and covariance and report them, along with the sample size. Do they match the theoretically predicted values ?

4. (10 points) Suppose random variable X has a uniform distribution over (0; ), where the param-eter is unknown. Consider a Pareto distribution prior on , with a scale parameter _m > 0 and a shape parameter > 1, as P ( ) / ( _m= ) for _m and P ( ) = 0 otherwise.

Find the maximum-likelihood estimate b^ML and the maximum-a-posteriori estimate b^MAP. Does b^MAP tend to b^ML as the sample size tends to infinity ? Is this desirable or not ?

Find an estimator of the mean of the posterior distribution b^{PosteriorMean}.

Does b^{PosteriorMean} tend to b^ML as the sample size tends to infinity ? Is this desirable or not ?

3