ML003 – MLE Estimation

When given a dataset, we often seek to model the distribution from which the data is generated. This involves two key steps:

Any statistic used to estimate an unknown parameter, denoted as \(\theta\), is called an estimator of \(\theta\). The estimate \(\hat{\boldsymbol{\theta}}\) is a random variable that approximates \(\boldsymbol{\theta}\) based on the data. There are four primary techniques for parameter estimation:

Let \((X_1, X_2, \dots, X_n)\) be \(n\) independent and identically distributed (i.i.d) random variables, where each \(X_i\) is drawn from a distribution \(F\) with expected value \(\mathrm{E}[X_i] = \mu\) and variance \(\mathrm{Var}(X_i) = \sigma^2\). Two common unbiased estimators are:

These estimators can be applied to data from any distribution to estimate its mean and variance. For example, if we assume the data follows a Gaussian distribution, we can use these estimators to estimate its parameters. However, not all distributions have mean and variance as their parameters, making this approach limited in its general applicability to parameter estimation.

Consider a sample of \(n\) independent and identically distributed (i.i.d.) random variables \((X_1, X_2, \dots, X_n)\). Since this forms a random vector, the joint probability density function (PDF) or probability mass function (PMF) of these random variables is given by: \(f(X_1, X_2, \dots, X_n)\). Since \(X_i\)'s are identical and independent, their distribution can be characterized by a parameter \(\theta\), and the joint distribution can be written as:

Observed Data Example:

Suppose each \(X_i\) was drawn from a distribution \(\texttt{Ber}(p)\) with an unknown parameter \(p\). Then the PMF of each \(X_i\) is \(f(X_i=x_i) = p^{x_i} (1-p)^{(1- x_i)}\), which means \(f(X_i =1) = p \text{ and } f(X_i =0) = 1-p\).

Let \(n=10\) and suppose we observe the data: \([0,0,1,1,1,1,1,1,1,1]\). For the given data, we get:

The above function is called the likelihood function, denoted as \(L(p)\). For each parameter value \(p\), we compute the likelihood. Thus, the likelihood function is a function of \(p\), given the observed data. By plotting the likelihood for different values of \(p\), we gain insights such as: "if \(p=0.4\), how likely is the observed data?"

Consider a sample of \(n\) i.i.d random variables \((X_1, X_2, \dots, X_n)\). Each \(X_i\) is drawn from a distribution with density (or mass) function \(f(x_i |\, \boldsymbol{\theta})\), where \(\boldsymbol{\theta} = (\theta_1, \theta_2, \dots, \theta_k)\). The PMF or PDF gives the likelihood of a single data point \(x_i\) given parameter \(\boldsymbol{\theta}\). Here we are given a set of observed i.i.d data points \((x_1, x_2, \dots, x_n)\) and we seek to answer:

How likely is the observed data \((x_1, x_2, \dots, x_n)\) given parameter \(\boldsymbol{\theta}\)?

The likelihood of all the observed data points is given by the likelihood function, denoted by \(L(\boldsymbol{\theta})\), which is defined as

The Maximum Likelihood Estimator (MLE) of \(\boldsymbol{\theta}\) is the value of \(\boldsymbol{\theta}\) that maximizes \(L(\boldsymbol{\theta})\), i.e., the value that makes the observed data look as likely as possible.

We know that \(\log\) is a strictly increasing function (i.e.,) if \(x_1 < x_2 \Leftrightarrow \log x_1 < \log x_2\). So for all \(x_1, x_2\) where \(x_1, x_2 >0 \text{ and } x_1 \ne x_2\), if \(f(x_1) < f(x_2)\), then \(\log f(x_1) < \log f(x_2)\), for any function \(f\).

Taking log (natural log) on both sides of the likelihood equation,

Since log is monotonic, maximizing \(L(\boldsymbol{\theta})\) is equivalent to maximizing \(LL(\boldsymbol{\theta})\), i.e., the argument that maximizes \(L(\boldsymbol{\theta})\) is the same as the argument that maximizes \(LL(\boldsymbol{\theta})\). We often maximize the log likelihood for numerical stability.