This course provides a comprehensive introduction to the mathematical foundations of statistics, progressing from classical parametric inference through Bayesian methods and into modern nonparametric and computational techniques. The course emphasizes the theoretical underpinnings of statistical estimation and inference, building rigorous mathematical understanding rather than focusing on applications or cookbook methods. By working through maximum likelihood estimation, information theory, and asymptotic analysis, students develop the conceptual tools needed to understand why standard statistical procedures work and when they can be trusted.

The first half of the course establishes the frequentist framework through the lens of maximum likelihood estimation. Beginning with the basic definition and properties of MLEs, the course develops the key theoretical results that justify their use: the Fisher information and Cramér-Rao bound provide measures of efficiency, while consistency and asymptotic normality guarantee that MLEs behave predictably in large samples. This progression from finite-sample properties to asymptotic theory culminates in practical inferential procedures like confidence intervals and hypothesis tests. The chapters on asymptotic inference demonstrate how these theoretical results translate into actionable statistical methods.

The second half broadens the statistical perspective by introducing Bayesian inference, decision theory, and modern computational and nonparametric methods. Students learn how prior distributions and loss functions shape Bayesian estimation, and how frequentist and Bayesian approaches relate through risk analysis and admissibility. The final chapters on resampling, Monte Carlo methods, and nonparametric statistics acknowledge the limitations of parametric assumptions and demonstrate how computational power enables statistical inference without strong structural assumptions. Throughout, the course emphasizes that sound statistical practice requires both theoretical understanding and awareness of practical limitations.

# Introduction

## Statistical Models and the Likelihood Principle

Statistics is concerned with the inverse problem of probability: given access to draws from an unknown probability distribution, make rigorous statements about that distribution. Where probability theory starts from a known distribution and asks what we expect to see, statistics starts from what we have seen and asks what distribution produced it. This chapter introduces the formal framework — statistical models, parameter spaces, and the three central goals of the discipline — before developing the likelihood principle, the most fundamental strategy for extracting information from data.

## The Statistical Framework

The first difficulty in statistics is that an arbitrary probability distribution is an infinite-dimensional object — specifying it completely would require knowing $F(t)$ for every $t \in \mathbb{R}$. No finite dataset can pin down an object that complex. The way out is to restrict attention to families of distributions parametrized by a finite-dimensional vector $\theta$: instead of searching over all possible distributions, we search over a tractable parameter space $\Theta$. This section sets up that framework precisely.

We work with a real-valued random variable $X$ on a probability space $(\Omega, \mathcal{F}, \mathbb{P})$. The distribution of $X$ is described by its cumulative distribution function

\begin{align*} F(t) = \mathbb{P}(\omega \in \Omega : X(\omega) \leq t), \quad t \in \mathbb{R}. \end{align*}

When $X$ is discrete, $F$ is related to the probability mass function $f$ by

\begin{align*} F(t) = \sum_{x \leq t} f(x), \end{align*}

and when $X$ is continuous, $F$ is related to the probability density function $f$ by

\begin{align*} F(t) = \int_{-\infty}^{t} f(s)\, ds. \end{align*}

Most statistical problems concern not a single observation but a sample: $n$ independent copies $X_1, \ldots, X_n$ of $X$, where $n$ is called the sample size. The distribution of each $X_i$ is the same unknown distribution, which we wish to learn. Rather than working with a completely unknown distribution — an infinite-dimensional object — statistics typically restricts attention to families of distributions parametrized by a finite-dimensional vector $\theta$.

[definition: Statistical Model] A **statistical model** for a sample from $X$ is a family \begin{align*} \{f(\theta, \cdot) : \theta \in \Theta\} \quad \text{or} \quad \{P_\theta : \theta \in \Theta\} \end{align*} of probability mass functions or probability density functions $f(\theta, \cdot)$, or of probability distributions $P_\theta$ for the law of $X$. The index set $\Theta$ is called the **parameter space**. [/definition]

The parameter space $\Theta$ encodes what values the unknown parameter $\theta$ is allowed to take. Both the shape of the model family and the choice of $\Theta$ are modelling decisions that the statistician makes before seeing data.

[example: Standard Statistical Models] The following models illustrate how the choice of $\Theta$ encodes modelling assumptions. (i) $\mathcal{N}(\theta, 1)$ with $\theta \in \Theta = \mathbb{R}$. The variance is treated as known; only the mean is unknown. This is the simplest Gaussian model and will serve as our running example throughout the course. (ii) $\mathcal{N}(\mu, \sigma^2)$ with $\theta = (\mu, \sigma^2) \in \Theta = \mathbb{R} \times (0, \infty)$. Both mean and variance are unknown, so we have a two-dimensional parameter. (iii) $\mathrm{Exp}(\theta)$ with $\theta \in \Theta = (0, \infty)$. An exponential model with unknown rate. The parameter space is open because a zero rate makes no probabilistic sense. (iv) $\mathcal{N}(\theta, 1)$ with $\theta \in \Theta = [-1, 1]$. The same Gaussian family as (i), but with the mean constrained to a compact set — perhaps because domain knowledge rules out means outside $[-1, 1]$. [/example]

Models (i) and (iv) use the same distributional family but impose different parameter spaces. This distinction is not cosmetic. The set $\Theta$ determines which estimators are admissible: an MLE for model (iv) must respect the constraint $\hat{\theta} \in [-1, 1]$, which can force a different answer than the unconstrained optimum. And if the true mean is $\theta_0 = 2$, then model (iv) is misspecified — no element of the family reproduces the true distribution, and the theory we develop simply does not apply. This is the chief danger of parametric modelling: a wrong assumption about $\Theta$ is not a small error in a coefficient, it is a categorical mistake about what distributions are even being considered.

A less obvious danger is the opposite extreme: allowing $\Theta$ to be too large, or abandoning a parametric restriction altogether. If we do not restrict the distribution at all — if we allow $F$ to be any cumulative distribution function — then we are in the nonparametric setting. Nonparametric estimation is possible (the empirical distribution function is the canonical example), but it comes at a serious statistical cost: with no structure to exploit, estimators converge far more slowly, and many questions that are easy in the parametric case become hard or impossible to answer with finite data. The parametric assumption is not a crutch — it is what makes efficient inference tractable.