Statistics is the science of learning from data under uncertainty. This course develops the theory and practice of parametric statistical inference — the art of using observations from a probability model to draw conclusions about unknown parameters that govern that model. We begin with a simple but powerful framework: you observe data drawn from some distribution, that distribution depends on parameters you don't know, and you want to use the data to estimate those parameters, test hypotheses about them, or make predictions. This framework unifies the entire course and motivates every major topic, from likelihood-based estimation to hypothesis tests to linear regression.

The course is built around three interconnected problems. First, how do we produce point estimates and interval estimates of unknown parameters? We explore bias, mean squared error, sufficiency, and the likelihood principle — tools that guide us toward estimators with desirable properties. Second, how do we make binary decisions about parameters in the face of uncertainty? Hypothesis testing, grounded in the Neyman-Pearson framework, answers this by balancing Type I and Type II errors and extending ideas from estimation to composite hypotheses and goodness-of-fit tests. Third, how do we estimate relationships between variables? Linear models are the workhorse of applied statistics, and we develop their theory from first principles, connecting them to estimation and testing through the lens of the normal distribution and the Gauss–Markov theorem.

These three chapters are not separate topics but chapters of a single story. Chapter 0 establishes the parametric framework and the "three questions of inference" that motivate the course. Chapter 1 gives you estimators — point and interval estimates derived from likelihood and Bayesian thinking. Chapter 2 teaches you to make decisions using those estimates, introducing hypothesis tests as a formal tool for binary choices. Chapter 3 applies all of this machinery to linear models, the most important and widely used probability model in practice, where estimation, testing, and inference collapse into a single elegant theory. By the end, you will see statistics not as a collection of recipes, but as a coherent body of reasoning about how data constrains our knowledge of unknown parameters.

# Introduction

Statistics is, at its core, a discipline of principled uncertainty. We observe data — numbers generated by some process in the world — and we want to draw conclusions about that process. This course is concerned with the formal mathematical theory that makes such conclusions rigorous. The question is not merely "what does the data suggest?" but "what can we legitimately infer, with quantified confidence, from the data we have seen?"

The three chapters that follow cover the three fundamental problems of statistical inference: estimating an unknown quantity from data (Estimation), bounding how far our estimate might be from the truth (Confidence Intervals, developed within the Estimation chapter), and deciding between competing claims about the world (Hypothesis Testing). A final chapter on Linear Models shows how these tools apply to the most widely used class of statistical models. This introduction sets up the common framework that underlies all three.

## The Parametric Framework

Here is the central difficulty statistics must overcome: any finite dataset is compatible with infinitely many different probability distributions. Suppose you record $n = 20$ counts of some phenomenon. A distribution with mean 3.1 fits the data; so does one with mean 3.2; so does a distribution of a completely different shape whose mean happens to coincide with the sample mean. Without additional structure, no amount of data can distinguish among them — you would need to specify infinitely many numbers to pin down an arbitrary distribution over the integers, and a finite sample provides only $n$ numbers. The data alone are simply insufficient to uniquely identify the data-generating process.

The parametric approach resolves this by restricting attention to a *known family* of distributions indexed by a finite-dimensional parameter. The key insight is that once we commit to a family — say, all Poisson distributions — the entire probability law is determined by a single number $\mu > 0$. The infinite search space collapses to a one-dimensional one, and a moderate sample can meaningfully pin down $\mu$. This is not a claim that the world is literally Poisson; it is a modelling commitment that makes inference tractable.

[definition: Parametric Family] A **parametric family** of distributions is a collection $\{f(\,\cdot\,;\theta) : \theta \in \Theta\}$, where $\Theta \subseteq \mathbb{R}^k$ is the **parameter space**, and for each $\theta \in \Theta$, $f(\,\cdot\,;\theta) : \mathcal{X} \to [0,\infty)$ is a probability density or mass function on the **sample space** $\mathcal{X}$. Here $\mathcal{X}$ is the set of values each observation can take: $\mathcal{X} = \{0, 1, 2, \ldots\}$ for count data, $\mathcal{X} = \mathbb{R}$ for continuous measurements, $\mathcal{X} = [0,1]$ for proportions. The unknown $\theta \in \Theta$ is the **parameter** of the model. When $k = 1$ we speak of a scalar parameter; when $k > 1$, a vector parameter. [/definition]

[example: Common Parametric Families] Three families appear repeatedly in this course, and they illustrate two structurally different ways the choice of family matters. **Poisson.** If $X \sim \operatorname{Pois}(\mu)$ with $\mu > 0$, then \begin{align*} f(x; \mu) = \frac{e^{-\mu}\mu^x}{x!}, \qquad x = 0, 1, 2, \ldots \end{align*} The parameter $\mu = \mathbb{E}[X]$ is the mean rate of occurrence. We know the data come from *some* Poisson distribution, but we do not know $\mu$. **Binomial.** If $X \sim \operatorname{Bin}(n, \theta)$ with $n$ known and $\theta \in (0,1)$ unknown, \begin{align*} f(x; \theta) = \binom{n}{x}\theta^x(1-\theta)^{n-x}, \qquad x = 0, 1, \ldots, n. \end{align*} **Normal.** If $X \sim N(\mu, \sigma^2)$ with one or both of $\mu \in \mathbb{R}$, $\sigma^2 > 0$ unknown, the parameter is $\theta = (\mu, \sigma^2) \in \mathbb{R} \times (0,\infty)$. Here $\Theta$ is two-dimensional. Now consider what the Poisson assumption buys us. Suppose we observe $n = 10$ counts: say $x = (2, 0, 3, 1, 4, 2, 1, 0, 3, 2)$. Under the Poisson model, the only free quantity is $\mu$, and these 10 numbers collectively pin it down: the maximum likelihood estimate (derived in Chapter 1) is simply $\hat{\mu} = \bar{x} = 1.8$. The parametric assumption has converted 10 numbers into a single inferential target. Without the Poisson assumption, those same 10 observations are compatible with any distribution on $\{0,1,2,\ldots\}$ that assigns positive probability to the values $\{0,1,2,3,4\}$. A geometric distribution, a negative binomial, a distribution that places all its mass on five arbitrary points — all are consistent with the data. The sample cannot distinguish among them because each introduces its own free parameters that can be tuned to fit the observations. The parametric assumption is the commitment that converts an underdetermined problem into a tractable one. Notice also that the choice of family has predictive consequences. If we believe the data are Poisson, then the probability of observing $X = 10$ in a future trial is $e^{-1.8}(1.8)^{10}/10! \approx 0.00003$. If instead we model the data as negative binomial with the same mean, the tail probabilities differ — possibly substantially. The family is not a neutral container; it shapes what the model predicts beyond the range of the observed data. [/example]

The parametric assumption is what makes inference possible. Without it, any continuous density on $\mathbb{R}$ — and there are uncountably many — is compatible with any finite sample, since the sample has probability zero under every continuous distribution in isolation. By restricting to a known family parameterised by $\theta \in \Theta \subseteq \mathbb{R}^k$, we reduce the infinite-dimensional estimation problem to a finite-dimensional one, where the data can genuinely discriminate among models. The choice of family is therefore the first and most consequential modelling decision.

## Simple Random Samples

A single observation from $f(\,\cdot\,;\theta)$ is rarely enough to say anything useful about $\theta$. But repetition raises an immediate difficulty: if our observations are not independent of each other, or if they come from different distributions, the bookkeeping becomes intractable. Yesterday's measurement might influence today's; a patient recruited early in a trial might differ systematically from one recruited later. These dependencies and inhomogeneities, if not accounted for, silently corrupt every downstream inference — an estimator derived under the i.i.d. assumption applied to dependent data can be badly biased, and confidence intervals can fail to cover the true parameter at anything near their nominal level. The simplest setting that avoids all these complications is one where each observation is a fresh, independent draw from the same distribution.

[definition: Simple Random Sample] A **simple random sample** of size $n$ from $f(\,\cdot\,;\theta)$ is a collection $X_1, X_2, \ldots, X_n$ of random variables that are independent and identically distributed (i.i.d.), each with density or mass function $f(\,\cdot\,;\theta)$. Formally, each $X_i : \Omega \to \mathcal{X}$ is a random variable on the underlying probability space $(\Omega, \mathcal{F}, \mathbb{P})$, taking values in the sample space $\mathcal{X}$. We write the sample as the vector $X = (X_1, \ldots, X_n)$. After observing the experiment, we obtain the **observed sample** $x = (x_1, \ldots, x_n)$, the realised values. [/definition]

[remark: Random Variables vs Observed Values] Throughout the course we maintain the convention that capital $X$ denotes the random vector (before observation) and lower-case $x$ denotes the observed numerical values. A function $T(X)$ is a random variable; $T(x)$ is a number. This distinction matters when we discuss the sampling distribution of estimators. [/remark]

Because the observations are i.i.d., the joint density of the sample factorises:

\begin{align*} f(x; \theta) = \prod_{i=1}^{n} f(x_i; \theta). \end{align*}

This product structure is what makes the likelihood function — defined properly in Chapter 1 — tractable to maximise. Independence turns a complicated multivariate density into a simple product; this is the mathematical payoff for the i.i.d. assumption.

## The Three Questions of Statistical Inference