[proofplan]
The regularity assumption provides a bias expansion $\mathbb{E}[T_n] = \theta + a(\theta)/n + O(1/n^2)$ for the full-sample estimator, and the same expansion applied to each leave-one-out estimator $T_{n-1,i}$ gives $\mathbb{E}[T_{n-1,i}] = \theta + a(\theta)/(n-1) + O(1/n^2)$. We form the jackknife estimator as a specific linear combination of $T_n$ and $\bar{T}_{n-1} = \frac{1}{n}\sum_{i=1}^n T_{n-1,i}$ designed to cancel the $O(1/n)$ term. Computing the expectation of this combination confirms that only the $O(1/n^2)$ remainder survives.
[/proofplan]
[step:Write the bias expansion for the full-sample estimator $T_n$]
The regularity assumption on the bias states that $\mathbb{E}_\theta[T_n]$ admits the uniform expansion
\begin{align*}
\mathbb{E}_\theta[T_n] = \theta + \frac{a(\theta)}{n} + \frac{b(\theta)}{n^2} + O\!\left(\frac{1}{n^3}\right),
\end{align*}
where $a(\theta)$ and $b(\theta)$ are functions of $\theta$ that do not depend on $n$, and the $O(1/n^3)$ remainder is uniform in $\theta$. The leading bias term is $a(\theta)/n$.
[/step]
[step:Write the bias expansion for each leave-one-out estimator $T_{n-1,i}$]
For each $i \in \{1, \ldots, n\}$, the leave-one-out estimator $T_{n-1,i}$ is computed from the sample $(X_1, \ldots, X_{i-1}, X_{i+1}, \ldots, X_n)$ of size $n-1$. Since the $X_j$ are i.i.d., $T_{n-1,i}$ has the same distribution as $T_{n-1}$ (an estimator computed from $n-1$ i.i.d. observations), so the same bias expansion holds with $n$ replaced by $n-1$:
\begin{align*}
\mathbb{E}_\theta[T_{n-1,i}] = \theta + \frac{a(\theta)}{n-1} + \frac{b(\theta)}{(n-1)^2} + O\!\left(\frac{1}{n^3}\right).
\end{align*}
Define the leave-one-out average $\bar{T}_{n-1} = \frac{1}{n}\sum_{i=1}^n T_{n-1,i}$. By linearity of expectation:
\begin{align*}
\mathbb{E}_\theta[\bar{T}_{n-1}] = \theta + \frac{a(\theta)}{n-1} + \frac{b(\theta)}{(n-1)^2} + O\!\left(\frac{1}{n^3}\right).
\end{align*}
[guided]
Each $T_{n-1,i}$ is computed from $n-1$ observations drawn i.i.d. from the same distribution. The regularity assumption is a statement about the bias of $T_m$ as a function of sample size $m$, so it applies equally well to $m = n-1$. The crucial point is that $a(\theta)$ is the same function regardless of sample size; only the denominator changes from $n$ to $n-1$.
Since the leave-one-out average $\bar{T}_{n-1}$ is a simple average of $n$ identically distributed terms, its expectation equals the common expectation $\mathbb{E}[T_{n-1,i}]$.
[/guided]
[/step]
[step:Form the jackknife estimator and cancel the $O(1/n)$ bias]
The jackknife bias-corrected estimator is defined as
\begin{align*}
\tilde{T}_{\mathrm{JACK}} = n T_n - (n-1) \bar{T}_{n-1}.
\end{align*}
Taking expectations:
\begin{align*}
\mathbb{E}_\theta[\tilde{T}_{\mathrm{JACK}}] &= n \mathbb{E}_\theta[T_n] - (n-1) \mathbb{E}_\theta[\bar{T}_{n-1}] \\
&= n\!\left(\theta + \frac{a(\theta)}{n} + \frac{b(\theta)}{n^2} + O\!\left(\frac{1}{n^3}\right)\right) - (n-1)\!\left(\theta + \frac{a(\theta)}{n-1} + \frac{b(\theta)}{(n-1)^2} + O\!\left(\frac{1}{n^3}\right)\right).
\end{align*}
Expanding term by term:
The $\theta$ terms: $n\theta - (n-1)\theta = \theta$.
The $a(\theta)$ terms: $n \cdot \frac{a(\theta)}{n} - (n-1) \cdot \frac{a(\theta)}{n-1} = a(\theta) - a(\theta) = 0$.
The $b(\theta)$ terms: $n \cdot \frac{b(\theta)}{n^2} - (n-1) \cdot \frac{b(\theta)}{(n-1)^2} = \frac{b(\theta)}{n} - \frac{b(\theta)}{n-1}$.
Now $\frac{1}{n} - \frac{1}{n-1} = \frac{n-1-n}{n(n-1)} = \frac{-1}{n(n-1)}$, so the $b(\theta)$ contribution is $\frac{-b(\theta)}{n(n-1)} = O(1/n^2)$.
The remaining terms contribute $n \cdot O(1/n^3) + (n-1) \cdot O(1/n^3) = O(1/n^2)$.
Combining:
\begin{align*}
\mathbb{E}_\theta[\tilde{T}_{\mathrm{JACK}}] = \theta + O\!\left(\frac{1}{n^2}\right).
\end{align*}
[guided]
The jackknife is a clever linear combination designed to kill the $a(\theta)/n$ term. Why does the particular choice $nT_n - (n-1)\bar{T}_{n-1}$ work?
Consider the general linear combination $\alpha T_n + \beta \bar{T}_{n-1}$. For unbiasedness up to $O(1/n^2)$, we need two conditions:
1. The $\theta$ coefficient is 1: $\alpha + \beta = 1$.
2. The $a(\theta)$ coefficient is 0: $\alpha \cdot \frac{1}{n} + \beta \cdot \frac{1}{n-1} = 0$.
From condition 2: $\alpha(n-1) + \beta n = 0$, so $\beta = -\alpha(n-1)/n$. Substituting into condition 1: $\alpha - \alpha(n-1)/n = 1$, giving $\alpha/n = 1$, so $\alpha = n$ and $\beta = -(n-1)$.
This yields $\tilde{T}_{\mathrm{JACK}} = nT_n - (n-1)\bar{T}_{n-1}$. The leading bias $a(\theta)/n$ cancels exactly, and the next-order term $b(\theta)$ is reduced from $O(1/n)$ to $O(1/n^2)$ because
\begin{align*}
n \cdot \frac{b(\theta)}{n^2} - (n-1) \cdot \frac{b(\theta)}{(n-1)^2} = b(\theta)\!\left(\frac{1}{n} - \frac{1}{n-1}\right) = \frac{-b(\theta)}{n(n-1)} = O\!\left(\frac{1}{n^2}\right).
\end{align*}
The conclusion is
\begin{align*}
\left|\mathbb{E}_\theta[\tilde{T}_{\mathrm{JACK}}] - \theta\right| = O\!\left(\frac{1}{n^2}\right),
\end{align*}
confirming that the jackknife reduces the bias from $O(1/n)$ to $O(1/n^2)$.
[/guided]
[/step]
