[proofplan]
We write the deleted normal matrix as a rank-one perturbation of the full normal matrix. The key algebraic input is the Sherman–Morrison inverse identity, which we verify directly for the matrix $X^\top X-x_i x_i^\top$. Substituting this inverse into the deleted normal equations gives the leave-one-out correction for $\hat{\beta}_{(i)}$. Multiplying by $x_i^\top$ then converts the coefficient formula into the deleted residual formula.
[/proofplan]
[step:Express the deleted normal equations as a rank-one update]
Define
\begin{align*}
A := X^\top X \in \mathbb{R}^{p\times p},
\qquad
C := A^{-1} \in \mathbb{R}^{p\times p},
\qquad
x := x_i \in \mathbb{R}^p,
\qquad
s := y_i \in \mathbb{R}.
\end{align*}
Since $X$ has full column rank, $A$ is invertible. By deleting the $i$-th row of $X$ and the $i$-th component of $y$, we have
\begin{align*}
X_{(i)}^\top X_{(i)} = A - x x^\top,
\qquad
X_{(i)}^\top y_{(i)} = X^\top y - x s.
\end{align*}
Also,
\begin{align*}
h_{ii} = x^\top Cx,
\qquad
\hat{\beta}=CX^\top y,
\qquad
e_i=s-x^\top \hat{\beta}.
\end{align*}
[/step]
[step:Invert the deleted normal matrix by verifying the Sherman–Morrison identity]
We claim that
\begin{align*}
(A-xx^\top)^{-1}
=
C+\frac{Cxx^\top C}{1-x^\top Cx}.
\end{align*}
The denominator is nonzero because $1-x^\top Cx=1-h_{ii}>0$. Define
\begin{align*}
M := C+\frac{Cxx^\top C}{1-x^\top Cx} \in \mathbb{R}^{p\times p},
\qquad
I_p := \operatorname{id}_{\mathbb{R}^p} \in \mathbb{R}^{p\times p}.
\end{align*}
Here $I_p$ denotes the $p \times p$ identity matrix. Then
\begin{align*}
(A-xx^\top)M
&=
AC
+
\frac{ACxx^\top C}{1-x^\top Cx}
-
xx^\top C
-
\frac{xx^\top Cxx^\top C}{1-x^\top Cx} \\
&=
I_p
+
\frac{xx^\top C}{1-x^\top Cx}
-
xx^\top C
-
\frac{x(x^\top Cx)x^\top C}{1-x^\top Cx} \\
&=
I_p
+
\frac{xx^\top C-(1-x^\top Cx)xx^\top C-(x^\top Cx)xx^\top C}{1-x^\top Cx} \\
&=
I_p.
\end{align*}
Thus $A-xx^\top$ has a right inverse $M$. Since it is a square matrix, this right inverse is its inverse, so $X_{(i)}^\top X_{(i)}=A-xx^\top$ is invertible and the displayed formula holds.
[guided]
The deleted normal matrix differs from the full normal matrix by removing exactly the contribution of the deleted row. Since the contribution of row $i$ to $X^\top X$ is $x_i x_i^\top$, this is a rank-one perturbation. We now verify the inverse formula directly, rather than treating it as a black box.
Let
\begin{align*}
M := C+\frac{Cxx^\top C}{1-x^\top Cx} \in \mathbb{R}^{p\times p},
\qquad
I_p := \operatorname{id}_{\mathbb{R}^p} \in \mathbb{R}^{p\times p}.
\end{align*}
Here $I_p$ denotes the $p \times p$ identity matrix. This matrix $M$ is well-defined because $1-x^\top Cx=1-h_{ii}>0$. We multiply $M$ by $A-xx^\top$ on the left:
\begin{align*}
(A-xx^\top)M
&=
(A-xx^\top)\left(C+\frac{Cxx^\top C}{1-x^\top Cx}\right) \\
&=
AC
+
\frac{ACxx^\top C}{1-x^\top Cx}
-
xx^\top C
-
\frac{xx^\top Cxx^\top C}{1-x^\top Cx}.
\end{align*}
Because $C=A^{-1}$, we have $AC=I_p$ and $ACx=x$. Also $x^\top Cx$ is a scalar, so
\begin{align*}
xx^\top Cxx^\top C
=
x(x^\top Cx)x^\top C
=
(x^\top Cx)xx^\top C.
\end{align*}
Substituting these identities gives
\begin{align*}
(A-xx^\top)M
&=
I_p
+
\frac{xx^\top C}{1-x^\top Cx}
-
xx^\top C
-
\frac{(x^\top Cx)xx^\top C}{1-x^\top Cx} \\
&=
I_p
+
\frac{xx^\top C-(1-x^\top Cx)xx^\top C-(x^\top Cx)xx^\top C}{1-x^\top Cx} \\
&=
I_p.
\end{align*}
Therefore $M$ is a right inverse of the square matrix $A-xx^\top$. A square matrix with a right inverse is invertible, and the right inverse equals the inverse. Hence
\begin{align*}
(A-xx^\top)^{-1}
=
C+\frac{Cxx^\top C}{1-x^\top Cx}.
\end{align*}
[/guided]
[/step]
[step:Substitute the inverse formula into the deleted estimator]
Using the deleted normal equations and the inverse formula just proved,
\begin{align*}
\hat{\beta}_{(i)}
&=
(A-xx^\top)^{-1}(X^\top y-xs) \\
&=
\left(C+\frac{Cxx^\top C}{1-x^\top Cx}\right)(X^\top y-xs) \\
&=
CX^\top y-Cxs
+
\frac{Cx\,x^\top CX^\top y}{1-x^\top Cx}
-
\frac{Cx\,x^\top Cx\,s}{1-x^\top Cx}.
\end{align*}
Since $CX^\top y=\hat{\beta}$, $x^\top C X^\top y=x^\top \hat{\beta}$, and $x^\top Cx=h_{ii}$, this becomes
\begin{align*}
\hat{\beta}_{(i)}
&=
\hat{\beta}
-
Cxs
+
\frac{Cx\,x^\top \hat{\beta}}{1-h_{ii}}
-
\frac{Cx\,h_{ii}s}{1-h_{ii}} \\
&=
\hat{\beta}
+
Cx\left(
-s+\frac{x^\top\hat{\beta}-h_{ii}s}{1-h_{ii}}
\right) \\
&=
\hat{\beta}
+
Cx\left(
\frac{-s(1-h_{ii})+x^\top\hat{\beta}-h_{ii}s}{1-h_{ii}}
\right) \\
&=
\hat{\beta}
-
\frac{Cx(s-x^\top\hat{\beta})}{1-h_{ii}} \\
&=
\hat{\beta}
-
\frac{(X^\top X)^{-1}x_i e_i}{1-h_{ii}}.
\end{align*}
[guided]
Now we insert the inverse of the deleted normal matrix into the formula for the deleted least-squares estimator. The deleted estimator is
\begin{align*}
\hat{\beta}_{(i)}
=
(X_{(i)}^\top X_{(i)})^{-1}X_{(i)}^\top y_{(i)}.
\end{align*}
Using
\begin{align*}
X_{(i)}^\top X_{(i)}=A-xx^\top,
\qquad
X_{(i)}^\top y_{(i)}=X^\top y-xs,
\end{align*}
and the inverse identity from the previous step, we obtain
\begin{align*}
\hat{\beta}_{(i)}
&=
\left(C+\frac{Cxx^\top C}{1-x^\top Cx}\right)(X^\top y-xs) \\
&=
CX^\top y-Cxs
+
\frac{Cx\,x^\top CX^\top y}{1-x^\top Cx}
-
\frac{Cx\,x^\top Cx\,s}{1-x^\top Cx}.
\end{align*}
Each term now has a regression interpretation. Since $CX^\top y=\hat{\beta}$, the scalar $x^\top C X^\top y$ equals $x^\top\hat{\beta}$, and $x^\top Cx=h_{ii}$. Therefore
\begin{align*}
\hat{\beta}_{(i)}
&=
\hat{\beta}
-
Cxs
+
\frac{Cx\,x^\top \hat{\beta}}{1-h_{ii}}
-
\frac{Cx\,h_{ii}s}{1-h_{ii}}.
\end{align*}
We collect the three terms containing $Cx$:
\begin{align*}
\hat{\beta}_{(i)}
&=
\hat{\beta}
+
Cx\left(
-s+\frac{x^\top\hat{\beta}-h_{ii}s}{1-h_{ii}}
\right) \\
&=
\hat{\beta}
+
Cx\left(
\frac{-s(1-h_{ii})+x^\top\hat{\beta}-h_{ii}s}{1-h_{ii}}
\right) \\
&=
\hat{\beta}
+
Cx\left(
\frac{x^\top\hat{\beta}-s}{1-h_{ii}}
\right).
\end{align*}
Because $e_i=s-x^\top\hat{\beta}$, the numerator $x^\top\hat{\beta}-s$ is $-e_i$. Returning to the original notation $C=(X^\top X)^{-1}$ and $x=x_i$, we get
\begin{align*}
\hat{\beta}_{(i)}
=
\hat{\beta}
-
\frac{(X^\top X)^{-1}x_i e_i}{1-h_{ii}}.
\end{align*}
[/guided]
[/step]
[step:Evaluate the deleted residual from the coefficient formula]
By definition,
\begin{align*}
e_{i(i)}
=
y_i-x_i^\top\hat{\beta}_{(i)}.
\end{align*}
Using the coefficient formula,
\begin{align*}
x_i^\top\hat{\beta}_{(i)}
&=
x_i^\top\hat{\beta}
-
\frac{x_i^\top (X^\top X)^{-1}x_i\, e_i}{1-h_{ii}} \\
&=
x_i^\top\hat{\beta}
-
\frac{h_{ii}e_i}{1-h_{ii}}.
\end{align*}
Hence
\begin{align*}
e_{i(i)}
&=
y_i-x_i^\top\hat{\beta}
+
\frac{h_{ii}e_i}{1-h_{ii}} \\
&=
e_i+\frac{h_{ii}e_i}{1-h_{ii}} \\
&=
\frac{e_i}{1-h_{ii}}.
\end{align*}
This is the claimed deleted residual formula.
[/step]
