[proofplan] We apply the second-order Taylor expansion of $f$ at the critical point $x_0$, so the first-order term vanishes and the sign of $f(x_0+v)-f(x_0)$ is controlled by the quadratic form associated to $Hf_{x_0}$ up to an $o(|v|^2)$ error. In the positive definite case, compactness of the unit sphere gives a uniform positive lower bound for this quadratic form, and the remainder is absorbed into that bound. The negative definite case is obtained by applying the positive definite argument to $-f$. In the indefinite case, Taylor expansion along one positive and one negative direction gives nearby points with function values respectively above and below $f(x_0)$, so no local extremum is possible. [/proofplan] [step:Choose a ball around $x_0$ on which Taylor expansion applies] For $a\in\mathbb R^n$ and $r>0$, write $B(a,r):=\{x\in\mathbb R^n:|x-a|<r\}$ for the open Euclidean ball centered at $a$ with radius $r$. Since $U$ is open and $x_0 \in U$, there exists $\rho>0$ such that $B(x_0,\rho)\subset U$. For every $v\in \mathbb R^n$ with $|v|<\rho$, the line segment from $x_0$ to $x_0+v$ is contained in $B(x_0,\rho)\subset U$. Define the matrix \begin{align*} A:=Hf_{x_0}\in \mathbb R^{n\times n}. \end{align*} By the second-order Taylor expansion for $C^2$ functions, [citetheorem:8606], there is a function \begin{align*} r:B(0,\rho)\to \mathbb R \end{align*} such that, for $|v|<\rho$, \begin{align*} f(x_0+v)=f(x_0)+\nabla f(x_0)\cdot v+\frac{1}{2}v^\top A v+r(v). \end{align*} Since $x_0$ is critical, $\nabla f(x_0)=0$, and therefore \begin{align*} f(x_0+v)-f(x_0)=\frac{1}{2}v^\top A v+r(v). \end{align*} The remainder satisfies \begin{align*} \lim_{v\to 0}\frac{|r(v)|}{|v|^2}=0. \end{align*} [/step] [step:Use positive definiteness to force nearby values above $f(x_0)$] Assume that $A$ is positive definite. Let \begin{align*} S:=\{w\in\mathbb R^n:|w|=1\} \end{align*} be the Euclidean unit sphere, and define \begin{align*} q:S\to \mathbb R,\qquad w\mapsto w^\top A w. \end{align*} The map $q$ is continuous, and positive definiteness gives $q(w)>0$ for every $w\in S$. Since $S$ is compact, $q$ attains a positive minimum. Define \begin{align*} \mu:=\min_{w\in S}q(w)>0. \end{align*} For every nonzero $v\in\mathbb R^n$, the vector $v/|v|$ lies in $S$, so \begin{align*} v^\top A v=|v|^2\left(\frac{v}{|v|}\right)^\top A\left(\frac{v}{|v|}\right)\ge \mu |v|^2. \end{align*} By the remainder estimate, there exists $\delta\in(0,\rho)$ such that \begin{align*} |r(v)|\le \frac{\mu}{4}|v|^2 \end{align*} whenever $0<|v|<\delta$. Hence, for $0<|v|<\delta$, \begin{align*} f(x_0+v)-f(x_0)\ge \frac{1}{2}\mu |v|^2-\frac{\mu}{4}|v|^2=\frac{\mu}{4}|v|^2>0. \end{align*} Thus every point $x\in B(x_0,\delta)$ with $x\ne x_0$ satisfies $f(x)>f(x_0)$. Therefore $x_0$ is a strict local minimum of $f$. [guided] Assume that $A=Hf_{x_0}$ is positive definite. The goal is to upgrade the pointwise sign condition $v^\top A v>0$ for $v\ne 0$ into a uniform lower bound of the form $v^\top A v\ge \mu |v|^2$ near the origin. That uniformity is what lets us compare the quadratic Taylor term against the small remainder $r(v)$. Let $S:=\{w\in\mathbb R^n:|w|=1\}$ be the Euclidean unit sphere, and define $q:S\to \mathbb R$ by $q(w)=w^\top A w$. The map $q$ is continuous because it is a polynomial in the coordinates of $w$. Since $A$ is positive definite, $q(w)>0$ for every $w\in S$. The sphere $S$ is compact, so $q$ attains a minimum on $S$; define \begin{align*} \mu:=\min_{w\in S}q(w). \end{align*} Because every value of $q$ on $S$ is positive, this minimum satisfies $\mu>0$. Now take any nonzero $v\in\mathbb R^n$. The vector $v/|v|$ lies in $S$, and the homogeneity of the quadratic form gives \begin{align*} v^\top A v=|v|^2\left(\frac{v}{|v|}\right)^\top A\left(\frac{v}{|v|}\right)\ge \mu |v|^2. \end{align*} This is the uniform lower bound we need. The Taylor expansion established in the first step applies because $f\in C^2(U)$, $B(x_0,\rho)\subset U$, and $x_0+v\in B(x_0,\rho)$ whenever $|v|<\rho$. With $A:=Hf_{x_0}$ and with $r:B(0,\rho)\to\mathbb R$ denoting the Taylor remainder, it gives \begin{align*} f(x_0+v)-f(x_0)=\frac{1}{2}v^\top A v+r(v) \end{align*} for $|v|<\rho$, because $\nabla f(x_0)=0$. The remainder satisfies \begin{align*} \lim_{v\to 0}\frac{|r(v)|}{|v|^2}=0. \end{align*} Using this limit with threshold $\mu/4$, there exists $\delta\in(0,\rho)$ such that \begin{align*} |r(v)|\le \frac{\mu}{4}|v|^2 \end{align*} whenever $0<|v|<\delta$. Combining the lower bound for the quadratic form with this estimate gives \begin{align*} f(x_0+v)-f(x_0)\ge \frac{1}{2}\mu |v|^2-\frac{\mu}{4}|v|^2=\frac{\mu}{4}|v|^2>0 \end{align*} for every $v$ with $0<|v|<\delta$. Therefore every $x=x_0+v\in B(x_0,\delta)$ with $x\ne x_0$ satisfies $f(x)>f(x_0)$, which is exactly the definition of a strict local minimum at $x_0$. [/guided] [/step] [step:Reduce the negative definite case to the positive definite case] Assume that $A=Hf_{x_0}$ is negative definite. Define \begin{align*} g:U\to\mathbb R,\qquad x\mapsto -f(x). \end{align*} Since $f\in C^2(U)$, the function $g$ belongs to $C^2(U)$ by the algebra of $C^2$ functions, [citetheorem:8608]. Moreover $\nabla g(x_0)=-\nabla f(x_0)=0$, so $x_0$ is a critical point of $g$, and \begin{align*} Hg_{x_0}=-Hf_{x_0}=-A. \end{align*} Since $A$ is negative definite, $-A$ is positive definite. Applying the positive definite case to $g$, there exists $\delta>0$ such that $g(x)>g(x_0)$ for every $x\in B(x_0,\delta)$ with $x\ne x_0$. Equivalently, \begin{align*} -f(x)>-f(x_0), \end{align*} so $f(x)<f(x_0)$ for every such $x$. Hence $x_0$ is a strict local maximum of $f$. [/step] [step:Use two signed directions to rule out a local extremum] Assume that $A$ is indefinite. Then there exist nonzero vectors $h,k\in\mathbb R^n$ such that \begin{align*} h^\top A h>0 \end{align*} and \begin{align*} k^\top A k<0. \end{align*} Define \begin{align*} \alpha:=h^\top A h>0 \end{align*} and \begin{align*} \beta:=k^\top A k<0. \end{align*} For $t\in\mathbb R$ with $|t|\,|h|<\rho$, the Taylor formula applied to $v=th$ gives \begin{align*} f(x_0+th)-f(x_0)=\frac{1}{2}t^2\alpha+r(th). \end{align*} Since $r(v)=o(|v|^2)$ as $v\to 0$, there exists $\delta_h>0$ such that $|t|<\delta_h$ and $t\ne 0$ imply \begin{align*} |r(th)|\le \frac{\alpha}{4}t^2. \end{align*} After decreasing $\delta_h$ if necessary so that $|t|\,|h|<\rho$, we obtain \begin{align*} f(x_0+th)-f(x_0)\ge \frac{\alpha}{2}t^2-\frac{\alpha}{4}t^2=\frac{\alpha}{4}t^2>0 \end{align*} for all $0<|t|<\delta_h$. Similarly, for $t\in\mathbb R$ with $|t|\,|k|<\rho$, \begin{align*} f(x_0+tk)-f(x_0)=\frac{1}{2}t^2\beta+r(tk). \end{align*} Because $\beta<0$, there exists $\delta_k>0$ such that $|t|<\delta_k$ and $t\ne 0$ imply \begin{align*} |r(tk)|\le \frac{-\beta}{4}t^2. \end{align*} After decreasing $\delta_k$ if necessary so that $|t|\,|k|<\rho$, we get \begin{align*} f(x_0+tk)-f(x_0)\le \frac{\beta}{2}t^2+\frac{-\beta}{4}t^2=\frac{\beta}{4}t^2<0 \end{align*} for all $0<|t|<\delta_k$. Every neighbourhood of $x_0$ contains points of the form $x_0+th$ with $0<|t|<\delta_h$ and points of the form $x_0+tk$ with $0<|t|<\delta_k$. The former have values strictly larger than $f(x_0)$, and the latter have values strictly smaller than $f(x_0)$. Hence $x_0$ is neither a local minimum nor a local maximum, so $x_0$ is not a local extremum of $f$. [/step]