[proofplan]
We use the defining matrix condition for the [orthogonal group](/page/Orthogonal%20Group): $A\in O(n)$ means $A^\top A=I_n$, where $I_n$ denotes the $n\times n$ identity matrix. In one direction, this condition is substituted directly into the Euclidean [inner product](/page/Inner%20Product) formula $\langle u,v\rangle=u^\top v$. In the reverse direction, preservation of all inner products implies that the [bilinear form](/page/Bilinear%20Form) associated to $A^\top A-I_n$ vanishes on every pair of vectors; evaluating on the standard basis then forces every matrix entry of $A^\top A-I_n$ to be zero.
[/proofplan]
[step:Use the orthogonality equation to preserve the Euclidean inner product]
Assume $A\in O(n)$. By the definition of the orthogonal group, this means
\begin{align*}
A^\top A=I_n.
\end{align*}
Let $x,y\in\mathbb{R}^n$. Since the Euclidean inner product is given by $\langle u,v\rangle=u^\top v$ for $u,v\in\mathbb{R}^n$, we compute
\begin{align*}
\langle Ax,Ay\rangle=(Ax)^\top Ay.
\end{align*}
Using the transpose identity $(Ax)^\top=x^\top A^\top$, this becomes
\begin{align*}
\langle Ax,Ay\rangle=x^\top A^\top Ay.
\end{align*}
Substituting $A^\top A=I_n$ gives
\begin{align*}
\langle Ax,Ay\rangle=x^\top I_n y.
\end{align*}
Since $I_n y=y$, we obtain
\begin{align*}
\langle Ax,Ay\rangle=x^\top y=\langle x,y\rangle.
\end{align*}
Thus $A$ preserves the Euclidean inner product on every pair of vectors.
[/step]
[step:Convert preservation of inner products into a vanishing bilinear form]
Conversely, assume that
\begin{align*}
\langle Ax,Ay\rangle=\langle x,y\rangle
\end{align*}
for every $x,y\in\mathbb{R}^n$. Define the matrix $B\in M_n(\mathbb{R})$ by
\begin{align*}
B:=A^\top A-I_n.
\end{align*}
For arbitrary $x,y\in\mathbb{R}^n$, the same inner product computation gives
\begin{align*}
\langle Ax,Ay\rangle=x^\top A^\top Ay.
\end{align*}
Also,
\begin{align*}
\langle x,y\rangle=x^\top I_n y.
\end{align*}
Subtracting these equalities and using the assumed preservation of inner products yields
\begin{align*}
x^\top By=0
\end{align*}
for every $x,y\in\mathbb{R}^n$.
[guided]
We now turn the geometric-looking hypothesis into a concrete matrix equation. The hypothesis says that applying $A$ to both vectors does not change their Euclidean inner product:
\begin{align*}
\langle Ax,Ay\rangle=\langle x,y\rangle
\end{align*}
for every $x,y\in\mathbb{R}^n$.
The standard Euclidean inner product on $\mathbb{R}^n$ is computed by the matrix formula $\langle u,v\rangle=u^\top v$. Therefore, for arbitrary $x,y\in\mathbb{R}^n$,
\begin{align*}
\langle Ax,Ay\rangle=(Ax)^\top Ay.
\end{align*}
Using the transpose rule $(Ax)^\top=x^\top A^\top$, this becomes
\begin{align*}
\langle Ax,Ay\rangle=x^\top A^\top Ay.
\end{align*}
On the other hand,
\begin{align*}
\langle x,y\rangle=x^\top I_n y.
\end{align*}
Define the matrix $B\in M_n(\mathbb{R})$ by
\begin{align*}
B:=A^\top A-I_n.
\end{align*}
This definition isolates exactly the obstruction to $A$ being orthogonal: proving $B=0$ is the same as proving $A^\top A=I_n$. Since the two inner products are equal by hypothesis, subtracting the two matrix expressions gives
\begin{align*}
x^\top A^\top Ay-x^\top I_n y=0.
\end{align*}
Factoring the common row vector $x^\top$ on the left and the common column vector $y$ on the right gives
\begin{align*}
x^\top(A^\top A-I_n)y=0.
\end{align*}
By the definition of $B$, this is
\begin{align*}
x^\top By=0
\end{align*}
for every $x,y\in\mathbb{R}^n$.
[/guided]
[/step]
[step:Evaluate on standard basis vectors to force the matrix obstruction to vanish]
For each $i\in\{1,\dots,n\}$, let $e_i\in\mathbb{R}^n$ denote the $i$-th standard basis vector. Since $x^\top By=0$ for every $x,y\in\mathbb{R}^n$, we may take $x=e_i$ and $y=e_j$ for arbitrary $i,j\in\{1,\dots,n\}$. This gives
\begin{align*}
e_i^\top B e_j=0.
\end{align*}
By the definition of matrix entries with respect to the standard basis,
\begin{align*}
e_i^\top B e_j=B_{ij}.
\end{align*}
Hence $B_{ij}=0$ for every $i,j\in\{1,\dots,n\}$. Therefore $B=0$, so
\begin{align*}
A^\top A-I_n=0.
\end{align*}
Thus
\begin{align*}
A^\top A=I_n.
\end{align*}
By the definition of $O(n)$, this implies $A\in O(n)$.
[/step]
[step:Combine the two implications]
We have proved that $A\in O(n)$ implies preservation of the Euclidean inner product on all pairs of vectors in $\mathbb{R}^n$, and that preservation of the Euclidean inner product on all pairs of vectors implies $A^\top A=I_n$, hence $A\in O(n)$. Therefore $A\in O(n)$ if and only if
\begin{align*}
\langle Ax,Ay\rangle=\langle x,y\rangle
\end{align*}
for every $x,y\in\mathbb{R}^n$.
[/step]
