[proofplan] We use the defining matrix condition for the [orthogonal group](/page/Orthogonal%20Group): $A\in O(n)$ means $A^\top A=I_n$, where $I_n$ denotes the $n\times n$ identity matrix. In one direction, this condition is substituted directly into the Euclidean [inner product](/page/Inner%20Product) formula $\langle u,v\rangle=u^\top v$. In the reverse direction, preservation of all inner products implies that the [bilinear form](/page/Bilinear%20Form) associated to $A^\top A-I_n$ vanishes on every pair of vectors; evaluating on the standard basis then forces every matrix entry of $A^\top A-I_n$ to be zero. [/proofplan]

[step:Use the orthogonality equation to preserve the Euclidean inner product] Assume $A\in O(n)$. By the definition of the orthogonal group, this means \begin{align*} A^\top A=I_n. \end{align*} Let $x,y\in\mathbb{R}^n$. Since the Euclidean inner product is given by $\langle u,v\rangle=u^\top v$ for $u,v\in\mathbb{R}^n$, we compute \begin{align*} \langle Ax,Ay\rangle=(Ax)^\top Ay. \end{align*} Using the transpose identity $(Ax)^\top=x^\top A^\top$, this becomes \begin{align*} \langle Ax,Ay\rangle=x^\top A^\top Ay. \end{align*} Substituting $A^\top A=I_n$ gives \begin{align*} \langle Ax,Ay\rangle=x^\top I_n y. \end{align*} Since $I_n y=y$, we obtain \begin{align*} \langle Ax,Ay\rangle=x^\top y=\langle x,y\rangle. \end{align*} Thus $A$ preserves the Euclidean inner product on every pair of vectors. [/step]

[step:Convert preservation of inner products into a vanishing bilinear form]Conversely, assume that \begin{align*} \langle Ax,Ay\rangle=\langle x,y\rangle \end{align*} for every $x,y\in\mathbb{R}^n$. Define the matrix $B\in M_n(\mathbb{R})$ by \begin{align*} B:=A^\top A-I_n. \end{align*} For arbitrary $x,y\in\mathbb{R}^n$, the same inner product computation gives \begin{align*} \langle Ax,Ay\rangle=x^\top A^\top Ay. \end{align*} Also, \begin{align*} \langle x,y\rangle=x^\top I_n y. \end{align*} Subtracting these equalities and using the assumed preservation of inner products yields \begin{align*} x^\top By=0 \end{align*} for every $x,y\in\mathbb{R}^n$.[/step]

[guided]We now turn the geometric-looking hypothesis into a concrete matrix equation. The hypothesis says that applying $A$ to both vectors does not change their Euclidean inner product: \begin{align*} \langle Ax,Ay\rangle=\langle x,y\rangle \end{align*} for every $x,y\in\mathbb{R}^n$. The standard Euclidean inner product on $\mathbb{R}^n$ is computed by the matrix formula $\langle u,v\rangle=u^\top v$. Therefore, for arbitrary $x,y\in\mathbb{R}^n$, \begin{align*} \langle Ax,Ay\rangle=(Ax)^\top Ay. \end{align*} Using the transpose rule $(Ax)^\top=x^\top A^\top$, this becomes \begin{align*} \langle Ax,Ay\rangle=x^\top A^\top Ay. \end{align*} On the other hand, \begin{align*} \langle x,y\rangle=x^\top I_n y. \end{align*} Define the matrix $B\in M_n(\mathbb{R})$ by \begin{align*} B:=A^\top A-I_n. \end{align*} This definition isolates exactly the obstruction to $A$ being orthogonal: proving $B=0$ is the same as proving $A^\top A=I_n$. Since the two inner products are equal by hypothesis, subtracting the two matrix expressions gives \begin{align*} x^\top A^\top Ay-x^\top I_n y=0. \end{align*} Factoring the common row vector $x^\top$ on the left and the common column vector $y$ on the right gives \begin{align*} x^\top(A^\top A-I_n)y=0. \end{align*} By the definition of $B$, this is \begin{align*} x^\top By=0 \end{align*} for every $x,y\in\mathbb{R}^n$.[/guided]

[step:Evaluate on standard basis vectors to force the matrix obstruction to vanish] For each $i\in\{1,\dots,n\}$, let $e_i\in\mathbb{R}^n$ denote the $i$-th standard basis vector. Since $x^\top By=0$ for every $x,y\in\mathbb{R}^n$, we may take $x=e_i$ and $y=e_j$ for arbitrary $i,j\in\{1,\dots,n\}$. This gives \begin{align*} e_i^\top B e_j=0. \end{align*} By the definition of matrix entries with respect to the standard basis, \begin{align*} e_i^\top B e_j=B_{ij}. \end{align*} Hence $B_{ij}=0$ for every $i,j\in\{1,\dots,n\}$. Therefore $B=0$, so \begin{align*} A^\top A-I_n=0. \end{align*} Thus \begin{align*} A^\top A=I_n. \end{align*} By the definition of $O(n)$, this implies $A\in O(n)$. [/step]

[step:Combine the two implications] We have proved that $A\in O(n)$ implies preservation of the Euclidean inner product on all pairs of vectors in $\mathbb{R}^n$, and that preservation of the Euclidean inner product on all pairs of vectors implies $A^\top A=I_n$, hence $A\in O(n)$. Therefore $A\in O(n)$ if and only if \begin{align*} \langle Ax,Ay\rangle=\langle x,y\rangle \end{align*} for every $x,y\in\mathbb{R}^n$. [/step]