[proofplan]
We identify $M_n(\mathbb{R})$ with $\mathbb{R}^{n^2}$ by listing matrix entries and use the Euclidean topology. The set $O(n)$ is the inverse image of the singleton $\{I_n\}$ under the continuous map $A\mapsto A^\top A$, so it is closed. Each orthogonal matrix has columns of Euclidean norm $1$, which bounds every entry and hence bounds the whole set in $\mathbb{R}^{n^2}$. The finite-dimensional [Heine-Borel theorem](/theorems/309) then gives compactness.
[/proofplan]
[step:Identify $O(n)$ as the inverse image of $I_n$ under a continuous matrix map]
Define the map
\begin{align*}
F:M_n(\mathbb{R})\to M_n(\mathbb{R})
\end{align*}
by $F(A)=A^\top A$ for every $A\in M_n(\mathbb{R})$. For $A=(A_{ij})_{1\leq i,j\leq n}$, the $(i,j)$-entry of $F(A)$ is
\begin{align*}
F(A)_{ij}=\sum_{k=1}^{n} A_{ki}A_{kj}.
\end{align*}
Thus each coordinate function $A\mapsto F(A)_{ij}$ is a polynomial in the matrix entries, so $F$ is continuous for the Euclidean topology on $M_n(\mathbb{R})\cong\mathbb{R}^{n^2}$.
By definition of the [orthogonal group](/page/Orthogonal%20Group),
\begin{align*}
O(n)=F^{-1}(\{I_n\}).
\end{align*}
The singleton $\{I_n\}$ is closed in the Euclidean space $M_n(\mathbb{R})$. Since $F$ is continuous, $O(n)$ is closed in $M_n(\mathbb{R})$.
[guided]
The point of this step is to express the algebraic condition $A^\top A=I_n$ as a closed condition in Euclidean space. Define
\begin{align*}
F:M_n(\mathbb{R})\to M_n(\mathbb{R})
\end{align*}
by $F(A)=A^\top A$ for every matrix $A\in M_n(\mathbb{R})$.
We verify continuity directly in coordinates. Write $A=(A_{ij})_{1\leq i,j\leq n}$. Matrix multiplication gives, for each pair of indices $1\leq i,j\leq n$,
\begin{align*}
F(A)_{ij}=(A^\top A)_{ij}=\sum_{k=1}^{n} A_{ki}A_{kj}.
\end{align*}
This is a finite sum of products of coordinate functions on $\mathbb{R}^{n^2}$. Products and finite sums of continuous real-valued functions are continuous, so every coordinate function of $F$ is continuous. Therefore $F$ is continuous as a map between finite-dimensional Euclidean spaces.
Now the definition of the orthogonal group is exactly
\begin{align*}
O(n)=\{A\in M_n(\mathbb{R}):A^\top A=I_n\}=F^{-1}(\{I_n\}).
\end{align*}
The set $\{I_n\}$ is a singleton in the Euclidean space $M_n(\mathbb{R})$, hence it is closed. Since the inverse image of a [closed set](/page/Closed%20Set) under a continuous map is closed, $O(n)$ is closed in $M_n(\mathbb{R})$.
[/guided]
[/step]
[step:Bound every matrix entry using the unit length of the columns]
Let $A=(A_{ij})_{1\leq i,j\leq n}\in O(n)$. For each $j\in\{1,\dots,n\}$, the equality $A^\top A=I_n$ gives
\begin{align*}
1=(A^\top A)_{jj}=\sum_{i=1}^{n} A_{ij}^2.
\end{align*}
Since each summand $A_{ij}^2$ is non-negative, we have $A_{ij}^2\leq 1$ for every $i,j$, and hence $|A_{ij}|\leq 1$ for every $i,j$.
Equivalently, under the standard identification $M_n(\mathbb{R})\cong\mathbb{R}^{n^2}$, every element of $O(n)$ lies in the cube $[-1,1]^{n^2}$. Therefore $O(n)$ is bounded in $M_n(\mathbb{R})$.
[/step]
[step:Apply finite-dimensional Heine-Borel to conclude compactness]
We have shown that $O(n)$ is closed and bounded in the finite-dimensional Euclidean space $M_n(\mathbb{R})\cong\mathbb{R}^{n^2}$. By the finite-dimensional [Heine-Borel theorem](/theorems/315), every closed and bounded subset of $\mathbb{R}^{n^2}$ is compact. Therefore $O(n)$ is compact as a subset of $M_n(\mathbb{R})$. This proves the theorem.
[/step]