[proofplan] We first average an arbitrary positive definite [inner product](/page/Inner%20Product) over the compact group to obtain a positive definite inner product invariant under the representation. Once such an invariant inner product is available, every invariant subspace has an invariant orthogonal complement. The proof then proceeds by induction on $\dim_{\mathbb K}V$: if $V$ is not irreducible, split it as $W\oplus W^\perp$ and decompose the two smaller invariant summands. [/proofplan]

[step:Average an inner product to make the representation unitary]Let \begin{align*} h_0:V\times V\to\mathbb K \end{align*} be a positive definite inner product when $\mathbb K=\mathbb R$, and a positive definite Hermitian inner product, linear in the first variable, when $\mathbb K=\mathbb C$. Let $\nu$ denote the normalized left Haar measure on the compact Lie group $G$, so $\nu(G)=1$. Define \begin{align*} h:V\times V&\to\mathbb K \end{align*} by \begin{align*} h(v,w)=\int_G h_0(\rho(g^{-1})v,\rho(g^{-1})w)\,d\nu(g). \end{align*} For fixed $v,w\in V$, the map \begin{align*} G&\to\mathbb K \end{align*} given by \begin{align*} g&\mapsto h_0(\rho(g^{-1})v,\rho(g^{-1})w) \end{align*} is continuous because inversion on $G$, the representation $\rho$, the action map $GL(V)\times V\to V$, and $h_0$ are continuous. Hence the integral is well-defined. Linearity in the first variable and conjugate-linearity in the second variable in the complex case, or bilinearity in the real case, follows from the corresponding property of $h_0$ and linearity of the integral. For $v\in V$, \begin{align*} h(v,v)=\int_G h_0(\rho(g^{-1})v,\rho(g^{-1})v)\,d\nu(g)\geq 0. \end{align*} If $v\neq 0$, then $\rho(g^{-1})v\neq 0$ for every $g\in G$, since $\rho(g^{-1})$ is invertible. Therefore the [continuous function](/page/Continuous%20Function) \begin{align*} g\mapsto h_0(\rho(g^{-1})v,\rho(g^{-1})v) \end{align*} is strictly positive on $G$, so its integral against the normalized Haar measure is positive. Thus $h$ is positive definite. We now prove $G$-invariance. Let $a\in G$ and $v,w\in V$. Since $\rho$ is a homomorphism, \begin{align*} \rho(g^{-1})\rho(a)=\rho(g^{-1}a). \end{align*} Thus \begin{align*} h(\rho(a)v,\rho(a)w)=\int_G h_0(\rho(g^{-1}a)v,\rho(g^{-1}a)w)\,d\nu(g). \end{align*} Under the substitution $k=a^{-1}g$, equivalently $g=ak$, left invariance of $\nu$ gives $d\nu(g)=d\nu(k)$, and \begin{align*} g^{-1}a=(ak)^{-1}a=k^{-1}. \end{align*} Hence \begin{align*} h(\rho(a)v,\rho(a)w)=\int_G h_0(\rho(k^{-1})v,\rho(k^{-1})w)\,d\nu(k)=h(v,w). \end{align*} So $h$ is a $G$-invariant positive definite inner product on $V$.[/step]

[guided]The purpose of this step is to replace an arbitrary inner product by one that is compatible with the [group action](/page/Group%20Action). Choose a positive definite form \begin{align*} h_0:V\times V\to\mathbb K, \end{align*} where this means a real inner product if $\mathbb K=\mathbb R$ and a positive definite Hermitian inner product, linear in the first variable, if $\mathbb K=\mathbb C$. Let $\nu$ be the normalized left Haar measure on the compact Lie group $G$, so $\nu(G)=1$. Define a new form \begin{align*} h:V\times V&\to\mathbb K \end{align*} by averaging the pullbacks of $h_0$: \begin{align*} h(v,w)=\int_G h_0(\rho(g^{-1})v,\rho(g^{-1})w)\,d\nu(g). \end{align*} This integral is legitimate because, for fixed $v,w\in V$, the integrand is the continuous map \begin{align*} G&\to\mathbb K \end{align*} defined by \begin{align*} g&\mapsto h_0(\rho(g^{-1})v,\rho(g^{-1})w). \end{align*} Continuity follows from the continuity of inversion on the Lie group $G$, the smoothness of $\rho$, the continuity of the evaluation action of $GL(V)$ on $V$, and the continuity of $h_0$. The algebraic properties of $h$ come directly from the same properties of $h_0$ and the linearity of integration. Thus $h$ is bilinear in the real case, and Hermitian with linear first variable in the complex case. To check positive definiteness, take $v\in V$. Then \begin{align*} h(v,v)=\int_G h_0(\rho(g^{-1})v,\rho(g^{-1})v)\,d\nu(g)\geq 0. \end{align*} If $v\neq 0$, every vector $\rho(g^{-1})v$ is nonzero because $\rho(g^{-1})\in GL(V)$. Therefore \begin{align*} h_0(\rho(g^{-1})v,\rho(g^{-1})v)>0 \end{align*} for every $g\in G$. Since the integrand is continuous and strictly positive on the compact group $G$, its integral against the normalized Haar measure is strictly positive. Hence $h(v,v)>0$ for $v\neq 0$. It remains to prove invariance. Fix $a\in G$ and $v,w\in V$. Using the homomorphism identity for $\rho$, we compute \begin{align*} h(\rho(a)v,\rho(a)w)=\int_G h_0(\rho(g^{-1})\rho(a)v,\rho(g^{-1})\rho(a)w)\,d\nu(g). \end{align*} Since $\rho(g^{-1})\rho(a)=\rho(g^{-1}a)$, this becomes \begin{align*} h(\rho(a)v,\rho(a)w)=\int_G h_0(\rho(g^{-1}a)v,\rho(g^{-1}a)w)\,d\nu(g). \end{align*} Now use the substitution $k=a^{-1}g$, or $g=ak$. Left invariance of Haar measure gives $d\nu(g)=d\nu(k)$, and the group calculation gives \begin{align*} g^{-1}a=(ak)^{-1}a=k^{-1}. \end{align*} Therefore \begin{align*} h(\rho(a)v,\rho(a)w)=\int_G h_0(\rho(k^{-1})v,\rho(k^{-1})w)\,d\nu(k)=h(v,w). \end{align*} Thus $h$ is invariant under the action of every $a\in G$.[/guided]

[step:Show orthogonal complements of subrepresentations are subrepresentations] Let $W\subset V$ be a $\rho$-invariant subspace. Define its $h$-orthogonal complement by \begin{align*} W^\perp=\{v\in V:h(v,w)=0\text{ for every }w\in W\}. \end{align*} We claim that $W^\perp$ is $\rho$-invariant. Let $a\in G$, let $v\in W^\perp$, and let $w\in W$. Since $W$ is $\rho$-invariant, $\rho(a^{-1})w\in W$. Using $G$-invariance of $h$ with the element $a^{-1}$, we have \begin{align*} h(\rho(a)v,w)=h(v,\rho(a^{-1})w)=0. \end{align*} Since this holds for every $w\in W$, we get $\rho(a)v\in W^\perp$. Thus $W^\perp$ is a subrepresentation. Because $h$ is positive definite and $V$ is finite-dimensional, the standard [orthogonal decomposition](/theorems/436) for finite-dimensional inner product spaces gives \begin{align*} V=W\oplus W^\perp. \end{align*} [/step]

[step:Induct on the dimension of the representation] We prove the theorem by induction on \begin{align*} d=\dim_{\mathbb K}V. \end{align*} If $d=0$, then $V$ is the empty direct sum of irreducible subrepresentations. If $d=1$, then $V$ is irreducible, because its only $\mathbb K$-linear subspaces are $0$ and $V$. Assume $d\geq 2$ and assume the theorem holds for all smooth representations of compact Lie groups on $\mathbb K$-vector spaces of dimension strictly smaller than $d$. If $V$ is irreducible, then \begin{align*} V=V \end{align*} is already a direct sum of one irreducible subrepresentation. Suppose instead that $V$ is not irreducible. Then there exists a $\rho$-invariant subspace $W\subset V$ such that \begin{align*} 0\neq W\neq V. \end{align*} By the previous step, \begin{align*} V=W\oplus W^\perp, \end{align*} and both $W$ and $W^\perp$ are $\rho$-invariant. Define the restricted representations \begin{align*} \rho_W:G&\to GL(W) \end{align*} and \begin{align*} \rho_{W^\perp}:G&\to GL(W^\perp) \end{align*} by \begin{align*} \rho_W(g)&=\rho(g)|_W \end{align*} and \begin{align*} \rho_{W^\perp}(g)&=\rho(g)|_{W^\perp}. \end{align*} These maps are smooth representations because they are restrictions of the smooth representation $\rho$ to invariant finite-dimensional subspaces. Since $0<\dim_{\mathbb K}W<d$ and $0<\dim_{\mathbb K}W^\perp<d$, the induction hypothesis gives irreducible subrepresentations $W_1,\dots,W_r\subset W$ and $U_1,\dots,U_s\subset W^\perp$ such that \begin{align*} W=W_1\oplus\cdots\oplus W_r \end{align*} and \begin{align*} W^\perp=U_1\oplus\cdots\oplus U_s. \end{align*} Combining these decompositions with $V=W\oplus W^\perp$ gives \begin{align*} V=W_1\oplus\cdots\oplus W_r\oplus U_1\oplus\cdots\oplus U_s. \end{align*} Each summand is irreducible as a subrepresentation of $V$, because the action on it is the same restricted action used in the decomposition of $W$ or $W^\perp$. This completes the induction and proves the theorem. [/step]