[proofplan] We prove the contrapositive. If $g$ and $h$ are not conjugate, their conjugacy classes are disjoint compact subsets of the compact [Hausdorff space](/page/Hausdorff%20Space) $G$, so a [continuous function](/page/Continuous%20Function) separates them. Averaging that function over conjugation produces a continuous class function taking different values at $g$ and $h$. Finally, Peter-Weyl density and [Schur's lemma](/theorems/2414) imply that finite linear combinations of irreducible characters are uniformly dense in the continuous class functions, so some irreducible character must distinguish $g$ from $h$. [/proofplan]

[step:Separate the two conjugacy classes by a continuous function] Let $\mu$ denote the normalized Haar probability measure on $G$. For $x\in G$, define the [conjugacy class](/page/Conjugacy%20Class) \begin{align*} \operatorname{Cl}_G(x):=\{axa^{-1}:a\in G\}. \end{align*} Assume, contrapositively, that $g$ and $h$ are not conjugate. Then \begin{align*} \operatorname{Cl}_G(g)\cap \operatorname{Cl}_G(h)=\varnothing. \end{align*} For each $x\in G$, the map \begin{align*} \gamma_x:G&\to G \end{align*} \begin{align*} a&\mapsto axa^{-1} \end{align*} is continuous. Since $G$ is compact, both $\operatorname{Cl}_G(g)$ and $\operatorname{Cl}_G(h)$ are compact. Since compact Lie groups are Hausdorff, these conjugacy classes are closed, disjoint subsets of the [normal topological space](/page/Normal%20Topological%20Space) $G$. By [Urysohn's lemma](/theorems/887), applied to the two closed sets $\operatorname{Cl}_G(g)$ and $\operatorname{Cl}_G(h)$, there exists a continuous function \begin{align*} F:G&\to [0,1] \end{align*} such that \begin{align*} F(x)=1\quad\text{for every }x\in \operatorname{Cl}_G(g) \end{align*} and \begin{align*} F(x)=0\quad\text{for every }x\in \operatorname{Cl}_G(h). \end{align*} Here Urysohn's lemma is being used as a standard topological separation result not yet resolved to a theorem entry in the wiki. [/step]

[step:Average the separating function over conjugation]Define the averaged function \begin{align*} f:G&\to \mathbb C \end{align*} \begin{align*} y&\mapsto \int_G F(aya^{-1})\,d\mu(a). \end{align*} The map $(a,y)\mapsto aya^{-1}$ from $G\times G$ to $G$ is continuous, and $F$ is continuous, so the integrand is continuous on the [compact space](/page/Compact%20Space) $G\times G$. Therefore $f$ is continuous. For every $b\in G$ and every $y\in G$, left invariance of $\mu$ gives \begin{align*} f(byb^{-1})=\int_G F(ab y b^{-1}a^{-1})\,d\mu(a). \end{align*} Using the substitution $c=ab$, the left invariance of Haar measure gives $d\mu(c)=d\mu(a)$, and hence \begin{align*} f(byb^{-1})=\int_G F(cyc^{-1})\,d\mu(c)=f(y). \end{align*} Thus $f$ is a continuous class function. Moreover, for every $a\in G$, the element $aga^{-1}$ belongs to $\operatorname{Cl}_G(g)$, so $F(aga^{-1})=1$. Since $\mu(G)=1$, \begin{align*} f(g)=\int_G 1\,d\mu(a)=1. \end{align*} Similarly, $aha^{-1}\in \operatorname{Cl}_G(h)$ for every $a\in G$, so \begin{align*} f(h)=\int_G 0\,d\mu(a)=0. \end{align*} Therefore $f(g)\ne f(h)$.[/step]

[guided]The purpose of averaging is to turn the separating function $F$ into a class function without losing the separation. Define \begin{align*} f:G&\to \mathbb C \end{align*} \begin{align*} y&\mapsto \int_G F(aya^{-1})\,d\mu(a). \end{align*} This integral is with respect to the normalized Haar probability measure $\mu$ on $G$. The function under the integral is continuous in both variables because multiplication, inversion, and $F$ are continuous. Since $G$ is compact, integration against the finite measure $\mu$ preserves continuity in the parameter $y$, so $f$ is continuous. We now check that $f$ is invariant under conjugation. Let $b\in G$ and $y\in G$. Then \begin{align*} f(byb^{-1})=\int_G F(ab y b^{-1}a^{-1})\,d\mu(a). \end{align*} The expression inside $F$ becomes a conjugate of $y$ by the element $c=ab$. Since $\mu$ is left-invariant, the substitution $c=ab$ preserves Haar measure: \begin{align*} d\mu(c)=d\mu(a). \end{align*} Therefore \begin{align*} f(byb^{-1})=\int_G F(cyc^{-1})\,d\mu(c)=f(y). \end{align*} This proves that $f$ is a class function. Finally, the averaging has not changed the values on the two conjugacy classes. For every $a\in G$, the element $aga^{-1}$ lies in $\operatorname{Cl}_G(g)$, where $F$ is identically $1$. Hence \begin{align*} f(g)=\int_G F(aga^{-1})\,d\mu(a)=\int_G 1\,d\mu(a)=1. \end{align*} Likewise, for every $a\in G$, the element $aha^{-1}$ lies in $\operatorname{Cl}_G(h)$, where $F$ is identically $0$. Thus \begin{align*} f(h)=\int_G F(aha^{-1})\,d\mu(a)=\int_G 0\,d\mu(a)=0. \end{align*} So the averaged function is continuous, conjugation-invariant, and still separates $g$ from $h$.[/guided]

[step:Show that averaged irreducible matrix coefficients are multiples of characters] Let $\pi:G\to U(V)$ be an irreducible finite-dimensional continuous unitary complex representation, and let $d:=\dim_{\mathbb C}V$. Let $(\cdot,\cdot)_V$ be the Hermitian [inner product](/page/Inner%20Product) on $V$, linear in the first argument. For $v,w\in V$, define the matrix coefficient \begin{align*} m_{v,w}:G&\to \mathbb C \end{align*} \begin{align*} y&\mapsto (\pi(y)v,w)_V. \end{align*} Define its conjugation average \begin{align*} P m_{v,w}:G&\to \mathbb C \end{align*} \begin{align*} y&\mapsto \int_G m_{v,w}(aya^{-1})\,d\mu(a). \end{align*} We claim that \begin{align*} P m_{v,w}(y)=\frac{(v,w)_V}{d}\chi_\pi(y) \end{align*} for every $y\in G$. Let $A_{v,w}:V\to V$ be the rank-one [linear map](/page/Linear%20Map) \begin{align*} A_{v,w}(u):=(u,w)_Vv. \end{align*} For every $y\in G$, \begin{align*} m_{v,w}(y)=\operatorname{tr}(A_{v,w}\pi(y)). \end{align*} Define \begin{align*} B_{v,w}:V&\to V \end{align*} \begin{align*} u&\mapsto \int_G \pi(a)^{-1}A_{v,w}\pi(a)u\,d\mu(a). \end{align*} Then \begin{align*} P m_{v,w}(y)=\operatorname{tr}(B_{v,w}\pi(y)). \end{align*} For each $b\in G$, invariance of Haar measure gives \begin{align*} \pi(b)^{-1}B_{v,w}\pi(b)=B_{v,w}. \end{align*} Thus $B_{v,w}$ commutes with $\pi(b)$ for every $b\in G$. Since $\pi$ is irreducible, Schur's lemma gives a scalar $\lambda\in\mathbb C$ such that \begin{align*} B_{v,w}=\lambda I_V. \end{align*} Here Schur's lemma is being used as a standard representation-theoretic result not yet resolved to a theorem entry in the wiki. Taking traces and using cyclicity of trace, \begin{align*} \operatorname{tr}(B_{v,w})=\int_G \operatorname{tr}(\pi(a)^{-1}A_{v,w}\pi(a))\,d\mu(a)=\int_G \operatorname{tr}(A_{v,w})\,d\mu(a)=\operatorname{tr}(A_{v,w}). \end{align*} Since $\operatorname{tr}(B_{v,w})=\lambda d$ and $\operatorname{tr}(A_{v,w})=(v,w)_V$, we have \begin{align*} \lambda=\frac{(v,w)_V}{d}. \end{align*} Therefore \begin{align*} P m_{v,w}(y)=\operatorname{tr}(\lambda I_V\pi(y))=\frac{(v,w)_V}{d}\chi_\pi(y). \end{align*} [/step]

[step:Approximate every continuous class function by irreducible characters] Let $C(G)$ denote the complex Banach [algebra of continuous functions](/theorems/197) $G\to\mathbb C$ with the [uniform norm](/page/Uniform%20Norm) \begin{align*} \|u\|_\infty:=\sup_{y\in G}|u(y)|. \end{align*} Let $C(G)^G$ denote the closed subspace of continuous class functions: \begin{align*} C(G)^G:=\{u\in C(G):u(aya^{-1})=u(y)\text{ for every }a,y\in G\}. \end{align*} Define the conjugation-averaging operator \begin{align*} P:C(G)&\to C(G)^G \end{align*} \begin{align*} u&\mapsto \left(y\mapsto \int_G u(aya^{-1})\,d\mu(a)\right). \end{align*} The operator $P$ is linear and satisfies \begin{align*} \|Pu\|_\infty\le \|u\|_\infty \end{align*} for every $u\in C(G)$, because $\mu(G)=1$. If $u\in C(G)^G$, then $Pu=u$. By the [Peter-Weyl Theorem][citetheorem:8833], the linear span of all irreducible unitary matrix coefficients is uniformly dense in $C(G)$. Let $u\in C(G)^G$ and let $\varepsilon>0$. Choose a finite linear combination $r$ of irreducible unitary matrix coefficients such that \begin{align*} \|u-r\|_\infty<\varepsilon. \end{align*} Since $Pu=u$ and $P$ is contractive, \begin{align*} \|u-Pr\|_\infty=\|P(u-r)\|_\infty\le \|u-r\|_\infty<\varepsilon. \end{align*} By the previous step, $Pr$ is a finite linear combination of irreducible characters. Therefore finite linear combinations of irreducible characters are uniformly dense in $C(G)^G$. [/step]

[step:Derive the contradiction from uniform approximation] Assume that \begin{align*} \chi_\pi(g)=\chi_\pi(h) \end{align*} for every irreducible finite-dimensional continuous unitary complex representation $\pi$ of $G$. Then every finite linear combination of irreducible characters has the same value at $g$ and $h$. Let $f\in C(G)^G$ be the continuous class function constructed above, so that $f(g)=1$ and $f(h)=0$. By the density just proved, for $\varepsilon:=1/3$ there exists a finite linear combination $q:G\to\mathbb C$ of irreducible characters such that \begin{align*} \|f-q\|_\infty<\frac{1}{3}. \end{align*} Since $q(g)=q(h)$, the triangle inequality gives \begin{align*} 1=|f(g)-f(h)|\le |f(g)-q(g)|+|q(h)-f(h)|<\frac{2}{3}, \end{align*} which is impossible. Hence the contraposition assumption was false. Therefore $g$ and $h$ are conjugate in $G$. [/step]