[proofplan]
We construct local orthonormal frames by applying the [Gram-Schmidt process](/theorems/435) smoothly to arbitrary smooth local frames. A local orthonormal frame identifies each orthonormal frame over the same base point with a unique orthogonal matrix, giving local trivialisations by $U\times O(r)$. These trivialisations are compatible with the right $O(r)$-action and have $O(r)$-valued transition functions, so they define a smooth principal $O(r)$-subbundle of the full frame bundle.
[/proofplan]
[step:Construct smooth local orthonormal frames by Gram-Schmidt]
Let $x_0\in M$. Since $E\to M$ is a smooth vector bundle of rank $r$, there is an open neighbourhood $U\subset M$ of $x_0$ and a smooth local frame $(e_1,\dots,e_r)$ on $U$, where each $e_i:U\to E$ is a smooth section and $(e_1(x),\dots,e_r(x))$ is a basis of $E_x$ for every $x\in U$.
Define sections $u_1,\dots,u_r:U\to E$ inductively. First set
\begin{align*}
u_1(x)=\frac{e_1(x)}{\sqrt{h_x(e_1(x),e_1(x))}}
\end{align*}
for $x\in U$. Since $e_1(x)\neq 0$ and $h_x$ is positive definite, the denominator is a positive smooth function on $U$, so $u_1$ is smooth.
Assume $u_1,\dots,u_{k-1}$ have been constructed for some $2\le k\le r$. Define the section $v_k:U\to E$ by
\begin{align*}
v_k(x)=e_k(x)-\sum_{j=1}^{k-1}h_x(e_k(x),u_j(x))u_j(x).
\end{align*}
For each $x\in U$, the vector $v_k(x)$ is the component of $e_k(x)$ orthogonal to $\operatorname{span}(u_1(x),\dots,u_{k-1}(x))$. It is nonzero, because otherwise $e_k(x)$ would lie in $\operatorname{span}(e_1(x),\dots,e_{k-1}(x))$, contradicting the [linear independence](/page/Linear%20Independence) of the local frame. Hence $h_x(v_k(x),v_k(x))>0$ for every $x\in U$. Define
\begin{align*}
u_k(x)=\frac{v_k(x)}{\sqrt{h_x(v_k(x),v_k(x))}}.
\end{align*}
The smoothness of $h$, of $e_k$, and of the previously constructed $u_j$ implies that $v_k$ and $u_k$ are smooth. By construction, $(u_1(x),\dots,u_r(x))$ is an $h_x$-[orthonormal basis](/page/Orthonormal%20Basis) of $E_x$ for every $x\in U$.
[guided]
We begin with an arbitrary smooth local frame and turn it into an orthonormal one without losing smoothness. Let $x_0\in M$. Since $E\to M$ is a smooth rank-$r$ vector bundle, there is an open neighbourhood $U\subset M$ of $x_0$ and smooth sections $e_i:U\to E$ for $1\le i\le r$ such that $(e_1(x),\dots,e_r(x))$ is a basis of $E_x$ for every $x\in U$.
The fibrewise Gram-Schmidt process is legitimate here because every denominator that appears is a positive smooth function. Define
\begin{align*}
u_1(x)=\frac{e_1(x)}{\sqrt{h_x(e_1(x),e_1(x))}}.
\end{align*}
The function $x\mapsto h_x(e_1(x),e_1(x))$ is smooth by smoothness of the bundle metric and of the section $e_1$. It is strictly positive because $e_1(x)\neq 0$ and $h_x$ is positive definite. Therefore its square root is smooth and positive, so $u_1:U\to E$ is a smooth section with $h_x(u_1(x),u_1(x))=1$.
Now suppose that smooth orthonormal sections $u_1,\dots,u_{k-1}:U\to E$ have been constructed, where $2\le k\le r$. Define
\begin{align*}
v_k(x)=e_k(x)-\sum_{j=1}^{k-1}h_x(e_k(x),u_j(x))u_j(x).
\end{align*}
This subtracts from $e_k(x)$ its projection onto the already constructed orthonormal span. For each $1\le i<k$, bilinearity of $h_x$ gives
\begin{align*}
h_x(v_k(x),u_i(x))=h_x(e_k(x),u_i(x))-\sum_{j=1}^{k-1}h_x(e_k(x),u_j(x))h_x(u_j(x),u_i(x)).
\end{align*}
Since the sections $u_1,\dots,u_{k-1}$ are orthonormal, $h_x(u_j(x),u_i(x))$ is $1$ when $j=i$ and $0$ otherwise, so this expression is $0$. Thus $v_k(x)$ is orthogonal to each earlier $u_i(x)$.
We also need $v_k(x)\neq 0$. If $v_k(x)=0$, then $e_k(x)$ lies in $\operatorname{span}(u_1(x),\dots,u_{k-1}(x))$. Each $u_j(x)$ lies in $\operatorname{span}(e_1(x),\dots,e_j(x))$, hence in $\operatorname{span}(e_1(x),\dots,e_{k-1}(x))$. Therefore $e_k(x)$ would lie in $\operatorname{span}(e_1(x),\dots,e_{k-1}(x))$, contradicting that $(e_1(x),\dots,e_r(x))$ is a basis of $E_x$.
Hence $h_x(v_k(x),v_k(x))>0$ for every $x\in U$. Since $v_k$ is built from smooth sections using the smooth bundle metric, $x\mapsto h_x(v_k(x),v_k(x))$ is a positive smooth function. Define
\begin{align*}
u_k(x)=\frac{v_k(x)}{\sqrt{h_x(v_k(x),v_k(x))}}.
\end{align*}
Then $u_k$ is smooth, has $h_x(u_k(x),u_k(x))=1$, and is orthogonal to $u_1(x),\dots,u_{k-1}(x)$. Induction gives a smooth local orthonormal frame $(u_1,\dots,u_r)$ on $U$.
[/guided]
[/step]
[step:Use a local orthonormal frame to trivialise the orthonormal frame bundle]
Let $(u_1,\dots,u_r)$ be a smooth local $h$-orthonormal frame on an [open set](/page/Open%20Set) $U\subset M$. Define, for each $x\in U$, the linear isomorphism $u_x:\mathbb{R}^r\to E_x$ by
\begin{align*}
u_x(a)=\sum_{i=1}^r a_i u_i(x)
\end{align*}
for $a=(a_1,\dots,a_r)\in\mathbb{R}^r$. Since $(u_1(x),\dots,u_r(x))$ is $h_x$-orthonormal, $u_x$ is an isometry from the Euclidean [inner product](/page/Inner%20Product) on $\mathbb{R}^r$ to $h_x$ on $E_x$.
Define a map $\Phi_U:\operatorname{Fr}_O(E,h)|_U\to U\times O(r)$ by
\begin{align*}
\Phi_U(p)=(x,u_x^{-1}\circ p)
\end{align*}
where $p:\mathbb{R}^r\to E_x$ is an orthonormal frame over $x\in U$. The composite $u_x^{-1}\circ p:\mathbb{R}^r\to\mathbb{R}^r$ is an isometry of Euclidean space, hence an element of $O(r)$.
Define $\Psi_U:U\times O(r)\to \operatorname{Fr}_O(E,h)|_U$ by
\begin{align*}
\Psi_U(x,A)=u_x\circ A.
\end{align*}
If $A\in O(r)$, then $u_x\circ A$ is a composition of Euclidean isometries into $(E_x,h_x)$, so it is an $h_x$-orthonormal frame. The identities $\Psi_U(\Phi_U(p))=p$ and $\Phi_U(\Psi_U(x,A))=(x,A)$ follow directly from composition of linear maps. Thus $\Phi_U$ is a bijection with inverse $\Psi_U$.
In the usual local trivialisation of $\operatorname{Fr}(E)|_U$ determined by the smooth frame $(u_1,\dots,u_r)$, the map $\Phi_U$ is the restriction of the smooth frame-bundle trivialisation
\begin{align*}
p\mapsto (x,u_x^{-1}\circ p)
\end{align*}
from $\operatorname{Fr}(E)|_U$ to $U\times GL(r,\mathbb{R})$. Therefore $\operatorname{Fr}_O(E,h)|_U$ is identified smoothly with the embedded submanifold $U\times O(r)\subset U\times GL(r,\mathbb{R})$.
[/step]
[step:Check that the restricted right action is the principal $O(r)$ action]
Let $p\in \operatorname{Fr}_O(E,h)$ lie over $x\in M$, and let $A\in O(r)$. The right action on $\operatorname{Fr}(E)$ is $p\cdot A=p\circ A$. Since both $p:\mathbb{R}^r\to E_x$ and $A:\mathbb{R}^r\to\mathbb{R}^r$ are isometries, $p\circ A$ is again an $h_x$-orthonormal frame. Thus the $GL(r,\mathbb{R})$ action restricts to a right $O(r)$ action on $\operatorname{Fr}_O(E,h)$.
In the local trivialisation $\Phi_U$ determined by the orthonormal frame $(u_1,\dots,u_r)$, if $\Phi_U(p)=(x,B)$ and $A\in O(r)$, then
\begin{align*}
\Phi_U(p\cdot A)=(x,u_x^{-1}\circ p\circ A)=(x,BA).
\end{align*}
Therefore the restricted action is ordinary right multiplication on the $O(r)$ factor. Since right multiplication of $O(r)$ on itself is free and transitive, each fibre of $\operatorname{Fr}_O(E,h)\to M$ is a principal homogeneous space for $O(r)$.
[/step]
[step:Verify that changes of orthonormal frame have orthogonal transition functions]
Let $(u_1,\dots,u_r)$ and $(\tilde u_1,\dots,\tilde u_r)$ be two smooth local $h$-orthonormal frames on open sets $U,\tilde U\subset M$. On $U\cap \tilde U$, define $u_x:\mathbb{R}^r\to E_x$ and $\tilde u_x:\mathbb{R}^r\to E_x$ from these frames as above. Define the transition map $G:U\cap \tilde U\to GL(r,\mathbb{R})$ by
\begin{align*}
G(x)=u_x^{-1}\circ \tilde u_x.
\end{align*}
Because $u_x$ and $\tilde u_x$ are both isometries, $G(x)$ is an isometry of Euclidean space, so $G(x)\in O(r)$ for every $x\in U\cap\tilde U$. The entries of $G(x)$ are
\begin{align*}
G_{ij}(x)=h_x(\tilde u_j(x),u_i(x)).
\end{align*}
Since $h$ and the sections $u_i,\tilde u_j$ are smooth, each $G_{ij}:U\cap\tilde U\to\mathbb{R}$ is smooth. Hence $G:U\cap\tilde U\to O(r)$ is smooth.
The corresponding transition between the two local trivialisations is
\begin{align*}
(x,A)\mapsto (x,G(x)^{-1}A).
\end{align*}
Thus the local product descriptions glue by smooth $O(r)$-valued transition functions.
[/step]
[step:Conclude that the orthonormal frames form a principal subbundle]
The preceding steps give an open cover of $M$ by sets $U$ over which $\operatorname{Fr}_O(E,h)|_U$ is smoothly identified with $U\times O(r)$. Under these identifications, the projection to $M$ is the first projection $U\times O(r)\to U$, and the right action is ordinary right multiplication by $O(r)$. The transition functions are smooth maps into $O(r)$ and are precisely the restrictions of the transition functions of $\operatorname{Fr}(E)$ to the subgroup $O(r)\subset GL(r,\mathbb{R})$.
Therefore $\operatorname{Fr}_O(E,h)\to M$ is a smooth principal $O(r)$-bundle. Since each local inclusion is the smooth embedding $U\times O(r)\subset U\times GL(r,\mathbb{R})$ and the right action is the restriction of the frame-bundle action, it is a principal $O(r)$-subbundle of $\operatorname{Fr}(E)\to M$.
[/step]