[guided]We want to relate $\int_M d\alpha \wedge \star\beta$ — the integral that will eventually become $\langle\langle d\alpha, \beta \rangle\rangle_g$ — to an integral involving $\alpha$ rather than $d\alpha$. The standard tool for moving a derivative off one factor and onto the other is integration by parts, and on a manifold without boundary, integration by parts is precisely Stokes' theorem applied to the Leibniz expansion of an exact form. We carry out this argument carefully.
Fix $\alpha \in \Omega^{p-1}_c(M)$ and $\beta \in \Omega^p_c(M)$. The wedge product $\alpha \wedge \star\beta$ then lies in $\Omega^{n-1}_c(M)$, since $\star\beta \in \Omega^{n-p}(M)$ and $(p-1) + (n-p) = n - 1$. Compact support of either factor forces compact support of the product. Now $d : \Omega^*(M) \to \Omega^{*+1}(M)$ is a graded derivation of degree $+1$, meaning it satisfies the Leibniz rule
\begin{align*}
d(\alpha \wedge \beta) = d\alpha \wedge \beta + (-1)^{|\alpha|} \alpha \wedge d\beta
\end{align*}
for any homogeneous forms. Specialising with $|\alpha| = p - 1$ and $\beta$ replaced by $\star\beta$:
\begin{align*}
d(\alpha \wedge \star\beta) = d\alpha \wedge \star\beta + (-1)^{|\alpha|} \alpha \wedge d(\star\beta) = d\alpha \wedge \star\beta + (-1)^{p-1} \alpha \wedge d(\star\beta).
\end{align*}
Both terms on the right have degree $n$ (the maximal degree on $M$), and both are compactly supported, so $d(\alpha \wedge \star\beta) \in \Omega^n_c(M)$ and is integrable.
Now we want to integrate $d(\alpha \wedge \star\beta)$ over $M$ — the answer should be zero, and this is the integration-by-parts step. Why zero? By Stokes' theorem, for a compactly supported $(n-1)$-form $\eta$ on an oriented $n$-manifold $M$,
\begin{align*}
\int_M d\eta = \int_{\partial M} \eta.
\end{align*}
We must check the hypotheses. Stokes' theorem requires (i) $M$ oriented (given), (ii) $\eta$ smooth and compactly supported (the form $\alpha \wedge \star\beta$ is smooth because $\alpha, \beta, \star$ all are, and compactly supported because $\alpha$ is). It remains to evaluate the boundary integral. Two scenarios make this vanish: either $M$ has empty boundary (a closed manifold), in which case $\partial M = \emptyset$ and the boundary integral is trivially zero; or $\alpha$ has support strictly inside the interior of $M$, in which case $\alpha \wedge \star\beta$ vanishes on $\partial M$ and the boundary integral is again zero. The hypothesis of the theorem ("$M$ closed, or $\alpha$ compactly supported in the interior") is exactly what we need. Thus
\begin{align*}
\int_M d(\alpha \wedge \star\beta) = \int_{\partial M} \alpha \wedge \star\beta = 0.
\end{align*}
Why does this hypothesis matter? If $M$ has non-empty boundary and $\alpha \wedge \star\beta$ does not vanish there, the boundary integral contributes a defect, and the resulting identity $\langle\langle d\alpha, \beta \rangle\rangle_g = \langle\langle \alpha, \delta\beta \rangle\rangle_g$ holds only modulo a boundary term. The clean adjoint identity is a closed-manifold (or compactly-supported) phenomenon.
Combining the vanishing with the Leibniz expansion:
\begin{align*}
0 = \int_M d(\alpha \wedge \star\beta) = \int_M d\alpha \wedge \star\beta + (-1)^{p-1} \int_M \alpha \wedge d(\star\beta).
\end{align*}
Solving for the first integral:
\begin{align*}
\int_M d\alpha \wedge \star\beta = -(-1)^{p-1} \int_M \alpha \wedge d(\star\beta) = (-1)^p \int_M \alpha \wedge d(\star\beta),
\end{align*}
using $-(-1)^{p-1} = (-1)^p$. The derivative has been transferred from $\alpha$ to $\star\beta$, at the cost of a sign $(-1)^p$. The next step rewrites $d(\star\beta)$ in terms of $\delta\beta$ to expose the adjoint structure.[/guided]