[guided]The finite remainder is the analytic point where the proof must check more than formal algebra. Define $R_N: \mathcal{S}(\mathbb{R}^n) \to \mathcal{S}'(\mathbb{R}^n)$ by the oscillatory kernel
\begin{align*}
K_{R_N}(x,y) = (2\pi)^{-n}\int_{\mathbb{R}^n}e^{i(x-y)\cdot \xi}r_N(x,y,\xi)\, d\mathcal{L}^n(\xi).
\end{align*}
The theorem we use is the finite amplitude reduction theorem for Kohn-Nirenberg kernels. In the form needed here, it says the following: if $h: \mathbb{R}^n_x \times \mathbb{R}^n_y \times \mathbb{R}^n_\xi \to \mathbb{C}$ is a smooth amplitude satisfying, for all multi-indices $\beta,\gamma,\delta \in \mathbb{N}_0^n$, estimates $|\partial_x^\beta\partial_y^\gamma\partial_\xi^\delta h(x,y,\xi)| \le C_{\beta\gamma\delta}\langle \xi\rangle^{m-|\delta|}$ uniformly in $x,y,\xi$, then the oscillatory kernel with amplitude $h$ equals the kernel of a Kohn-Nirenberg operator with a left symbol in $S^m_{1,0}$ modulo a smoothing kernel. If the amplitude is $(y-x)^\alpha h(x,y,\xi)$ with $h$ satisfying those same order-$m$ estimates, then the reduced left symbol belongs to $S^{m-|\alpha|}_{1,0}$, because the factor $(y-x)^\alpha$ is converted into $|\alpha|$ integrations by parts in $\xi$.
We verify its hypotheses. The formula for $r_N$ is a finite sum over $|\alpha|=N$ of
\begin{align*}
(y-x)^\alpha N\alpha!^{-1}\int_0^1(1-t)^{N-1}\partial_x^\alpha\overline{a(x+t(y-x),\xi)}\, d\mathcal{L}^1(t).
\end{align*}
Because $a \in S^m_{1,0}$, the derivatives of $\overline a$ satisfy the same symbol estimates. When $\partial_x^\beta\partial_y^\gamma\partial_\xi^\delta$ falls on $\partial_x^\alpha\overline{a(x+t(y-x),\xi)}$, the chain rule produces only finitely many derivatives $\partial_x^{\alpha+\rho}\partial_\xi^\delta\overline a$ with $|\rho| \le |\beta|+|\gamma|$, multiplied by powers of $t$ and $1-t$. Since $0 \le t \le 1$, those coefficients are uniformly bounded, so the integrand is bounded by a constant times $\langle \xi\rangle^{m-|\delta|}$ uniformly in $x,y,\xi,t$. The interval $[0,1]$ has finite $\mathcal{L}^1$-measure, so the same estimate holds after integration in $t$. Thus the non-polynomial part of the amplitude satisfies the required uniform two-space amplitude estimates of order $m$.
Now the polynomial factor is converted into derivatives of the phase. Since $|\alpha|=N$,
\begin{align*}
(y-x)^\alpha e^{i(x-y)\cdot \xi} = (-1)^N D_\xi^\alpha e^{i(x-y)\cdot \xi}.
\end{align*}
Integrating by parts in $\xi$ transfers these $N$ derivatives to the order-$m$ amplitude. In the symbol class $S^m_{1,0}$, each $\xi$-derivative lowers order by one, so the reduced left symbol belongs to $S^{m-N}_{1,0}$. Therefore there is a map $c_N: \mathbb{R}^n_x \times \mathbb{R}^n_\xi \to \mathbb{C}$ with
\begin{align*}
c_N \in S^{m-N}_{1,0}(\mathbb{R}^n_x \times \mathbb{R}^n_\xi)
\end{align*}
such that $A^* - \operatorname{Op}(b_N+c_N)$ has a smooth Schwartz kernel. This is exactly the finite-order adjoint expansion needed for the asymptotic argument.[/guided]