[proofplan]
We prove that each branch of the proposed decision tree is sound, meaning that it either gives a necessary obstruction to solvability or transforms solutions into equivalent or further solutions. Congruence testing follows from reducing an integral equation modulo an integer. The homogeneous quadratic case is treated by the line-through-a-point construction on a projective conic. Pell-type equations are rewritten as norm equations in a quadratic field, and units preserve the norm by multiplicativity. Finally, descent is justified by the well-ordering of the positive integers.
[/proofplan]
[step:Reduce integral solutions to congruence solutions]
Fix an integer $m \geq 2$, and let
\begin{align*}
\pi_m: \mathbb{Z} &\to \mathbb{Z}/m\mathbb{Z}
\end{align*}
be the quotient map. Since $Q$ has integer coefficients, applying $\pi_m$ coefficientwise gives a homogeneous quadratic form
\begin{align*}
Q_m: (\mathbb{Z}/m\mathbb{Z})^3 &\to \mathbb{Z}/m\mathbb{Z}.
\end{align*}
If $(x,y,z) \in \mathbb{Z}^3$ satisfies $Q(x,y,z)=0$, then
\begin{align*}
Q_m(\pi_m(x),\pi_m(y),\pi_m(z))
= \pi_m(Q(x,y,z))
= \pi_m(0)
=0.
\end{align*}
Here an admissible residue solution means a residue triple satisfying every extra condition imposed on the integral solutions and preserved by reduction modulo $m$; for example, if one is searching for a nonzero primitive integral solution, then admissibility means that the reduced triple is not excluded by the corresponding reduced nonzero or primitivity condition. Thus failure of the congruence $Q_m(X,Y,Z)=0$ to have an admissible residue solution rules out an integral solution of $Q(x,y,z)=0$ with those same admissibility conditions.
Similarly, for the equation $x^2-Dy^2=N$, define
\begin{align*}
F: \mathbb{Z}^2 &\to \mathbb{Z} \\
(x,y) &\mapsto x^2-Dy^2-N.
\end{align*}
If $(x,y) \in \mathbb{Z}^2$ satisfies $F(x,y)=0$, then
\begin{align*}
\pi_m(x)^2-\pi_m(D)\pi_m(y)^2-\pi_m(N)=0
\end{align*}
in $\mathbb{Z}/m\mathbb{Z}$. Therefore every integral solution gives a solution modulo $m$. Taking $m$ to be small primes or $8$ is a finite necessary test.
[/step]
[step:Parametrize a rational conic from one rational point]
Define the projective zero locus of $Q$ by
\begin{align*}
C_Q(\mathbb{Q}) := \{[x:y:z] \in \mathbb{P}^2(\mathbb{Q}) : Q(x,y,z)=0\}.
\end{align*}
Assume $P=[x_0:y_0:z_0] \in C_Q(\mathbb{Q})$. Let $\mathcal{L}_P(\mathbb{Q})$ denote the set of projective lines in $\mathbb{P}^2(\mathbb{Q})$ passing through $P$. For any rational point $R=[x_1:y_1:z_1] \in C_Q(\mathbb{Q})$ with $R \neq P$, there is a unique projective line $L_{P,R} \in \mathcal{L}_P(\mathbb{Q})$ passing through $P$ and $R$; its defining linear equation has rational coefficients because both points have rational homogeneous coordinates. Hence every rational point of $C_Q$ lies on at least one rational line through $P$.
Conversely, fix a rational line $L \in \mathcal{L}_P(\mathbb{Q})$. Choose a rational parametrisation
\begin{align*}
\ell: \mathbb{P}^1(\mathbb{Q}) &\to L(\mathbb{Q})
\end{align*}
of this line with $\ell(t_0)=P$ for some $t_0 \in \mathbb{P}^1(\mathbb{Q})$. Define
\begin{align*}
q_L: \mathbb{P}^1(\mathbb{Q}) &\to \mathbb{Q} \\
t &\mapsto Q(\ell(t)).
\end{align*}
After choosing an affine coordinate on $\mathbb{P}^1$ away from the point at infinity, $q_L$ is represented by a polynomial over $\mathbb{Q}$ of degree at most $2$.
There are two cases. If $q_L$ is the zero polynomial, then $Q$ vanishes identically on $L$, so the whole rational line $L(\mathbb{Q})$ is contained in $C_Q(\mathbb{Q})$. This covers the degenerate possibility that $L$ is a component of the conic, and also covers the zero form $Q=0$.
If $q_L$ is not the zero polynomial, then $L$ is not contained in $C_Q$. Since $P \in C_Q(\mathbb{Q})$, the parameter $t_0$ is a rational root of $q_L$. Dividing $q_L$ by the corresponding linear factor over $\mathbb{Q}$ leaves a polynomial over $\mathbb{Q}$ of degree at most $1$, so any other intersection parameter is rational. If this remaining factor has no new root, then the line is tangent at $P$ and the intersection point is $P$ with multiplicity two. Thus every rational line through $P$ either lies inside the degenerate conic or returns the second rational intersection point, and every rational point of $C_Q$ occurs from the line joining it to $P$.
[/step]
[step:Rewrite Pell-type equations as norm equations]
Since $D$ is not a square in $\mathbb{Z}$, the element $\sqrt{D}$ does not lie in $\mathbb{Q}$, so $K=\mathbb{Q}(\sqrt{D})$ is a quadratic field. Define the nontrivial field automorphism
\begin{align*}
\sigma: K &\to K \\
a+b\sqrt{D} &\mapsto a-b\sqrt{D}
\end{align*}
for $a,b \in \mathbb{Q}$. The field norm is the map
\begin{align*}
N_{K/\mathbb{Q}}: K &\to \mathbb{Q} \\
\alpha &\mapsto \alpha \sigma(\alpha).
\end{align*}
For integers $x,y \in \mathbb{Z}$, put $\alpha=x+y\sqrt{D} \in K$. Then
\begin{align*}
N_{K/\mathbb{Q}}(\alpha)
&=(x+y\sqrt{D})(x-y\sqrt{D}) \\
&=x^2-Dy^2.
\end{align*}
Hence the Diophantine equation $x^2-Dy^2=N$ is exactly the norm equation
\begin{align*}
N_{K/\mathbb{Q}}(x+y\sqrt{D})=N.
\end{align*}
When $x+y\sqrt{D}$ is not the most natural integral representative in the order $\mathbb{Z}[\sqrt{D}]$, one replaces that order by the full ring of integers $\mathcal{O}_K$; the same norm identity holds because $N_{K/\mathbb{Q}}$ is defined on the field $K$ and restricts to an integer-valued norm on $\mathcal{O}_K$.
[/step]
[step:Generate further norm solutions by multiplying by units]
Let $\varepsilon \in \mathcal{O}_K^\times$ be a unit satisfying $N_{K/\mathbb{Q}}(\varepsilon)=1$. For an integer $n \in \mathbb{Z}$, define
\begin{align*}
M_{\varepsilon,n}: K &\to K \\
\alpha &\mapsto \varepsilon^n\alpha.
\end{align*}
If $\alpha \in \mathcal{O}_K$ satisfies $N_{K/\mathbb{Q}}(\alpha)=N$, then multiplicativity of the field norm gives
\begin{align*}
N_{K/\mathbb{Q}}(M_{\varepsilon,n}(\alpha))
&=N_{K/\mathbb{Q}}(\varepsilon^n\alpha) \\
&=N_{K/\mathbb{Q}}(\varepsilon)^n N_{K/\mathbb{Q}}(\alpha) \\
&=1^n N \\
&=N.
\end{align*}
Thus every power of a norm-$1$ unit sends a solution of the norm equation to another solution of the same norm equation. This proves the soundness of the unit-generation step used for Pell-type recurrence.
[/step]
[step:Justify descent by well-ordering]
Let $\mathcal{S}$ be a set of hypothetical nonzero integral solutions to one of the quadratic equations under study. Suppose there is a complexity map
\begin{align*}
c: \mathcal{S} &\to \mathbb{N}
\end{align*}
and suppose that every solution $s \in \mathcal{S}$ can be transformed into another solution $s' \in \mathcal{S}$ satisfying
\begin{align*}
c(s') < c(s).
\end{align*}
If $\mathcal{S}$ were nonempty, then the set $c(\mathcal{S}) \subset \mathbb{N}$ would be a nonempty set of positive integers. By the well-ordering property of $\mathbb{N}$, it would contain a least element $c(s_0)$ for some $s_0 \in \mathcal{S}$. Applying the descent transformation to $s_0$ gives $s_1 \in \mathcal{S}$ with
\begin{align*}
c(s_1)<c(s_0),
\end{align*}
contradicting the minimality of $c(s_0)$. Therefore no such solution set $\mathcal{S}$ exists. This proves that a valid descent step, meaning a transformation satisfying the displayed strict complexity decrease on all hypothetical nonzero solutions, can rule out solutions after local tests and parametrisation have failed to decide the equation. A refined factorisation argument is sound only when it supplies such an obstruction or such a descent transformation explicitly.
[/step]