[proofplan]
The three candidate values $a, c, a - |b| + c$ are attained at the primitive pairs $(1, 0)$, $(0, 1)$, and $(1, \pm 1)$ respectively. To show no smaller proper representation exists, we bound $f(x, y)$ from below for primitive pairs $(x, y)$ with at least one coordinate zero, and for primitive pairs with both coordinates nonzero. The zero-coordinate cases reduce to $f(\pm 1, 0) = a$ and $f(0, \pm 1) = c$. For the nonzero case, we use the reduced inequalities $|b| \leq a \leq c$ and symmetry in $(x, y)$ (WLOG $|x| \geq |y|$) to chain $ax^2 + bxy + cy^2 \geq (a - |b|)x^2 + cy^2 \geq (a - |b|) + c$. The reduced hypothesis enters twice: once to drop $|b|$ in the quadratic estimate, and once to conclude $a \leq c \leq a - |b| + c$, so the three values are listed in non-decreasing order.
[/proofplan]
[step:Evaluate $f$ at the pairs $(1, 0)$, $(0, 1)$, $(1, 1)$, $(1, -1)$ to realise the three candidate values]
We compute:
\begin{align*}
f(1, 0) &= a \cdot 1^2 + b \cdot 1 \cdot 0 + c \cdot 0^2 = a, \\
f(0, 1) &= a \cdot 0^2 + b \cdot 0 \cdot 1 + c \cdot 1^2 = c, \\
f(1, 1) &= a + b + c, \\
f(1, -1) &= a - b + c.
\end{align*}
The pairs $(1, 0)$ and $(0, 1)$ are primitive (both gcds equal $1$), so $a$ and $c$ are properly represented. The pairs $(1, 1)$ and $(1, -1)$ are also primitive. Choosing the sign of the second coordinate to match the sign of $-b$, we have
\begin{align*}
\min\{f(1, 1), f(1, -1)\} = a - |b| + c,
\end{align*}
so $a - |b| + c$ is properly represented at $(1, \pm 1)$, where the sign is $+1$ if $b \leq 0$ and $-1$ if $b > 0$.
[/step]
[step:Bound $f(x, y)$ from below for primitive pairs with $y = 0$ or $x = 0$]
Suppose $(x, y) \in \mathbb{Z}^2$ is primitive ($\gcd(x, y) = 1$) with $y = 0$. Then $\gcd(x, 0) = |x| = 1$, so $x = \pm 1$, and
\begin{align*}
f(x, 0) = ax^2 = a \cdot 1 = a.
\end{align*}
Symmetrically, if $x = 0$, then $\gcd(0, y) = |y| = 1$, so $y = \pm 1$, and $f(0, y) = c$.
Thus the only properly represented values arising from pairs with a zero coordinate are $a$ (from $x = \pm 1$, $y = 0$) and $c$ (from $x = 0$, $y = \pm 1$).
[/step]
[step:Bound $f(x, y)$ from below for primitive pairs with both coordinates nonzero]
Assume now $(x, y) \in \mathbb{Z}^2$ with $\gcd(x, y) = 1$ and $x, y \neq 0$, so $|x|, |y| \geq 1$. We will show
\begin{align*}
f(x, y) \geq a - |b| + c.
\end{align*}
By symmetry of $f$ in the sense that $f(x, y) = f(-x, -y)$, we may replace $(x, y)$ by $(-x, -y)$ if needed and assume WLOG that the cross-term $bxy$ is negative or zero, i.e. $bxy \leq 0$; in particular $|bxy| = -bxy$. (If $b = 0$, the cross term vanishes and the sign choice is immaterial.) More concretely: we use the crude estimate
\begin{align*}
bxy \geq -|b| \cdot |x| \cdot |y|,
\end{align*}
which holds without any sign arrangement because $|bxy| \leq |b| \cdot |x| \cdot |y|$ implies $bxy \geq -|bxy| \geq -|b||x||y|$. Substituting into $f$:
\begin{align*}
f(x, y) = ax^2 + bxy + cy^2 \geq ax^2 - |b| \cdot |x| \cdot |y| + cy^2. \tag{$\ast$}
\end{align*}
Now we split by whether $|x| \geq |y|$ or $|x| \leq |y|$.
**Case $|x| \geq |y| \geq 1$.** Then $|x| \cdot |y| \leq x^2$ (since $|y| \leq |x|$ and $|x| \geq 1$, so $|x| \cdot |y| \leq |x| \cdot |x| = x^2$). So $-|b| \cdot |x| \cdot |y| \geq -|b| x^2$, and ($\ast$) gives
\begin{align*}
f(x, y) \geq ax^2 - |b| x^2 + cy^2 = (a - |b|) x^2 + c y^2.
\end{align*}
We now use reducedness: $|b| \leq a$, so $a - |b| \geq 0$, and the first term is non-negative. Moreover, since $|x| \geq 1$, we have $x^2 \geq 1$, and since $|y| \geq 1$, we have $y^2 \geq 1$. Hence
\begin{align*}
f(x, y) \geq (a - |b|) \cdot 1 + c \cdot 1 = a - |b| + c.
\end{align*}
**Case $|y| \geq |x| \geq 1$.** By the symmetric estimate $|x| \cdot |y| \leq y^2$, ($\ast$) gives
\begin{align*}
f(x, y) \geq ax^2 + (c - |b|) y^2.
\end{align*}
By reducedness, $c \geq a \geq |b|$, so $c - |b| \geq 0$. Since $x^2, y^2 \geq 1$,
\begin{align*}
f(x, y) \geq a \cdot 1 + (c - |b|) \cdot 1 = a - |b| + c.
\end{align*}
In either case, $f(x, y) \geq a - |b| + c$.
[guided]
We want to show that any primitive pair $(x, y)$ with both coordinates nonzero produces a value of $f$ of at least $a - |b| + c$. The idea is that $|b|$ can be subtracted from whichever of $a$ or $c$ appears "more" in the expression — i.e., we route the bound according to which of $|x|, |y|$ is larger.
**Step 1: control the cross-term.** The cross-term $bxy$ can be either positive or negative depending on the signs of $b$, $x$, $y$. Rather than split into cases, we use the triangle-inequality bound
\begin{align*}
bxy \geq -|b| \cdot |x| \cdot |y|,
\end{align*}
which is valid regardless of signs. (This absorbs any potential "gain" from $bxy > 0$ into a universal lower bound.) Substituting into $f(x, y) = ax^2 + bxy + cy^2$:
\begin{align*}
f(x, y) \geq ax^2 - |b| \cdot |x| \cdot |y| + cy^2. \tag{$\ast$}
\end{align*}
**Step 2: bound $|x| \cdot |y|$ by the larger of $x^2, y^2$.** Both sides are integers. We split on the comparison of $|x|$ and $|y|$:
- **If $|x| \geq |y| \geq 1$:** then $|x| \cdot |y| \leq |x|^2 = x^2$. So ($\ast$) gives
\begin{align*}
f(x, y) \geq ax^2 - |b| x^2 + cy^2 = (a - |b|)x^2 + c y^2.
\end{align*}
From reducedness $|b| \leq a$, the coefficient $a - |b|$ is non-negative. Because $|x| \geq 1$ we have $x^2 \geq 1$; because $|y| \geq 1$ we have $y^2 \geq 1$. Hence
\begin{align*}
f(x, y) \geq (a - |b|) + c = a - |b| + c.
\end{align*}
- **If $|y| \geq |x| \geq 1$:** symmetrically, $|x| \cdot |y| \leq y^2$, and ($\ast$) gives
\begin{align*}
f(x, y) \geq ax^2 + (c - |b|)y^2.
\end{align*}
Reducedness gives $c \geq a \geq |b|$, so $c - |b| \geq 0$. Again $x^2 \geq 1$ and $y^2 \geq 1$, so
\begin{align*}
f(x, y) \geq a + (c - |b|) = a - |b| + c.
\end{align*}
In either case the lower bound $a - |b| + c$ holds. The role of reducedness is to guarantee that the "subtracted" coefficient ($a - |b|$ or $c - |b|$) is non-negative, so that replacing $x^2$ or $y^2$ by $1$ is a genuine decrease of the right-hand side.
[/guided]
[/step]
[step:Order the three values $a \leq c \leq a - |b| + c$ and assemble the conclusion]
The reduced inequalities $|b| \leq a \leq c$ imply
\begin{align*}
a \leq c \quad \text{(directly)},
\end{align*}
and
\begin{align*}
c \leq c + (a - |b|) = a - |b| + c,
\end{align*}
where the last inequality uses $a - |b| \geq 0$ (from $|b| \leq a$). Hence
\begin{align*}
a \leq c \leq a - |b| + c.
\end{align*}
Combining this ordering with the results of the previous three steps:
- Every proper representation uses a primitive pair $(x, y) \in \mathbb{Z}^2$.
- If one coordinate is zero, the value is $a$ or $c$.
- If both coordinates are nonzero, the value is at least $a - |b| + c$.
- All three values $a$, $c$, $a - |b| + c$ are attained (at $(1, 0)$, $(0, 1)$, $(1, \pm 1)$).
Therefore the three smallest properly represented integers, listed in non-decreasing order, are exactly
\begin{align*}
a, \qquad c, \qquad a - |b| + c.
\end{align*}
This completes the proof.
[/step]