[proofplan]
We verify the group axioms for $\operatorname{Aut}(G)$ with operation given by composition of maps. The main points are that the composite of two bijective homomorphisms is again a bijective homomorphism, the identity map on $G$ is an automorphism, and the inverse function of an automorphism is again a homomorphism. Associativity is inherited from associativity of ordinary function composition.
[/proofplan]
[step:Define the composition operation on automorphisms]
Let $\operatorname{Aut}(G)$ be the set of all bijective group homomorphisms $\varphi: G \to G$. Define the binary operation
\begin{align*}
\circ: \operatorname{Aut}(G) \times \operatorname{Aut}(G) &\to \operatorname{Aut}(G) \\
(\varphi,\psi) &\mapsto \varphi \circ \psi
\end{align*}
provided that $\varphi \circ \psi$ is again an automorphism. For elements $g \in G$, the composite map is defined by
\begin{align*}
(\varphi \circ \psi)(g)=\varphi(\psi(g)).
\end{align*}
We now verify that this operation is well-defined and satisfies the group axioms.
[/step]
[step:Show that the composite of two automorphisms is an automorphism]
Let
\begin{align*}
\varphi: G &\to G,\\
\psi: G &\to G
\end{align*}
be elements of $\operatorname{Aut}(G)$. Since $\varphi$ and $\psi$ are bijections, their composite $\varphi \circ \psi: G \to G$ is a bijection by the standard property of [bijective functions](/page/Bijection). It remains to verify the homomorphism property.
Let $a,b \in G$. Since $\psi$ is a group homomorphism, $\psi(a \cdot b)=\psi(a)\cdot \psi(b)$. Since $\varphi$ is a group homomorphism, applying $\varphi$ to this product gives
\begin{align*}
(\varphi \circ \psi)(a \cdot b)
&= \varphi(\psi(a \cdot b))\\
&= \varphi(\psi(a)\cdot \psi(b))\\
&= \varphi(\psi(a))\cdot \varphi(\psi(b))\\
&= (\varphi \circ \psi)(a)\cdot(\varphi \circ \psi)(b).
\end{align*}
Thus $\varphi \circ \psi$ is a group homomorphism. Therefore $\varphi \circ \psi \in \operatorname{Aut}(G)$, so composition is closed on $\operatorname{Aut}(G)$.
[guided]
Take two arbitrary automorphisms
\begin{align*}
\varphi: G &\to G,\\
\psi: G &\to G.
\end{align*}
To prove that their composite belongs to $\operatorname{Aut}(G)$, we must check exactly the two defining conditions of an automorphism: bijectivity and preservation of the group operation.
First, $\varphi$ and $\psi$ are bijections by definition of $\operatorname{Aut}(G)$. The composite of bijections is a bijection, so $\varphi \circ \psi: G \to G$ is bijective.
Second, we verify the homomorphism property. Let $a,b \in G$. The map $\psi$ preserves multiplication, so
\begin{align*}
\psi(a \cdot b)=\psi(a)\cdot \psi(b).
\end{align*}
The map $\varphi$ also preserves multiplication, so applying $\varphi$ to the product on the right gives
\begin{align*}
(\varphi \circ \psi)(a \cdot b)
&= \varphi(\psi(a \cdot b))\\
&= \varphi(\psi(a)\cdot \psi(b))\\
&= \varphi(\psi(a))\cdot \varphi(\psi(b))\\
&= (\varphi \circ \psi)(a)\cdot(\varphi \circ \psi)(b).
\end{align*}
This proves that $\varphi \circ \psi$ is a group homomorphism. Since it is both bijective and a homomorphism from $G$ to $G$, it is an automorphism. Hence composition is closed on $\operatorname{Aut}(G)$.
[/guided]
[/step]
[step:Use associativity of function composition]
Let
\begin{align*}
\varphi: G &\to G,\\
\psi: G &\to G,\\
\theta: G &\to G
\end{align*}
be elements of $\operatorname{Aut}(G)$. For every $g \in G$,
\begin{align*}
((\varphi \circ \psi)\circ \theta)(g)
&= (\varphi \circ \psi)(\theta(g))\\
&= \varphi(\psi(\theta(g)))\\
&= \varphi((\psi \circ \theta)(g))\\
&= (\varphi \circ (\psi \circ \theta))(g).
\end{align*}
Since the two maps agree on every element $g \in G$, we have
\begin{align*}
(\varphi \circ \psi)\circ \theta=\varphi \circ (\psi \circ \theta).
\end{align*}
Thus composition is associative on $\operatorname{Aut}(G)$.
[/step]
[step:Identify the identity automorphism]
Define the identity map
\begin{align*}
\operatorname{id}_G: G &\to G\\
g &\mapsto g.
\end{align*}
The map $\operatorname{id}_G$ is bijective, with inverse function itself. For $a,b \in G$,
\begin{align*}
\operatorname{id}_G(a\cdot b)
&= a\cdot b\\
&= \operatorname{id}_G(a)\cdot \operatorname{id}_G(b),
\end{align*}
so $\operatorname{id}_G$ is a group homomorphism. Hence $\operatorname{id}_G \in \operatorname{Aut}(G)$.
For every $\varphi \in \operatorname{Aut}(G)$ and every $g \in G$,
\begin{align*}
(\operatorname{id}_G \circ \varphi)(g)
&= \operatorname{id}_G(\varphi(g))\\
&= \varphi(g),
\end{align*}
and
\begin{align*}
(\varphi \circ \operatorname{id}_G)(g)
&= \varphi(\operatorname{id}_G(g))\\
&= \varphi(g).
\end{align*}
Therefore $\operatorname{id}_G \circ \varphi=\varphi$ and $\varphi \circ \operatorname{id}_G=\varphi$, so $\operatorname{id}_G$ is the identity element for composition on $\operatorname{Aut}(G)$.
[/step]
[step:Show that the inverse function of an automorphism is an automorphism]
Let $\varphi \in \operatorname{Aut}(G)$. Since $\varphi: G \to G$ is bijective, it has an inverse function
\begin{align*}
\varphi^{-1}: G &\to G.
\end{align*}
The inverse function $\varphi^{-1}$ is bijective. It remains to prove that $\varphi^{-1}$ is a group homomorphism.
Let $x,y \in G$. Since $\varphi$ is surjective, there exist $a,b \in G$ such that
\begin{align*}
x=\varphi(a), \qquad y=\varphi(b).
\end{align*}
By definition of $\varphi^{-1}$, this gives $a=\varphi^{-1}(x)$ and $b=\varphi^{-1}(y)$. Since $\varphi$ is a homomorphism,
\begin{align*}
xy
&= \varphi(a)\cdot \varphi(b)\\
&= \varphi(a\cdot b).
\end{align*}
Applying $\varphi^{-1}$ to both sides yields
\begin{align*}
\varphi^{-1}(xy)
&= a\cdot b\\
&= \varphi^{-1}(x)\cdot \varphi^{-1}(y).
\end{align*}
Thus $\varphi^{-1}$ is a group homomorphism, and hence $\varphi^{-1}\in \operatorname{Aut}(G)$.
Moreover,
\begin{align*}
\varphi^{-1}\circ \varphi=\operatorname{id}_G,
\qquad
\varphi\circ \varphi^{-1}=\operatorname{id}_G,
\end{align*}
by the defining property of the inverse function. Therefore every element of $\operatorname{Aut}(G)$ has an inverse in $\operatorname{Aut}(G)$ under composition.
[guided]
Let $\varphi \in \operatorname{Aut}(G)$. We need to find its inverse with respect to composition. Because $\varphi$ is an automorphism, it is a bijective map $\varphi: G \to G$, so it has an inverse function
\begin{align*}
\varphi^{-1}: G &\to G.
\end{align*}
This inverse function is automatically bijective. The only non-formal point is that $\varphi^{-1}$ must preserve the group operation.
Let $x,y \in G$. Since $\varphi$ is surjective, we may write $x=\varphi(a)$ and $y=\varphi(b)$ for some elements $a,b \in G$. These elements satisfy $a=\varphi^{-1}(x)$ and $b=\varphi^{-1}(y)$ by the definition of inverse function. Now use the homomorphism property of $\varphi$:
\begin{align*}
xy
&= \varphi(a)\cdot \varphi(b)\\
&= \varphi(a\cdot b).
\end{align*}
Applying $\varphi^{-1}$ to this equality gives
\begin{align*}
\varphi^{-1}(xy)
&= \varphi^{-1}(\varphi(a\cdot b))\\
&= a\cdot b\\
&= \varphi^{-1}(x)\cdot \varphi^{-1}(y).
\end{align*}
Thus $\varphi^{-1}$ is a group homomorphism. Since it is also bijective, $\varphi^{-1}$ is an automorphism of $G$.
Finally, the equations
\begin{align*}
\varphi^{-1}\circ \varphi=\operatorname{id}_G,
\qquad
\varphi\circ \varphi^{-1}=\operatorname{id}_G
\end{align*}
are exactly the defining equations for the inverse function of a bijection. Therefore $\varphi^{-1}$ is the inverse of $\varphi$ inside $\operatorname{Aut}(G)$ under composition.
[/guided]
[/step]
[step:Conclude that the automorphisms form a group]
We have proved that composition is a closed binary operation on $\operatorname{Aut}(G)$, that composition is associative, that $\operatorname{id}_G$ is an identity element, and that every $\varphi \in \operatorname{Aut}(G)$ has inverse $\varphi^{-1}\in \operatorname{Aut}(G)$. These are precisely the group axioms for the operation $\circ$. Hence $\operatorname{Aut}(G)$ is a group under composition.
[/step]