Let $G$ be a connected, simply connected three-dimensional Lie group (unimodular or non-unimodular) equipped with a left-invariant (Riemannian or Lorentzian) metric $g$. By definition, the isometry group $Isom(G, g)$ contains $G$ itself, acting by left translations. It turns out that, generically, $Isom(G, g)$ is actually equal to $G$, and the natural question then becomes to classify those special metrics for which this is not the case. Using Lie-theoretical methods, we present a unified approach to obtain all pairs $(G, g)$ whose full isometry group $Isom(G, g)$ has dimension greater than or equal to four. As a consequence, we determine, for every pair $(G, g)$, up to automorphism and scaling, the dimension of $Isom(G, g)$, which can be three, four, or six. (Joint work with S. Chaib and A. Zeghib).