[guided]Define the characteristic polynomial of the matrix $B$:
\begin{align*}
f(x) = \det(xI_n - B) \in R[x],
\end{align*}
where $x$ is an indeterminate and the entries of $xI_n - B$ lie in $R[x]$ (diagonal entries are $x - b_{ii}$, off-diagonal entries are $-b_{ij}$, all of which are in $R[x]$). Hence $\det(xI_n - B) \in R[x]$.
We claim $f$ is monic of degree $n$. By the Leibniz formula,
\begin{align*}
f(x) = \sum_{\sigma \in S_n} \operatorname{sgn}(\sigma) \prod_{i=1}^n (xI_n - B)_{i, \sigma(i)}.
\end{align*}
For $\sigma = \operatorname{id}$, the product is $\prod_i (x - b_{ii})$, a product of $n$ monic linear factors, with leading term $x^n$. For any other $\sigma$, there are at least two indices $i$ where $\sigma(i) \neq i$, so at least two factors in the product are off-diagonal entries $-b_{i,\sigma(i)}$ which do not depend on $x$. Therefore the product has $x$-degree at most $n - 2$. Can any other permutation also contribute a degree-$n$ term? No — only the identity has all $n$ factors contain $x$, and only those factors are linear in $x$ with leading coefficient $1$. So
\begin{align*}
f(x) = x^n + c_{n-1} x^{n-1} + \cdots + c_0, \qquad c_i \in R,
\end{align*}
is monic of degree $n$ with coefficients in $R$.
Now apply the ring homomorphism $R[x] \to S$, $x \mapsto s$ (which exists because $S$ is an $R$-algebra via the inclusion $R \subseteq S$). The image $f(s) \in S$ equals $\det(sI_n - B)$, and by the previous step this determinant is $0$. Therefore
\begin{align*}
f(s) = 0 \quad \text{in } S,
\end{align*}
so $s$ satisfies the monic polynomial $f \in R[x]$: $s$ is integral over $R$.
Since $s \in S$ was arbitrary, every element of $S$ is integral over $R$. By definition, $S$ is integral over $R$.
As a side remark: this is the classical **determinant trick**, the module-theoretic generalisation of the Cayley-Hamilton theorem. The conclusion is quantitative: the integrality degree is at most the size of the generating set. In particular, if $S$ is generated by $n$ elements as an $R$-module, every element of $S$ satisfies a monic polynomial of degree at most $n$.[/guided]