[proofplan]
We fix a rank-one direction $B=a\otimes \xi$ and prove [Jensen's inequality](/theorems/9) for the one-variable function $\tau\mapsto f(A+\tau B)$. The main device is a one-dimensional sawtooth whose derivative takes only two values, so that the induced vector-valued perturbation has gradients alternating between the two rank-one states. Since quasiconvexity requires zero boundary trace, we insert the oscillation into the interior of a large cube and cut it off near the boundary; the boundary layer contributes a small error because its relative measure tends to zero and all gradients remain in a fixed compact set. Letting the cube size tend to infinity gives the desired convexity inequality.
[/proofplan]
[step:Reduce rank-one convexity to Jensen's inequality along one fixed rank-one line]
Fix $A\in\mathbb R^{m\times n}$ and a rank-one matrix $B\in\mathbb R^{m\times n}$. If $B=0$, then $\tau\mapsto f(A+\tau B)$ is constant and hence convex. Assume $B\ne 0$. Choose $a\in\mathbb R^m\setminus\{0\}$ and $\xi\in\mathbb R^n\setminus\{0\}$ such that
\begin{align*}
B=a\otimes \xi.
\end{align*}
Let $s,t\in\mathbb R$ and $\lambda\in[0,1]$, and define
\begin{align*}
r:=\lambda s+(1-\lambda)t.
\end{align*}
It is enough to prove
\begin{align*}
f(A+rB)\le \lambda f(A+sB)+(1-\lambda)f(A+tB).
\end{align*}
The endpoint cases $\lambda=0$ and $\lambda=1$ are identities, so assume $0<\lambda<1$.
[/step]
[step:Build a periodic sawtooth with the prescribed average slope]
Define the $1$-periodic function $h:\mathbb R\to\mathbb R$ by choosing $h(0)=0$, requiring $h$ to be affine with slope $s-r$ on $(0,\lambda)$ and affine with slope $t-r$ on $(\lambda,1)$, extending continuously to $[0,1]$, and then extending periodically to all of $\mathbb R$. Equivalently, for $\mathcal L^1$-a.e. $\theta\in(0,1)$,
\begin{align*}
h'(\theta)=(s-r)\mathbb{1}_{(0,\lambda)}(\theta)+(t-r)\mathbb{1}_{(\lambda,1)}(\theta)
\end{align*}
The mean of $h'$ over one period is zero because
\begin{align*}
\int_0^1 h'(\theta)\,d\mathcal L^1(\theta)=\lambda(s-r)+(1-\lambda)(t-r)=0
\end{align*}
Thus the periodic extension is well-defined and Lipschitz. Define
\begin{align*}
M_h:=\sup_{\theta\in\mathbb R}|h(\theta)|
\end{align*}
and
\begin{align*}
M_1:=\max\{|s-r|,|t-r|\}
\end{align*}
Then $M_h<\infty$ and $|h'|\le M_1$ $\mathcal L^1$-a.e. on $\mathbb R$.
[guided]
The purpose of the sawtooth is to encode a convex combination as an average of gradients. We want the gradient perturbation to take the value $(s-r)B$ on a fraction $\lambda$ of the domain and $(t-r)B$ on a fraction $1-\lambda$ of the domain. The average perturbation should be zero, because quasiconvexity will be applied at the base matrix $A+rB$.
Define $h:\mathbb R\to\mathbb R$ to be $1$-periodic and Lipschitz, with $h(0)=0$, affine with slope $s-r$ on $(0,\lambda)$, and affine with slope $t-r$ on $(\lambda,1)$. Thus, for $\mathcal L^1$-a.e. $\theta\in(0,1)$,
\begin{align*}
h'(\theta)=(s-r)\mathbb{1}_{(0,\lambda)}(\theta)+(t-r)\mathbb{1}_{(\lambda,1)}(\theta)
\end{align*}
The compatibility condition for a periodic [Lipschitz function](/page/Lipschitz%20Function) is that the integral of its derivative over one period vanish. Here this holds exactly:
\begin{align*}
\int_0^1 h'(\theta)\,d\mathcal L^1(\theta)=\lambda(s-r)+(1-\lambda)(t-r)=\lambda s+(1-\lambda)t-r=0
\end{align*}
Thus the sawtooth returns to its starting height after one period. Since $h'$ is bounded, $h$ is Lipschitz; since $h$ is periodic and continuous, its supremum
\begin{align*}
M_h:=\sup_{\theta\in\mathbb R}|h(\theta)|
\end{align*}
is finite. We also define
\begin{align*}
M_1:=\max\{|s-r|,|t-r|\}
\end{align*}
so that $|h'|\le M_1$ almost everywhere. These two constants will control the oscillating perturbations and the boundary cutoffs.
[/guided]
[/step]
[step:Insert the sawtooth into large cubes with zero boundary trace]
Let $e:=\xi/|\xi|$. For each integer $N\ge 2$, define the cube
\begin{align*}
Q_N:=(-N,N)^n\subset\mathbb R^n.
\end{align*}
Choose a cutoff function $\eta_N\in W^{1,\infty}_0(Q_N)$ such that $0\le \eta_N\le 1$, $\eta_N=1$ on $Q_{N-1}$, and $|\nabla\eta_N|\le 2$ $\mathcal L^n$-a.e. on $Q_N$. Define the map $\varphi_N:Q_N\to\mathbb R^m$ by
\begin{align*}
\varphi_N(x)=|\xi|\eta_N(x)a h(e\cdot x).
\end{align*}
Since $h$ is Lipschitz and $x\mapsto e\cdot x$ is smooth, the Sobolev chain rule gives $h(e\cdot x)\in W^{1,\infty}(Q_N)$ with weak gradient $h'(e\cdot x)e$ for $\mathcal L^n$-a.e. $x\in Q_N$. Since $\eta_N\in W^{1,\infty}_0(Q_N)$ and $h$ is bounded, $\varphi_N\in W^{1,\infty}_0(Q_N;\mathbb R^m)$. By the product rule for Lipschitz Sobolev functions, for $\mathcal L^n$-a.e. $x\in Q_N$,
\begin{align*}
\nabla\varphi_N(x)=|\xi|a h(e\cdot x)\otimes \nabla\eta_N(x)+\eta_N(x)a\otimes \xi\, h'(e\cdot x).
\end{align*}
On $Q_{N-1}$, where $\eta_N=1$ and $\nabla\eta_N=0$ a.e., this simplifies to $\nabla\varphi_N(x)=h'(e\cdot x)B$. Therefore, on $Q_{N-1}$, the matrix $A+rB+\nabla\varphi_N(x)$ equals $A+sB$ on the layers where $h'(e\cdot x)=s-r$, and equals $A+tB$ on the layers where $h'(e\cdot x)=t-r$.
[/step]
[step:Apply quasiconvexity and isolate the boundary-layer error]
By quasiconvexity applied on $Q_N$ with base matrix $A+rB$ and [test function](/page/Test%20Function) $\varphi_N\in W^{1,\infty}_0(Q_N;\mathbb R^m)$,
\begin{align*}
f(A+rB)\mathcal L^n(Q_N)
\le
\int_{Q_N} f(A+rB+\nabla\varphi_N(x))\,d\mathcal L^n(x).
\end{align*}
Define the compact set
\begin{align*}
K:=\left\{A+rB+C\in\mathbb R^{m\times n}: |C|\le |\xi||a|M_1+2|\xi||a|M_h\right\}.
\end{align*}
For $\mathcal L^n$-a.e. $x\in Q_N$, the gradient formula gives
\begin{align*}
|\nabla\varphi_N(x)|
\le
|\xi||a|M_1+2|\xi||a|M_h,
\end{align*}
so $A+rB+\nabla\varphi_N(x)\in K$. Since $f$ is continuous, define
\begin{align*}
C_K:=\sup_{C\in K}|f(C)|<\infty.
\end{align*}
Splitting the integral over $Q_{N-1}$ and $Q_N\setminus Q_{N-1}$ gives
\begin{align*}
f(A+rB)\mathcal L^n(Q_N)
\le
\int_{Q_{N-1}} f(A+rB+h'(e\cdot x)B)\,d\mathcal L^n(x)
+
C_K\mathcal L^n(Q_N\setminus Q_{N-1}).
\end{align*}
[/step]
[step:Compute the interior layer average and let the boundary error vanish]
Define the bounded $1$-periodic [measurable function](/page/Measurable%20Function) $F:\mathbb R\to\mathbb R$ by
\begin{align*}
F(\theta)=f(A+rB+h'(\theta)B).
\end{align*}
Its one-period average is
\begin{align*}
\int_0^1 F(\theta)\,d\mathcal L^1(\theta)=\lambda f(A+sB)+(1-\lambda)f(A+tB),
\end{align*}
because $h'=s-r$ on $(0,\lambda)$ and $h'=t-r$ on $(\lambda,1)$.
[claim:Periodic directional averages over expanding cubes converge to the period average]
For every bounded $1$-periodic measurable function $F:\mathbb R\to\mathbb R$ and every unit vector $e\in\mathbb R^n$,
\begin{align*}
\lim_{R\to\infty}\frac{1}{\mathcal L^n((-R,R)^n)}\int_{(-R,R)^n}F(e\cdot x)\,d\mathcal L^n(x)=\int_0^1F(\theta)\,d\mathcal L^1(\theta).
\end{align*}
[/claim]
[proof]
Let $A_R$ denote the averaging functional
\begin{align*}
A_R(G):=\frac{1}{\mathcal L^n((-R,R)^n)}\int_{(-R,R)^n}G(e\cdot x)\,d\mathcal L^n(x)
\end{align*}
for bounded $1$-periodic [measurable functions](/page/Measurable%20Functions) $G:\mathbb R\to\mathbb C$. First suppose $G(\theta)=e^{2\pi i k\theta}$ for an integer $k\in\mathbb Z$. The function $x\mapsto G(e\cdot x)$ is bounded on $(-R,R)^n$, hence integrable with respect to $\mathcal L^n$, so [Fubini's theorem](/theorems/2961) applies. If $k=0$, then $A_R(G)=1$. If $k\ne0$, choose an index $j\in\{1,\dots,n\}$ such that $e_j\ne0$. Fubini's theorem gives a product of one-dimensional averages, and the $j$th factor is
\begin{align*}
\frac{1}{2R}\int_{-R}^{R}e^{2\pi i k e_j x_j}\,d\mathcal L^1(x_j).
\end{align*}
Its absolute value is at most $(\pi |k e_j|R)^{-1}$, so it tends to $0$ as $R\to\infty$. Thus the assertion holds for trigonometric monomials and, by linearity, for trigonometric polynomials.
It remains to pass from trigonometric polynomials to bounded measurable functions. Choose $j\in\{1,\dots,n\}$ with $e_j\ne0$. For every nonnegative $1$-periodic measurable function $G:\mathbb R\to[0,\infty)$ and every $R\ge1$, fixing all variables except $x_j$ gives
\begin{align*}
\frac{1}{2R}\int_{-R}^{R}G(e\cdot x)\,d\mathcal L^1(x_j)\le \left(1+\frac{1}{2|e_j|}\right)\int_0^1G(\theta)\,d\mathcal L^1(\theta).
\end{align*}
Indeed, for the fixed values of the other coordinates, we can write $e\cdot x=e_jx_j+\gamma$ for some constant $\gamma\in\mathbb R$ depending on those coordinates. The interval $e_j[-R,R]+\gamma$ has length $2R|e_j|$ modulo period $1$, so it covers each point of the torus at most $\lceil 2R|e_j|\rceil$ times, and $\lceil 2R|e_j|\rceil/(2R|e_j|)\le 1+1/(2|e_j|)$. Integrating this estimate over the remaining $n-1$ variables yields
\begin{align*}
|A_R(G)|\le C_e\int_0^1|G(\theta)|\,d\mathcal L^1(\theta),\qquad C_e:=1+\frac{1}{2|e_j|},
\end{align*}
for every bounded $1$-periodic measurable $G:\mathbb R\to\mathbb C$.
Let $F:\mathbb R\to\mathbb R$ be bounded, measurable, and $1$-periodic. Since trigonometric polynomials are dense in $L^1(0,1)$, for every $\varepsilon>0$ choose a trigonometric polynomial $P:\mathbb R\to\mathbb C$ with
\begin{align*}
\int_0^1|F(\theta)-P(\theta)|\,d\mathcal L^1(\theta)<\varepsilon.
\end{align*}
The preceding bound gives $|A_R(F-P)|\le C_e\varepsilon$ for all $R\ge1$, while the polynomial case gives
\begin{align*}
\lim_{R\to\infty}A_R(P)=\int_0^1P(\theta)\,d\mathcal L^1(\theta).
\end{align*}
Therefore
\begin{align*}
\limsup_{R\to\infty}\left|A_R(F)-\int_0^1F(\theta)\,d\mathcal L^1(\theta)\right|\le (C_e+1)\varepsilon.
\end{align*}
Letting $\varepsilon\downarrow0$ proves the claim.
[/proof]
Applying the claim with $R=N-1$ gives
\begin{align*}
\lim_{N\to\infty}
\frac{1}{\mathcal L^n(Q_{N-1})}
\int_{Q_{N-1}} f(A+rB+h'(e\cdot x)B)\,d\mathcal L^n(x)
=
\lambda f(A+sB)+(1-\lambda)f(A+tB).
\end{align*}
Also,
\begin{align*}
\lim_{N\to\infty}
\frac{\mathcal L^n(Q_N\setminus Q_{N-1})}{\mathcal L^n(Q_N)}
=
0,
\qquad
\lim_{N\to\infty}
\frac{\mathcal L^n(Q_{N-1})}{\mathcal L^n(Q_N)}
=
1.
\end{align*}
After dividing the quasiconvexity estimate by $\mathcal L^n(Q_N)$, the interior term is
\begin{align*}
\frac{\mathcal L^n(Q_{N-1})}{\mathcal L^n(Q_N)}\cdot\frac{1}{\mathcal L^n(Q_{N-1})}\int_{Q_{N-1}} f(A+rB+h'(e\cdot x)B)\,d\mathcal L^n(x).
\end{align*}
The first factor tends to $1$, the second factor tends to $\lambda f(A+sB)+(1-\lambda)f(A+tB)$, and the boundary-layer term tends to $0$. Passing to the limit yields
\begin{align*}
f(A+rB)\le \lambda f(A+sB)+(1-\lambda)f(A+tB).
\end{align*}
[/step]
[step:Conclude convexity along every rank-one direction]
We have proved that for arbitrary $s,t\in\mathbb R$ and every $\lambda\in[0,1]$,
\begin{align*}
f(A+(\lambda s+(1-\lambda)t)B)
\le
\lambda f(A+sB)+(1-\lambda)f(A+tB).
\end{align*}
Thus the map $g_{A,B}:\mathbb R\to\mathbb R$ defined by
\begin{align*}
g_{A,B}(\tau)=f(A+\tau B)
\end{align*}
is convex. Since $A\in\mathbb R^{m\times n}$ and the rank-one matrix $B\in\mathbb R^{m\times n}$ were arbitrary, $f$ is rank-one convex.
[/step]