Existence of nontrivial solutions for a fractional p & q $p\&q$ -Laplacian equation with sandwich-type and sign-changing nonlinearities

Abstract In this paper, we deal with the following fractional p & q $p\&q$ -Laplacian problem: { ( − Δ ) p s u + ( − Δ ) q s u = λ a ( x ) | u | θ − 2 u + μ b ( x ) | u | r − 2 u in Ω , u ( x ) = 0 in R N ∖ Ω , $$ \left \{ \textstyle\begin{array}{l@{\quad }l} (-\Delta )_{p}^{s}u +(-\Delta )_{q}^{s}u =\lambda a(x)|u|^{\theta -2}u+ \mu b(x)|u|^{r-2}u&\text{in}\;\ \Omega , \\ u(x)=0 &\text{in}\;\ \mathbb{R}^{N}\setminus \Omega , \end{array}\displaystyle \right . $$ where Ω ⊂ R N $\Omega \subset \mathbb{R}^{N}$ is a bounded domain with smooth boundary, s ∈ ( 0 , 1 ) $s\in (0,1)$ , ( − Δ ) m s $(-\Delta )_{m}^{s}$ ( m ∈ { p , q } ) $(m\in \{p,q\})$ is the fractional m-Laplacian operator, p , q , r , θ ∈ ( 1 , p s ∗ ] $p,q,r,\theta \in (1,p_{s}^{*}]$ , p s ∗ = N p N − s p $p_{s}^{*}=\frac{Np}{N-sp}$ , λ , μ > 0 $\lambda , \mu >0$ , and the weights a ( x ) $a(x)$ and b ( x ) $b(x)$ are possibly sign changing. Using the concentration compactness principle for fractional Sobolev spaces and the Ekeland variational principle, we prove that the problem admits a nonnegative solution for the critical case r = p s ∗ $r=p_{s}^{*}$ . Moreover, for the subcritical case r < p s ∗ $r< p_{s}^{*}$ , we obtain two existence results by applying the Ekeland variational principle and the mountain pass theorem.