Existence of nontrivial solution for Schrödinger–Poisson systems with indefinite steep potential well

In this paper, we study a class of nonlinear Schrodinger–Poisson systems with indefinite steep potential well: $$\begin{aligned} \left\{ \begin{array}{l@{\quad }l} -\Delta u+V_{\lambda }(x)u+K(x)\phi u=|u|^{p-2}u &{} \text { in }\mathbb {R}^{3},\\ -\Delta \phi =K\left( x\right) u^{2} &{} \ \text {in }\mathbb {R}^{3}, \end{array} \right. \end{aligned}$$ where $$30$$ and $$ K(x)\ge 0$$ for all $$x\in \mathbb {R}^{3}$$ . We require that $$a\in C( \mathbb {R}^{3}) $$ is nonnegative and has a potential well $$\Omega _{a}$$ , namely $$a(x)\equiv 0$$ for $$x\in \Omega _{a}$$ and $$a(x)>0$$ for $$x\in \mathbb {R}^{3}\setminus \overline{\Omega _{a}}$$ . Unlike most other papers on this problem, we allow that $$b\in C(\mathbb {R}^{3}) $$ is unbounded below and sign-changing. By introducing some new hypotheses on the potentials and applying the method of penalized functions, we obtain the existence of nontrivial solutions for $$\lambda $$ sufficiently large. Furthermore, the concentration behavior of the nontrivial solution is also described as $$\lambda \rightarrow \infty $$ .