On the boundary theory of subordinate killed L\'evy processes

Let $Z$ be a subordinate Brownian motion in ${\mathbb R}^d$, $d\ge 2$, via a subordinator with Laplace exponent $\phi$. We kill the process $Z$ upon exiting a bounded open set $D\subset {\mathbb R}^d$ to obtain the killed process $Z^D$, and then we subordinate the process $Z^D$ by a subordinator with Laplace exponent $\psi$. The resulting process is denoted by $Y^D$. Both $\phi$ and $\psi$ are assumed to satisfy certain weak scaling conditions at infinity. We study the potential theory of $Y^D$, in particular the boundary theory. First, in case that $D$ is a $\kappa$-fat bounded open set, we show that the Harnack inequality holds. If, in addition, $D$ satisfies the local exterior volume condition, then we prove the Carleson estimate. In case $D$ is a smooth open set and the lower weak scaling index of $\psi$ is strictly larger than $1/2$, we establish the boundary Harnack principle with explicit decay rate near the boundary of $D$. On the other hand, when $\psi(\lambda)=\lambda^{\gamma}$ with $\gamma\in (0,1/2]$, we show that the boundary Harnack principle near the boundary of $D$ fails for any bounded $C^{1,1}$ open set $D$. Our results give the first example where the Carleson estimate holds true, but the boundary Harnack principle does not. One of the main ingredients in the proofs is the sharp two-sided estimates of the Green function of $Y^D$. Under an additional condition on $\psi$, we establish sharp two-sided estimates of the jumping kernel of $Y^D$ which exhibit some unexpected boundary behavior. We also prove a boundary Harnack principle for non-negative functions harmonic in a smooth open set $E$ strictly contained in $D$, showing that the behavior of $Y^D$ in the interior of $D$ is determined by the composition $\psi\circ \phi$.
Comment: A few typos corrected. Accepted for publication in Potential Analysis (55 pp). arXiv admin note: text overlap with arXiv:1610.00872