Nilpotent orbits and some small unitary representations of indefinite orthogonal groups

For <f>2&les;m&les;l/2</f>, let <f>G</f> be a simply connected Lie group with <f>g0=so(2m,2l−2m)</f> as Lie algebra, let <f>g=k⊕p</f> be the complexification of the usual Cartan decomposition, let <f>K</f> be the analytic subgroup with Lie algebra <f>k∩g0</f>, and let <f>U(g)</f> be the universal enveloping algebra of <f>g</f>. This work examines the unitarity and <f>K</f> spectrum of representations in the “analytic continuation” of discrete series of <f>G</f>, relating these properties to orbits in the nilpotent radical of a certain parabolic subalgebra of <f>g</f>.The roots with respect to the usual compact Cartan subalgebra are all <f>±ei±ej</f> with <f>1&les;i<j&les;l</f>. In the usual positive system of roots, the simple root <f>em−em+1</f> is noncompact and the other simple roots are compact. Let <f>q=l⊕u</f> be the parabolic subalgebra of <f>g</f> for which <f>em−em+1</f> contributes to <f>u</f> and the other simple roots contribute to <f>l</f>, let <f>L</f> be the analytic subgroup of <f>G</f> with Lie algebra <f>l∩g0</f>, let <f>LC=Intg(l)</f>, let <f>2δ(u)</f> be the sum of the roots contributing to <f>u</f>, and let <f>q¯=l⊕u¯</f> be the parabolic subalgebra opposite to <f>q</f>.The members of <f>u∩p</f> are nilpotent members of <f>g</f>. The group <f>LC</f> acts on <f>u∩p</f> with finitely many orbits, and the topological closure of each orbit is an irreducible algebraic variety. If <f>Y</f> is one of these varieties, let <f>R(Y)</f> be the dual coordinate ring of <f>Y</f>; this is a quotient of the algebra of symmetric tensors on <f>u∩p</f> that carries a fully reducible representation of <f>LC</f>.For <f>s∈Z</f>, let <f>λs=∑lower limit k=1, upper limit m (−l+<NU>s</NU>/2)ek</f>. Then <f>λs</f> defines a one-dimensional <f>(l,L)</f> module <f>Cλs</f>. Extend this to a <f>(q¯,L)</f> module by having <f>u¯</f> act by 0, and define <f>N(λs+2δ(u))=U(g)⊗q¯Cλs+2δ(u)</f>. Let <f>N′(λs+2δ(u))</f> be the unique irreducible quotient of <f>N(λs+2δ(u))</f>. The representations under study are <f>πs=ΠS(N(λs+2δ(u)))</f> and <f>πs′=ΠS(N′(λs+2δ(u)))</f>, where <f>S=dim(u∩k)</f> and <f>ΠS</f> is the <f>S</f>th derived Bernstein functor.For <f>s>2l−2</f>, it is known that <f>πs=πs′</f> and that <f>πs′</f> is in the discrete series. Enright, Parthsarathy, Wallach, and Wolf showed for <f>m&les;s&les;2l−2</f> that <f>πs=πs′</f> and that <f>πs′</f> is still unitary. The present paper shows that <f>πs′</f> is unitary for <f>0&les;s&les;m−1</f> even though <f>πs≠πs′</f>, and it relates the <f>K</f> spectrum of the representations <f>πs′</f> to the representation of <f>LC</f> on a suitable <f>R(Y)</f> with <f>Y</f> depending on <f>s</f>. Use of a branching formula of D. E. Littlewood allows one to obtain an explicit multiplicity formula for each <f>K</f> type in <f>πs′</f>; the variety <f>Y</f> is indispensable in the proof. The chief tools involved are an idea of B. Gross and Wallach, a geometric interpretation of Littlewood's theorem, and some estimates of norms.It is shown further that the natural invariant Hermitian form on <f>πs′</f> does not make <f>πs′</f> unitary for <f>s<0</f> and that the <f>K</f> spectrum of <f>πs′</f> in these cases is not related in the above way to the representation of <f>LC</f> on any <f>R(Y)</f>.A final section of the paper treats in similar fashion the simply connected Lie group with Lie algebra <f>g0=so(2m,2l−2m+1)</f>, <f>2&les;m&les;l/2</f>. [Copyright &y& Elsevier]