Fourier Coefficients of Automorphic Forms and Arthur Classification A DISSERTATION SUBMITTED TO THE FACULTY OF THE GRADUATE SCHOOL OF THE UNIVERSITY OF MINNESOTA BY Baiying Liu IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF Doctor of Philosophy Prof. Dr. Dihua Jiang May, 2013 (cid:13)c Baiying Liu 2013 ALL RIGHTS RESERVED Acknowledgements There are many people that have earned my gratitude for their contribution to my time in graduate school. First and foremost, I would like to take this opportunity to express my deepest grat- itude to my advisor Prof. Dihua Jiang, for introducing me to the topics of Fourier coef- ficients of automorphic forms, automorphic descent, constructions of square-integrable automorphic representations, and representations of p-adic groups, for sharing with me his wonderful ideas and insights to various problems of mathematics, for his constant encouragement and support. I would like to thank Prof. James Arthur, Prof. David Ginzburg, Prof. David Soudry for helpful conversations and communications. And I would like to thank Prof. Freydoon Shahidi for helpful comments on the paper of Fourier coefficients of automor- phic forms of GL . n I also would like to thank my committee members, Prof. Paul Garrett, Prof. Kai- Wen Lan and Prof. Richard McGehee, for reviewing my thesis and serving on the committee of my thesis defence. I really appreciate their help. Finally, I would like to thank my academic elder brothers Dr. Lei Zhang and Dr. Xin Shen, for their helpful discussion and comments. i Abstract Fourier coefficients play important roles in the study of both classical modular forms and automorphic forms. For example, it is a well-known theorem of Shalika and Piatetski-Shapiro that cuspidal automorphic forms of GL (A) are globally generic, n that is, have non-degenerate Whittaker-Fourier coefficients, which is proved by taking Fourier expansion. For general connected reductive groups, there is a framework of attaching Fourier coefficients to nilpotent orbits. For general linear groups and classical groups, nilpotent orbits are parametrized by partitions. Given any automorphic repre- sentation π of general linear groups or classical groups, characterizing the set nm(π) of maximal partitions with corresponding nilpotent orbits providing non-vanishing Fourier coefficients is an interesting question, and has applications in automorphic descent and construction of endoscopic lifting. In this thesis, first, we extend the Fourier expansion of cuspidal automorphic forms of GL (A) to any automorphic form occurring in the discrete spectrum of GL (A). n n Then,wedeterminethesetnm(π)foranyresidualrepresentation∆(τ,m)ofGL (A) 2mn (withτ anirreducibleunitarycuspidalautomorphicrepresentationofGL (A))andcer- 2n tainresidualrepresentationE ofSp (A)constructedfrom∆(τ,m)byJiang,Liu ∆(τ,m) 4mn and Zhang. Next, we consider certain set of irreducible cuspidal automorphic representations of Sp (A) which are nearly equivalent to the residual representation E . We show 4mn ∆(τ,m) that this set decomposes naturally into two disjoint sets, corresponding to certain sets of irreducible cuspidal automorphic representations of S(cid:102)p (A) and S(cid:102)p (A), 4mn−2n 4mn+2n respectively. This extends the Ginzburg-Jiang-Soudry correspondences between certain automorphic forms on Sp4n(A) and S(cid:102)p2n(A). At last, we recall Arthur’s classification of the discrete spectrum and a conjecture of Jiang towards understanding Fourier coefficients of automorphic forms in automorphic L2-packets, and briefly discuss the relation between them and the above results in this thesis. ii Contents Acknowledgements i Abstract ii 1 Introduction 1 1.1 On Fourier Coefficients of Automorphic Forms of GL . . . . . . . . . . 2 n 1.2 OnExtensionofGinzburg-Jiang-SoudryCorrespondencestoCertainAu- tomorphic Forms on Sp4mn(A) and S(cid:102)p4mn±2n(A) . . . . . . . . . . . . . 4 1.3 On Arthur Classification and Jiang’s Conjecture . . . . . . . . . . . . . 7 2 On Fourier Coefficients of Automorphic Forms of GL 9 n 2.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 2.2 Discrete Spectrum of GL . . . . . . . . . . . . . . . . . . . . . . . . . . 11 n 2.2.1 Structure of discrete spectrum . . . . . . . . . . . . . . . . . . . 11 2.2.2 Fourier expansion for cuspidal automorphic forms . . . . . . . . 12 2.3 Fourier Expansion for the Discrete Spectrum . . . . . . . . . . . . . . . 13 2.3.1 Families of Fourier coefficients . . . . . . . . . . . . . . . . . . . 14 2.3.2 Fourier expansion: step one . . . . . . . . . . . . . . . . . . . . . 15 2.3.3 Fourier expansion: step two . . . . . . . . . . . . . . . . . . . . . 18 2.4 Proof of Lemma 2.3.2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 2.5 Fourier Coefficients for GL . . . . . . . . . . . . . . . . . . . . . . . . . 26 n 2.5.1 Fourier coefficients for GL . . . . . . . . . . . . . . . . . . . . . 27 n 2.5.2 Fourier coefficients for the discrete spectrum of GL . . . . . . . 33 n 2.6 Proof of Theorem 2.5.5 . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 iii 3 On Fourier Coefficients of Automorphic Forms of Symplectic Groups 51 3.1 Fourier Coefficients Automorphic Forms of Symplectic Groups Attached to Nilpotent Orbits . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51 3.2 Fourier-Jacobi Coefficients and Automorphic Descent . . . . . . . . . . . 61 4 On Extension of Ginzburg-Jiang-Soudry Correspondences to Certain Automorphic Forms on Sp4mn(A) and S(cid:102)p4mn±2n(A) 63 4.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64 4.2 Proof of Theorem 4.1.1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69 4.3 Proof of Lemma 4.2.4 . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81 4.3.1 ω -term . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83 0 4.3.2 ω -term . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84 1 4.4 Proof of Part (1) of Theorem 4.1.2 . . . . . . . . . . . . . . . . . . . . . 86 4.5 Proof of Theorem 4.4.6 . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106 4.6 Proof of Part (2) of Theorem 4.1.2 . . . . . . . . . . . . . . . . . . . . . 110 4.7 Irreducibility of Certain Descent Representations . . . . . . . . . . . . . 110 5 On Arthur Classification and Jiang’s Conjecture 115 References 117 iv Chapter 1 Introduction In the study of classical modular forms, it is known that the Fourier coefficients carry significantarithmeticinformation. Forexample,theSato–TateConjecturesaysthat,for certain holomorphic cusp forms corresponding to (non-CM) elliptic curves over rational field, the angles corresponding to their Fourier coefficients distribute in a certain way, this is related to the number of points of the reductions modulo primes of elliptic curves (this conjecture has been proved by Barnet-Lamb, Geraghty, Harris and Taylor [BLGHT11]). Fourier coefficients also play an important role in the study of automorphic forms. For example, a basic and fundamental result in the theory of automorphic forms for GL (A) is that cuspidal automorphic forms are globally generic, that is, have non- n vanishing Whittaker-Fourier coefficients, due to Shalika [S74] and Piatetski-Shapiro [PS79] independently. This has been used to prove the strong multiplicity one theo- rem for cuspidal automorphic representations of GL (A). n As another example, for classical groups, Ginzburg, Rallis and Soudry ([GRS11]) developed the theory of automorphic descent by studying certain Fourier coefficients of special type residual representations, which produces the inverse of the Langlands functorial transfers from classical groups to the general linear groups. 1 2 1.1 On Fourier Coefficients of Automorphic Forms of GL n It is a well-known theorem of Shalika and Piatetski-Shapiro that cuspidal automorphic forms ϕ of GL (A) are globally generic, that is, have non-vanishing Whittaker-Fourier n coefficients. ThistheoremisprovedusingtheFourierexpansionofϕalongthestandard maximal unipotent subgroup U of GL . Since it is not abelian, Shalika and Piatetski- n n Shapiro’s idea is to consider abelian subgroups of U consisting of only column elements n and take the Fourier expansion column-by-column. Theorem 1.1.1 (Shalika [S74], Piatetski-Shapiro [PS79]). Assume that ϕ is a cuspidal automorphic form of GL (A). Then, n (cid:88) ϕ(g) = Wψ(ϕ,ι (γ)g), (1.1) n γ∈Un−1(F)\GLn−1(F) (cid:32) (cid:33) γ 0 where ι (γ) = , and n 0 1 (cid:90) Wψ(ϕ,g) := ϕ(ug)ψ−1(u)du, Un(k)\Un(A) Un with ψ (u) := ψ(u +u +···+u ). (1.2) Un 1,2 2,3 n−1,n Moreover, the Fourier expansion in (1.1) is absolutely convergent and uniformly con- verges on any compact set in g. Wψ(ϕ,g) is called a non-degenerate Whittaker Fourier coefficient of ϕ. In [JL12], joint with Jiang, we extend Theorem 1.1.1 to any automorphic forms of GL (A) occurring in the discrete spectrum. n The explicit construction of the discrete spectrum of GL (A) was conjectured by n Jacquet ([J84]) and then proved by Moeglin and Waldspurger ([MW89]). It turns out that an irreducible automorphic representation π of GL (A) occurring in the discrete n spectrum is parametrized by a pair (τ,b) with τ an irreducible unitary cuspidal auto- morphic representation of GL (A), for some pair a,b of integers such that n = ab. In a particular, if π is also cuspidal, then b = 1. The irreducible automorphic representation parametrized by (τ,b) can be denoted by E (or ∆(τ,b) sometimes), which is called (τ,b) a Speh residual representation if b > 1. For more discussion, see Section 2.2.1. 3 Theorem 1.1.2 (Jiang and Liu [JL12]). Assume that ϕ ∈ E , n = ab. Then, (τ,b) ϕ(g) = (cid:88)Wψ (ϕ,γg), (1.3) (τ,b) γ where γ is a certain rational matrix which will be specified in Theorem 2.3.3, and (cid:90) W(ψτ,b)(ϕ,g) := ϕ(ug)ψ(cid:103)Un−1(u)du, Un(k)\Un(A) with n−1 b−1 (cid:88) (cid:88) ψ(cid:103)Un(u) := ψ( ui,i+1− uja,ja+1). (1.4) i=1 j=1 Moreover, the Fourier expansion in (1.3) is absolutely convergent and uniformly con- verges on any compact set in g. Wψ (ϕ,g) is called a degenerate Whittaker Fourier coefficient of ϕ if b > 1. (τ,b) Note that after comparing the characters in (1.2) and (1.4), we can see that the character occurring in the Fourier expansion of ϕ ∈ E only involves “simple roots (τ,b) inside the cuspidal support”. Also note that the Fourier coefficients in both (1.1) and (1.3)havethesameintegrationdomain. Whenb = 1,Theorem1.1.2reducestoTheorem 1.1.1. If one takes an arbitrary automorphic form of GL (A), one can still apply the idea n of Shalika and Piatetski-Shapiro to do the Fourier expansion, but the resulting formula might be complicated. Such a general formula can be found in [[Y93], Proposition 2.1.3]. However, Theorem 1.1.2 gives the explicit summation domain for γ in terms of the cuspidal support of the residual representation E . (τ,b) For classical groups, on one hand, it is very hard to get a Fourier expansion as above, due to the complicate group structures. On the other hand, there exist non- generic cuspidal automorphic forms in general. Therefore, the idea of “taking off simple roots outside of cuspidal support” doesn’t work, since the degenerate Whittaker Fourier coefficients like Wψ (ϕ,g) will all be killed by cuspidality. (τ,b) For general linear groups and classical groups, there is a general framework of at- taching Fourier coefficients to nilpotent orbits which are classified by partitions, gener- alizing the Whittaker-Fourier coefficients. See Section 2.5.1 (for GL ) and Section 3.1 n (for Sp ), for explicit discussions. Also, see [[J12], Section 4] for more discussions and 2n related applications. 4 Given an automorphic representation π, let n(π) be the set of all partitions such that there is a corresponding nilpotent orbit providing non-vanishing Fourier coefficient to π. Let nm(π) be the subset of n(π) consisting of only maximal elements under the natural ordering of partitions. Characterizing the set nm(π) is an interesting question, and has applications to automorphic descent and endoscopic constructions. In Sections 2.5 and 2.6, we will figure out the set nm(E ) (n = ab). Note that (τ,b) Ginzburg in [G06] gives a sketch of a proof of this result (Proposition 5.3, [G06]) with an argument combining local and global methods. We give here a global proof with full details. Theorem 1.1.3 (Ginzburg [G06], Jiang and Liu [JL12]). nm(E ) = {[ab]}. (τ,b) Now, forE , wehavetwokindsofFouriercoefficients, oneisWψ (ϕ,g), obtained (τ,b) (τ,b) from the Fourier expansion, the other one is attached to the nilpotent orbit correspond- ing to the partition [ab]. One may ask that what is the relation between them? It turns out that they share the same non-vanishing property. 1.2 On Extension of Ginzburg-Jiang-Soudry Correspon- dences to Certain Automorphic Forms on Sp (A) and 4mn S(cid:102)p (A) 4mn±2n Let τ be an irreducible unitary cuspidal automorphic representation of GL (A), with 2n the properties that L(s,τ,∧2) has a simple pole at s = 1, and L(1,τ) (cid:54)= 0. 2 Let P = M N be the maximal parabolic subgroup of Sp with Levi subgroup M r r r 2l r isomorphic to GL ×Sp . Using the normalization in [Sh10], the group XSp2l of all r 2l−2r Mr continueshomomorphismsfromM (A)toC×, whichistrivialonM (A)1 (see[MW95]), r r will be identified with C by s → λ . s Let∆(τ,m)beaSpehresidualrepresentationinthediscretespectrumofGL (A). 2mn For any φ ∈ A(N (A)M (F)\Sp (A)) , following [L76] and [MW95], an 2mn 2mn 4mn ∆(τ,m) residual Eisenstein series can be defined by (cid:88) E(φ,s)(g) = λ φ(γg). s γ∈P2mn(F)\Sp4mn(F)