Pattern-avoiding polytopes.

Two well-known polytopes whose vertices are indexed by permutations in the symmetric group S n are the permutohedron P n and the Birkhoff polytope B n . We consider polytopes P n ( Π ) and B n ( Π ) , whose vertices correspond to the permutations in S n avoiding a set of patterns Π . For various choices of Π , we explore the Ehrhart polynomials and h ∗ -vectors of these polytopes as well as other aspects of their combinatorial structure. For P n ( Π ) , we consider all subsets Π ⊆ S 3 and are able to provide results in most cases. To illustrate, P n ( 123 , 132 ) is a Pitman–Stanley polytope, the number of interior lattice points in P n ( 132 , 312 ) is a derangement number, and the normalized volume of P n ( 123 , 231 , 312 ) is the number of trees on n vertices. The polytopes B n ( Π ) seem much more difficult to analyze, so we focus on four particular choices of Π . First we show that the B n ( 231 , 321 ) is exactly the Chan–Robbins–Yuen polytope. Next we prove that for any Π containing { 123 , 312 } we have h ∗ ( B n ( Π ) ) = 1 . Finally, we study B n ( 132 , 312 ) and B ˜ n ( 123 ) , where the tilde indicates that we choose vertices corresponding to alternating permutations avoiding the pattern 123. In both cases we use order complexes of posets and techniques from toric algebra to construct regular, unimodular triangulations of the polytopes. The posets involved turn out to be isomorphic to the lattices of Young diagrams contained in a certain shape, and this permits us to give an exact expression for the normalized volumes of the corresponding polytopes via the hook formula. Finally, Stanley’s theory of ( P , ω ) -partitions allows us to show that their h ∗ -vectors are symmetric and unimodal. Various questions and conjectures are presented throughout. [ABSTRACT FROM AUTHOR]