Markov invariants, plethysms, and phylogenetics (the long version)

We explore model based techniques of phylogenetic tree inference exercising Markov invariants. Markov invariants are group invariant polynomials and are distinct from what is known in the literature as phylogenetic invariants, although we establish a commonality in some special cases. We show that the simplest Markov invariant forms the foundation of the Log-Det distance measure. We take as our primary tool group representation theory, and show that it provides a general framework for analysing Markov processes on trees. From this algebraic perspective, the inherent symmetries of these processes become apparent, and focusing on plethysms, we are able to define Markov invariants and give existence proofs. We give an explicit technique for constructing the invariants, valid for any number of character states and taxa. For phylogenetic trees with three and four leaves, we demonstrate that the corresponding Markov invariants can be fruitfully exploited in applied phylogenetic studies.
Comment: 39 pages, 10 figures, 2 tables, 3 appendices. Long arxiv version includes extended introduction, subsection on mixed-weight invariants, 3rd appendix on K3ST model and a more relaxed pace with additional discussion throughout. "Short version" is to appear in Journal of Theoretical Biology. v4: Sequence length in simulation was corrected from N=1000 to N=10000