We introduce mod $l$ Milnor invariants of a Galois element associated to Ihara's Galois representation on the pro-$l$ fundamental group of a punctured projective line ($l$ being a prime number), as arithmetic analogues of Milnor invariants of a pure braid. We then show that triple quadratic (resp. cubic) residue symbols of primes in the rational (resp. Eisenstein) number field are expressed by mod $2$ (resp. mod $3$) triple Milnor invariants of Frobenius elements. For this, we introduce dilogarithmic mod $l$ Heisenberg ramified covering ${\cal D}^{(l)}$ of $\mathbb{P}^1$, which may be regarded as a higher analog of the dilogarithmic function, for the gerbe associated to the mod $l$ Heisenberg group, and we study the monodromy transformations of certain functions on ${\cal D}^{(l)}$ along the pro-$l$ longitudes of Frobenius elements for $l=2,3$., Comment: 33 pages, 1 figure