The name of the & character, ampersand, is short for and per se and, meaning and by itself and. People used to recite it at the end of the alphabet. About that much there's no doubt.
But of the two ands in that phrase, which one designates the ampersand? Is it and per se & or & per se and? It seemed clear to me that the former is the correct reading, so I did a little desultory research.
Most sources say the derivation is & per se and, but the story of reciting the alphabet is firmly established and I don't see how and, the conjunction, could possibly appear at the end. The Morris Dictionary of Word and Phrase Origins takes my point of view.
An alternative adopted by the American Heritage and Merriam-Webster dictionaries is that the words and per se and are to be construed as & by itself [means] 'and', but that seems far more strained to me than the natural x, y, z, and per se &.
So I suppose you can say what you like.