Transformer Concept Representations Anti-Concentrate in Spectral Tails

Analysis of 17 models shows that transformer concept representations systematically anti-concentrate in the spectral tail of the unembedding covariance. This reveals a dual geometry where activation-space concept directions occupy low-variance directions while unembedding-row contrasts concentrate in high-variance directions.