{"id":40881,"date":"2026-09-17T11:15:54","date_gmt":"2026-09-17T11:15:54","guid":{"rendered":"https:\/\/lamarr-institute.org\/publication\/grokking-is-a-geometric-phase-transition-archetypal-analysis-reveals-causal-structure-in-transformer-representations\/"},"modified":"2026-09-17T11:15:54","modified_gmt":"2026-09-17T11:15:54","slug":"grokking-is-a-geometric-phase-transition-archetypal-analysis-reveals-causal-structure-in-transformer-representations","status":"publish","type":"publication","link":"https:\/\/lamarr-institute.org\/de\/publication\/grokking-is-a-geometric-phase-transition-archetypal-analysis-reveals-causal-structure-in-transformer-representations\/","title":{"rendered":"Grokking Is a\u00a0Geometric Phase Transition: Archetypal Analysis Reveals Causal Structure in\u00a0Transformer Representations"},"content":{"rendered":"<p>We show that grokking, the sudden generalization of overfit neural networks, coincides with a geometric phase transition in representation space. Using Archetypal Analysis (AA), we decompose residual-stream activations of single-layer Transformers trained on modular arithmetic into convex combinations of extreme points. Before grokking, activations are diffuse. At the transition, they reorganize into a simplex whose P vertices correspond one-to-one with output classes, and this co-emergence serves as a mechanistic signature of grokking. The simplex structure is then causally validated by direct intervention: swapping a sample\u2019s AA mixture weights to those of a target class redirects the model\u2019s prediction with 98\u2013100\\% accuracy, while k-means ( ${\\sim }6\\%$ ) and unstructured baselines fail. The decomposition requires no label supervision: overcomplete AA with behavior-based merging recovers exactly P groups with 100\\% label alignment across all seeds. Archetype permutations further reveal cyclic group structure, implementing $y \\mapsto y{+}d \\bmod P$ at 80\u201385\\% with $k=P$ and 100\\% with the behavior-merged $k=2P{\\rightarrow }P$ representation, connecting the simplex geometry to the Fourier mode theory of grokking.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>We show that grokking, the sudden generalization of overfit neural networks, coincides with a geometric phase transition in representation space. Using Archetypal Analysis (AA), we decompose residual-stream activations of single-layer Transformers trained on modular arithmetic into convex combinations of extreme points. Before grokking, activations are diffuse. At the transition, they reorganize into a simplex whose P vertices correspond one-to-one with output classes, and this co-emergence serves as a mechanistic signature [&hellip;]<\/p>\n","protected":false},"author":14,"featured_media":0,"template":"","meta":{"_acf_changed":false,"footnotes":""},"publication-type":[32],"class_list":["post-40881","publication","type-publication","status-publish","hentry","publication-type-inproceedings"],"acf":[],"publishpress_future_workflow_manual_trigger":{"enabledWorkflows":[]},"_links":{"self":[{"href":"https:\/\/lamarr-institute.org\/de\/wp-json\/wp\/v2\/publication\/40881","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/lamarr-institute.org\/de\/wp-json\/wp\/v2\/publication"}],"about":[{"href":"https:\/\/lamarr-institute.org\/de\/wp-json\/wp\/v2\/types\/publication"}],"author":[{"embeddable":true,"href":"https:\/\/lamarr-institute.org\/de\/wp-json\/wp\/v2\/users\/14"}],"version-history":[{"count":0,"href":"https:\/\/lamarr-institute.org\/de\/wp-json\/wp\/v2\/publication\/40881\/revisions"}],"wp:attachment":[{"href":"https:\/\/lamarr-institute.org\/de\/wp-json\/wp\/v2\/media?parent=40881"}],"wp:term":[{"taxonomy":"publication-type","embeddable":true,"href":"https:\/\/lamarr-institute.org\/de\/wp-json\/wp\/v2\/publication-type?post=40881"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}