Journal Article

PUBDB-2025-01273

http://join2-wiki.gsi.de/foswiki/pub/Main/Artwork/join2_logo100x88.png Analysis of mean-field models arising from self-attention dynamics in transformer architectures with layer normalization

Burger, M. (Corresponding author)DESY* ; Kabri, S.DESY* ; Korolev, Y. ; Roith, T.Extern*DESY* ; Weigand, L.DESY*

2025
Royal Soc. London

This record in other databases:    

Please use a persistent id in citations: doi:10.1098/rsta.2024.0233  doi:10.3204/PUBDB-2025-01273

Abstract: The aim of this paper is to provide a mathematical analysis of transformer architectures using aself-attention mechanism with layer normalization. In particular, observed patterns in such architecturesresembling either clusters or uniform distributions pose a number of challenging mathematical questions.We focus on a special case that admits a gradient flow formulation in the spaces of probability measureson the unit sphere under a special metric, which allows us to give at least partial answers in a rigorousway. The arising mathematical problems resemble those recently studied in aggregation equations, butwith additional challenges emerging from restricting the dynamics to the sphere and the particular formof the interaction energy.We provide a rigorous framework for studying the gradient flow, which also suggests a possible metricgeometry to study the general case (i.e. one that is not described by a gradient flow). We further analyzethe stationary points of the induced self-attention dynamics. The latter are related to stationary pointsof the interaction energy in the Wasserstein geometry, and we further discuss energy minimizers andmaximizers in different parameter settings.

Note: ISSN 1471-2962 not unique: **2 hits**.

---

Contributing Institute(s):

1. Computational Imaging (FS-CI)

Research Program(s):

1. 623 - Data Management and Analysis (POF4-623) (POF4-623)
2. DFG project G:(GEPRIS)464101359 - Deep-Learning basierte Regularisierung inverser Probleme (464101359) (464101359)
3. DFG project G:(GEPRIS)464101190 - Theoretischer Grundlagen des Unsicherheits-robusten Deep Learning für Inverse Probleme (464101190) (464101190)

Experiment(s):

1. No specific instrument

Appears in the scientific report 2025

---

Database coverage:
; ; ; Clarivate Analytics Master Journal List ; Current Contents - Physical, Chemical and Earth Sciences ; Ebsco Academic Search ; Essential Science Indicators ; IF >= 5 ; JCR ; National-Konsortium ; PubMed Central ; SCOPUS ; Science Citation Index Expanded ; Web of Science Core Collection

---