Transformer-XL

E701503

Transformer variant language model architecture neural network architecture

Transformer-XL is a neural network architecture for language modeling that extends the Transformer with segment-level recurrence and relative positional encodings to better capture long-range dependencies.

Try in SPARQL Jump to: Surface forms Disambiguation Statements Elicitation Referenced by

All labels observed (2)

Label	Occurrences
Transformer-XL canonical	3
Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context	1

How this entity was disambiguated

This entity first appeared as the object of triple T7874884 — resolving that mention is where its identity was fixed. The disambiguator weighed these candidate entities and picked the highlighted one (or “None”, minting a new entity). This is how homonymy is resolved: the same surface form can point to different entities.

NED1 Entity disambiguation (via context triple) gpt-5-mini-2025-08-07

Target entity: Transformer-XL
Context triple: [Layer Normalization, usedIn, Transformer-XL]

A. XLNet
XLNet is a generalized autoregressive pretraining model for natural language processing that improves on BERT by leveraging permutation-based language modeling to better capture bidirectional context.
B. Longformer
Longformer is a transformer-based neural network architecture designed for efficient processing of very long sequences using sparse attention mechanisms.
C. Attention Is All You Need
"Attention Is All You Need" is the landmark 2017 research paper that introduced the Transformer architecture and revolutionized modern natural language processing and sequence modeling.
D. XLM-R
XLM-R is a multilingual transformer-based language model (XLM-RoBERTa) designed for cross-lingual understanding and natural language processing across many languages.
E. DeBERTa
DeBERTa is a transformer-based language model developed by Microsoft that improves upon BERT and RoBERTa using disentangled attention and enhanced mask decoder mechanisms for superior natural language understanding.
F. None of above. chosen
G. Unsure - the case is ambiguous/there is not enough information to decide.

NED2 Entity disambiguation (via description) gpt-5-mini-2025-08-07

Target entity: Transformer-XL
Target entity description: Transformer-XL is a neural network architecture for language modeling that extends the Transformer with segment-level recurrence and relative positional encodings to better capture long-range dependencies.

A. XLNet
XLNet is a generalized autoregressive pretraining model for natural language processing that improves on BERT by leveraging permutation-based language modeling to better capture bidirectional context.
B. Longformer
Longformer is a transformer-based neural network architecture designed for efficient processing of very long sequences using sparse attention mechanisms.
C. Attention Is All You Need
"Attention Is All You Need" is the landmark 2017 research paper that introduced the Transformer architecture and revolutionized modern natural language processing and sequence modeling.
D. XLM-R
XLM-R is a multilingual transformer-based language model (XLM-RoBERTa) designed for cross-lingual understanding and natural language processing across many languages.
E. DeBERTa
DeBERTa is a transformer-based language model developed by Microsoft that improves upon BERT and RoBERTa using disentangled attention and enhanced mask decoder mechanisms for superior natural language understanding.
F. None of above. chosen

Statements (48)

Predicate	Object
instanceOf	Transformer variant ⓘ language model architecture ⓘ neural network architecture ⓘ
addressesLimitationOf	standard Transformer context length ⓘ
aimsTo	capture long-range dependencies ⓘ
appliedTo	character-level language modeling ⓘ word-level language modeling ⓘ
benchmarkedOn	Enwik8 ⓘ One Billion Word Benchmark ⓘ WikiText-103 ⓘ
describedInPaper	Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context ⓘ linked to: Transformer-XL
designedFor	language modeling ⓘ
developedAt	Carnegie Mellon University ⓘ linked to: CMU Google Brain ⓘ
evaluationSpeedupReason	reuse of cached hidden states ⓘ
extends	Transformer ⓘ
hasFullName	Transformer eXtra Long ⓘ
hasKeyConcept	decoupling positional encoding from absolute positions ⓘ reusing hidden states from previous segments ⓘ
improves	evaluation efficiency ⓘ modeling of long-term dependencies ⓘ training efficiency for long sequences ⓘ
improvesMetric	perplexity on language modeling benchmarks ⓘ
influenced	later long-context Transformer architectures ⓘ
introducedFeature	relative positional encodings ⓘ segment-level recurrence ⓘ
memoryMechanismType	segment-level recurrence over hidden states ⓘ
outperforms	standard Transformer on long-context language modeling benchmarks ⓘ
paperPublishedAt	ACL 2019 ⓘ linked to: Annual Meeting of the Association for Computational Linguistics
positionalEncodingType	relative positional encoding ⓘ
proposedBy	Jaime Carbonell ⓘ Quoc V. Le ⓘ Ruslan Salakhutdinov ⓘ William W. Cohen ⓘ Yiming Yang ⓘ Zhilin Yang ⓘ Zihang Dai ⓘ
proposedIn	2019 ⓘ
reduces	context fragmentation ⓘ
supports	longer effective context than vanilla Transformer ⓘ
trainingObjective	autoregressive language modeling ⓘ
uses	layer normalization ⓘ memory mechanism ⓘ multi-head attention ⓘ position-wise feed-forward networks ⓘ relative positional embeddings ⓘ residual connections ⓘ self-attention ⓘ

How these facts were elicited

Referenced by (4)

Full triples — surface form annotated when it differs from this entity's canonical label.

Layer Normalization → usedIn → Transformer-XL ⓘ

XLNet → relatedTo → Transformer-XL ⓘ

XLNet → extends → Transformer-XL ⓘ

Transformer-XL → describedInPaper → Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context ⓘ

linked to: Transformer-XL

All labels observed (2)

How this entity was disambiguated Show

Statements (48)

How these facts were elicited Show

Referenced by (4)

How this entity was disambiguated

How these facts were elicited