Transformer (neural network architecture)

URI: https://gptkb.org/entity/Transformer_(neural_network_architecture)

GPTKB entity

Statements (104)

Predicate	Object
gptkbp:instanceOf	gptkb:convolutional_neural_network
gptkbp:activatedBy	gptkb:ReLU
gptkbp:architecture	encoder-decoder
gptkbp:component	self-attention layer normalization multi-head attention positional encoding residual connection feed-forward network
gptkbp:decoderComponent	stacked decoder layers
gptkbp:encoderComponent	stacked encoder layers
gptkbp:features	attention mechanism scalability parallelization long-range dependency modeling
https://www.w3.org/2000/01/rdf-schema#label	Transformer (neural network architecture)
gptkbp:input	integer sequence
gptkbp:inputEmbedding	word embedding positional encoding
gptkbp:inspiredBy	gptkb:BART gptkb:T5 gptkb:BERT gptkb:ALBERT gptkb:DistilBERT gptkb:GPT gptkb:RoBERTa gptkb:Vision_Transformer gptkb:XLNet
gptkbp:introduced	gptkb:Illia_Polosukhin gptkb:Łukasz_Kaiser gptkb:Aidan_N._Gomez gptkb:Ashish_Vaswani gptkb:Jakob_Uszkoreit gptkb:Llion_Jones gptkb:Niki_Parmar gptkb:Attention_Is_All_You_Need gptkb:Noam_Shazeer
gptkbp:introducedIn	2017
gptkbp:limitation	quadratic memory complexity inefficiency with very long sequences
gptkbp:openSource	gptkb:TensorFlow gptkb:PyTorch gptkb:Hugging_Face_Transformers
gptkbp:output	integer sequence
gptkbp:outputEmbedding	linear projection
gptkbp:publishedAtConference	gptkb:NeurIPS_2017
gptkbp:replacedBy	gptkb:long_short-term_memory recurrent neural network gated recurrent unit
gptkbp:type	scaled dot-product attention
gptkbp:usedFor	machine translation natural language processing image processing speech processing question answering text summarization
gptkbp:variant	gptkb:Informer gptkb:BART gptkb:Electra gptkb:DETR gptkb:Pegasus gptkb:T5 gptkb:Funnel_Transformer gptkb:Switch_Transformer gptkb:XLM-R gptkb:mT5 gptkb:GShard gptkb:Swin_Transformer gptkb:BEiT gptkb:LayoutLM gptkb:Perceiver gptkb:Perceiver_IO gptkb:ERNIE gptkb:BERT gptkb:ALBERT gptkb:BigBird gptkb:DeBERTa gptkb:DistilBERT gptkb:GPT gptkb:Longformer gptkb:MiniLM gptkb:MobileBERT gptkb:RoBERTa gptkb:Sparse_Transformer gptkb:TinyBERT gptkb:Transformer-XL gptkb:Vision_Transformer gptkb:XLNet actor Reformer Data2Vec Linformer Music Transformer BigGAN-Transformer ConvBERT Graph Transformer LongT5 Speech-Transformer SwinIR TabTransformer Universal Transformer ViT-GPT2
gptkbp:bfsParent	gptkb:Łukasz_Kaiser
gptkbp:bfsLayer	7