Transformer (in some NLP tasks)

GPTKB entity

Predicate	Object
gptkbp:instanceOf	gptkb:convolutional_neural_network
gptkbp:advantage	long-range dependency modeling high memory usage parallelizable training slow inference for long sequences
gptkbp:decoderUses	self-attention encoder-decoder attention
gptkbp:encoderUses	self-attention
gptkbp:hasComponent	decoder encoder
gptkbp:hasVariant	gptkb:BigBird gptkb:Longformer gptkb:Vision_Transformer actor Reformer Linformer
https://www.w3.org/2000/01/rdf-schema#label	Transformer (in some NLP tasks)
gptkbp:implementedIn	gptkb:TensorFlow gptkb:JAX gptkb:PyTorch
gptkbp:input	sequence of tokens
gptkbp:inspiredBy	gptkb:T5 gptkb:BERT gptkb:GPT gptkb:RoBERTa gptkb:XLNet
gptkbp:introduced	gptkb:Vaswani_et_al.
gptkbp:introducedIn	2017
gptkbp:limitation	quadratic time complexity with sequence length
gptkbp:openSource	gptkb:OpenNMT gptkb:Fairseq gptkb:Hugging_Face_Transformers
gptkbp:output	sequence of tokens
gptkbp:publishedIn	gptkb:Attention_Is_All_You_Need
gptkbp:replacedBy	recurrent neural networks gated recurrent units long short-term memory networks
gptkbp:trainer	backpropagation
gptkbp:usedFor	machine translation question answering language modeling text summarization
gptkbp:uses	feed-forward neural network multi-head attention positional encoding self-attention mechanism
gptkbp:bfsParent	gptkb:LSTM
gptkbp:bfsLayer	7