Google Switch Transformer

URI: https://gptkb.org/entity/Google_Switch_Transformer
GPTKB entity
Statements (29)
Predicate Object
gptkbp:instanceOf gptkb:convolutional_neural_network
gptkbp:announced 2021
gptkbp:author gptkb:Noam_Shazeer
gptkb:Barret_Zoph
gptkb:William_Fedus
gptkbp:citation 1000+
gptkbp:developedBy gptkb:Google
gptkbp:enables scaling to trillion-parameter models
gptkbp:feature activates only a subset of model parameters per input
https://www.w3.org/2000/01/rdf-schema#label Google Switch Transformer
gptkbp:improves gptkb:T5
gptkb:GPT-3
dense transformer models
gptkbp:notablePublication gptkb:Switch_Transformers:_Scaling_to_Trillion_Parameter_Models_with_Simple_and_Efficient_Sparsity
gptkbp:openSource no
gptkbp:organization gptkb:Google_Research
gptkbp:parameter 1.6 trillion
gptkbp:publishedIn gptkb:arXiv
gptkbp:purpose natural language processing
language modeling
gptkbp:reduces computational cost
gptkbp:relatedTo gptkb:T5
gptkb:GPT-3
gptkb:Mixture_of_Experts
gptkbp:type gptkb:model
sparse model
gptkbp:uses gptkb:Mixture_of_Experts
gptkbp:bfsParent gptkb:Mixture_of_Experts
gptkbp:bfsLayer 6