Google BERT

GPTKB entity

Statements (50)
Predicate Object
gptkbp:instanceOf large language model
gptkbp:architecture gptkb:transformation
gptkbp:attentionMechanism self-attention
gptkbp:author gptkb:Jacob_Devlin
gptkb:Kenton_Lee
gptkb:Ming-Wei_Chang
gptkb:Kristina_Toutanova
gptkbp:citation over 50,000 (as of 2024)
gptkbp:developedBy gptkb:Google
gptkbp:fineTuning task-specific
gptkbp:fullName gptkb:Bidirectional_Encoder_Representations_from_Transformers
gptkbp:github https://github.com/google-research/bert
https://www.w3.org/2000/01/rdf-schema#label Google BERT
gptkbp:impact state-of-the-art on GLUE
state-of-the-art on SQuAD
gptkbp:influenced gptkb:ERNIE
gptkb:ALBERT
gptkb:DistilBERT
gptkb:RoBERTa
gptkb:XLNet
gptkbp:input gptkb:text
gptkbp:inputLengthLimit 512 tokens
gptkbp:language English
gptkbp:license Apache 2.0
gptkbp:notablePublication gptkb:BERT:_Pre-training_of_Deep_Bidirectional_Transformers_for_Language_Understanding
gptkbp:openSource true
gptkbp:output contextualized word embeddings
gptkbp:parameter 110 million (BERT Base)
340 million (BERT Large)
gptkbp:predecessor gptkb:OpenAI_GPT
gptkb:ELMo
gptkbp:pretrainingTasks masked language modeling
next sentence prediction
gptkbp:publishedIn gptkb:NAACL_2019
gptkbp:releaseDate 2018
gptkbp:successor gptkb:ALBERT
gptkb:DistilBERT
gptkb:RoBERTa
gptkbp:supports transfer learning
multi-task learning
gptkbp:tokenizerType gptkb:WordPiece
gptkbp:trainer gptkb:Wikipedia
gptkb:BookCorpus
gptkbp:usedFor question answering
sentiment analysis
text classification
named entity recognition
language understanding
gptkbp:bfsParent gptkb:Retrieval-Augmented_Generation_(RAG)
gptkbp:bfsLayer 6