Bidirectional Encoder Representations from Transformers

URI: https://gptkb.org/entity/Bidirectional_Encoder_Representations_from_Transformers

GPTKB entity

Statements (47)

Predicate	Object
gptkbp:instanceOf	gptkb:model large language model
gptkbp:abbreviation	gptkb:BERT
gptkbp:application	natural language processing question answering sentiment analysis named entity recognition
gptkbp:architecture	gptkb:transformation
gptkbp:author	gptkb:Jacob_Devlin gptkb:Kenton_Lee gptkb:Ming-Wei_Chang gptkb:Kristina_Toutanova
gptkbp:basedOn	transformer architecture
gptkbp:category	gptkb:machine_learning deep learning natural language processing
gptkbp:citation	over 50,000 (as of 2024)
gptkbp:developer	gptkb:Google
gptkbp:fineTuning	task-specific
https://www.w3.org/2000/01/rdf-schema#label	Bidirectional Encoder Representations from Transformers
gptkbp:influenced	gptkb:ALBERT gptkb:DistilBERT gptkb:RoBERTa gptkb:XLNet
gptkbp:input	gptkb:text
gptkbp:language	gptkb:Chinese English
gptkbp:license	gptkb:Apache_License_2.0
gptkbp:notableFor	improving state-of-the-art in NLP tasks
gptkbp:notablePublication	gptkb:BERT:_Pre-training_of_Deep_Bidirectional_Transformers_for_Language_Understanding https://arxiv.org/abs/1810.04805
gptkbp:openSource	true
gptkbp:output	contextual word embeddings
gptkbp:parameter	110 million (BERT Base) 340 million (BERT Large)
gptkbp:predecessor	gptkb:OpenAI_GPT gptkb:ELMo
gptkbp:pretrainingTask	masked language modeling next sentence prediction
gptkbp:publicationYear	2018
gptkbp:repository	https://github.com/google-research/bert
gptkbp:tokenization	gptkb:WordPiece
gptkbp:trainer	gptkb:English_Wikipedia gptkb:BooksCorpus
gptkbp:uses	self-attention mechanism
gptkbp:bfsParent	gptkb:large_language_model
gptkbp:bfsLayer	5