RoBERTa: A Robustly Optimized BERT Pretraining Approach
GPTKB entity
Statements (33)
| Predicate | Object |
|---|---|
| gptkbp:instanceOf |
gptkb:academic_journal
large language model |
| gptkbp:arXivID |
1907.11692
|
| gptkbp:author |
gptkb:Mike_Lewis
gptkb:Omer_Levy gptkb:Danqi_Chen gptkb:Jingfei_Du gptkb:Mandar_Joshi gptkb:Myle_Ott gptkb:Yinhan_Liu gptkb:Veselin_Stoyanov gptkb:Luke_Zettlemoyer gptkb:Naman_Goyal |
| gptkbp:basedOn |
gptkb:BERT
|
| gptkbp:citation |
high
|
| gptkbp:developedBy |
gptkb:Facebook_AI
|
| https://www.w3.org/2000/01/rdf-schema#label |
RoBERTa: A Robustly Optimized BERT Pretraining Approach
|
| gptkbp:improves |
gptkb:BERT_on_RACE
gptkb:BERT_on_SQuAD BERT on GLUE benchmark BERT pretraining methodology |
| gptkbp:language |
English
|
| gptkbp:openSource |
true
|
| gptkbp:publicationYear |
2019
|
| gptkbp:publishedIn |
gptkb:arXiv
|
| gptkbp:removes |
Next Sentence Prediction objective
|
| gptkbp:repository |
https://github.com/pytorch/fairseq
|
| gptkbp:uses |
dynamic masking
larger mini-batches more data longer training |
| gptkbp:bfsParent |
gptkb:RoBERTa
|
| gptkbp:bfsLayer |
6
|