Statements (51)
| Predicate | Object |
|---|---|
| gptkbp:instanceOf |
Language modeling task
|
| gptkbp:application |
Code generation
Text generation Dialogue systems Autoregressive modeling |
| gptkbp:architecture |
Unidirectional transformer
|
| gptkbp:contrastsWith |
gptkb:Masked_language_modeling
Bidirectional language modeling |
| gptkbp:dataRequirement |
Large text corpora
|
| gptkbp:directionA |
Left-to-right
|
| gptkbp:example |
gptkb:GPT-2
gptkb:GPT-3 gptkb:GPT-4 gptkb:LLaMA gptkb:BLOOM gptkb:OPT |
| https://www.w3.org/2000/01/rdf-schema#label |
Causal language modeling
|
| gptkbp:input |
Sequence of tokens
|
| gptkbp:lossFunction |
Cross-entropy loss
|
| gptkbp:output |
Probability distribution over vocabulary
Logits for each token |
| gptkbp:predicts |
Next token
|
| gptkbp:relatedTo |
gptkb:Masked_language_modeling
|
| gptkbp:tokenization |
gptkb:WordPiece
Byte pair encoding Subword tokenization |
| gptkbp:trainingObjective |
Maximize likelihood of next word
|
| gptkbp:usedBy |
gptkb:Transformer_models
GPT models |
| gptkbp:usedFor |
Chatbots
Conversational AI Paraphrasing Creative writing Code completion Text completion Autocompletion Dialogue generation Email drafting Machine translation (autoregressive) Poetry generation Pretraining large language models Question answering (generative) Script generation Story generation Summarization (autoregressive) Text expansion Text infilling |
| gptkbp:usedIn |
Natural language processing
Machine learning |
| gptkbp:bfsParent |
gptkb:Language_modeling
|
| gptkbp:bfsLayer |
7
|