Statements (24)
| Predicate | Object |
|---|---|
| gptkbp:instanceOf |
gptkb:software
|
| gptkbp:availableOn |
gptkb:PyPI
|
| gptkbp:developedBy |
gptkb:Hugging_Face
|
| gptkbp:documentation |
https://huggingface.co/docs/tokenizers
|
| gptkbp:feature |
normalization
customizable pipelines post-processing fast tokenization pre-tokenization |
| gptkbp:firstReleased |
2019
|
| https://www.w3.org/2000/01/rdf-schema#label |
Hugging Face tokenizers
|
| gptkbp:integratesWith |
gptkb:Transformers_library
|
| gptkbp:license |
gptkb:Apache_License_2.0
|
| gptkbp:programmingLanguage |
gptkb:Python
gptkb:Rust |
| gptkbp:repository |
https://github.com/huggingface/tokenizers
|
| gptkbp:supports |
gptkb:WordPiece
gptkb:BPE gptkb:SentencePiece Unigram |
| gptkbp:usedFor |
natural language processing
text tokenization |
| gptkbp:bfsParent |
gptkb:BPE
|
| gptkbp:bfsLayer |
7
|