Statements (107)
| Predicate | Object |
|---|---|
| gptkbp:instance_of |
gptkb:Model
|
| gptkbp:bfsLayer |
4
|
| gptkbp:bfsParent |
gptkb:BERT
|
| gptkbp:application |
gptkb:software
|
| gptkbp:architectural_style |
gptkb:Transformers_character
|
| gptkbp:community_support |
gptkb:municipality
Open-source Research papers Git Hub repositories |
| gptkbp:developed_by |
gptkb:Huawei
|
| gptkbp:features |
Transfer learning
Contextual embeddings Masked language modeling Next sentence prediction Multi-task learning Layer normalization Positional encoding Self-attention mechanism |
| gptkbp:goal |
Maintain performance
Reduce model size |
| gptkbp:has_achievements |
state-of-the-art results
|
| gptkbp:has_variants |
gptkb:Tiny_BERT-4
gptkb:Tiny_BERT-6 gptkb:Tiny_BERT-8 |
| https://www.w3.org/2000/01/rdf-schema#label |
Tiny BERT
|
| gptkbp:impact |
Industry applications
AI development Research advancements Chatbot development Text processing tasks |
| gptkbp:improves |
BERT's performance
|
| gptkbp:input_output |
512 tokens
|
| gptkbp:is_available_on |
gptkb:Hugging_Face_Model_Hub
gptkb:Py_Torch_Hub gptkb:Tensor_Flow_Hub |
| gptkbp:is_compared_to |
gptkb:Distil_BERT
ALBERT |
| gptkbp:is_evaluated_by |
gptkb:MNLI_benchmark
gptkb:RTE_benchmark gptkb:GLUE_benchmark gptkb:Co_LA_benchmark gptkb:QQP_benchmark gptkb:WNLI_benchmark Accuracy F1 score S Qu AD benchmark STS-B benchmark |
| gptkbp:is_optimized_for |
gptkb:smartphone
edge devices |
| gptkbp:is_part_of |
transformer models
Tiny BERT family |
| gptkbp:is_popular_in |
gptkb:Research_Institute
industry applications |
| gptkbp:is_used_for |
gptkb:academic_research
gptkb:film_production_company language translation dialog systems content moderation chatbots information retrieval transfer learning spam detection feature extraction text generation text mining text summarization data annotation social media analysis named entity recognition semantic similarity customer support automation text entailment user intent prediction |
| gptkbp:is_used_in |
question answering
sentiment analysis text classification language understanding |
| gptkbp:language |
English
|
| gptkbp:performance |
Faster than BERT
|
| gptkbp:provides_information_on |
gptkb:Wikipedia
gptkb:Common_Crawl gptkb:Book_Corpus |
| gptkbp:purpose |
gptkb:software
|
| gptkbp:related_to |
gptkb:Artificial_Intelligence
gptkb:BERT Deep learning Machine learning NLP models |
| gptkbp:release_date |
gptkb:2019
gptkb:2020 |
| gptkbp:resolution |
768 dimensions
|
| gptkbp:size |
14 million parameters
|
| gptkbp:speed |
gptkb:BERT
|
| gptkbp:successor |
gptkb:Tiny_BERT_2.0
|
| gptkbp:supports |
multiple languages
|
| gptkbp:training |
Knowledge Distillation
large text corpora English text |
| gptkbp:tuning |
specific tasks
Task-specific datasets |
| gptkbp:type |
Lightweight model
|
| gptkbp:use_case |
Question answering
Sentiment analysis Text classification Named entity recognition |
| gptkbp:uses |
knowledge distillation
|
| gptkbp:weight |
gptkb:BERT
|