Statements (18)
| Predicate | Object |
|---|---|
| gptkbp:instanceOf |
activation function
|
| gptkbp:activationFunctionType |
gated linear unit variant
|
| gptkbp:advantage |
improves model performance
increases training stability |
| gptkbp:component |
feedforward neural networks
|
| gptkbp:form |
SwiGLU(x) = x1 * sigmoid(x2)
|
| gptkbp:fullName |
gptkb:Switch_Gated_Linear_Unit
|
| https://www.w3.org/2000/01/rdf-schema#label |
SwiGLU
|
| gptkbp:introducedIn |
gptkb:Shazeer_2020
|
| gptkbp:relatedTo |
gptkb:ReLU
gptkb:GLU gptkb:GeGLU |
| gptkbp:usedBy |
gptkb:Google_PaLM
gptkb:LLaMA gptkb:OpenAI_GPT-3.5 |
| gptkbp:usedIn |
transformer models
|
| gptkbp:bfsParent |
gptkb:Mixtral_8x7B
|
| gptkbp:bfsLayer |
6
|