Adam

E701497

Adam is a widely used stochastic optimization algorithm in machine learning that combines ideas from momentum and adaptive learning rates to efficiently train deep neural networks.

All labels observed (1)

Label Occurrences
Adam canonical 6

How this entity was disambiguated

Statements (46)

Predicate Object
instanceOf optimization algorithm
stochastic optimization method
abbreviationFor Adaptive Moment Estimation
appliedIn computer vision
natural language processing
reinforcement learning
speech recognition
basedOn adaptive learning rates
momentum
stochastic gradient descent
commonVariant AMSGrad
AdamW
linked to: Adam optimizer
comparedWith AdaGrad
RMSProp
SGD with momentum
defaultHyperparameter beta1 = 0.9
beta2 = 0.999
epsilon = 1e-8
learning rate = 0.001
describedIn Adam: A Method for Stochastic Optimization
field deep learning
machine learning
hasProperty computationally efficient
handles sparse gradients
memory efficient
scale invariant to gradient magnitudes
suitable for high-dimensional parameter spaces
suitable for large datasets
implementedIn JAX
Keras
PyTorch
TensorFlow
introducedIn 2014
optimizationType first-order method
performs bias correction of moment estimates
proposedBy Diederik P. Kingma
Jimmy Ba
publishedAt International Conference on Learning Representations
linked to: ICLR
publishedIn 2015
updates parameters with element-wise adaptive learning rates
usedFor stochastic optimization
training deep neural networks
uses exponentially decaying averages of past gradients
exponentially decaying averages of past squared gradients
first moment estimates of gradients
second moment estimates of gradients

How these facts were elicited

Referenced by (6)

Full triples — surface form annotated when it differs from this entity's canonical label.