S. Bengio, Y. Bengio, and J. Cloutier. On the search for new learning rules for ANNs. Neural Processing Letters,
2(4):26–30, 1995.
Y. Bengio, S. Bengio, and J. Cloutier. Learning a synaptic learning rule. Université de Montréal, Département
d’informatique et de recherche opérationnelle, 1990.
Y. Bengio, N. Boulanger-Lewandowski, and R. Pascanu. Advances in optimizing recurrent networks. In
International Conference on Acoustics, Speech and Signal Processing, pages 8624–8628. IEEE, 2013.
F. Bobolas. brain-neurons, 2009. URL
https://www.flickr.com/photos/fbobolas/3822222947
. Cre-
ative Commons Attribution-ShareAlike 2.0 Generic.
N. E. Cotter and P. R. Conwell. Fixed-weight networks can learn. In International Joint Conference on Neural
Networks, pages 553–559, 1990.
C. Daniel, J. Taylor, and S. Nowozin. Learning step size controllers for robust neural network training. In
Association for the Advancement of Artificial Intelligence, 2016.
J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database.
In Computer Vision and Pattern Recognition, pages 248–255. IEEE, 2009.
D. L. Donoho. Compressed sensing. Transactions on Information Theory, 52(4):1289–1306, 2006.
J. Duchi, E. Hazan, and Y. Singer. Adaptive subgradient methods for online learning and stochastic optimization.
Journal of Machine Learning Research, 12:2121–2159, 2011.
L. A. Feldkamp and G. V. Puskorius. A signal processing framework based on dynamic neural networks
with application to problems in adaptation, filtering, and classification. Proceedings of the IEEE, 86(11):
2259–2277, 1998.
L. A. Gatys, A. S. Ecker, and M. Bethge. A neural algorithm of artistic style. arXiv Report 1508.06576, 2015.
A. Graves, G. Wayne, and I. Danihkela. Neural Turing machines. arXiv Report 1410.5401, 2014.
S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural computation, 9(8):1735–1780, 1997.
S. Hochreiter, A. S. Younger, and P. R. Conwell. Learning to learn using gradient descent. In International
Conference on Artificial Neural Networks, pages 87–94. Springer, 2001.
D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In International Conference on Learning
Representations, 2015.
A. Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
B. M. Lake, T. D. Ullman, J. B. Tenenbaum, and S. J. Gershman. Building machines that learn and think like
people. arXiv Report 1604.00289, 2016.
T. Maley. neuron, 2011. URL
https://www.flickr.com/photos/taylortotz101/6280077898
. Creative
Commons Attribution 2.0 Generic.
J. Martens and R. Grosse. Optimizing neural networks with Kronecker-factored approximate curvature. In
International Conference on Machine Learning, pages 2408–2417, 2015.
G. L. Nemhauser and L. A. Wolsey. Integer and combinatorial optimization. John Wiley & Sons, 1988.
Y. Nesterov. A method of solving a convex programming problem with convergence rate o (1/k2). In Soviet
Mathematics Doklady, volume 27, pages 372–376, 1983.
J. Nocedal and S. Wright. Numerical optimization. Springer Science & Business Media, 2006.
M. Riedmiller and H. Braun. A direct adaptive method for faster backpropagation learning: The RPROP
algorithm. In International Conference on Neural Networks, pages 586–591, 1993.
T. P. Runarsson and M. T. Jonsson. Evolution and design of distributed learning rules. In IEEE Symposium on
Combinations of Evolutionary Computation and Neural Networks, pages 59–63. IEEE, 2000.
A. Santoro, S. Bartunov, M. Botvinick, D. Wierstra, and T. Lillicrap. Meta-learning with memory-augmented
neural networks. In International Conference on Machine Learning, 2016.
J. Schmidhuber. Evolutionary principles in self-referential learning; On learning how to learn: The meta-meta-...
hook. PhD thesis, Institut f. Informatik, Tech. Univ. Munich, 1987.
J. Schmidhuber. Learning to control fast-weight memories: An alternative to dynamic recurrent networks.
Neural Computation, 4(1):131–139, 1992.
J. Schmidhuber. A neural network that embeds its own meta-levels. In International Conference on Neural
Networks, pages 407–412. IEEE, 1993.
J. Schmidhuber, J. Zhao, and M. Wiering. Shifting inductive bias with success-story algorithm, adaptive levin
search, and incremental self-improvement. Machine Learning, 28(1):105–130, 1997.
N. N. Schraudolph. Local gain adaptation in stochastic gradient descent. In International Conference on
Artificial Neural Networks, volume 2, pages 569–574, 1999.
R. S. Sutton. Adapting bias by gradient descent: An incremental version of delta-bar-delta. In Association for
the Advancement of Artificial Intelligence, pages 171–176, 1992.
S. Thrun and L. Pratt. Learning to learn. Springer Science & Business Media, 1998.
T. Tieleman and G. Hinton. Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent
magnitude. COURSERA: Neural Networks for Machine Learning, 4:2, 2012.
P. Tseng. An incremental gradient (-projection) method with momentum term and adaptive stepsize rule. Journal
on Optimization, 8(2):506–531, 1998.
D. H. Wolpert and W. G. Macready. No free lunch theorems for optimization. Transactions on Evolutionary
Computation, 1(1):67–82, 1997.
9