# A guide to recurrent neural networks and backpropagation

@inproceedings{Bodn2001AGT, title={A guide to recurrent neural networks and backpropagation}, author={Mikael Bod{\'e}n}, year={2001} }

This paper provides guidance to some of the concepts surrounding recurrent neural networks. Contrary to feedforward networks, recurrent networks can be sensitive, and be adapted to past inputs. Backpropagation learning is described for feedforward networks, adapted to suit our (probabilistic) modeling needs, and extended to cover recurrent networks. The aim of this brief paper is to set the scene for applying and understanding recurrent neural networks.

#### 226 Citations

Extensions of recurrent neural network language model

- Computer Science
- 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)
- 2011

Several modifications of the original recurrent neural network language model are presented, showing approaches that lead to more than 15 times speedup for both training and testing phases and possibilities how to reduce the amount of parameters in the model. Expand

Recurrent Neural Networks and Related Models

- Computer Science
- 2015

This chapter presents the state-space formulation of the basic RNN as a nonlinear dynamical system, where the recurrent matrix governing the system dynamics is largely unstructured and analyzes the RNNAs a bottom-up, discriminative, dynamic system model against the top-down, generative counterpart of dynamic system. Expand

A Resource Efficient Localized Recurrent Neural Network Architecture and Learning Algorithm

- Computer Science
- 2006

This thesis introduces TRTRL, an e¢ cient, low-complexity online learning algorithm for recurrent neural networks based on the real-time recurrent learning (RTRL) algorithm, whereby the sensitivity set of each neuron is reduced to weights associated either with its input or output links. Expand

Problem and Strategy: Overfitting in Recurrent Cycles of Internal Symmetry Networks by Back Propagation

- Computer Science
- 2009 International Conference on Computational Intelligence and Natural Computing
- 2009

Overfitting in recurrent cycles of Internal Symmetry Networks is analyzed and back propagation is trained for an image processing task. Expand

Artificial neural networks in artificial time series prediction benchmark

artificial neural networks. The characteristic samples of artificial neural network types were selected to be compared in numerous simulations, while influences of key parameters are studied. The… Expand

Fully Bayesian Recurrent Neural Networks for Safe Reinforcement Learning

- Computer Science, Mathematics
- ArXiv
- 2019

This work builds on Probabilistic Backpropagation to introduce a fully Bayesian Recurrent Neural Network architecture, and demonstrates that the proposed method significantly outperforms a popular approach for obtaining model uncertainties in collision avoidance tasks. Expand

Generating weights and generating vectors to map complex functions with artificial neural networks

- Computer Science
- 2008 IEEE International Joint Conference on Neural Networks (IEEE World Congress on Computational Intelligence)
- 2008

A set of vectors are generated from the transformed derived nets that are then used to train an ANN to map one-to-many tasks and the associated rotational symmetries performed are specified. Expand

Large scale recurrent neural network on GPU

- Computer Science
- 2014 International Joint Conference on Neural Networks (IJCNN)
- 2014

The experiment results of the Microsoft Research Sentence Completion Challenge demonstrate that the large scale recurrent network without class layer is able to beat the traditional class-based modest-size recurrent network and achieve an accuracy of 47%, the best result achieved by a single recurrent neural network on the same dataset. Expand

Recurrent neural network based language model

- Computer Science
- INTERSPEECH
- 2010

Results indicate that it is possible to obtain around 50% reduction of perplexity by using mixture of several RNN LMs, compared to a state of the art backoff language model. Expand

rnn : Recurrent Library for Torch

- Computer Science
- ArXiv
- 2015

The rnn package provides components for implementing a wide range of Recurrent Neural Networks. It is built withing the framework of the Torch distribution for use with the nn package. The components… Expand

#### References

SHOWING 1-10 OF 20 REFERENCES

A Learning Algorithm for Continually Running Fully Recurrent Neural Networks

- Computer Science
- Neural Computation
- 1989

The exact form of a gradient-following learning algorithm for completely recurrent networks running in continually sampled time is derived and used as the basis for practical algorithms for temporal… Expand

Learning long-term dependencies with gradient descent is difficult

- Computer Science, Medicine
- IEEE Trans. Neural Networks
- 1994

This work shows why gradient based learning algorithms face an increasingly difficult problem as the duration of the dependencies to be captured increases, and exposes a trade-off between efficient learning by gradient descent and latching on information for long periods. Expand

Context-free and context-sensitive dynamics in recurrent neural networks

- Computer Science
- Connect. Sci.
- 2000

The dynamics in recurrent neural networks that process context-free languages can also be employed in processing some context-sensitive languages, and this continuity of mechanism between language classes contributes to the understanding of neural networks in modelling language learning and processing. Expand

The Dynamics of Discrete-Time Computation, with Application to Recurrent Neural Networks and Finite State Machine Extraction

- Computer Science, Medicine
- Neural Computation
- 1996

It is shown that an RNN performing a finite state computation must organize its state space to mimic the states in the minimal deterministic finite state machine that can perform that computation, and a precise description of the attractor structure of such systems is given. Expand

A Recurrent Neural Network that Learns to Count

- Computer Science
- Connect. Sci.
- 1999

This research employs standard backpropagation training techniques for a recurrent neural network in the task of learning to predict the next character in a simple deterministic CFL (DCFL), and shows that an RNN can learn to recognize the structure of a simple DCFL. Expand

Backpropagation: the basic theory

- Computer Science
- 1995

Since the publication of the PDP volumes in 1986, learning by backpropagation has become the most popular method of training neural networks because of the underlying simplicity and relative power of the algorithm. Expand

Fool's Gold: Extracting Finite State Machines from Recurrent Network Dynamics

- Computer Science
- NIPS
- 1993

How sensitivity to initial conditions and discrete measurements can trick these extraction methods to return illusory finite state descriptions is described. Expand

Finite State Automata and Simple Recurrent Networks

- Computer Science, Mathematics
- Neural Computation
- 1989

A network architecture introduced by Elman (1988) for predicting successive elements of a sequence and shows that long distance sequential contingencies can be encoded by the network even if only subtle statistical properties of embedded strings depend on the early information. Expand

Learning and Extracting Finite State Automata with Second-Order Recurrent Neural Networks

- Computer Science
- Neural Computation
- 1992

It is shown that a recurrent, second-order neural network using a real-time, forward training algorithm readily learns to infer small regular grammars from positive and negative string training samples, and many of the neural net state machines are dynamically stable, that is, they correctly classify many long unseen strings. Expand

Learning to predict a context-free language: analysis of dynamics in recurrent hidden units

- Computer Science
- 1999

It is concluded that any gradient-based learning method will experience difficulty in learning the language due to the nature of the space, and that a more promising approach to improving learning performance may be to make weight changes in a non-independent manner. Expand