Introduction to Recurrent Neural Networks

Recurrent Neural Networks (RNNs) work a bit different from regular neural networks. In neural network the information flows in one direction from input to output. However in RNN information is fed back into the system after each step. Think of it like reading a sentence, when you’re trying to predict the next word you don’t just look at the current word but also need to remember the words that came before to make accurate guess.

RNNs allow the network to “remember” past information by feeding the output from one step into next step. This helps the network understand the context of what has already happened and make better predictions based on that. For example when predicting the next word in a sentence the RNN uses the previous words to help decide what word is most likely to come next.

Recurrent Neural Network

This image showcases the basic architecture of RNN and the feedback loop mechanism where the output is passed back as input for the next time step.

How RNN Differs from Feedforward Neural Networks?

Feedforward Neural Networks (FNNs) process data in one direction from input to output without retaining information from previous inputs. This makes them suitable for tasks with independent inputs like image classification. However FNNs struggle with sequential data since they lack memory.

Recurrent Neural Networks (RNNs) solve this by incorporating loops that allow information from previous steps to be fed back into the network. This feedback enables RNNs to remember prior inputs making them ideal for tasks where context is important.

Recurrent Vs Feedforward networks

Key Components of RNNs

1. Recurrent Neurons

The fundamental processing unit in RNN is a Recurrent Unit. Recurrent units hold a hidden state that maintains information about previous inputs in a sequence. Recurrent units can “remember” information from prior steps by feeding back their hidden state, allowing them to capture dependencies across time.

Recurrent Neuron

2. RNN Unfolding

RNN unfolding or unrolling is the process of expanding the recurrent structure over time steps. During unfolding each step of the sequence is represented as a separate layer in a series illustrating how information flows across each time step.

This unrolling enables backpropagation through time (BPTT) a learning process where errors are propagated across time steps to adjust the network’s weights enhancing the RNN’s ability to learn dependencies within sequential data.

RNN Unfolding

Recurrent Neural Network Architecture

RNNs share similarities in input and output structures with other deep learning architectures but differ significantly in how information flows from input to output. Unlike traditional deep neural networks, where each dense layer has distinct weight matrices, RNNs use shared weights across time steps, allowing them to remember information over sequences.

In RNNs, the hidden state [Tex]H_i[/Tex]​ is calculated for every input [Tex]X_i[/Tex]​ to retain sequential dependencies. The computations follow these core formulas:

1. Hidden State Calculation:

[Tex]h = \sigma(U \cdot X + W \cdot h_{t-1} + B)[/Tex]

Here, [Tex]h[/Tex] represents the current hidden state, [Tex]U[/Tex] and [Tex]W[/Tex] are weight matrices, and [Tex]B[/Tex] is the bias.

2. Output Calculation:

[Tex]Y = O(V \cdot h + C)[/Tex]

The output [Tex]Y[/Tex] is calculated by applying [Tex]O[/Tex], an activation function, to the weighted hidden state, where [Tex]V[/Tex] and [Tex]C[/Tex] represent weights and bias.

3. Overall Function:

[Tex]Y = f(X, h, W, U, V, B, C)[/Tex]

This function defines the entire RNN operation, where the state matrix [Tex]S[/Tex] holds each element [Tex]s_i[/Tex] representing the network’s state at each time step [Tex]i[/Tex].

Recurrent Neural Architecture

How does RNN work?

At each time step RNNs process units with a fixed activation function. These units have an internal hidden state that acts as memory that retains information from previous time steps. This memory allows the network to store past knowledge and adapt based on new inputs.

Updating the Hidden State in RNNs

The current hidden state [Tex]h_t[/Tex]​ depends on the previous state [Tex]h_{t-1}[/Tex]​ and the current input [Tex]x_t[/Tex]​, and is calculated using the following relations:

1. State Update:

[Tex]h_t = f(h_{t-1}, x_t)[/Tex]

where:

• [Tex]h_t[/Tex]​ is the current state
• [Tex]h_{t-1}[/Tex]​ is the previous state
• [Tex]x_t[/Tex] is the input at the current time step

2. Activation Function Application:

[Tex]h_t = \tanh(W_{hh} \cdot h_{t-1} + W_{xh} \cdot x_t)[/Tex]

Here, [Tex]W_{hh}[/Tex]​ is the weight matrix for the recurrent neuron, and [Tex]W_{xh}[/Tex]​ is the weight matrix for the input neuron.

3. Output Calculation:

[Tex]y_t = W_{hy} \cdot h_t[/Tex]

where [Tex]y_t[/Tex]​ is the output and [Tex]W_{hy}[/Tex]​ is the weight at the output layer.

These parameters are updated using backpropagation. However, since RNN works on sequential data here we use an updated backpropagation which is known as backpropagation through time. 

Backpropagation Through Time (BPTT) in RNNs

Since RNNs process sequential data Backpropagation Through Time (BPTT) is used to update the network’s parameters. The loss function L(θ) depends on the final hidden state [Tex]h_3[/Tex] and each hidden state relies on preceding ones forming a sequential dependency chain:

[Tex]h_3[/Tex] depends on [Tex] \text{ depends on } h_2, \, h_2 \text{ depends on } h_1, \, \dots, \, h_1 \text{ depends on } h_0[/Tex]​.

Backpropagation Through Time (BPTT) In RNN

In BPTT, gradients are backpropagated through each time step. This is essential for updating network parameters based on temporal dependencies.

1. Simplified Gradient Calculation:
   [Tex]\frac{\partial L(\theta)}{\partial W} = \frac{\partial L (\theta)}{\partial h_3} \cdot \frac{\partial h_3}{\partial W}[/Tex]
2. Handling Dependencies in Layers:
   Each hidden state is updated based on its dependencies:
   [Tex]h_3 = \sigma(W \cdot h_2 + b)[/Tex]
   The gradient is then calculated for each state, considering dependencies from previous hidden states.
3. Gradient Calculation with Explicit and Implicit Parts: The gradient is broken down into explicit and implicit parts summing up the indirect paths from each hidden state to the weights.
   [Tex]\frac{\partial h_3}{\partial W} = \frac{\partial h_3^{+}}{\partial W} + \frac{\partial h_3}{\partial h_2} \cdot \frac{\partial h_2^{+}}{\partial W}[/Tex]
4. Final Gradient Expression:
   The final derivative of the loss function with respect to the weight matrix W is computed:
   [Tex]\frac{\partial L(\theta)}{\partial W} = \frac{\partial L(\theta)}{\partial h_3} \cdot \sum_{k=1}^{3} \frac{\partial h_3}{\partial h_k} \cdot \frac{\partial h_k}{\partial W}[/Tex]

This iterative process is the essence of backpropagation through time.

Types Of Recurrent Neural Networks

There are four types of RNNs based on the number of inputs and outputs in the network:

1. One-to-One RNN

This is the simplest type of neural network architecture where there is a single input and a single output. It is used for straightforward classification tasks such as binary classification where no sequential data is involved.

One to One RNN

2. One-to-Many RNN

In a One-to-Many RNN the network processes a single input to produce multiple outputs over time. This is useful in tasks where one input triggers a sequence of predictions (outputs). For example in image captioning a single image can be used as input to generate a sequence of words as a caption.

One to Many RNN

3. Many-to-One RNN

The Many-to-One RNN receives a sequence of inputs and generates a single output. This type is useful when the overall context of the input sequence is needed to make one prediction. In sentiment analysis the model receives a sequence of words (like a sentence) and produces a single output like positive, negative or neutral.

Many to One RNN

4. Many-to-Many RNN

The Many-to-Many RNN type processes a sequence of inputs and generates a sequence of outputs. In language translation task a sequence of words in one language is given as input, and a corresponding sequence in another language is generated as output.

Many to Many RNN

Variants of Recurrent Neural Networks (RNNs)

There are several variations of RNNs, each designed to address specific challenges or optimize for certain tasks:

1. Vanilla RNN

This simplest form of RNN consists of a single hidden layer where weights are shared across time steps. Vanilla RNNs are suitable for learning short-term dependencies but are limited by the vanishing gradient problem, which hampers long-sequence learning.

2. Bidirectional RNNs

Bidirectional RNNs process inputs in both forward and backward directions, capturing both past and future context for each time step. This architecture is ideal for tasks where the entire sequence is available, such as named entity recognition and question answering.

3. Long Short-Term Memory Networks (LSTMs)

Long Short-Term Memory Networks (LSTMs) introduce a memory mechanism to overcome the vanishing gradient problem. Each LSTM cell has three gates:

• Input Gate: Controls how much new information should be added to the cell state.
• Forget Gate: Decides what past information should be discarded.
• Output Gate: Regulates what information should be output at the current step. This selective memory enables LSTMs to handle long-term dependencies, making them ideal for tasks where earlier context is critical.

4. Gated Recurrent Units (GRUs)

Gated Recurrent Units (GRUs) simplify LSTMs by combining the input and forget gates into a single update gate and streamlining the output mechanism. This design is computationally efficient, often performing similarly to LSTMs, and is useful in tasks where simplicity and faster training are beneficial.

Implementing a Text Generator Using Recurrent Neural Networks (RNNs)

In this section, we create a character-based text generator using Recurrent Neural Network (RNN) in TensorFlow and Keras. We’ll implement an RNN that learns patterns from a text sequence to generate new text character-by-character.

Step 1: Import Necessary Libraries

We start by importing essential libraries for data handling and building the neural network.

import numpy as np
import tensorflow as tf
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import SimpleRNN, Dense

Step 2: Define the Input Text and Prepare Character Set

We define the input text and identify unique characters in the text which we’ll encode for our model.

text = "This is GeeksforGeeks a software training institute"
chars = sorted(list(set(text)))
char_to_index = {char: i for i, char in enumerate(chars)}
index_to_char = {i: char for i, char in enumerate(chars)}

Step 3: Create Sequences and Labels

To train the RNN, we need sequences of fixed length (seq_length) and the character following each sequence as the label.

seq_length = 3
sequences = []
labels = []

for i in range(len(text) - seq_length):
    seq = text[i:i + seq_length]
    label = text[i + seq_length]
    sequences.append([char_to_index[char] for char in seq])
    labels.append(char_to_index[label])

X = np.array(sequences)
y = np.array(labels)

Step 4: Convert Sequences and Labels to One-Hot Encoding

For training, we convert X and y into one-hot encoded tensors.

X_one_hot = tf.one_hot(X, len(chars))
y_one_hot = tf.one_hot(y, len(chars))

Step 5: Build the RNN Model

We create a simple RNN model with a hidden layer of 50 units and a Dense output layer with softmax activation.

model = Sequential()
model.add(SimpleRNN(50, input_shape=(seq_length, len(chars)), activation='relu'))
model.add(Dense(len(chars), activation='softmax'))

Step 6: Compile and Train the Model

We compile the model using the categorical_crossentropy loss and train it for 100 epochs.

model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])
model.fit(X_one_hot, y_one_hot, epochs=100)

Output:

Epoch 1/100
2/2 ━━━━━━━━━━━━━━━━━━━━ 4s 23ms/step – accuracy: 0.0243 – loss: 2.9043
Epoch 2/100
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 14ms/step – accuracy: 0.0139 – loss: 2.8720
Epoch 3/100
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 10ms/step – accuracy: 0.0243 – loss: 2.8454
.
.
.
Epoch 99/100
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 9ms/step – accuracy: 0.8889 – loss: 0.5060
Epoch 100/100
2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 9ms/step – accuracy: 0.9236 – loss: 0.4934

Step 7: Generate New Text Using the Trained Model

After training, we use a starting sequence to generate new text character-by-character.

start_seq = "This is G"
generated_text = start_seq

for i in range(50):
    x = np.array([[char_to_index[char] for char in generated_text[-seq_length:]]])
    x_one_hot = tf.one_hot(x, len(chars))
    prediction = model.predict(x_one_hot)
    next_index = np.argmax(prediction)
    next_char = index_to_char[next_index]
    generated_text += next_char

print("Generated Text:")
print(generated_text)

Output:

Generated Text: This is Geeks a software training instituteais is is is is

Complete Code

import numpy as np
import tensorflow as tf
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import SimpleRNN, Dense

text = "This is GeeksforGeeks a software training institute"
chars = sorted(list(set(text)))
char_to_index = {char: i for i, char in enumerate(chars)}
index_to_char = {i: char for i, char in enumerate(chars)}

seq_length = 3
sequences = []
labels = []

for i in range(len(text) - seq_length):
    seq = text[i:i + seq_length]
    label = text[i + seq_length]
    sequences.append([char_to_index[char] for char in seq])
    labels.append(char_to_index[label])

X = np.array(sequences)
y = np.array(labels)

X_one_hot = tf.one_hot(X, len(chars))
y_one_hot = tf.one_hot(y, len(chars))

model = Sequential()
model.add(SimpleRNN(50, input_shape=(seq_length, len(chars)), activation='relu'))
model.add(Dense(len(chars), activation='softmax'))

model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])
model.fit(X_one_hot, y_one_hot, epochs=100)

start_seq = "This is G"
generated_text = start_seq

for i in range(50):
    x = np.array([[char_to_index[char] for char in generated_text[-seq_length:]]])
    x_one_hot = tf.one_hot(x, len(chars))
    prediction = model.predict(x_one_hot)
    next_index = np.argmax(prediction)
    next_char = index_to_char[next_index]
    generated_text += next_char

print("Generated Text:")
print(generated_text)

Advantages of Recurrent Neural Networks

• Sequential Memory: RNNs retain information from previous inputs, making them ideal for time-series predictions where past data is crucial. This capability is often called Long Short-Term Memory (LSTM).
• Enhanced Pixel Neighborhoods: RNNs can be combined with convolutional layers to capture extended pixel neighborhoods improving performance in image and video data processing.

Limitations of Recurrent Neural Networks (RNNs)

While RNNs excel at handling sequential data, they face two main training challenges i.e., vanishing gradient and exploding gradient problem:

1. Vanishing Gradient: During backpropagation, gradients diminish as they pass through each time step, leading to minimal weight updates. This limits the RNN’s ability to learn long-term dependencies, which is crucial for tasks like language translation.
2. Exploding Gradient: Sometimes, gradients grow uncontrollably, causing excessively large weight updates that destabilize training. Gradient clipping is a common technique to manage this issue.

These challenges can hinder the performance of standard RNNs on complex, long-sequence tasks.

Applications of Recurrent Neural Networks

RNNs are used in various applications where data is sequential or time-based:

• Time-Series Prediction: RNNs excel in forecasting tasks, such as stock market predictions and weather forecasting.
• Natural Language Processing (NLP): RNNs are fundamental in NLP tasks like language modeling, sentiment analysis, and machine translation.
• Speech Recognition: RNNs capture temporal patterns in speech data, aiding in speech-to-text and other audio-related applications.
• Image and Video Processing: When combined with convolutional layers, RNNs help analyze video sequences, facial expressions, and gesture recognition.

GeeksforGeeks

Improve

Article Tags :

• AI-ML-DS
• Deep Learning
• AI-ML-DS With Python
• Neural Network