Chapter 2: Multilayer Perceptron and Backpropagation

The Core Algorithm of Deep Learning - Backpropagation

📖 Reading Time: 30-35 minutes 📊 Difficulty: Beginner to Intermediate 💻 Code Examples: 15 📝 Exercises: 5

Learning Objectives

By reading this chapter, you will be able to:

2.1 Structure of the Multilayer Perceptron (MLP)

What is an MLP

A multilayer perceptron (MLP) is a neural network that combines multiple layers of perceptrons.

graph LR x1[Input Layer
x1] --> h1[Hidden Layer
h1] x2[Input Layer
x2] --> h1 x1 --> h2[Hidden Layer
h2] x2 --> h2 h1 --> y1[Output Layer
y1] h2 --> y1 style x1 fill:#e3f2fd style x2 fill:#e3f2fd style h1 fill:#fff3e0 style h2 fill:#fff3e0 style y1 fill:#e8f5e9

Types of Layers

Layer Type Role Description
Input Layer Input Layer Layer that receives the data (not trained)
Hidden Layer Hidden Layer Layer that performs feature extraction (trained)
Output Layer Output Layer Layer that produces the final result (trained)

Equations of a Two-Layer Neural Network

The computation from input $\mathbf{x} = [x_1, x_2]^T$ to output $y$:

Layer 1 (input → hidden layer):

$$ \mathbf{h} = \sigma(\mathbf{W}^{(1)} \mathbf{x} + \mathbf{b}^{(1)}) $$

Layer 2 (hidden layer → output):

$$ y = \sigma(\mathbf{W}^{(2)} \mathbf{h} + b^{(2)}) $$

Here, $\sigma$ is an activation function (such as the sigmoid function).

Basic Structure of the Python Implementation

import numpy as np

def sigmoid(x):
    """Sigmoid function"""
    return 1 / (1 + np.exp(-x))

class TwoLayerNet:
    """Two-layer neural network"""

    def __init__(self, input_size, hidden_size, output_size):
        """
        Args:
            input_size: Number of neurons in the input layer
            hidden_size: Number of neurons in the hidden layer
            output_size: Number of neurons in the output layer
        """
        # Initialize weights (randomly)
        self.W1 = np.random.randn(input_size, hidden_size)
        self.b1 = np.zeros(hidden_size)

        self.W2 = np.random.randn(hidden_size, output_size)
        self.b2 = np.zeros(output_size)

    def forward(self, x):
        """
        Forward propagation

        Args:
            x: Input data (n_samples, input_size)

        Returns:
            Output (n_samples, output_size)
        """
        # Layer 1
        self.z1 = np.dot(x, self.W1) + self.b1
        self.a1 = sigmoid(self.z1)

        # Layer 2
        self.z2 = np.dot(self.a1, self.W2) + self.b2
        self.a2 = sigmoid(self.z2)

        return self.a2

# Test
net = TwoLayerNet(input_size=2, hidden_size=3, output_size=1)
x = np.array([[0, 0], [0, 1], [1, 0], [1, 1]])
output = net.forward(x)

print("=== Output right after initialization ===")
print(output)

2.2 Loss Functions

What is a Loss Function

A loss function quantifies the difference between the neural network's predictions and the true values. The goal of training is to minimize this loss.

Mean Squared Error (MSE)

A loss function commonly used for regression problems:

$$ L = \frac{1}{n} \sum_{i=1}^{n} (y_i - \hat{y}_i)^2 $$

Here, $y_i$ is the true value and $\hat{y}_i$ is the predicted value.

def mean_squared_error(y_true, y_pred):
    """
    Mean Squared Error (MSE)

    Args:
        y_true: True labels (n_samples,)
        y_pred: Predicted values (n_samples,)

    Returns:
        MSE value
    """
    return np.mean((y_true - y_pred) ** 2)

# Example
y_true = np.array([0, 1, 1, 0])
y_pred = np.array([0.1, 0.9, 0.8, 0.2])
loss = mean_squared_error(y_true, y_pred)
print(f"MSE: {loss:.4f}")  # 0.0125

Cross-Entropy Loss

A loss function used for classification problems:

$$ L = -\frac{1}{n} \sum_{i=1}^{n} \left[ y_i \log(\hat{y}_i) + (1-y_i) \log(1-\hat{y}_i) \right] $$

def binary_cross_entropy(y_true, y_pred):
    """
    Binary Cross-Entropy

    Args:
        y_true: True labels (n_samples,)
        y_pred: Predicted probabilities (n_samples,)

    Returns:
        BCE value
    """
    # Clip with a small value for numerical stability
    epsilon = 1e-15
    y_pred = np.clip(y_pred, epsilon, 1 - epsilon)

    return -np.mean(y_true * np.log(y_pred) +
                    (1 - y_true) * np.log(1 - y_pred))

# Example
loss_ce = binary_cross_entropy(y_true, y_pred)
print(f"Cross-Entropy: {loss_ce:.4f}")  # 0.1625

2.3 Gradient Descent

Basic Idea

Gradient descent is an algorithm for minimizing the loss function. It updates parameters little by little in the opposite direction of the gradient (derivative) of the loss function.

graph TD A[Initial Parameters] --> B[Compute Loss] B --> C[Compute Gradient] C --> D[Update Parameters] D --> E{Converged?} E -->|No| B E -->|Yes| F[Training Complete] style A fill:#e3f2fd style B fill:#fff3e0 style C fill:#f3e5f5 style D fill:#e8f5e9 style F fill:#c8e6c9

Update Rule

Update of parameter $w$:

$$ w \leftarrow w - \eta \frac{\partial L}{\partial w} $$

Here, $\eta$ is the learning rate.

Python Implementation

def gradient_descent_demo():
    """Gradient descent demo"""

    # Simple function: f(x) = x^2
    def f(x):
        return x ** 2

    # Derivative: f'(x) = 2x
    def df(x):
        return 2 * x

    # Initial value and learning rate
    x = 10.0
    learning_rate = 0.1
    n_iterations = 20

    print("=== Gradient Descent Demo ===")
    print(f"Goal: find the minimum of f(x) = x^2")
    print(f"Initial value: x = {x}")
    print()

    for i in range(n_iterations):
        grad = df(x)
        x = x - learning_rate * grad

        if i % 5 == 0:
            print(f"Iteration {i:2d}: x = {x:8.4f}, f(x) = {f(x):8.4f}, grad = {grad:8.4f}")

    print()
    print(f"Final result: x = {x:.4f}, f(x) = {f(x):.4f}")
    print(f"Theoretical value: x = 0.0000, f(x) = 0.0000")

gradient_descent_demo()

Output:

=== Gradient Descent Demo ===
Goal: find the minimum of f(x) = x^2
Initial value: x = 10.0

Iteration  0: x =   8.0000, f(x) =  64.0000, grad =  20.0000
Iteration  5: x =   2.6214, f(x) =   6.8718, grad =   5.2429
Iteration 10: x =   0.8590, f(x) =   0.7379, grad =   1.7179
Iteration 15: x =   0.2815, f(x) =   0.0792, grad =   0.5630

Final result: x = 0.0922, f(x) = 0.0085
Theoretical value: x = 0.0000, f(x) = 0.0000

Effect of the Learning Rate

def compare_learning_rates():
    """Compare different learning rates"""
    def f(x):
        return x ** 2

    def df(x):
        return 2 * x

    learning_rates = [0.01, 0.1, 0.5, 0.9]
    x_init = 10.0
    n_iterations = 10

    print("=== Learning Rate Comparison ===")
    for lr in learning_rates:
        x = x_init
        for _ in range(n_iterations):
            x = x - lr * df(x)

        print(f"Learning rate η={lr:.2f} → final value x={x:8.4f}, f(x)={f(x):8.4f}")

compare_learning_rates()

Output:

=== Learning Rate Comparison ===
Learning rate η=0.01 → final value x=   8.1707, f(x)=  66.7604
Learning rate η=0.10 → final value x=   0.0922, f(x)=   0.0085
Learning rate η=0.50 → final value x=   0.0098, f(x)=   0.0001
Learning rate η=0.90 → final value x= -10.0000, f(x)= 100.0000 (diverged!)

Important: If the learning rate is too large, training diverges; if too small, convergence is slow!

2.4 Backpropagation

Why Backpropagation is Needed

In multilayer networks, we need to compute the gradients of the parameters in each layer. Backpropagation is a method that efficiently computes gradients from the output layer back toward the input layer.

Chain Rule

Differentiation of composite functions:

$$ \frac{\partial L}{\partial w} = \frac{\partial L}{\partial y} \cdot \frac{\partial y}{\partial z} \cdot \frac{\partial z}{\partial w} $$

This is the mathematical foundation of backpropagation.

Backpropagation in a Two-Layer Network

Forward computation:

$$ \begin{align} z^{(1)} &= W^{(1)} x + b^{(1)} \\ a^{(1)} &= \sigma(z^{(1)}) \\ z^{(2)} &= W^{(2)} a^{(1)} + b^{(2)} \\ y &= \sigma(z^{(2)}) \\ L &= \frac{1}{2}(y - t)^2 \end{align} $$

Backward computation (gradients):

$$ \begin{align} \frac{\partial L}{\partial y} &= y - t \\ \frac{\partial L}{\partial z^{(2)}} &= \frac{\partial L}{\partial y} \cdot \sigma'(z^{(2)}) \\ \frac{\partial L}{\partial W^{(2)}} &= \frac{\partial L}{\partial z^{(2)}} \cdot (a^{(1)})^T \\ \frac{\partial L}{\partial b^{(2)}} &= \frac{\partial L}{\partial z^{(2)}} \\ \frac{\partial L}{\partial a^{(1)}} &= (W^{(2)})^T \cdot \frac{\partial L}{\partial z^{(2)}} \\ \frac{\partial L}{\partial z^{(1)}} &= \frac{\partial L}{\partial a^{(1)}} \cdot \sigma'(z^{(1)}) \\ \frac{\partial L}{\partial W^{(1)}} &= \frac{\partial L}{\partial z^{(1)}} \cdot x^T \\ \frac{\partial L}{\partial b^{(1)}} &= \frac{\partial L}{\partial z^{(1)}} \end{align} $$

Complete Implementation

import numpy as np

def sigmoid(x):
    """Sigmoid function"""
    return 1 / (1 + np.exp(-np.clip(x, -500, 500)))

def sigmoid_derivative(x):
    """Derivative of the sigmoid function"""
    s = sigmoid(x)
    return s * (1 - s)

class TwoLayerNetWithBackprop:
    """Two-layer neural network with backpropagation"""

    def __init__(self, input_size, hidden_size, output_size, learning_rate=0.1):
        """
        Args:
            input_size: Size of the input layer
            hidden_size: Size of the hidden layer
            output_size: Size of the output layer
            learning_rate: Learning rate
        """
        # Initialize weights (He initialization)
        self.W1 = np.random.randn(input_size, hidden_size) * np.sqrt(2.0 / input_size)
        self.b1 = np.zeros(hidden_size)

        self.W2 = np.random.randn(hidden_size, output_size) * np.sqrt(2.0 / hidden_size)
        self.b2 = np.zeros(output_size)

        self.learning_rate = learning_rate

    def forward(self, x):
        """Forward propagation"""
        # Layer 1
        self.z1 = np.dot(x, self.W1) + self.b1
        self.a1 = sigmoid(self.z1)

        # Layer 2
        self.z2 = np.dot(self.a1, self.W2) + self.b2
        self.a2 = sigmoid(self.z2)

        return self.a2

    def backward(self, x, y_true, y_pred):
        """
        Backpropagation

        Args:
            x: Input data
            y_true: True labels
            y_pred: Predicted values
        """
        batch_size = x.shape[0]

        # Gradient at the output layer
        delta2 = (y_pred - y_true) * sigmoid_derivative(self.z2)

        # Gradients of layer 2 weights and bias
        dW2 = np.dot(self.a1.T, delta2) / batch_size
        db2 = np.sum(delta2, axis=0) / batch_size

        # Gradient at the hidden layer
        delta1 = np.dot(delta2, self.W2.T) * sigmoid_derivative(self.z1)

        # Gradients of layer 1 weights and bias
        dW1 = np.dot(x.T, delta1) / batch_size
        db1 = np.sum(delta1, axis=0) / batch_size

        # Update parameters
        self.W1 -= self.learning_rate * dW1
        self.b1 -= self.learning_rate * db1
        self.W2 -= self.learning_rate * dW2
        self.b2 -= self.learning_rate * db2

    def train(self, x, y_true, epochs=1000, verbose=True):
        """
        Training loop

        Args:
            x: Training data
            y_true: True labels
            epochs: Number of epochs
            verbose: Show progress
        """
        losses = []

        for epoch in range(epochs):
            # Forward propagation
            y_pred = self.forward(x)

            # Compute loss
            loss = np.mean((y_true - y_pred) ** 2)
            losses.append(loss)

            # Backpropagation
            self.backward(x, y_true, y_pred)

            # Show progress
            if verbose and (epoch % 100 == 0 or epoch == epochs - 1):
                print(f"Epoch {epoch:4d}: Loss = {loss:.6f}")

        return losses

# Test on the XOR problem
print("=== Training on the XOR Problem ===")

# Prepare data
X = np.array([[0, 0], [0, 1], [1, 0], [1, 1]])
y = np.array([[0], [1], [1], [0]])

# Create and train the network
net = TwoLayerNetWithBackprop(input_size=2, hidden_size=4, output_size=1, learning_rate=0.5)
losses = net.train(X, y, epochs=5000, verbose=True)

# Final results
print("\n=== Final Predictions ===")
predictions = net.forward(X)
for i in range(len(X)):
    pred_label = 1 if predictions[i] > 0.5 else 0
    print(f"Input: {X[i]} → Prediction: {predictions[i][0]:.4f} → Label: {pred_label} (True: {y[i][0]})")

Example output:

=== Training on the XOR Problem ===
Epoch    0: Loss = 0.259762
Epoch  100: Loss = 0.249876
Epoch  200: Loss = 0.249011
Epoch  300: Loss = 0.246863
...
Epoch 4900: Loss = 0.000625
Epoch 4999: Loss = 0.000612

=== Final Predictions ===
Input: [0 0] → Prediction: 0.0247 → Label: 0 (True: 0)
Input: [0 1] → Prediction: 0.9753 → Label: 1 (True: 1)
Input: [1 0] → Prediction: 0.9751 → Label: 1 (True: 1)
Input: [1 1] → Prediction: 0.0254 → Label: 0 (True: 0)

Success! With the multilayer perceptron and backpropagation, we solved the XOR problem!

2.5 Visualizing the Learning Curve

import matplotlib.pyplot as plt

def plot_learning_curve(losses):
    """Plot the learning curve"""
    plt.figure(figsize=(10, 6))
    plt.plot(losses, linewidth=2)
    plt.xlabel('Epoch', fontsize=12)
    plt.ylabel('Loss (MSE)', fontsize=12)
    plt.title('Learning Curve for the XOR Problem', fontsize=14, fontweight='bold')
    plt.grid(True, alpha=0.3)
    plt.yscale('log')  # Logarithmic scale
    plt.show()

plot_learning_curve(losses)

Visualizing the Decision Boundary

def plot_decision_boundary(net, X, y):
    """Visualize the decision boundary"""
    # Create grid
    x_min, x_max = -0.5, 1.5
    y_min, y_max = -0.5, 1.5
    xx, yy = np.meshgrid(np.linspace(x_min, x_max, 200),
                         np.linspace(y_min, y_max, 200))

    # Predict at each point
    Z = net.forward(np.c_[xx.ravel(), yy.ravel()])
    Z = Z.reshape(xx.shape)

    # Plot
    plt.figure(figsize=(10, 8))

    # Background colors (decision boundary)
    plt.contourf(xx, yy, Z, levels=20, cmap='RdYlBu', alpha=0.8)
    plt.colorbar(label='Predicted value')

    # Data points
    plt.scatter(X[y.flatten()==0][:, 0], X[y.flatten()==0][:, 1],
                s=200, c='blue', marker='o', edgecolors='k', linewidths=2,
                label='Class 0')
    plt.scatter(X[y.flatten()==1][:, 0], X[y.flatten()==1][:, 1],
                s=200, c='red', marker='s', edgecolors='k', linewidths=2,
                label='Class 1')

    plt.xlim(x_min, x_max)
    plt.ylim(y_min, y_max)
    plt.xlabel('x1', fontsize=14)
    plt.ylabel('x2', fontsize=14)
    plt.title('Decision Boundary for the XOR Problem (Multilayer Perceptron)', fontsize=16, fontweight='bold')
    plt.legend(fontsize=12)
    plt.grid(True, alpha=0.3)
    plt.show()

plot_decision_boundary(net, X, y)

2.6 Mini-Batch Training

Types of Batch Training

Method Batch Size Characteristics
Batch Gradient Descent All data Stable but slow
Stochastic Gradient Descent (SGD) 1 sample Fast but unstable
Mini-Batch Gradient Descent Tens to hundreds Well balanced (practical) 
def create_mini_batches(X, y, batch_size):
    """
    Create mini-batches

    Args:
        X: Input data
        y: Labels
        batch_size: Batch size

    Yields:
        Tuple of (X_batch, y_batch)
    """
    n_samples = X.shape[0]
    indices = np.random.permutation(n_samples)

    for start_idx in range(0, n_samples, batch_size):
        end_idx = min(start_idx + batch_size, n_samples)
        batch_indices = indices[start_idx:end_idx]

        yield X[batch_indices], y[batch_indices]

# Example of mini-batch training
def train_with_minibatch(net, X, y, epochs=1000, batch_size=2):
    """Mini-batch training"""
    losses = []

    for epoch in range(epochs):
        epoch_loss = 0
        n_batches = 0

        for X_batch, y_batch in create_mini_batches(X, y, batch_size):
            # Forward propagation
            y_pred = net.forward(X_batch)

            # Loss
            loss = np.mean((y_batch - y_pred) ** 2)
            epoch_loss += loss
            n_batches += 1

            # Backpropagation
            net.backward(X_batch, y_batch, y_pred)

        avg_loss = epoch_loss / n_batches
        losses.append(avg_loss)

        if epoch % 100 == 0:
            print(f"Epoch {epoch:4d}: Loss = {avg_loss:.6f}")

    return losses

# Test
print("\n=== Mini-Batch Training ===")
net_mini = TwoLayerNetWithBackprop(input_size=2, hidden_size=4, output_size=1, learning_rate=0.5)
losses_mini = train_with_minibatch(net_mini, X, y, epochs=2000, batch_size=2)

2.7 Chapter Summary

What We Learned

  1. Structure of the Multilayer Perceptron

    • Input layer, hidden layer, output layer
    • Each layer has weights $W$ and bias $b$
    • Activation functions introduce nonlinearity

  2. Loss Functions

    • MSE: regression problems
    • Cross-Entropy: classification problems

  3. Gradient Descent

    • $w \leftarrow w - \eta \frac{\partial L}{\partial w}$
    • Importance of the learning rate $\eta$

  4. Backpropagation

    • Efficient gradient computation via the chain rule
    • Backward computation from the output layer to the input layer
    • Complete implementation with NumPy

  5. Solving the XOR Problem

    • Multiple layers solve nonlinear problems
    • Actually trained and achieved 100% accuracy

Key Equations

Concept Equation
Forward Propagation $y = \sigma(W^{(2)} \sigma(W^{(1)} x + b^{(1)}) + b^{(2)})$
MSE Loss $L = \frac{1}{n} \sum_{i=1}^{n} (y_i - \hat{y}_i)^2$
Gradient Descent $w \leftarrow w - \eta \frac{\partial L}{\partial w}$
Chain Rule $\frac{\partial L}{\partial w} = \frac{\partial L}{\partial y} \cdot \frac{\partial y}{\partial z} \cdot \frac{\partial z}{\partial w}$

To the Next Chapter

In Chapter 3, we will learn about activation functions and optimization:

Exercises

Problem 1 (Difficulty: easy)

Determine whether each of the following statements is true or false.

  1. A multilayer perceptron has hidden layers
  2. Backpropagation computes from the input layer toward the output layer
  3. A learning rate that is too large can cause divergence
  4. The XOR problem can be solved with a single-layer perceptron
Sample Answer
  1. True - This is the definition of an MLP
  2. False - Backpropagation goes from the output layer to the input layer
  3. True - A large learning rate causes oscillation and divergence
  4. False - XOR is not linearly separable; multiple layers are required

Problem 2 (Difficulty: medium)

Derive the derivative of the sigmoid function. Also, implement it in Python.

Hint

Sigmoid function: $\sigma(x) = \frac{1}{1 + e^{-x}}$

Use the quotient rule for differentiation.

Sample Answer

Derivation:

$$ \begin{align} \sigma(x) &= \frac{1}{1 + e^{-x}} \\ \sigma'(x) &= \frac{d}{dx} (1 + e^{-x})^{-1} \\ &= -(1 + e^{-x})^{-2} \cdot (-e^{-x}) \\ &= \frac{e^{-x}}{(1 + e^{-x})^2} \\ &= \frac{1}{1 + e^{-x}} \cdot \frac{e^{-x}}{1 + e^{-x}} \\ &= \sigma(x) \cdot \frac{e^{-x}}{1 + e^{-x}} \\ &= \sigma(x) \cdot \frac{1 + e^{-x} - 1}{1 + e^{-x}} \\ &= \sigma(x) \cdot (1 - \sigma(x)) \end{align} $$

Python implementation:

def sigmoid(x):
    return 1 / (1 + np.exp(-x))

def sigmoid_derivative(x):
    s = sigmoid(x)
    return s * (1 - s)

# Test
x_test = np.linspace(-5, 5, 100)
plt.figure(figsize=(10, 6))
plt.plot(x_test, sigmoid(x_test), label='σ(x)', linewidth=2)
plt.plot(x_test, sigmoid_derivative(x_test), label="σ'(x)", linewidth=2)
plt.xlabel('x', fontsize=12)
plt.ylabel('y', fontsize=12)
plt.title('Sigmoid Function and Its Derivative', fontsize=14)
plt.legend(fontsize=12)
plt.grid(True, alpha=0.3)
plt.show()

Problem 3 (Difficulty: medium)

Implement a three-layer neural network (input layer, two hidden layers, output layer).

Sample Answer 
class ThreeLayerNet:
    """Three-layer neural network"""

    def __init__(self, input_size, hidden1_size, hidden2_size, output_size, learning_rate=0.1):
        # Initialize weights
        self.W1 = np.random.randn(input_size, hidden1_size) * 0.1
        self.b1 = np.zeros(hidden1_size)

        self.W2 = np.random.randn(hidden1_size, hidden2_size) * 0.1
        self.b2 = np.zeros(hidden2_size)

        self.W3 = np.random.randn(hidden2_size, output_size) * 0.1
        self.b3 = np.zeros(output_size)

        self.learning_rate = learning_rate

    def forward(self, x):
        """Forward propagation"""
        # Layer 1
        self.z1 = np.dot(x, self.W1) + self.b1
        self.a1 = sigmoid(self.z1)

        # Layer 2
        self.z2 = np.dot(self.a1, self.W2) + self.b2
        self.a2 = sigmoid(self.z2)

        # Layer 3
        self.z3 = np.dot(self.a2, self.W3) + self.b3
        self.a3 = sigmoid(self.z3)

        return self.a3

    def backward(self, x, y_true, y_pred):
        """Backpropagation"""
        batch_size = x.shape[0]

        # Output layer
        delta3 = (y_pred - y_true) * sigmoid_derivative(self.z3)
        dW3 = np.dot(self.a2.T, delta3) / batch_size
        db3 = np.sum(delta3, axis=0) / batch_size

        # Second hidden layer
        delta2 = np.dot(delta3, self.W3.T) * sigmoid_derivative(self.z2)
        dW2 = np.dot(self.a1.T, delta2) / batch_size
        db2 = np.sum(delta2, axis=0) / batch_size

        # First hidden layer
        delta1 = np.dot(delta2, self.W2.T) * sigmoid_derivative(self.z1)
        dW1 = np.dot(x.T, delta1) / batch_size
        db1 = np.sum(delta1, axis=0) / batch_size

        # Update parameters
        self.W1 -= self.learning_rate * dW1
        self.b1 -= self.learning_rate * db1
        self.W2 -= self.learning_rate * dW2
        self.b2 -= self.learning_rate * db2
        self.W3 -= self.learning_rate * dW3
        self.b3 -= self.learning_rate * db3

# Test
print("=== Three-Layer Neural Network ===")
net3 = ThreeLayerNet(input_size=2, hidden1_size=4, hidden2_size=4, output_size=1, learning_rate=0.5)

# Train on the XOR problem
for epoch in range(3000):
    y_pred = net3.forward(X)
    net3.backward(X, y, y_pred)

    if epoch % 500 == 0:
        loss = np.mean((y - y_pred) ** 2)
        print(f"Epoch {epoch:4d}: Loss = {loss:.6f}")

# Final results
print("\n=== Final Predictions ===")
final_pred = net3.forward(X)
for i in range(len(X)):
    print(f"Input: {X[i]} → Prediction: {final_pred[i][0]:.4f} (True: {y[i][0]})")

Problem 4 (Difficulty: hard)

Implement a neural network that learns the three logic gates AND, OR, and NAND simultaneously (multi-task learning).

Hint
Sample Answer 
# Prepare data
X = np.array([[0, 0], [0, 1], [1, 0], [1, 1]])
y_multi = np.array([
    [0, 0, 1],  # AND=0, OR=0, NAND=1
    [0, 1, 1],  # AND=0, OR=1, NAND=1
    [0, 1, 1],  # AND=0, OR=1, NAND=1
    [1, 1, 0]   # AND=1, OR=1, NAND=0
])

# Multi-task network
net_multi = TwoLayerNetWithBackprop(input_size=2, hidden_size=6, output_size=3, learning_rate=0.5)

print("=== Multi-Task Learning (AND, OR, NAND) ===")

# Training
for epoch in range(5000):
    y_pred = net_multi.forward(X)
    net_multi.backward(X, y_multi, y_pred)

    if epoch % 1000 == 0:
        loss = np.mean((y_multi - y_pred) ** 2)
        print(f"Epoch {epoch:4d}: Loss = {loss:.6f}")

# Final results
print("\n=== Final Predictions ===")
print("Input | AND pred | OR pred | NAND pred | AND true | OR true | NAND true")
print("-" * 75)
final_pred = net_multi.forward(X)
for i in range(len(X)):
    print(f"{X[i]} | {final_pred[i][0]:.4f}  | {final_pred[i][1]:.4f}  | {final_pred[i][2]:.4f}   | "
          f"{y_multi[i][0]}       | {y_multi[i][1]}       | {y_multi[i][2]}")

Problem 5 (Difficulty: hard)

Implement learning rate scheduling (gradually decreasing the learning rate).

Sample Answer 
def learning_rate_decay(initial_lr, epoch, decay_rate=0.95, decay_step=100):
    """
    Learning rate decay

    Args:
        initial_lr: Initial learning rate
        epoch: Current epoch
        decay_rate: Decay rate
        decay_step: Decay step

    Returns:
        Decayed learning rate
    """
    return initial_lr * (decay_rate ** (epoch // decay_step))

class TwoLayerNetWithLRScheduling(TwoLayerNetWithBackprop):
    """Network with learning rate scheduling"""

    def __init__(self, input_size, hidden_size, output_size, initial_lr=0.5, decay_rate=0.95):
        super().__init__(input_size, hidden_size, output_size, initial_lr)
        self.initial_lr = initial_lr
        self.decay_rate = decay_rate

    def train_with_scheduling(self, X, y, epochs=5000):
        """Training with learning rate scheduling"""
        losses = []

        for epoch in range(epochs):
            # Update learning rate
            self.learning_rate = learning_rate_decay(
                self.initial_lr, epoch, self.decay_rate, decay_step=500
            )

            # Forward propagation
            y_pred = self.forward(X)

            # Loss
            loss = np.mean((y - y_pred) ** 2)
            losses.append(loss)

            # Backpropagation
            self.backward(X, y, y_pred)

            if epoch % 500 == 0:
                print(f"Epoch {epoch:4d}: LR = {self.learning_rate:.6f}, Loss = {loss:.6f}")

        return losses

# Test
print("=== Learning Rate Scheduling ===")
net_sched = TwoLayerNetWithLRScheduling(input_size=2, hidden_size=4, output_size=1,
                                        initial_lr=1.0, decay_rate=0.9)
losses_sched = net_sched.train_with_scheduling(X, y, epochs=5000)

References

  1. Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986). "Learning representations by back-propagating errors." Nature, 323(6088), 533-536.
  2. LeCun, Y., Bengio, Y., & Hinton, G. (2015). "Deep learning." Nature, 521(7553), 436-444.
  3. Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep Learning. MIT Press.
  4. Saito, K. (2016). Deep Learning from Scratch. O'Reilly Japan.
← Previous Chapter: Perceptron Next Chapter: Activation Functions and Optimization →

Disclaimer