Chapter 3: Activation Functions and Optimization

Learning Objectives

By reading this chapter, you will be able to:

• ✅ Understand the characteristics of various activation functions (Sigmoid, ReLU, Leaky ReLU, ELU, Swish)
• ✅ Explain the Vanishing Gradient Problem and its countermeasures
• ✅ Implement advanced optimization algorithms (Momentum, AdaGrad, RMSprop, Adam)
• ✅ Understand the importance of learning rate scheduling
• ✅ Apply weight initialization strategies (Xavier, He initialization)

3.1 Activation Functions

The Role of Activation Functions

Activation functions introduce nonlinearity into neural networks. Without activation functions, no matter how many layers are stacked, the result is only a linear transformation, and complex patterns cannot be learned.

"Activation functions are a key element that determines the 'expressive power' of a neural network."

3.1.1 Sigmoid Function

Formula:

$$ \sigma(x) = \frac{1}{1 + e^{-x}} $$

Derivative:

$$ \sigma'(x) = \sigma(x)(1 - \sigma(x)) $$

import numpy as np
import matplotlib.pyplot as plt

def sigmoid(x):
    """Sigmoid function"""
    return 1 / (1 + np.exp(-np.clip(x, -500, 500)))

def sigmoid_derivative(x):
    """Derivative of sigmoid"""
    s = sigmoid(x)
    return s * (1 - s)

# Visualization
x = np.linspace(-10, 10, 200)
y = sigmoid(x)
dy = sigmoid_derivative(x)

plt.figure(figsize=(12, 5))

plt.subplot(1, 2, 1)
plt.plot(x, y, linewidth=2, label='σ(x)')
plt.xlabel('x', fontsize=12)
plt.ylabel('σ(x)', fontsize=12)
plt.title('Sigmoid Function', fontsize=14, fontweight='bold')
plt.grid(True, alpha=0.3)
plt.legend()

plt.subplot(1, 2, 2)
plt.plot(x, dy, linewidth=2, color='red', label="σ'(x)")
plt.xlabel('x', fontsize=12)
plt.ylabel("σ'(x)", fontsize=12)
plt.title('Derivative of Sigmoid', fontsize=14, fontweight='bold')
plt.grid(True, alpha=0.3)
plt.legend()

plt.tight_layout()
plt.show()

Characteristics:

• ✅ Output range: (0, 1)
• ✅ Smooth and differentiable
• ❌ Vanishing gradient problem: when $|x|$ is large, the derivative approaches 0
• ❌ Output is not zero-centered (slows convergence during training)

3.1.2 tanh Function (Hyperbolic Tangent)

Formula:

$$ \tanh(x) = \frac{e^x - e^{-x}}{e^x + e^{-x}} $$

Derivative:

$$ \tanh'(x) = 1 - \tanh^2(x) $$

def tanh(x):
    """tanh function"""
    return np.tanh(x)

def tanh_derivative(x):
    """Derivative of tanh"""
    return 1 - np.tanh(x) ** 2

# Visualization
y_tanh = tanh(x)
dy_tanh = tanh_derivative(x)

plt.figure(figsize=(12, 5))

plt.subplot(1, 2, 1)
plt.plot(x, y_tanh, linewidth=2, color='green')
plt.xlabel('x', fontsize=12)
plt.ylabel('tanh(x)', fontsize=12)
plt.title('tanh Function', fontsize=14, fontweight='bold')
plt.grid(True, alpha=0.3)

plt.subplot(1, 2, 2)
plt.plot(x, dy_tanh, linewidth=2, color='red')
plt.xlabel('x', fontsize=12)
plt.ylabel("tanh'(x)", fontsize=12)
plt.title('Derivative of tanh', fontsize=14, fontweight='bold')
plt.grid(True, alpha=0.3)

plt.tight_layout()
plt.show()

Characteristics:

• ✅ Output range: (-1, 1)
• ✅ Zero-centered output (better than Sigmoid)
• ❌ Vanishing gradient problem remains

3.1.3 ReLU (Rectified Linear Unit)

Formula:

$$ \text{ReLU}(x) = \max(0, x) $$

Derivative:

$$ \text{ReLU}'(x) = \begin{cases} 1 & \text{if } x > 0 \\ 0 & \text{if } x \leq 0 \end{cases} $$

def relu(x):
    """ReLU function"""
    return np.maximum(0, x)

def relu_derivative(x):
    """Derivative of ReLU"""
    return (x > 0).astype(float)

# Visualization
y_relu = relu(x)
dy_relu = relu_derivative(x)

plt.figure(figsize=(12, 5))

plt.subplot(1, 2, 1)
plt.plot(x, y_relu, linewidth=2, color='purple')
plt.xlabel('x', fontsize=12)
plt.ylabel('ReLU(x)', fontsize=12)
plt.title('ReLU Function', fontsize=14, fontweight='bold')
plt.grid(True, alpha=0.3)

plt.subplot(1, 2, 2)
plt.plot(x, dy_relu, linewidth=2, color='red')
plt.xlabel('x', fontsize=12)
plt.ylabel("ReLU'(x)", fontsize=12)
plt.title('Derivative of ReLU', fontsize=14, fontweight='bold')
plt.grid(True, alpha=0.3)

plt.tight_layout()
plt.show()

Characteristics:

• ✅ Very fast to compute (only a max operation)
• ✅ Greatly mitigates the vanishing gradient problem
• ✅ Currently the most widely used activation function
• ❌ Dying ReLU problem: neurons die on negative inputs (zero gradient)

3.1.4 Leaky ReLU

Formula:

$$ \text{Leaky ReLU}(x) = \begin{cases} x & \text{if } x > 0 \\ \alpha x & \text{if } x \leq 0 \end{cases} $$

Typically, $\alpha = 0.01$

def leaky_relu(x, alpha=0.01):
    """Leaky ReLU function"""
    return np.where(x > 0, x, alpha * x)

def leaky_relu_derivative(x, alpha=0.01):
    """Derivative of Leaky ReLU"""
    return np.where(x > 0, 1.0, alpha)

# Visualization
y_leaky = leaky_relu(x)
dy_leaky = leaky_relu_derivative(x)

plt.figure(figsize=(12, 5))

plt.subplot(1, 2, 1)
plt.plot(x, y_leaky, linewidth=2, color='orange')
plt.xlabel('x', fontsize=12)
plt.ylabel('Leaky ReLU(x)', fontsize=12)
plt.title('Leaky ReLU Function (α=0.01)', fontsize=14, fontweight='bold')
plt.grid(True, alpha=0.3)

plt.subplot(1, 2, 2)
plt.plot(x, dy_leaky, linewidth=2, color='red')
plt.xlabel('x', fontsize=12)
plt.ylabel("Leaky ReLU'(x)", fontsize=12)
plt.title('Derivative of Leaky ReLU', fontsize=14, fontweight='bold')
plt.grid(True, alpha=0.3)

plt.tight_layout()
plt.show()

Characteristics:

• ✅ Solves the Dying ReLU problem
• ✅ A small gradient flows even for negative inputs

3.1.5 ELU (Exponential Linear Unit)

Formula:

$$ \text{ELU}(x) = \begin{cases} x & \text{if } x > 0 \\ \alpha (e^x - 1) & \text{if } x \leq 0 \end{cases} $$

def elu(x, alpha=1.0):
    """ELU function"""
    return np.where(x > 0, x, alpha * (np.exp(x) - 1))

def elu_derivative(x, alpha=1.0):
    """Derivative of ELU"""
    return np.where(x > 0, 1.0, alpha * np.exp(x))

# Visualization
y_elu = elu(x)
dy_elu = elu_derivative(x)

plt.figure(figsize=(12, 5))

plt.subplot(1, 2, 1)
plt.plot(x, y_elu, linewidth=2, color='brown')
plt.xlabel('x', fontsize=12)
plt.ylabel('ELU(x)', fontsize=12)
plt.title('ELU Function (α=1.0)', fontsize=14, fontweight='bold')
plt.grid(True, alpha=0.3)

plt.subplot(1, 2, 2)
plt.plot(x, dy_elu, linewidth=2, color='red')
plt.xlabel('x', fontsize=12)
plt.ylabel("ELU'(x)", fontsize=12)
plt.title('Derivative of ELU', fontsize=14, fontweight='bold')
plt.grid(True, alpha=0.3)

plt.tight_layout()
plt.show()

Characteristics:

• ✅ Smooth and differentiable
• ✅ Output mean close to zero
• ❌ High computational cost of the exponential function

Comparison of Activation Functions

3.2 Vanishing Gradient Problem

The Essence of the Problem

In deep networks, gradients become exponentially smaller during backpropagation.

By the chain rule:

$$ \frac{\partial L}{\partial w^{(1)}} = \frac{\partial L}{\partial w^{(10)}} \cdot \frac{\partial w^{(10)}}{\partial w^{(9)}} \cdot \ldots \cdot \frac{\partial w^{(2)}}{\partial w^{(1)}} $$

If $|\frac{\partial w^{(l)}}{\partial w^{(l-1)}}| < 1$ at each layer, the gradient vanishes.

def demonstrate_vanishing_gradient():
    """Demonstration of the vanishing gradient problem"""

    # Sigmoid network (10 layers)
    def forward_sigmoid_deep(x, n_layers=10):
        a = x
        activations = [a]

        for _ in range(n_layers):
            z = a * 0.5  # Simplified weight
            a = sigmoid(z)
            activations.append(a)

        return activations

    # Compute gradients
    x = np.array([1.0])
    activations = forward_sigmoid_deep(x, n_layers=10)

    # Compute the gradient at each layer
    gradients = []
    grad = 1.0

    for i in range(len(activations) - 1, 0, -1):
        a = activations[i]
        grad = grad * (a * (1 - a)) * 0.5  # Chain rule
        gradients.append(grad)

    gradients = gradients[::-1]

    # Plot
    plt.figure(figsize=(10, 6))
    plt.plot(range(1, len(gradients) + 1), gradients, marker='o',
             linewidth=2, markersize=8, label='Gradient magnitude')
    plt.xlabel('Layer depth', fontsize=12)
    plt.ylabel('Gradient', fontsize=12)
    plt.title('Visualization of the Vanishing Gradient Problem (Sigmoid, 10 layers)', fontsize=14, fontweight='bold')
    plt.yscale('log')
    plt.grid(True, alpha=0.3)
    plt.legend()
    plt.show()

    print("=== Gradient at each layer ===")
    for i, grad in enumerate(gradients, 1):
        print(f"Layer {i}: {grad:.10f}")

demonstrate_vanishing_gradient()

Countermeasures

1. Use ReLU: gradient is 1 (when $x > 0$)
2. Batch Normalization: normalize the inputs of each layer
3. Residual Connection: shortcut for gradients
4. Proper initialization: Xavier, He initialization

3.3 Optimization Algorithms

3.3.1 SGD (Stochastic Gradient Descent)

Update rule:

$$ w \leftarrow w - \eta \frac{\partial L}{\partial w} $$

class SGD:
    """Stochastic gradient descent"""

    def __init__(self, learning_rate=0.01):
        self.learning_rate = learning_rate

    def update(self, params, grads):
        """
        Update parameters

        Args:
            params: dictionary of parameters {'W1': ..., 'b1': ...}
            grads: dictionary of gradients {'W1': ..., 'b1': ...}
        """
        for key in params.keys():
            params[key] -= self.learning_rate * grads[key]

3.3.2 Momentum

Update rule:

$$ \begin{align} v &\leftarrow \beta v - \eta \frac{\partial L}{\partial w} \\ w &\leftarrow w + v \end{align} $$

Typically, $\beta = 0.9$

class Momentum:
    """Momentum optimization"""

    def __init__(self, learning_rate=0.01, momentum=0.9):
        self.learning_rate = learning_rate
        self.momentum = momentum
        self.velocity = None

    def update(self, params, grads):
        if self.velocity is None:
            self.velocity = {}
            for key, val in params.items():
                self.velocity[key] = np.zeros_like(val)

        for key in params.keys():
            self.velocity[key] = self.momentum * self.velocity[key] - self.learning_rate * grads[key]
            params[key] += self.velocity[key]

Characteristics:

• ✅ Takes past gradients into account
• ✅ Suppresses oscillation and accelerates convergence

3.3.3 AdaGrad (Adaptive Gradient)

Update rule:

$$ \begin{align} h &\leftarrow h + \left(\frac{\partial L}{\partial w}\right)^2 \\ w &\leftarrow w - \frac{\eta}{\sqrt{h} + \epsilon} \frac{\partial L}{\partial w} \end{align} $$

class AdaGrad:
    """AdaGrad optimization"""

    def __init__(self, learning_rate=0.01):
        self.learning_rate = learning_rate
        self.h = None
        self.epsilon = 1e-8

    def update(self, params, grads):
        if self.h is None:
            self.h = {}
            for key, val in params.items():
                self.h[key] = np.zeros_like(val)

        for key in params.keys():
            self.h[key] += grads[key] ** 2
            params[key] -= self.learning_rate * grads[key] / (np.sqrt(self.h[key]) + self.epsilon)

Characteristics:

• ✅ Adjusts the learning rate per parameter
• ❌ Learning rate gradually becomes too small

3.3.4 RMSprop

Update rule:

$$ \begin{align} h &\leftarrow \beta h + (1 - \beta) \left(\frac{\partial L}{\partial w}\right)^2 \\ w &\leftarrow w - \frac{\eta}{\sqrt{h} + \epsilon} \frac{\partial L}{\partial w} \end{align} $$

class RMSprop:
    """RMSprop optimization"""

    def __init__(self, learning_rate=0.01, decay_rate=0.99):
        self.learning_rate = learning_rate
        self.decay_rate = decay_rate
        self.h = None
        self.epsilon = 1e-8

    def update(self, params, grads):
        if self.h is None:
            self.h = {}
            for key, val in params.items():
                self.h[key] = np.zeros_like(val)

        for key in params.keys():
            self.h[key] = self.decay_rate * self.h[key] + (1 - self.decay_rate) * grads[key] ** 2
            params[key] -= self.learning_rate * grads[key] / (np.sqrt(self.h[key]) + self.epsilon)

Characteristics:

• ✅ Improved version of AdaGrad
• ✅ Exponential moving average mitigates learning-rate decay

3.3.5 Adam (Adaptive Moment Estimation)

Update rule:

$$ \begin{align} m &\leftarrow \beta_1 m + (1 - \beta_1) \frac{\partial L}{\partial w} \\ v &\leftarrow \beta_2 v + (1 - \beta_2) \left(\frac{\partial L}{\partial w}\right)^2 \\ \hat{m} &\leftarrow \frac{m}{1 - \beta_1^t} \\ \hat{v} &\leftarrow \frac{v}{1 - \beta_2^t} \\ w &\leftarrow w - \frac{\eta}{\sqrt{\hat{v}} + \epsilon} \hat{m} \end{align} $$

class Adam:
    """Adam optimization (most recommended)"""

    def __init__(self, learning_rate=0.001, beta1=0.9, beta2=0.999):
        self.learning_rate = learning_rate
        self.beta1 = beta1
        self.beta2 = beta2
        self.m = None
        self.v = None
        self.t = 0
        self.epsilon = 1e-8

    def update(self, params, grads):
        if self.m is None:
            self.m = {}
            self.v = {}
            for key, val in params.items():
                self.m[key] = np.zeros_like(val)
                self.v[key] = np.zeros_like(val)

        self.t += 1

        for key in params.keys():
            # First moment (mean)
            self.m[key] = self.beta1 * self.m[key] + (1 - self.beta1) * grads[key]

            # Second moment (variance)
            self.v[key] = self.beta2 * self.v[key] + (1 - self.beta2) * (grads[key] ** 2)

            # Bias correction
            m_hat = self.m[key] / (1 - self.beta1 ** self.t)
            v_hat = self.v[key] / (1 - self.beta2 ** self.t)

            # Parameter update
            params[key] -= self.learning_rate * m_hat / (np.sqrt(v_hat) + self.epsilon)

Characteristics:

• ✅ Combines the strengths of Momentum and RMSprop
• ✅ Currently the most widely used optimization algorithm
• ✅ Requires almost no hyperparameter tuning

Comparison of Optimization Algorithms

def compare_optimizers():
    """Comparison of optimization algorithms"""

    # Test function: f(x, y) = x^2 + 10*y^2 (ellipse)
    def f(x, y):
        return x ** 2 + 10 * y ** 2

    def grad_f(x, y):
        return np.array([2*x, 20*y])

    # Initial values
    init_pos = (-7.0, 2.0)
    learning_rate = 0.1
    iterations = 30

    # Optimize with each optimization algorithm
    optimizers = {
        'SGD': SGD(learning_rate=learning_rate),
        'Momentum': Momentum(learning_rate=learning_rate),
        'AdaGrad': AdaGrad(learning_rate=learning_rate),
        'RMSprop': RMSprop(learning_rate=learning_rate),
        'Adam': Adam(learning_rate=learning_rate)
    }

    trajectories = {}

    for name, optimizer in optimizers.items():
        pos = np.array(init_pos)
        params = {'pos': pos}
        trajectory = [pos.copy()]

        for _ in range(iterations):
            grads = {'pos': grad_f(pos[0], pos[1])}
            optimizer.update(params, grads)
            pos = params['pos']
            trajectory.append(pos.copy())

        trajectories[name] = np.array(trajectory)

    # Plot
    plt.figure(figsize=(12, 10))

    # Draw contour lines
    x = np.linspace(-8, 2, 100)
    y = np.linspace(-3, 3, 100)
    X, Y = np.meshgrid(x, y)
    Z = f(X, Y)

    plt.contour(X, Y, Z, levels=20, alpha=0.3)

    # Trajectory of each optimization algorithm
    colors = ['blue', 'green', 'red', 'purple', 'orange']
    for (name, trajectory), color in zip(trajectories.items(), colors):
        plt.plot(trajectory[:, 0], trajectory[:, 1], marker='o',
                 label=name, color=color, linewidth=2, markersize=4)

    plt.plot(0, 0, 'r*', markersize=20, label='Optimal solution')
    plt.xlabel('x', fontsize=12)
    plt.ylabel('y', fontsize=12)
    plt.title('Comparison of Optimization Algorithms', fontsize=14, fontweight='bold')
    plt.legend(fontsize=10)
    plt.grid(True, alpha=0.3)
    plt.show()

compare_optimizers()

3.4 Weight Initialization

Why Initialization Matters

If the initial weights are inappropriate:

• ❌ Vanishing or exploding gradients
• ❌ Training fails to progress
• ❌ Getting stuck in local optima

3.4.1 Xavier Initialization

Formula (for Sigmoid, tanh):

$$ W \sim \mathcal{N}\left(0, \sqrt{\frac{2}{n_{\text{in}} + n_{\text{out}}}}\right) $$

def xavier_init(n_in, n_out):
    """Xavier initialization"""
    return np.random.randn(n_in, n_out) * np.sqrt(2.0 / (n_in + n_out))

# Example
W = xavier_init(100, 50)
print(f"Xavier initialization: mean={W.mean():.4f}, std={W.std():.4f}")

3.4.2 He Initialization

Formula (for ReLU):

$$ W \sim \mathcal{N}\left(0, \sqrt{\frac{2}{n_{\text{in}}}}\right) $$

def he_init(n_in, n_out):
    """He initialization (for ReLU)"""
    return np.random.randn(n_in, n_out) * np.sqrt(2.0 / n_in)

# Example
W = he_init(100, 50)
print(f"He initialization: mean={W.mean():.4f}, std={W.std():.4f}")

Comparison of Initialization Methods

3.5 Chapter Summary

What We Learned

1. Activation Functions

   • ReLU: the current default
   • Leaky ReLU: countermeasure for Dying ReLU
   • Sigmoid/tanh: suffer from the vanishing gradient problem

2. Vanishing Gradient Problem

   • A challenge for deep networks
   • Countermeasures: ReLU, Batch Norm, proper initialization

3. Optimization Algorithms

   • Adam: most recommended
   • Momentum: accelerates convergence
   • SGD: basic but slow

4. Weight Initialization

   • ReLU → He initialization
   • Sigmoid/tanh → Xavier initialization

Recommended Settings