Optimization for Machine Learning

Build a working understanding of the algorithms that train and tune machine learning models — implemented from scratch and with production tools like SciPy and Optuna.

Series Overview

This series is a practical, 3-chapter guide to optimization, the engine underneath every machine learning model. It progresses from the gradient-based methods that train neural networks, through gradient-free metaheuristics for problems where derivatives are unavailable, to Bayesian optimization for tuning models when each evaluation is expensive.

Optimization is the process of finding parameter values that minimize (or maximize) an objective function. In machine learning it appears at two levels: training, where optimizers like SGD and Adam adjust model parameters to reduce a loss, and hyperparameter tuning, where methods like Bayesian optimization search over configurations. This series covers both, pairing the mathematics with runnable Python so you can see exactly how each method behaves on real loss landscapes.

Features:

• ✅ Theory to Practice: Each method is derived, then implemented and run
• ✅ Implementation-Focused: 30 executable Python code examples across the series
• ✅ From Scratch and With Tools: Hand-coded gradient descent alongside SciPy and Optuna
• ✅ Visual Intuition: Loss landscapes, convergence paths, and acquisition functions
• ✅ Practical Guidance: When to reach for each family of methods

Total Learning Time: 1.5-2 hours (including code execution and exercises)

How to Study

Recommended Study Order

For Beginners (new to optimization):
- Chapter 1 → Chapter 2 → Chapter 3 (all chapters recommended)
- Time required: 1.5-2 hours

For Intermediate learners (comfortable with gradient descent):
- Chapter 2 → Chapter 3
- Time required: 50-60 minutes

For Specific Topic Enhancement:
- Gradient descent, Adam, learning-rate schedules: Chapter 1
- Simulated annealing, genetic algorithms, PSO: Chapter 2
- Gaussian processes, acquisition functions, Optuna: Chapter 3

Chapter Details

Chapter 1: Optimization Fundamentals

Difficulty: Beginner-Intermediate
Reading Time: 25-30 minutes
Code Examples: 10

Learning Content

1. What is Optimization in Machine Learning? - Objective functions, minima, and the training loop
2. Convexity and Loss Landscapes - Convex vs. non-convex objectives, local minima, saddle points
3. Gradient Descent from Scratch - The update rule, step size, and convergence
4. Stochastic Gradient Descent and Mini-Batching - Trading noise for speed
5. Momentum, RMSProp, and Adam - Adaptive optimizers and why they work
6. Learning-Rate Schedules - Step decay, cosine annealing, and warmup

Learning Objectives

• ✅ Explain convexity and read a loss landscape
• ✅ Implement gradient descent from scratch
• ✅ Understand the trade-offs of stochastic gradient descent and mini-batching
• ✅ Implement Momentum, RMSProp, and Adam
• ✅ Choose and implement learning-rate schedules

Read Chapter 1 →

Chapter 2: Metaheuristic Optimization

Difficulty: Intermediate
Reading Time: 25-30 minutes
Code Examples: 9

Learning Content

1. Why Gradient-Free Optimization? - Discrete, noisy, and multimodal objectives
2. Simulated Annealing - The Metropolis criterion and cooling schedules
3. Genetic Algorithms - Selection, crossover, and mutation
4. Particle Swarm Optimization - Swarm dynamics and convergence
5. Differential Evolution with SciPy - A robust, practical global optimizer
6. Practical Guidance - Choosing a metaheuristic and setting a fair budget

Learning Objectives

• ✅ Recognize when gradient methods fail and gradient-free search is needed
• ✅ Implement simulated annealing with the Metropolis criterion
• ✅ Implement a genetic algorithm
• ✅ Implement particle swarm optimization and visualize convergence
• ✅ Compare metaheuristics fairly using equal evaluation budgets

Read Chapter 2 →

Chapter 3: Bayesian Optimization and Surrogate Models

Difficulty: Intermediate-Advanced
Reading Time: 25-30 minutes
Code Examples: 11

Learning Content

1. The Cost Problem of Hyperparameter Search - When every evaluation is a full training run
2. Gaussian Process Surrogates - Modeling the objective with uncertainty
3. Acquisition Functions - Expected Improvement and the exploration-exploitation trade-off
4. Practical Bayesian Optimization with Optuna - TPE, samplers, and pruning
5. Search-Space Design and Pitfalls - Log scales, conditional spaces, and common mistakes

Learning Objectives

• ✅ Explain why Bayesian optimization suits expensive objectives
• ✅ Understand Gaussian process surrogates and predictive uncertainty
• ✅ Interpret acquisition functions such as Expected Improvement
• ✅ Run Bayesian optimization with Optuna
• ✅ Design effective search spaces and avoid common pitfalls

Read Chapter 3 →

Overall Learning Outcomes

Upon completing this series, you will have acquired the following skills and knowledge:

Knowledge Level (Understanding)

• ✅ Explain how gradient-based optimizers train machine learning models
• ✅ Distinguish convex from non-convex optimization problems
• ✅ Understand the principles behind metaheuristics and Bayesian optimization
• ✅ Explain the exploration-exploitation trade-off

Practical Skills (Doing)

• ✅ Implement gradient descent, Momentum, RMSProp, and Adam from scratch
• ✅ Implement simulated annealing, genetic algorithms, and PSO
• ✅ Use SciPy differential evolution and Optuna Bayesian optimization
• ✅ Visualize loss landscapes, convergence paths, and acquisition functions

Application Ability (Applying)

• ✅ Choose the right optimization family for a given problem
• ✅ Tune hyperparameters efficiently under a limited evaluation budget
• ✅ Design search spaces that make optimization tractable

Prerequisites

To effectively study this series, the following knowledge is recommended:

Required (Must Have)

• ✅ Python Basics: Variables, functions, loops, and classes
• ✅ NumPy: Arrays and vectorized operations
• ✅ Calculus Basics: Derivatives and gradients
• ✅ Machine Learning Basics: Loss functions and model training (recommended)

Recommended (Nice to Have)

• 💡 Matplotlib: For reproducing the visualizations
• 💡 Probability & Statistics: For the Gaussian process material in Chapter 3
• 💡 scikit-learn: For the hyperparameter-tuning examples

Technologies and Tools Used

Main Libraries

• NumPy 1.24+ - Numerical computation and from-scratch implementations
• SciPy 1.10+ - Differential evolution and optimization utilities
• Optuna 3.4+ - Bayesian optimization and hyperparameter search
• Matplotlib 3.7+ - Visualization of landscapes and convergence
• scikit-learn 1.3+ - Models used in the tuning examples

Development Environment

• Python 3.8+ - Programming language
• Jupyter Notebook / Lab - Recommended for running the examples interactively

Let's Get Started!

Ready to begin? Start with Chapter 1 and build optimization from the ground up.

Chapter 1: Optimization Fundamentals →

Next Steps

After completing this series, we recommend continuing with the following related topics:

Related Series

• 🎯 AutoML Introduction - Automated hyperparameter optimization and model selection
• 🎯 Neural Networks Introduction - Where gradient-based optimizers are applied

Update History

• 2026-07-09: v1.0 first edition published