Chapter 2: Fundamentals of Machine Learning

Concepts, Methods, and Ecosystem

📖 Reading Time: 20-25 min 📊 Difficulty: Beginner–Intermediate 💻 Code Examples: 5 📝 Exercises: 4

In this chapter, you will learn the basic concepts of machine learning, the three learning types, major algorithms, and a practical workflow. You will understand the technical terminology and prepare for implementation.

Learning Objectives

After completing this chapter, you will be able to:

2.1 Definition and Classification of Machine Learning

2.1.1 What is Machine Learning?

Machine Learning (ML) is a general term for computer programs that automatically learn patterns from data to make predictions and decisions.

Classic definition (Arthur Samuel, 1959):

"The field of study that gives computers the ability to learn without being explicitly programmed."

Formal definition (Tom Mitchell, 1997):

"A program that learns from experience E with respect to some task T and performance measure P, if its performance at task T, as measured by P, improves with experience E."

Understanding through a concrete example:

If the system learns from past email data and improves its accuracy in judging whether new emails are spam, then we can say it is "doing machine learning."

The Relationship Between AI, ML, and DL

graph TD A[Artificial Intelligence AI
Artificial Intelligence] --> B[Machine Learning ML
Machine Learning] B --> C[Deep Learning DL
Deep Learning] A1[Systems that exhibit
intelligent behavior] -.-> A B1[Learning patterns
from data] -.-> B C1[Learning using
neural networks] -.-> C style A fill:#e3f2fd style B fill:#fff3e0 style C fill:#f3e5f5

Containment relationship:

2.1.2 The Three Learning Types of Machine Learning

Machine learning is broadly divided into three categories based on the type of data and the learning method.

1. Supervised Learning

Definition: A method that learns from labeled data (pairs of inputs and correct answers).

How it works:

Input data (features) + correct labels → model training → prediction

Concrete examples:

Major algorithms:

2. Unsupervised Learning

Definition: A method that discovers hidden patterns or structure from unlabeled data.

How it works:

Input data only (no answers) → model → pattern discovery

Concrete examples:

Major algorithms:

3. Reinforcement Learning

Definition: A method that learns behavior to maximize rewards through trial and error.

How it works:

Agent → action → environment → reward/state feedback → learning

Key concepts:

Concrete examples:

Comparison of the Three Learning Types

Learning Type Data Objective Applications Difficulty
Supervised Learning Labeled Prediction / classification Spam detection, price prediction, image recognition Medium
Unsupervised Learning Unlabeled Pattern discovery Customer analysis, anomaly detection, data compression High
Reinforcement Learning Reward signal Action optimization Game AI, robotics, autonomous driving Highest

Flowchart for Choosing a Learning Type

graph TD A[Problem to solve] --> B{Is there
ground-truth data?} B -->|Yes| C{What is the output?} B -->|No| D{What is the goal?} C -->|Continuous value
e.g. price| E[Regression
Linear Regression] C -->|Category
e.g. spam/not spam| F[Classification
Logistic Regression
Decision Tree] D -->|Grouping| G[Clustering
K-means] D -->|Dimensionality reduction| H[Dimensionality Reduction
PCA] D -->|Anomaly detection| I[Anomaly Detection
Isolation Forest] J{Optimize through
trial and error?} -->|Yes| K[Reinforcement Learning
Q-learning
DQN] style E fill:#e8f5e9 style F fill:#e8f5e9 style G fill:#fff3e0 style H fill:#fff3e0 style I fill:#fff3e0 style K fill:#f3e5f5

2.2 Glossary of Key Machine Learning Terms (20 Terms)

Here are the essential technical terms for learning machine learning, organized by category. Understand each term as a four-part set: Japanese, English, definition, and a concrete example.

Basic Terms (8 Terms)

Term (Japanese) Term (English) Definition and Example
1. 特徴量 Feature An attribute or property of the data that serves as input to the model. For house price prediction, this includes "size, age, distance to station," etc. The quality of features greatly influences model performance. Designing appropriate features is called "feature engineering."
2. ラベル Label The ground-truth data in supervised learning. For spam detection, the labels "spam"/"not spam"; for house price prediction, the numeric "price." Because collecting labeled data is costly, the amount of data is often limited.
3. 訓練データ Training Data The dataset used to train the model. It is common to use 70-80% of all data for training. The more training data there is, the more complex patterns the model can learn, but the risk of overfitting also increases.
4. テストデータ Test Data Unseen data used to evaluate the performance of a trained model. It uses 20-30% of all data. Important: test data is never used for training and is used only for final evaluation. If performance is high on the test data, we can judge that generalization performance is high.
5. モデル Model A formula or program that represents a mapping from inputs to outputs. For linear regression, the formula $y = wx + b$; for a neural network, a multi-layer computation graph. Through training, the model's parameters (weights w, bias b) are optimized.
6. 予測 Prediction The value that a trained model outputs for new input data. For regression, a continuous value (e.g., a price of 35 million yen); for classification, a class (e.g., spam). Improving prediction accuracy is the main goal of machine learning.
7. 損失関数 Loss Function A function that quantifies the difference (error) between the predicted value and the correct answer. Mean squared error (MSE) is commonly used for regression, and cross-entropy for classification. The goal of learning is to minimize the loss function. A smaller loss means higher prediction accuracy.
8. 過学習 Overfitting A phenomenon in which the model fits the training data too closely and its predictive performance on unseen data declines. It occurs when training accuracy is high but test accuracy is low. Countermeasures: increase the amount of data, simplify the model, regularization, cross-validation, etc. It is one of the most important problems to watch for in machine learning.

Method Terms (7 Terms)

Term (Japanese) Term (English) Definition and Example
9. 回帰 Regression A task of predicting continuous values. Numeric predictions such as house prices, stock prices, and temperatures. Representative methods: linear regression, ridge regression, random forest. The output is a concrete numeric value such as "35 million yen" or "25.3°C."
10. 分類 Classification A task of predicting a category (class). Spam detection, disease diagnosis, image recognition, etc. There are binary classification (Yes/No) and multi-class classification (cat/dog/bird). The output is a label such as "spam," "benign," or "cat."
11. クラスタリング Clustering Unsupervised learning that groups data without ground-truth labels. Used for customer segmentation, document classification, image compression, etc. In K-means, you specify the number of groups K in advance, and the data is automatically split into K groups.
12. 交差検証 Cross-Validation A method that splits the data into K parts and repeats K times the operation of using one part for testing and the rest for training. K-fold cross-validation (K=5 or 10 is common) allows accurate evaluation of the model's generalization performance. It is especially effective when data is scarce.
13. ハイパーパラメータ Hyperparameter Adjustable settings that a human sets before training. Learning rate, tree depth, number of neurons, etc. Unlike model parameters (weights w, bias b), they are not updated during training. The optimal values are searched for with Grid Search or Random Search.
14. 正則化 Regularization A method that imposes a penalty on model complexity to prevent overfitting. L1 regularization (Lasso) pushes parameters toward zero, and L2 regularization (Ridge) suppresses them to small values. Regularization improves generalization performance.
15. アンサンブル Ensemble A method that combines the predictions of multiple models to improve accuracy. Bagging (Random Forest), boosting (XGBoost), stacking, etc. Following the principle "two heads are better than one," it often achieves higher accuracy than a single model.

Evaluation Terms (5 Terms)

Term (Japanese) Term (English) Definition and Example
16. 精度 Accuracy The proportion of all predictions that are correct. Formula: $\text{Accuracy} = \frac{\text{number correct}}{\text{total number of data}}$. If 85 out of 100 are correct, the accuracy is 85%. Simple, but it can be misleading when the data is imbalanced (spam 1% vs. not spam 99%).
17. 再現率 Recall The proportion of actual positive cases that were correctly detected. Formula: $\text{Recall} = \frac{\text{true positives}}{\text{true positives + false negatives}}$. Emphasized when "not missing sick people" is important in disease diagnosis. The higher the recall, the fewer the misses.
18. 適合率 Precision Of the cases the model predicted as positive, the proportion that were actually positive. Formula: $\text{Precision} = \frac{\text{true positives}}{\text{true positives + false positives}}$. Emphasized when "not misclassifying non-spam as spam" is important in spam detection. The higher the precision, the fewer the false detections.
19. F1スコア F1 Score The harmonic mean of precision and recall. Formula: $\text{F1} = 2 \times \frac{\text{Precision} \times \text{Recall}}{\text{Precision} + \text{Recall}}$. A metric that balances both; a good score is not obtained if only one of them is high. It is more useful than accuracy when the data is imbalanced.
20. AUC-ROC AUC-ROC The area under the ROC curve (horizontal axis: false positive rate, vertical axis: true positive rate). 0.5 means completely random, and 1.0 means perfect classification. It is a threshold-independent evaluation metric that indicates the overall performance of a classifier. An AUC of 0.9 or higher is evaluated as an excellent model.

2.3 Comparison of Major Frameworks

For implementing machine learning, it is important to choose a framework according to your purpose. Here we compare four major frameworks in detail.

2.3.1 scikit-learn

Characteristics: The most widely used general-purpose machine learning library in Python.

Strengths:

Weaknesses:

Where to use it:

2.3.2 PyTorch

Characteristics: A deep learning framework developed by Meta (formerly Facebook).

Strengths:

Weaknesses:

Where to use it:

2.3.3 TensorFlow / Keras

Characteristics: A production-oriented deep learning framework developed by Google (Keras is the high-level API).

Strengths:

Weaknesses:

Where to use it:

2.3.4 XGBoost / LightGBM

Characteristics: Fast libraries specialized in gradient boosting.

Strengths:

Weaknesses:

Where to use it:

Framework Comparison Table

Framework Strong Area Learning Difficulty Speed Production GPU Support
scikit-learn Classical ML, tabular data Easy Medium △ ×
PyTorch Deep learning, R&D Medium High (GPU) ○ ◎
TensorFlow/Keras Production, large-scale systems Medium–High High (GPU) ◎ ◎
XGBoost/LightGBM Tabular data, competitions Medium High (CPU) ○ △

Framework Selection Flowchart

graph TD A[What type of data
are you handling?] --> B{Tabular data
CSV/Excel} A --> C{Image, text, or audio} B --> D{Data size} D -->|Small–Medium
up to 100k rows| E[scikit-learn
beginner-friendly] D -->|Large
over 100k rows| F[XGBoost/LightGBM
high accuracy, fast] C --> G{What is the goal?} G -->|Research / experiments| H[PyTorch
flexibility-focused] G -->|Production deployment| I[TensorFlow/Keras
scalability-focused] style E fill:#e8f5e9 style F fill:#e8f5e9 style H fill:#f3e5f5 style I fill:#fff3e0

2.4 The Machine Learning Ecosystem

Machine learning projects consist of multiple steps, from data collection to production operation. It is important to understand the overall picture.

graph LR A[Data source
DB/API/files] --> B[Data collection
Scraping/ETL] B --> C[Data storage
Data Lake/DWH] C --> D[Preprocessing
Cleaning/Normalization] D --> E[Feature engineering
Transform/Select/Generate] E --> F[Model training
Learning/Tuning] F --> G[Evaluation
Accuracy validation] G --> H{Performance OK?} H -->|No| I[Hyperparameter tuning
Add features] I --> F H -->|Yes| J[Deploy
Production] J --> K[Monitoring
Performance monitoring] K --> L[Feedback
Add data] L --> B style A fill:#e3f2fd style F fill:#fff3e0 style J fill:#e8f5e9 style K fill:#ffebee

The role of each component:

2.5 The Machine Learning Workflow in Detail (7 Steps)

An actual machine learning project proceeds through the following seven steps. Understand the details, time estimates, and points to note for each step.

Step 0: Problem Formulation (The most important, and often overlooked)

Objective: Define the problem to be solved in ML terms.

What to do:

Time estimate: 1-2 weeks (including discussions with stakeholders)

Common failures:

Concrete example:

Business challenge: We want to reduce customer churn.

ML problem formulation:

Step 1: Data Collection

Objective: Gather the data needed for learning.

Data sources:

Guidelines for the amount of data needed:

Time estimate: 1-4 weeks

Points to note:

Step 2: Exploratory Data Analysis (EDA)

Objective: Understand the characteristics of the data and discover problems.

What to do:

Time estimate: A few days to 1 week

Tools: pandas, matplotlib, seaborn, pandas-profiling

Step 3: Preprocessing / Data Cleaning

Objective: Transform the data into a form that ML models can handle.

What to do:

1. Handling Missing Values

2. Handling Outliers

3. Scaling (Normalization / Standardization)

4. Encoding Categorical Variables

Time estimate: A few days to 1 week

Points to note:

Step 4: Feature Engineering

Objective: Create features that improve the model's prediction accuracy.

Methods:

Time estimate: A few days to 2 weeks

Concrete example:

In house price prediction, create new features from "size" and "building age":

Step 5: Model Selection and Training

Objective: Choose an appropriate algorithm and train it.

Strategy:

  1. Baseline model: The simplest model (linear regression, logistic regression)
  2. Comparison of multiple models: Decision tree, random forest, XGBoost, SVM
  3. Selecting the best model: Evaluate performance with cross-validation

Time estimate: A few hours to a few days

Tools: scikit-learn, XGBoost, LightGBM

Step 6: Evaluation and Hyperparameter Tuning

Objective: Maximize the model's performance.

Evaluation methods:

Tuning methods:

Time estimate: A few days to 1 week

Step 7: Model Interpretation and Validation

Objective: Confirm that the model is working correctly.

What to do:

Time estimate: A few days to 1 week

Time Estimate for the Entire Workflow

gantt title Timeline of a Machine Learning Project dateFormat YYYY-MM-DD section Preparation Problem formulation :a1, 2024-01-01, 14d section Data Data collection :a2, 2024-01-15, 21d EDA :a3, 2024-02-05, 7d Preprocessing :a4, 2024-02-12, 7d section Model Feature engineering :a5, 2024-02-19, 14d Model training :a6, 2024-03-04, 3d Evaluation & tuning :a7, 2024-03-07, 7d Validation :a8, 2024-03-14, 7d

Entire project: 2-3 months (small to medium scale), 6-12 months (large scale)

2.6 A Deep Dive into Features

A feature is an attribute of the data that serves as input to a machine learning model. By designing appropriate features, model accuracy can improve dramatically.

2.6.1 Types of Features

1. Numerical Features

Continuous values: Numbers that can be divided infinitely finely

Discrete values: Integer values

2. Categorical Features

Nominal: No ordering relationship

Ordinal: Has an ordering relationship

3. Text Features

Transformation methods:

4. Time Series Features

Derived features:

5. Image Features

Transformation methods:

2.6.2 Numericalizing Categorical Variables (Code Example)

"""
Two methods for converting categorical variables to numbers

Purpose: Machine learning models cannot handle categories directly, so numericalization is required
Audience: Beginners
Runtime: about 3 seconds
"""

import pandas as pd
from sklearn.preprocessing import LabelEncoder

# 1. Prepare sample data
df = pd.DataFrame({
    'color': ['red', 'blue', 'green', 'red', 'blue'],
    'size': ['S', 'M', 'L', 'M', 'S'],
    'price': [100, 150, 200, 120, 90]
})

print("Original data:")
print(df)
print("\n" + "="*50 + "\n")

# 2. Label Encoding
# Convert each category to an integer (red->0, blue->1, green->2)
le = LabelEncoder()
df['color_label'] = le.fit_transform(df['color'])

print("After Label Encoding:")
print(df[['color', 'color_label']])
print("\nNote: an ordering relationship is introduced (blue=1 > red=0), but in reality there is no ordering")
print("\n" + "="*50 + "\n")

# 3. One-Hot Encoding
# Convert each category into an independent binary column
df_onehot = pd.get_dummies(df, columns=['size'], prefix='size')

print("After One-Hot Encoding:")
print(df_onehot)
print("\nThe three columns size_S, size_M, size_L are created; only the matching column is 1, the others are 0")

# Expected output:
# Original data:
#   color size  price
# 0   red    S    100
# 1  blue    M    150
# 2 green    L    200
# 3   red    M    120
# 4  blue    S     90
#
# After Label Encoding:
#   color  color_label
# 0   red            2
# 1  blue            0
# 2 green            1
# 3   red            2
# 4  blue            0
#
# After One-Hot Encoding:
#   color  price  color_label  size_L  size_M  size_S
# 0   red    100            2       0       0       1
# 1  blue    150            0       0       1       0
# 2 green    200            1       1       0       0
# 3   red    120            2       0       1       0
# 4  blue     90            0       0       0       1

2.6.3 Scaling Numerical Features (Code Example)

"""
Scaling numerical features (standardization / normalization)

Purpose: Unify features of different scales to stabilize model training
Audience: Beginners
Runtime: about 3 seconds
"""

import numpy as np
from sklearn.preprocessing import StandardScaler, MinMaxScaler

# 1. Sample data (housing data)
data = np.array([
    [50, 10, 500],   # size(m^2), building age, distance to station(m)
    [60, 15, 300],
    [70, 5, 800],
    [80, 20, 200],
    [90, 8, 600]
])

print("Original data:")
print("  size(m^2)  age  distance(m)")
print(data)
print("\nNote: the scales are completely different (size:50-90, age:5-20, distance:200-800)")
print("\n" + "="*50 + "\n")

# 2. Standardization
# Transform to mean 0, standard deviation 1
scaler_std = StandardScaler()
data_std = scaler_std.fit_transform(data)

print("After standardization (mean 0, std 1):")
print(data_std)
print("\nMean of each column:", data_std.mean(axis=0))
print("Std of each column:", data_std.std(axis=0))
print("\n" + "="*50 + "\n")

# 3. Normalization
# Transform to the 0-1 range
scaler_norm = MinMaxScaler()
data_norm = scaler_norm.fit_transform(data)

print("After normalization (0-1 range):")
print(data_norm)
print("\nMin of each column:", data_norm.min(axis=0))
print("Max of each column:", data_norm.max(axis=0))

# Expected output:
# Original data:
#   size(m^2)  age  distance(m)
# [[ 50  10 500]
#  [ 60  15 300]
#  [ 70   5 800]
#  [ 80  20 200]
#  [ 90   8 600]]
#
# After standardization (mean 0, std 1):
# [[-1.41 -0.39  0.00]
#  [-0.71  0.78 -1.00]
#  [ 0.00 -1.17  1.50]
#  [ 0.71  1.56 -1.50]
#  [ 1.41 -0.78  0.50]]
#
# After normalization (0-1 range):
# [[0.00 0.33 0.50]
#  [0.25 0.67 0.17]
#  [0.50 0.00 1.00]
#  [0.75 1.00 0.00]
#  [1.00 0.20 0.67]]

2.6.4 Feature Selection and Importance (Code Example)

"""
Feature selection and importance evaluation

Purpose: Determine which features are important for prediction and remove unnecessary features
Audience: Beginner to intermediate
Runtime: about 5 seconds
"""

import pandas as pd
from sklearn.datasets import load_diabetes
from sklearn.ensemble import RandomForestRegressor
import matplotlib.pyplot as plt

# 1. Load sample data (diabetes dataset)
diabetes = load_diabetes()
X = pd.DataFrame(diabetes.data, columns=diabetes.feature_names)
y = diabetes.target

print("Dataset info:")
print(f"Number of samples: {X.shape[0]}, number of features: {X.shape[1]}")
print(f"Features: {list(X.columns)}")
print("\n" + "="*50 + "\n")

# 2. Compute feature importance with a random forest
model = RandomForestRegressor(n_estimators=100, random_state=42)
model.fit(X, y)

# 3. Retrieve feature importances
importances = pd.DataFrame({
    'feature': X.columns,
    'importance': model.feature_importances_
}).sort_values('importance', ascending=False)

print("Feature importance ranking:")
print(importances)
print("\n" + "="*50 + "\n")

# 4. Compute correlation coefficients
correlations = X.corrwith(pd.Series(y)).abs().sort_values(ascending=False)

print("Correlation coefficients with the target variable (absolute value):")
print(correlations)

# Expected output:
# Dataset info:
# Number of samples: 442, number of features: 10
# Features: ['age', 'sex', 'bmi', 'bp', 's1', 's2', 's3', 's4', 's5', 's6']
#
# Feature importance ranking:
#    feature  importance
# 2      bmi    0.289345
# 8       s5    0.201928
# 4       s1    0.127654
# ...
#
# Correlation coefficients with the target variable (absolute value):
# bmi    0.586450
# s5     0.565883
# bp     0.441484
# ...

2.6.5 A Practical Example of Feature Engineering (Code Example)

"""
Feature engineering: creating new features

Purpose: Create more predictive derived features from existing features
Audience: Intermediate
Runtime: about 3 seconds
"""

import pandas as pd
import numpy as np

# 1. Sample data (house price prediction)
df = pd.DataFrame({
    'area': [50, 60, 70, 80, 90],
    'building_age': [10, 15, 5, 20, 8],
    'station_distance': [500, 300, 800, 200, 600],
    'rooms': [2, 3, 3, 4, 2],
    'price': [3000, 3500, 4200, 4500, 3800]  # in 10k yen
})

print("Original features:")
print(df)
print("\n" + "="*50 + "\n")

# 2. Create derived features

# (1) Interaction term: product of area and building age
df['area_x_building_age'] = df['area'] * df['building_age']

# (2) Ratio: price per unit area
df['price_per_area'] = df['price'] / df['area']

# (3) Polynomial: square of building age (aging deterioration is nonlinear)
df['building_age_sq'] = df['building_age'] ** 2

# (4) Categorization: convert station distance to "near/far"
df['near_station_flag'] = (df['station_distance'] < 400).astype(int)

# (5) Domain knowledge: area per room
df['area_per_room'] = df['area'] / df['rooms']

# (6) Log transform: log of station distance (reduce the impact of long distances)
df['log_station_distance'] = np.log1p(df['station_distance'])

print("After feature engineering:")
print(df)
print("\n" + "="*50 + "\n")

print("Statistics of the new features:")
print(df[['area_x_building_age', 'price_per_area', 'near_station_flag', 'area_per_room']].describe())

# Expected output:
# Original features:
#    area  building_age  station_distance  rooms  price
# 0    50            10               500      2   3000
# 1    60            15               300      3   3500
# 2    70             5               800      3   4200
# 3    80            20               200      4   4500
# 4    90             8               600      2   3800
#
# After feature engineering:
#    area  building_age  station_distance  rooms  price  area_x_building_age  price_per_area  building_age_sq  near_station_flag  area_per_room  log_station_distance
# 0    50            10               500      2   3000                  500            60.0              100                  0           25.0                 6.215
# 1    60            15               300      3   3500                  900            58.3              225                  1           20.0                 5.704
# 2    70             5               800      3   4200                  350            60.0               25                  0           23.3                 6.685
# 3    80            20               200      4   4500                 1600            56.2              400                  1           20.0                 5.298
# 4    90             8               600      2   3800                  720            42.2               64                  0           45.0                 6.397

Chapter Summary

In this chapter, you systematically learned the fundamentals of machine learning. You acquired the following:

Knowledge and Skills Acquired

Bridge to the Next Chapter

In Chapter 2, you learned the theory and concepts of machine learning. In the next Chapter 3, you will actually write Python code and implement six machine learning models. Starting from setting up the environment, you will experience data preprocessing, model training, and evaluation hands-on.

What you will learn in Chapter 3:

Exercises

Exercise 1 (Difficulty: easy)

Explain the difference between supervised learning and unsupervised learning, with concrete examples.

Hint

Focus on the presence or absence of "correct labels." Supervised learning requires pairs of inputs and correct answers, whereas unsupervised learning discovers patterns without correct answers.

Sample Answer

Supervised Learning

Definition: A method that learns from labeled data (pairs of inputs and correct answers).

Concrete examples:

Unsupervised Learning

Definition: A method that discovers hidden patterns or structure from unlabeled data.

Concrete examples:

Main Differences

Item Supervised Learning Unsupervised Learning
Data Labeled Unlabeled
Objective Prediction / classification Pattern discovery
Evaluation Accuracy, F1 score, etc. Silhouette coefficient, etc.

Exercise 2 (Difficulty: easy)

Explain what overfitting is and why it is a problem. Also, list three methods to prevent overfitting.

Hint

Focus on the performance gap between the training data and the test data. If training accuracy is high but test accuracy is low, overfitting is likely.

Sample Answer

What is Overfitting?

Definition: A phenomenon in which the model fits the training data too closely and its predictive performance on unseen data declines.

Symptoms:

Causes:

Why it is a problem:

Methods to Prevent Overfitting

  1. Increase the amount of data: Increasing the training data lets the model learn general patterns
  2. Regularization: Constrain parameters with L1 regularization (Lasso) or L2 regularization (Ridge)
  3. Cross-validation: Accurately evaluate generalization performance with K-fold cross-validation
  4. Simplify the model: Limit the depth of decision trees, reduce the number of layers in a neural network
  5. Dropout: Randomly disable some nodes in a neural network
  6. Early Stopping: Stop training when the validation error begins to increase

Exercise 3 (Difficulty: medium)

You want to predict customer purchases on an e-commerce site. For the following two problem settings, determine whether each is a regression problem or a classification problem, and answer with the reasons.

Hint

If the output is a continuous value (a number), it is regression; if it is a category (Yes/No, high/medium/low, etc.), it is classification.

Sample Answer

(A) You want to predict the amount a customer will spend next month

Answer: Regression problem

Reason:

Applicable models:

(B) You want to predict whether a customer will make a purchase within one month

Answer: Classification problem

Reason:

Applicable models:

Business Perspective

Exercise 4 (Difficulty: medium)

In feature engineering, explain two methods for numericalizing the categorical variable "prefecture." Also state the advantages and disadvantages of each.

Hint

The representative methods for numericalizing categorical variables are Label Encoding and One-Hot Encoding. Focus on the presence or absence of an ordering relationship.

Sample Answer

Method 1: Label Encoding

Method: Assign a unique integer to each prefecture

Tokyo    → 0
Osaka    → 1
Fukuoka  → 2
Hokkaido → 3
...

Advantages:

Disadvantages:

Method 2: One-Hot Encoding

Method: Convert each prefecture into an independent binary column

Tokyo    → [1, 0, 0, 0, ...] (Tokyo column is 1, others are 0)
Osaka    → [0, 1, 0, 0, ...] (Osaka column is 1, others are 0)
Fukuoka  → [0, 0, 1, 0, ...] (Fukuoka column is 1, others are 0)
Hokkaido → [0, 0, 0, 1, ...] (Hokkaido column is 1, others are 0)

Advantages:

Disadvantages:

Usage Guide

Condition Recommended Method
Few categories (<100) One-Hot Encoding
Many categories (>100) Label Encoding + tree-based model
Using a linear model One-Hot Encoding required
Using a tree-based model Either works (Label Encoding is more efficient)

Recommendation for Prefectures

Recommended: One-Hot Encoding

References

  1. Mitchell, T. M. (1997). Machine Learning. McGraw-Hill. ISBN: 0070428077
  2. Bishop, C. M. (2006). Pattern Recognition and Machine Learning. Springer. ISBN: 0387310738
  3. Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep Learning. MIT Press. https://www.deeplearningbook.org/
  4. scikit-learn Documentation. (2024). https://scikit-learn.org/stable/
  5. Koichi Kato (2018). Machine Learning at Work. O'Reilly Japan. ISBN: 4873118255
  6. Sebastian Raschka (2019). Python Machine Learning (3rd ed.). Packt Publishing. ISBN: 1789955750
  7. Aurélien Géron (2019). Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow (2nd ed.). O'Reilly Media. ISBN: 1492032646
← Chapter 1: Why Machine Learning Now Series Contents Chapter 3: Python Hands-on →

Disclaimer