Chapter 4: Overview of Machine Learning

Welcome to the world of machine learning - the foundation of AI that learns from data

Introduction

Machine Learning is a technology that automatically learns patterns from data to make predictions and decisions. Using the knowledge of Python, NumPy, and Pandas you have acquired so far, it is finally time to step into the world of machine learning.

In this chapter, you will learn the following:

• What machine learning is - definitions and applications
• Supervised learning (regression and classification)
• Unsupervised learning (clustering and dimensionality reduction)
• Splitting data into training and test sets
• The basics of using scikit-learn
• Implementing practical machine learning models

Definition of Machine Learning
"The field of study that gives computers the ability to learn without being explicitly programmed." - Arthur Samuel (1959)

1. What Is Machine Learning

1.1 Definition of Machine Learning

Machine learning is a technology that automatically discovers patterns from large amounts of data and uses those patterns to make predictions and decisions about new data.

Traditional Programming vs. Machine Learning

1.2 Applications of Machine Learning

• Image recognition: Face recognition, object detection, medical image diagnosis
• Natural language processing: Translation, chatbots, sentiment analysis
• Speech recognition: Voice assistants (Siri, Alexa)
• Recommendation systems: Recommending videos, music, and products (Netflix, Amazon)
• Finance: Stock price prediction, fraud detection, credit scoring
• Healthcare: Disease diagnosis, drug discovery, genomic analysis
• Autonomous driving: Obstacle detection, route planning

2. Types of Machine Learning

2.1 Supervised Learning

The model learns from labeled data and makes predictions about new data.

• Classification: Predicting a category (e.g., spam or not, disease diagnosis)
• Regression: Predicting a numerical value (e.g., house prices, sales forecasting)

2.2 Unsupervised Learning

The model discovers hidden patterns and structures in unlabeled data.

• Clustering: Grouping data into clusters (e.g., customer segmentation)
• Dimensionality Reduction: Reducing the number of features in the data (e.g., data visualization)

2.3 Reinforcement Learning

The model learns actions that maximize a reward through interaction with an environment (e.g., game AI, robot control).

3. Fundamentals of Supervised Learning

3.1 The Difference Between Classification and Regression

Example 1: The Difference Between Classification and Regression

import numpy as np
import pandas as pd

# Classification example: Iris species classification
# Input: petal length, width -> Output: species (Setosa, Versicolor, Virginica)
classification_data = {
    'petal_length': [1.4, 4.7, 5.1],
    'petal_width': [0.2, 1.4, 2.3],
    'species': ['Setosa', 'Versicolor', 'Virginica']  # Category
}
print("Classification data:")
print(pd.DataFrame(classification_data))

# Regression example: House price prediction
# Input: area, number of rooms -> Output: price (numerical)
regression_data = {
    'area_sqm': [50, 70, 90],
    'rooms': [2, 3, 4],
    'price_10k_yen': [3000, 4200, 5500]  # Continuous value
}
print("\nRegression data:")
print(pd.DataFrame(regression_data))

4. Training Data and Test Data

4.1 Why Is Splitting Necessary?

A machine learning model is trained on the training data and evaluated on the test data. If you train and evaluate on the same data, you cannot determine the model's true performance.

• Training Data: Used to train the model (typically 70-80%)
• Test Data: Used to evaluate the model (typically 20-30%)

Example 2: Splitting the Data

from sklearn.model_selection import train_test_split
import numpy as np

# Sample data
X = np.array([[1, 2], [3, 4], [5, 6], [7, 8], [9, 10]])
y = np.array([0, 1, 0, 1, 0])

print("Original data:")
print("X (features):")
print(X)
print("y (labels):", y)

# Split into training and test data (80% training, 20% test)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

print("\nTraining data:")
print("X_train:")
print(X_train)
print("y_train:", y_train)

print("\nTest data:")
print("X_test:")
print(X_test)
print("y_test:", y_test)

print("\nData sizes:")
print(f"Training data: {len(X_train)} samples, Test data: {len(X_test)} samples")

5. The Basics of scikit-learn

scikit-learn is the most widely used machine learning library in Python.

5.1 The Basic scikit-learn Workflow

1. Prepare the data
2. Select and create a model
3. Train the model (fit)
4. Make predictions (predict)
5. Evaluate (score)

Example 3: The Basics of Using scikit-learn

from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.neighbors import KNeighborsClassifier
from sklearn.metrics import accuracy_score

# 1. Prepare the data (Iris dataset)
iris = load_iris()
X = iris.data  # Features (petal length, width, etc.)
y = iris.target  # Labels (species)

print("Data shapes:")
print("X:", X.shape)  # (150, 4) = 150 samples, 4 features
print("y:", y.shape)  # (150,) = 150 labels

# 2. Split into training and test data
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.3, random_state=42
)

print("\nTraining data: {} samples".format(len(X_train)))
print("Test data: {} samples".format(len(X_test)))

# 3. Create the model (k-nearest neighbors)
model = KNeighborsClassifier(n_neighbors=3)

# 4. Train the model
model.fit(X_train, y_train)
print("\nModel training complete")

# 5. Make predictions
y_pred = model.predict(X_test)
print("\nPredictions (first 10):", y_pred[:10])
print("True labels (first 10):", y_test[:10])

# 6. Evaluate
accuracy = accuracy_score(y_test, y_pred)
print("\nAccuracy: {:.2f}%".format(accuracy * 100))

# Or
score = model.score(X_test, y_test)
print("Score: {:.2f}%".format(score * 100))

6. Implementing a Regression Problem

6.1 Linear Regression

Linear regression is a model that approximates the relationship between inputs and outputs with a straight line.

Equation: \( y = w_1 x_1 + w_2 x_2 + ... + w_n x_n + b \)

Example 4: Implementing Linear Regression

from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error, r2_score
import numpy as np
import matplotlib.pyplot as plt

# Create sample data (house price prediction)
np.random.seed(42)
n_samples = 100

# Feature: area (50-150 sqm)
area = np.random.uniform(50, 150, n_samples)

# Target variable: price = 30 * area + noise
price = 30 * area + np.random.normal(0, 200, n_samples)

# Reshape data into a 2D array
X = area.reshape(-1, 1)
y = price

print("Data samples:")
for i in range(5):
    print(f"Area: {X[i][0]:.1f} sqm -> Price: {y[i]:.0f} (10k yen)")

# Split into training and test data
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

# Create and train the model
model = LinearRegression()
model.fit(X_train, y_train)

print("\nLearned parameters:")
print(f"Slope (coefficient): {model.coef_[0]:.2f}")
print(f"Intercept: {model.intercept_:.2f}")

# Make predictions
y_pred = model.predict(X_test)

# Evaluate
mse = mean_squared_error(y_test, y_pred)
rmse = np.sqrt(mse)
r2 = r2_score(y_test, y_pred)

print("\nEvaluation metrics:")
print(f"Mean squared error (MSE): {mse:.2f}")
print(f"Root mean squared error (RMSE): {rmse:.2f}")
print(f"Coefficient of determination (R^2): {r2:.3f}")

# Visualization
plt.figure(figsize=(10, 6))
plt.scatter(X_test, y_test, alpha=0.5, label='Actual price')
plt.plot(X_test, y_pred, color='red', linewidth=2, label='Prediction')
plt.xlabel('Area (sqm)')
plt.ylabel('Price (10k yen)')
plt.title('House Price Prediction (Linear Regression)')
plt.legend()
plt.grid(True, alpha=0.3)
# plt.savefig('linear_regression.png')
# plt.show()

print("\nGraph created.")

7. Implementing a Classification Problem

7.1 Logistic Regression

Logistic regression is a fundamental method for performing binary classification (0 or 1).

Example 5: Implementing Logistic Regression

from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score, confusion_matrix, classification_report
from sklearn.datasets import load_iris
import numpy as np

# Prepare the data (Iris dataset, using only 2 classes)
iris = load_iris()
X = iris.data[:100]  # First 100 samples (2 classes)
y = iris.target[:100]

print("Data shapes:")
print("X:", X.shape)
print("y:", y.shape)
print("Classes:", np.unique(y))

# Split into training and test data
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.3, random_state=42
)

# Create and train the model
model = LogisticRegression(max_iter=200)
model.fit(X_train, y_train)

# Make predictions
y_pred = model.predict(X_test)

# Prediction probabilities
y_pred_proba = model.predict_proba(X_test)

print("\nPredictions (first 5):")
for i in range(5):
    print(f"Prediction: {y_pred[i]}, True: {y_test[i]}, Probability: {y_pred_proba[i]}")

# Evaluate
accuracy = accuracy_score(y_test, y_pred)
print(f"\nAccuracy: {accuracy:.2%}")

# Confusion matrix
cm = confusion_matrix(y_test, y_pred)
print("\nConfusion matrix:")
print(cm)

# Detailed report
print("\nClassification report:")
print(classification_report(y_test, y_pred, target_names=['Class 0', 'Class 1']))

8. Practical Example: Multiclass Classification with the Iris Dataset

Example 6: A Complete Implementation of 3-Class Classification

from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier
from sklearn.metrics import accuracy_score, confusion_matrix
import pandas as pd
import numpy as np

# Load the data
iris = load_iris()
X = iris.data
y = iris.target

print("=== Iris Dataset ===")
print("Feature names:", iris.feature_names)
print("Class names:", iris.target_names)
print("Data shape:", X.shape)

# Convert the data to a DataFrame
df = pd.DataFrame(X, columns=iris.feature_names)
df['species'] = y
df['species_name'] = df['species'].map({
    0: 'setosa', 1: 'versicolor', 2: 'virginica'
})

print("\nFirst 5 rows of the data:")
print(df.head())

print("\nNumber of samples per class:")
print(df['species_name'].value_counts())

# Statistics
print("\nFeature statistics:")
print(df.describe())

# Split into training and test data
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.3, random_state=42, stratify=y
)

print(f"\nTraining data: {len(X_train)} samples")
print(f"Test data: {len(X_test)} samples")

# Create the model (decision tree)
model = DecisionTreeClassifier(max_depth=3, random_state=42)

# Train
model.fit(X_train, y_train)

# Make predictions
y_pred = model.predict(X_test)

# Evaluate
accuracy = accuracy_score(y_test, y_pred)
print(f"\nAccuracy: {accuracy:.2%}")

# Confusion matrix
cm = confusion_matrix(y_test, y_pred)
print("\nConfusion matrix:")
cm_df = pd.DataFrame(
    cm,
    index=['setosa', 'versicolor', 'virginica'],
    columns=['setosa', 'versicolor', 'virginica']
)
print(cm_df)

# Accuracy per class
print("\nResults per class:")
for i, name in enumerate(iris.target_names):
    class_mask = (y_test == i)
    class_accuracy = accuracy_score(y_test[class_mask], y_pred[class_mask])
    print(f"{name}: {class_accuracy:.2%}")

# Feature importances
print("\nFeature importances:")
feature_importance = pd.DataFrame({
    'feature': iris.feature_names,
    'importance': model.feature_importances_
}).sort_values('importance', ascending=False)
print(feature_importance)

9. Fundamentals of Unsupervised Learning

9.1 Clustering

Clustering is a method that automatically groups similar data together.

Example 7: K-Means Clustering

from sklearn.cluster import KMeans
from sklearn.datasets import load_iris
import matplotlib.pyplot as plt
import numpy as np

# Prepare the data
iris = load_iris()
X = iris.data[:, :2]  # Use only the first 2 features (for visualization)

# K-Means clustering (3 clusters)
kmeans = KMeans(n_clusters=3, random_state=42)
clusters = kmeans.fit_predict(X)

print("Clustering results:")
print("Cluster each sample belongs to:", clusters)

# Number of samples per cluster
unique, counts = np.unique(clusters, return_counts=True)
print("\nNumber of samples per cluster:")
for cluster, count in zip(unique, counts):
    print(f"Cluster {cluster}: {count} samples")

# Cluster centers
print("\nCluster centers:")
print(kmeans.cluster_centers_)

# Visualization
plt.figure(figsize=(10, 6))
scatter = plt.scatter(X[:, 0], X[:, 1], c=clusters, cmap='viridis', alpha=0.6)
plt.scatter(kmeans.cluster_centers_[:, 0],
           kmeans.cluster_centers_[:, 1],
           s=300, c='red', marker='X', edgecolors='black',
           label='Center')
plt.xlabel(iris.feature_names[0])
plt.ylabel(iris.feature_names[1])
plt.title('K-Means Clustering')
plt.colorbar(scatter)
plt.legend()
plt.grid(True, alpha=0.3)
# plt.savefig('kmeans_clustering.png')
# plt.show()

print("\nClustering complete.")

9.2 Dimensionality Reduction

Dimensionality reduction is a method that reduces the number of features in the data to make visualization and processing easier.

Example 8: PCA (Principal Component Analysis)

from sklearn.decomposition import PCA
from sklearn.datasets import load_iris
import matplotlib.pyplot as plt

# Prepare the data
iris = load_iris()
X = iris.data  # 4 dimensions
y = iris.target

print("Original data shape:", X.shape)  # (150, 4)

# Reduce to 2 dimensions with PCA
pca = PCA(n_components=2)
X_pca = pca.fit_transform(X)

print("Shape after reduction:", X_pca.shape)  # (150, 2)

# Explained variance ratio
print("\nExplained variance ratio of each principal component:")
print(pca.explained_variance_ratio_)
print(f"Cumulative explained variance ratio: {sum(pca.explained_variance_ratio_):.2%}")

# Visualization
plt.figure(figsize=(10, 6))
colors = ['red', 'green', 'blue']
for i, color in enumerate(colors):
    mask = (y == i)
    plt.scatter(X_pca[mask, 0], X_pca[mask, 1],
               c=color, label=iris.target_names[i], alpha=0.6)

plt.xlabel(f'1st Principal Component (variance ratio: {pca.explained_variance_ratio_[0]:.2%})')
plt.ylabel(f'2nd Principal Component (variance ratio: {pca.explained_variance_ratio_[1]:.2%})')
plt.title('Dimensionality Reduction via PCA')
plt.legend()
plt.grid(True, alpha=0.3)
# plt.savefig('pca_visualization.png')
# plt.show()

print("\nPCA complete.")

10. The Flow of a Machine Learning Project

Example 9: A Complete Machine Learning Workflow

from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.preprocessing import StandardScaler
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score, classification_report
import pandas as pd
import numpy as np

print("=== Complete Workflow of a Machine Learning Project ===\n")

# 1. Data collection
print("1. Data collection")
iris = load_iris()
X, y = iris.data, iris.target
print(f"Data size: {X.shape}")

# 2. Data exploration
print("\n2. Data exploration")
df = pd.DataFrame(X, columns=iris.feature_names)
df['target'] = y
print(df.describe())
print("\nClass distribution:")
print(df['target'].value_counts())

# 3. Data preprocessing
print("\n3. Data preprocessing")
# Split into training and test data
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)

# Standardization
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)
print("Standardization complete")

# 4. Model selection
print("\n4. Model selection and training")
model = RandomForestClassifier(n_estimators=100, random_state=42)
model.fit(X_train_scaled, y_train)
print("Training complete")

# 5. Cross-validation
print("\n5. Cross-validation (5-fold)")
cv_scores = cross_val_score(model, X_train_scaled, y_train, cv=5)
print(f"Score for each fold: {cv_scores}")
print(f"Mean score: {cv_scores.mean():.3f} ± {cv_scores.std():.3f}")

# 6. Evaluate on the test data
print("\n6. Evaluate on the test data")
y_pred = model.predict(X_test_scaled)
accuracy = accuracy_score(y_test, y_pred)
print(f"Test accuracy: {accuracy:.2%}")

print("\nDetailed report:")
print(classification_report(y_test, y_pred,
                          target_names=iris.target_names))

# 7. Feature importances
print("7. Feature importances")
feature_importance = pd.DataFrame({
    'feature': iris.feature_names,
    'importance': model.feature_importances_
}).sort_values('importance', ascending=False)
print(feature_importance)

print("\n=== Project complete ===")

Example 10: Overfitting and Generalization Performance

from sklearn.model_selection import learning_curve
from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import load_iris
import numpy as np
import matplotlib.pyplot as plt

# Prepare the data
iris = load_iris()
X, y = iris.data, iris.target

# Compare decision trees of different depths
depths = [1, 2, 3, 5, 10, 20]

results = []
for depth in depths:
    model = DecisionTreeClassifier(max_depth=depth, random_state=42)

    # Compute the learning curve
    train_sizes, train_scores, val_scores = learning_curve(
        model, X, y, cv=5, train_sizes=np.linspace(0.1, 1.0, 10)
    )

    train_mean = train_scores.mean(axis=1)[-1]
    val_mean = val_scores.mean(axis=1)[-1]

    results.append({
        'depth': depth,
        'train_score': train_mean,
        'val_score': val_mean,
        'overfitting': train_mean - val_mean
    })

# Display the results
print("Relationship between depth and overfitting:")
print("=" * 60)
for r in results:
    print(f"Depth {r['depth']:2d}: train={r['train_score']:.3f}, "
          f"validation={r['val_score']:.3f}, "
          f"overfitting={r['overfitting']:.3f}")

# Find the optimal depth
best = max(results, key=lambda x: x['val_score'])
print(f"\nOptimal depth: {best['depth']}")
print(f"Validation score: {best['val_score']:.3f}")

# Visualization
depths_list = [r['depth'] for r in results]
train_scores = [r['train_score'] for r in results]
val_scores = [r['val_score'] for r in results]

plt.figure(figsize=(10, 6))
plt.plot(depths_list, train_scores, 'o-', label='Training score')
plt.plot(depths_list, val_scores, 's-', label='Validation score')
plt.xlabel('Decision tree depth')
plt.ylabel('Score')
plt.title('Relationship Between Model Complexity and Performance')
plt.legend()
plt.grid(True, alpha=0.3)
# plt.savefig('overfitting_analysis.png')
# plt.show()

print("\nOverfitting analysis complete.")

Summary

In this chapter, you learned the fundamentals of machine learning:

• ✅ Definition of machine learning: A technology that learns from data to make predictions
• ✅ Types: Supervised learning, unsupervised learning, reinforcement learning
• ✅ Supervised learning: Classification (predicting categories) and regression (predicting numbers)
• ✅ Data splitting: Training data and test data
• ✅ scikit-learn: The basic fit, predict, score workflow
• ✅ Implementation: Linear regression, logistic regression, decision trees
• ✅ Unsupervised learning: Clustering, dimensionality reduction
• ✅ Evaluation: Accuracy, confusion matrix, overfitting

Next steps: Now that you have completed this series, you are ready to move on to more specialized machine learning series (Introduction to Supervised Learning, Introduction to Neural Networks)!

Exercises

# Solution example
from sklearn.model_selection import train_test_split
import numpy as np

# Create data (imbalanced classes)
X = np.arange(100).reshape(-1, 1)
y = np.array([0]*30 + [1]*70)  # Class 0: 30 samples, Class 1: 70 samples

print("Original class distribution:", np.bincount(y))

# Without stratify
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)
print("\nWithout stratify:")
print("Training:", np.bincount(y_train))
print("Test:", np.bincount(y_test))

# With stratify
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)
print("\nWith stratify:")
print("Training:", np.bincount(y_train))
print("Test:", np.bincount(y_test))

# Solution example
from sklearn.linear_model import LinearRegression
from sklearn.preprocessing import PolynomialFeatures
from sklearn.metrics import mean_squared_error, r2_score
import numpy as np

# Generate nonlinear data
np.random.seed(42)
X = np.linspace(0, 10, 100).reshape(-1, 1)
y = 0.5 * X**2 + X + 2 + np.random.normal(0, 5, (100, 1)).flatten()

# Linear regression
model_linear = LinearRegression()
model_linear.fit(X, y)
y_pred_linear = model_linear.predict(X)

# Polynomial regression (degree 2)
poly = PolynomialFeatures(degree=2)
X_poly = poly.fit_transform(X)
model_poly = LinearRegression()
model_poly.fit(X_poly, y)
y_pred_poly = model_poly.predict(X_poly)

# Evaluate
print("Linear regression:")
print(f"RMSE: {np.sqrt(mean_squared_error(y, y_pred_linear)):.2f}")
print(f"R^2: {r2_score(y, y_pred_linear):.3f}")

print("\nPolynomial regression:")
print(f"RMSE: {np.sqrt(mean_squared_error(y, y_pred_poly)):.2f}")
print(f"R^2: {r2_score(y, y_pred_poly):.3f}")

# Solution example
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.neighbors import KNeighborsClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.metrics import accuracy_score, confusion_matrix

# Prepare the data
iris = load_iris()
X, y = iris.data, iris.target
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.3, random_state=42
)

# k-NN
knn = KNeighborsClassifier(n_neighbors=3)
knn.fit(X_train, y_train)
y_pred_knn = knn.predict(X_test)

print("k-NN (k=3):")
print(f"Accuracy: {accuracy_score(y_test, y_pred_knn):.2%}")
print("Confusion matrix:")
print(confusion_matrix(y_test, y_pred_knn))

# Decision tree
tree = DecisionTreeClassifier(max_depth=3, random_state=42)
tree.fit(X_train, y_train)
y_pred_tree = tree.predict(X_test)

print("\nDecision tree:")
print(f"Accuracy: {accuracy_score(y_test, y_pred_tree):.2%}")
print("Confusion matrix:")
print(confusion_matrix(y_test, y_pred_tree))

# Solution example
from sklearn.cluster import KMeans
from sklearn.datasets import load_iris
import matplotlib.pyplot as plt

# Prepare the data
iris = load_iris()
X = iris.data

# Compute inertia while varying the number of clusters
inertias = []
k_range = range(2, 11)

for k in k_range:
    kmeans = KMeans(n_clusters=k, random_state=42)
    kmeans.fit(X)
    inertias.append(kmeans.inertia_)

# Visualize with the elbow method
plt.figure(figsize=(10, 6))
plt.plot(k_range, inertias, 'bo-')
plt.xlabel('Number of clusters k')
plt.ylabel('Inertia')
plt.title('Determining the Number of Clusters with the Elbow Method')
plt.grid(True, alpha=0.3)
# plt.savefig('elbow_method.png')
# plt.show()

print("Inertia values:")
for k, inertia in zip(k_range, inertias):
    print(f"k={k}: {inertia:.2f}")

# Solution example
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC
from sklearn.metrics import classification_report, confusion_matrix
import numpy as np

# (1) Data splitting
iris = load_iris()
X, y = iris.data, iris.target
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)

print("(1) Data splitting:")
print(f"Training: {X_train.shape}, Test: {X_test.shape}")

# (2) Standardization
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

print("\n(2) Standardization:")
print(f"Training data mean: {X_train_scaled.mean(axis=0)}")
print(f"Training data standard deviation: {X_train_scaled.std(axis=0)}")

# (3) Model training
model = SVC(kernel='rbf', random_state=42)
model.fit(X_train_scaled, y_train)

print("\n(3) Model training complete")

# (4) Cross-validation
cv_scores = cross_val_score(model, X_train_scaled, y_train, cv=5)
print("\n(4) Cross-validation:")
print(f"Score for each fold: {cv_scores}")
print(f"Mean: {cv_scores.mean():.3f} ± {cv_scores.std():.3f}")

# (5) Test evaluation
y_pred = model.predict(X_test_scaled)

print("\n(5) Test evaluation:")
print(f"Test accuracy: {model.score(X_test_scaled, y_test):.2%}")
print("\nConfusion matrix:")
print(confusion_matrix(y_test, y_pred))
print("\nDetailed report:")
print(classification_report(y_test, y_pred,
                          target_names=iris.target_names))