Chapter 2: PI Fundamentals - Concepts, Methods, and Ecosystem

We organize the characteristics of predominantly time-series data such as temperature, pressure, and flow rate, and cover the basics of preprocessing and feature design. We also survey how to handle anomaly detection and seasonality.

💡 Supplement: Deciding how to handle missing values and outliers first stabilizes the downstream stages. Lag features and moving averages are the basic first techniques to try.

Learning Objectives

By reading this chapter, you will be able to: - Explain the definition of PI and how it differs from related fields (quality engineering, design of experiments, etc.) - Understand the major data types in chemical processes (time series, batch, quality data) - Explain the 5-step basic PI workflow (from problem formulation to implementation) in detail - Understand the types and importance of process descriptors (physicochemical parameters, equipment characteristics) - Correctly use the 20 technical terms that appear frequently in the PI domain

2.1 What Is PI: Definition and Related Fields

2.1.1 Origin and History of Process Informatics

The term Process Informatics (PI) began to be used in the early 2000s. It gained particular attention in the context of Industry 4.0 (the Fourth Industrial Revolution, proposed in Germany in 2011) [1].

The four pillars of Industry 4.0: 1. IoT (Internet of Things): Data collection via sensor networks 2. Big Data: Accumulation and management of large volumes of data 3. AI/Machine Learning: Knowledge extraction and prediction from data 4. Cyber-Physical Systems: Integration of the digital space and the physical world

PI can be described as the application of these technologies to the chemical process industry.

2.1.2 Definition

Process Informatics (PI) is an academic field that fuses chemical process engineering with data science. It is a set of methods that leverage large volumes of process data and information-science techniques such as machine learning to achieve process optimization, quality control, anomaly detection, and predictive maintenance.

A concise definition:

"The science of optimizing and making chemical processes smart with the power of data and AI."

Core elements: 1. Data: Process data, quality data, operational data 2. Modeling: Machine learning, statistical models, physical models 3. Optimization: Bayesian optimization, multi-objective optimization, evolutionary computation 4. Implementation: Process control, real-time monitoring

2.1.3 Comparison with Related Fields

PI is related to several fields, but each has a different focus.

What makes PI distinctive: - Integrated approach: Integrates quality, control, and optimization - Data-driven: Learns from large volumes of measured data - Real-time capability: Online analysis and feedback control - Multi-objective optimization: Considers yield, quality, cost, and environment simultaneously

2.1.4 Conventional Process Development vs. PI

2.2 PI Glossary: 20 Essential Terms

Here we organize, by category, the technical terms that appear frequently when studying PI.

Basic Terms (1-7)

Method Terms (8-14)

Application Terms (15-20)

Tips for learning the terms: - First, prioritize understanding 1-7 (basic terms) - Study the details of 8-14 (method terms) at the intermediate level - Apply 15-20 (application terms) in practical projects

2.3 Types and Characteristics of Process Data

Chemical processes generate diverse types of data. Understanding the characteristics of each is important for applying PI methods.

2.3.1 Time Series Data

Characteristics: - Data measured continuously along the time axis - Sampling frequency: every 1 second to 1 minute - Enormous data volume (1 sensor × 1 day = 86,400 data points)

Typical examples:

Analysis methods: - Time series forecasting: ARIMA, Prophet, LSTM (deep learning) - Anomaly detection: Control charts, One-Class SVM - Trend analysis: Moving averages, seasonal decomposition

Challenges: - High noise (sensor accuracy, influence of the external environment) - Missing values (sensor failures, communication errors) - Enormous data volume (1 plant × 1 year = several TB)

2.3.2 Batch Data

Characteristics: - One data point per batch - Each batch is an independent manufacturing unit - Conditions differ subtly from batch to batch

Typical example (pharmaceutical batches):

Analysis methods: - Regression analysis: Linear regression, random forest - Batch-to-batch variation analysis: ANOVA, principal component analysis (PCA) - Optimization: Bayesian optimization, response surface methodology

Challenges: - Few samples (tens to hundreds of batches per year) - Need to capture subtle differences between batches - Dependence on the raw material lot

2.3.3 Quality Data

Characteristics: - Data representing product quality - Often measured offline (GC, HPLC, NMR, etc.) - Measurement takes time and cost

Typical examples:

Analysis methods: - Quality prediction (soft sensor): Estimate quality from process variables - Quality control: Control charts, Cp/Cpk - Out-of-spec determination: Classification models (SVM, logistic regression)

Challenges: - Low measurement frequency (not real-time) - Measurement error is present - High cost

2.3.4 Operational Data

Characteristics: - Data representing the operating state of the plant - Operator action history, alarms, event logs

Typical example:

2025-10-16 09:15:32 | High-temperature alarm | Reactor R-101 | Temperature 120°C (upper limit 115°C)
2025-10-16 09:16:05 | Operator action | Increase cooling water flow | 50 L/min → 70 L/min
2025-10-16 09:18:20 | Alarm cleared | Reactor R-101 | Temperature 112°C

Analysis methods: - Event log analysis: Extraction of anomaly-occurrence patterns - Alarm frequency analysis: Distinguishing true alarms vs. false alarms - Root cause analysis: Tracing chains of alarms

2.3.5 Data Integration and Preprocessing

In PI, we integrate these different types of data and put them to use.

Preprocessing steps:

1. Data cleaning: Removal of outliers and missing values
2. Time-axis unification: Synchronizing data with different sampling frequencies
3. Normalization: Scaling data of different magnitudes to 0-1 or standardizing it
4. Feature engineering: Deriving new variables (e.g., temperature × time)

import pandas as pd
import numpy as np

# Example of time series data preprocessing
def preprocess_process_data(df):
    # 1. Outlier removal (3-sigma method)
    df = df[(np.abs(df['temperature'] - df['temperature'].mean())
             <= 3 * df['temperature'].std())]

    # 2. Missing value interpolation (linear interpolation)
    df = df.interpolate(method='linear')

    # 3. Moving average (noise removal)
    df['temperature_smooth'] = df['temperature'].rolling(window=10).mean()

    # 4. Normalization (0-1)
    df['temperature_norm'] = (df['temperature'] - df['temperature'].min()) / \
                             (df['temperature'].max() - df['temperature'].min())

    return df

2.4 The PI Ecosystem: The Flow of Data

PI is not a standalone technology but an ecosystem in which multiple elements work together. The following diagram shows the flow of data in PI.

How to read the diagram: 1. Data collection layer: Collects data from sensors, analyzers, and operators 2. Data management layer: Stores data and performs preprocessing 3. Analysis layer: Builds machine learning models and runs optimization 4. Implementation layer: Uses prediction results for process control and visualization 5. Feedback: Returns measured data back into the system for continuous improvement

Key points: - Data circulates (feedback loop) - Each layer plays a specialized role - Real-time capability is important (from seconds to minutes)

2.5 The Basic PI Workflow: 5 Steps in Detail

Chapter 1 introduced an overview, but here we explain the practical 5-step workflow in detail.

2.5.1 The Big Picture

2.5.2 Step 0: Problem Formulation

What to do: - Clearly define the problem you want to solve - Concretize the objective function and constraints - Set the criteria for success (KPIs)

Concrete example: Improving the yield of a chemical reaction process

Poor problem formulation:

"We want to increase the yield."

Good problem formulation:

"Maximize the yield of chemical reaction process X under the following conditions: - Objective: Improve the yield from the current 70% to 80% or higher - Constraints: - Temperature: 80-120°C (safe range) - Pressure: 1-5 atm - Reaction time: ≤6 hours (maintain productivity) - Purity: ≥98% (quality specification) - Cost: within +10% of current - KPIs (success criteria): - Achieve yield ≥80% for 10 consecutive batches - Batch-to-batch variation ≤2% - Deadline: within 6 months"

Problem-formulation checklist: - [ ] Is the objective defined quantitatively? - [ ] Are the constraints clear? - [ ] Are the KPIs measurable? - [ ] Is the deadline realistic? - [ ] Have the stakeholders agreed?

Time estimate: 1-2 weeks (including discussions with stakeholders)

2.5.3 Step 1: Data Collection

What to do: - Collect past operational data - Check data quality (missing values, outliers) - Conduct additional experiments as needed

Priority of data sources:

1. Existing plant data (most efficient) - Export time series data from the DCS/SCADA - Typically, months to years of data have accumulated

2. Laboratory data (complementary) - Bench-scale experimental results - Detailed quality analysis data

3. New experiments (as needed) - Add experiments for conditions that existing data lacks

Concrete example: Data collection for a reaction process

import pandas as pd

# Example of exporting data from a DCS
process_data = pd.read_csv('dcs_export.csv', parse_dates=['timestamp'])

print(f"Data period: {process_data['timestamp'].min()} - {process_data['timestamp'].max()}")
print(f"Number of data points: {len(process_data):,}")
print(f"Number of variables: {process_data.shape[1]}")

# Basic statistics
print("\nBasic statistics:")
print(process_data[['temperature', 'pressure', 'flow_rate', 'yield']].describe())

# Check for missing values
print("\nMissing values:")
print(process_data.isnull().sum())

Expected output:

Data period: 2023-01-01 - 2024-12-31
Number of data points: 31,536,000
Number of variables: 50

Basic statistics:
       temperature  pressure  flow_rate    yield
count    31536000  31536000   31536000  1200.0
mean         95.3      3.2       45.2    70.3
std           8.7      0.8       12.3     3.1
...

Missing values:
temperature        120
pressure            85
flow_rate          156
yield                0

Time estimate: 1 week (identifying, exporting, and quality-checking the data)

2.5.4 Step 2: Model Building

What to do: - Train a machine learning model using the collected data - Select appropriate descriptors (features) - Evaluate and optimize the model's performance

Sub-steps:

2.1 Descriptor design

Convert process variables into a form suited to machine learning models.

Examples of descriptors:

2.2 Model selection

2.3 Model training and evaluation

from sklearn.ensemble import RandomForestRegressor
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.metrics import mean_absolute_error, r2_score

# Data split
X = df[['temperature', 'pressure', 'catalyst_conc', 'residence_time']]
y = df['yield']

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

# Model training
model = RandomForestRegressor(n_estimators=100, random_state=42)
model.fit(X_train, y_train)

# Prediction and evaluation
y_pred = model.predict(X_test)
mae = mean_absolute_error(y_test, y_pred)
r2 = r2_score(y_test, y_pred)

print(f"MAE: {mae:.2f}%")
print(f"R²: {r2:.3f}")

# Cross-validation
cv_scores = cross_val_score(model, X, y, cv=5,
                            scoring='neg_mean_absolute_error')
print(f"CV MAE: {-cv_scores.mean():.2f} ± {cv_scores.std():.2f}%")

Performance guidelines: - R² > 0.8: Good - R² > 0.9: Excellent - R² < 0.7: The model needs to be reconsidered

Time estimate: 2 weeks (descriptor design, model training, tuning)

2.5.5 Step 3: Optimization

What to do: - Use the trained model to search for optimal operating conditions - Apply methods such as Bayesian optimization and multi-objective optimization

Example of Bayesian optimization:

from skopt import gp_minimize
from skopt.space import Real

# Objective function (maximize yield = minimize negative yield)
def objective(params):
    temperature, pressure, catalyst_conc = params
    predicted_yield = model.predict([[temperature, pressure, catalyst_conc, 60]])
    return -predicted_yield[0]  # Convert to a minimization problem

# Definition of the search space
space = [
    Real(80, 120, name='temperature'),     # Temperature range
    Real(1, 5, name='pressure'),           # Pressure range
    Real(0.5, 3, name='catalyst_conc'),    # Catalyst concentration range
]

# Run Bayesian optimization
result = gp_minimize(
    objective,
    space,
    n_calls=20,      # 20 evaluations
    random_state=42
)

print(f"Optimal conditions:")
print(f"  Temperature: {result.x[0]:.1f}°C")
print(f"  Pressure: {result.x[1]:.2f} atm")
print(f"  Catalyst concentration: {result.x[2]:.2f}%")
print(f"  Predicted yield: {-result.fun:.1f}%")

Example of multi-objective optimization (yield vs. cost):

from pymoo.algorithms.moo.nsga2 import NSGA2
from pymoo.optimize import minimize
from pymoo.core.problem import Problem

class ProcessOptimizationProblem(Problem):
    def __init__(self):
        super().__init__(
            n_var=3,           # 3 variables
            n_obj=2,           # 2 objectives (yield, cost)
            n_constr=0,
            xl=[80, 1, 0.5],   # Lower bounds
            xu=[120, 5, 3]     # Upper bounds
        )

    def _evaluate(self, X, out, *args, **kwargs):
        # Objective 1: Maximize yield (→ minimize negative yield)
        yield_pred = model_yield.predict(X)
        f1 = -yield_pred

        # Objective 2: Minimize cost
        cost = X[:, 0] * 0.1 + X[:, 1] * 5 + X[:, 2] * 20  # Simplified example
        f2 = cost

        out["F"] = np.column_stack([f1, f2])

# Optimize with the NSGA-II algorithm
algorithm = NSGA2(pop_size=100)
problem = ProcessOptimizationProblem()

result = minimize(
    problem,
    algorithm,
    ('n_gen', 50),  # 50 generations
    verbose=False
)

print(f"Number of Pareto-optimal solutions: {len(result.F)}")

Time estimate: 1-2 weeks (selecting the optimization method, running it, and analyzing results)

2.5.6 Step 4: Implementation and Validation

What to do: - Test the conditions obtained from optimization in the actual plant - Analyze the discrepancy between predictions and measurements - Implement in stages (pilot → commercial plant)

Implementation steps:

1. Pilot test (1-2 batches) - Confirm safety - Check the degree of agreement with predictions

2. Small-scale deployment (10 batches) - Confirm reproducibility - Evaluate batch-to-batch variation

3. Full rollout (continuous operation) - Adopt as the standard operating condition - Continuous monitoring

Validation checklist: - [ ] Does it meet safety standards? - [ ] Does it meet quality specifications? - [ ] Does the measured value agree with the predicted yield within ±5%? - [ ] Is the batch-to-batch variation within the acceptable range? - [ ] Can operators run it?

Time estimate: 1-2 months (pilot testing, phased deployment)

2.5.7 Continuous Improvement

What to do: - Collect data after implementation - Retrain the model periodically - Explore room for further optimization

The continuous improvement cycle:

Initial model (R² = 0.85)
  ↓ Add 10 batches of data
Model retraining (R² = 0.88)
  ↓ Add 20 batches of data
Model retraining (R² = 0.91)
  ↓ Discover new room for optimization

Time estimate: Ongoing (retraining every 3 months is recommended)

2.6 Process Descriptors in Detail

2.6.1 Types of Descriptors

1. Physicochemical parameters

2. Equipment characteristics

3. Operating conditions

2.6.2 Automatic Generation of Descriptors

import pandas as pd

# Calculation of basic descriptors
def generate_descriptors(df):
    # Derived variables
    df['residence_time'] = df['reactor_volume'] / df['flow_rate']  # minutes

    # Interaction term
    df['temp_pressure'] = df['temperature'] * df['pressure']

    # Time series feature (moving average)
    df['temp_ma_10min'] = df['temperature'].rolling(window=10).mean()

    # Rate of change
    df['temp_change_rate'] = df['temperature'].diff()

    return df

2.7 Summary

What You Learned in This Chapter

1. The definition and positioning of PI - Optimizing chemical processes with data and AI - Differences from related fields (QE, DoE, SPC, process control)

2. 20 PI terms - Basic terms, method terms, application terms - Understanding the technical terms that appear frequently in practice

3. Types of process data - Time series data, batch data, quality data, operational data - Characteristics and analysis methods of each data type

4. The PI ecosystem - Data collection → management → analysis → implementation → feedback - The role of each layer and how they work together

5. The 5-step PI workflow - Step 0: Problem formulation - Step 1: Data collection - Step 2: Model building - Step 3: Optimization - Step 4: Implementation and validation - The continuous improvement cycle

6. Process descriptors - Physicochemical parameters, equipment characteristics, operating conditions - Methods for automatically generating descriptors

On to the Next Chapter

Chapter 3 covers practical PI methods using Python: - Environment setup (three options) - 35 executable code examples - A chemical reactor optimization project

Exercises

Problem 1 (Difficulty: easy)

Explain the differences between PI and related fields (quality engineering, design of experiments, statistical process control).

Problem 2 (Difficulty: medium)

For each of the three types of chemical process data—time series data, batch data, and quality data—explain its characteristics and the analysis methods suited to it.

Problem 3 (Difficulty: medium)

Explain why Step 0 (problem formulation) of the PI workflow is the most important, giving a concrete example.

Data Licensing and Access

Process Databases (Common with Chapter 1)

Industrial process data: - AIChE DIPPR Database: Thermophysical property data (see Chapter 1 for details) - NIST Chemistry WebBook: A free thermochemical database

Ensuring Code Reproducibility

Environment information: - Python: 3.9 or later - pandas >= 1.3.0, numpy >= 1.21.0, scikit-learn >= 1.0.0

Libraries for the PI workflow: - skopt (scikit-optimize): >= 0.9.0 - Installation: pip install scikit-optimize - Use: Bayesian optimization (Gaussian Process)

• pymoo: >= 0.6.0
• Installation: pip install pymoo
• Use: Multi-objective optimization (NSGA-II, etc.)

Practical Pitfalls

Pitfall 1: Multicollinearity of descriptors

Adding interaction terms introduces multicollinearity:

# Example of the problem
df['temp_pressure'] = df['temperature'] * df['pressure']
df['temp_squared'] = df['temperature'] ** 2

# Multicollinearity check
from statsmodels.stats.outliers_influence import variance_inflation_factor
vif_data = pd.DataFrame()
vif_data["feature"] = X.columns
vif_data["VIF"] = [variance_inflation_factor(X.values, i) for i in range(len(X.columns))]
print(vif_data)
# VIF > 10 indicates multicollinearity

# Countermeasure: dimensionality reduction with principal component analysis (PCA)
from sklearn.decomposition import PCA
pca = PCA(n_components=0.95)  # Retain 95% of the variance
X_reduced = pca.fit_transform(X)

Pitfall 2: Autocorrelation of time series data

Process data is temporally correlated:

# Checking autocorrelation
from statsmodels.graphics.tsaplots import plot_acf
plot_acf(df['temperature'], lags=50)
plt.show()

# Countermeasure: use a differenced series
df['temp_diff'] = df['temperature'].diff()

Pitfall 3: Independence of batch data

Batches are assumed to be independent, but in reality they depend on the raw material lot:

# Group by raw material lot
for lot_id, group in df.groupby('raw_material_lot'):
    # Train a model per lot, or add the lot as a feature
    pass

End-of-Chapter Checklist (45 items)

1. Definition of PI and Related Fields (10 items)

• [ ] 1.1 Can explain the concise definition of PI
• [ ] 1.2 Can list the four pillars of Industry 4.0
• [ ] 1.3 Can state the four core elements of PI (data, modeling, optimization, implementation)
• [ ] 1.4 Can explain the difference from quality engineering (QE)
• [ ] 1.5 Can state the relationship to design of experiments (DoE)
• [ ] 1.6 Can explain the difference from statistical process control (SPC)
• [ ] 1.7 Understands the interplay with process control
• [ ] 1.8 Can explain what makes PI distinctive (integrated approach, data-driven, etc.)
• [ ] 1.9 Can draw the Mermaid diagram of conventional process development vs. PI
• [ ] 1.10 Understands the scope of PI (yield, quality, cost, environment)

2. The 20 PI Terms (10 items)

• [ ] 2.1 Can define basic terms 1-7 (process variable, descriptor, etc.)
• [ ] 2.2 Can explain method terms 8-14 (Bayesian optimization, multi-objective optimization, etc.)
• [ ] 2.3 Can state application terms 15-20 (digital twin, predictive maintenance, etc.)
• [ ] 2.4 Can explain the difference between batch and continuous processes
• [ ] 2.5 Understands the mechanism of Bayesian optimization (Gaussian Process)
• [ ] 2.6 Can explain the meaning of the Pareto front
• [ ] 2.7 Can give three uses of soft sensors
• [ ] 2.8 Can state the difference between DCS and SCADA
• [ ] 2.9 Can explain the definition of a digital twin
• [ ] 2.10 Understands the learning priority of the terms (basic → method → application)

3. Types of Process Data (10 items)

• [ ] 3.1 Can explain the characteristics of time series data (sampling frequency, data volume)
• [ ] 3.2 Can list three typical analysis methods for time series data
• [ ] 3.3 Can state the characteristics of batch data (independence, number of samples)
• [ ] 3.4 Can give three causes of batch-to-batch variation
• [ ] 3.5 Understands the measurement methods for quality data (HPLC, GC-MS, etc.)
• [ ] 3.6 Can explain the challenges of quality data (frequency, cost)
• [ ] 3.7 Can state how to utilize operational data (event logs)
• [ ] 3.8 Can explain the four steps of data preprocessing
• [ ] 3.9 Can implement an outlier removal method (3-sigma method)
• [ ] 3.10 Can appropriately choose a missing value interpolation method (linear interpolation)

4. The 5-Step PI Workflow (10 items)

• [ ] 4.1 Can explain the importance of Step 0 (problem formulation)
• [ ] 4.2 Understands the difference between good and poor problem formulation
• [ ] 4.3 Can list the five items of the problem-formulation checklist
• [ ] 4.4 Can state the priority of data sources in Step 1 (data collection)
• [ ] 4.5 Understands the importance of descriptor design in Step 2 (model building)
• [ ] 4.6 Remembers the model performance guidelines (R² > 0.8 good, >0.9 excellent)
• [ ] 4.7 Can write the Bayesian optimization code example for Step 3 (optimization)
• [ ] 4.8 Can explain the three stages of Step 4 (implementation/validation): pilot → small-scale → full
• [ ] 4.9 Can list the five items of the validation checklist
• [ ] 4.10 Understands the continuous improvement cycle (retraining every 3 months)

5. Process Descriptors (5 items)

• [ ] 5.1 Can explain the importance of the five physicochemical parameters (temperature, pressure, etc.)
• [ ] 5.2 Understands equipment-characteristic descriptors (reactor volume, stirring speed, etc.)
• [ ] 5.3 Can state operating-condition descriptors (heating rate, feeding order, etc.)
• [ ] 5.4 Can write Python code for automatic descriptor generation
• [ ] 5.5 Understands the physical meaning of interaction terms

References

1. Venkatasubramanian, V. (2019). "The promise of artificial intelligence in chemical engineering: Is it here, finally?" AIChE Journal, 65(2), 466-478. DOI: 10.1002/aic.16489

2. Lee, J. H., Shin, J., & Realff, M. J. (2018). "Machine learning: Overview of the recent progresses and implications for the process systems engineering field." Computers & Chemical Engineering, 114, 111-121. DOI: 10.1016/j.compchemeng.2017.10.008

3. Seborg, D. E., Edgar, T. F., Mellichamp, D. A., & Doyle III, F. J. (2016). Process Dynamics and Control (4th ed.). Wiley.

4. The Society of Chemical Engineers, Japan (ed.) (2020). Chemical Engineering Handbook, 8th Edition. Maruzen Publishing.

5. scikit-optimize Documentation. (2024). "Bayesian Optimization." URL: https://scikit-optimize.github.io/stable/

6. pymoo Documentation. (2024). "Multi-objective Optimization." URL: https://pymoo.org/

Author Information

This article was created as part of the MI Knowledge Hub project under Dr. Yusuke Hashimoto of Tohoku University.

Version: 1.1

Change Log - 2025-10-19: v1.1 Quality improvements - Added data licensing and access information - Ensured code reproducibility (skopt, pymoo) - Added three practical pitfalls (multicollinearity, autocorrelation, batch independence) - Added a 45-item end-of-chapter checklist (5 categories) - Added two references (scikit-optimize, pymoo) - 2025-10-16: v1.0 First edition created

License: CC BY 4.0