7.2b Data Preparation and Feature Engineering: preprocessing, tokenization, normalization¶

📘 Context Introduction¶

Before an AI model can learn anything useful, raw data must be transformed into a format the model can understand. This process—called data preparation and feature engineering—is often the most time-consuming part of any AI project. For new engineers, think of it like preparing ingredients before cooking: you wash, chop, and measure everything first so the recipe (the model) works correctly.

This section covers three core techniques: preprocessing, tokenization, and normalization. Each plays a distinct role in making data clean, structured, and mathematically usable.

⚙️ What is Data Preprocessing?¶

Data preprocessing is the first step—cleaning and organizing raw data to remove noise and inconsistencies. Without it, models can produce unreliable or biased results.

Key activities include: - Handling missing values — filling gaps (e.g., with the mean or median) or removing incomplete rows - Removing duplicates — ensuring no repeated data points skew training - Correcting errors — fixing typos, mislabeled categories, or outliers - Formatting consistency — converting dates, currencies, or text to a uniform style

Why it matters: Garbage in, garbage out. Preprocessing ensures the data entering your pipeline is as clean as possible.

🛠️ Tokenization — Breaking Text into Pieces¶

Tokenization is the process of splitting text into smaller units called tokens (words, subwords, or characters). This is essential for natural language processing (NLP) models, which cannot understand raw text—only numbers.

Common tokenization approaches: - Word tokenization — splits on spaces and punctuation (e.g., "Hello world" becomes ["Hello", "world"]) - Subword tokenization — breaks rare words into common pieces (e.g., "unhappiness" becomes ["un", "happiness"]) - Character tokenization — splits into individual characters (e.g., "cat" becomes ["c", "a", "t"])

After tokenization, each token is mapped to a unique integer ID (a vocabulary index), which the model can process mathematically.

Example flow: 1. Raw sentence: "AI is powerful" 2. Tokenized: ["AI", "is", "powerful"] 3. Mapped to IDs: [45, 12, 89]

📊 Visual Representation: Feature Engineering and Dataset Tokenization¶

This diagram displays how cleaned datasets are preprocessed through tokenization, normalization, and tensor encoding before execution.

📊 Normalization — Scaling Data to a Common Range¶

Normalization adjusts numerical values so they fall within a similar scale—typically between 0 and 1, or with a mean of 0 and standard deviation of 1. This prevents features with larger numeric ranges (e.g., income in thousands) from dominating features with smaller ranges (e.g., age in years).

Two common methods:

Why it matters: Many AI models (especially neural networks) converge faster and perform better when all input features are on a similar scale.

🧩 Putting It All Together — A Simple Workflow¶

Here is how preprocessing, tokenization, and normalization fit into a typical pipeline:

1. Collect raw data (text, images, numbers)
2. Preprocess — clean missing values, remove duplicates, fix errors
3. Tokenize (for text) — split into tokens, map to IDs
4. Normalize (for numbers) — scale features to a common range
5. Feed into model — ready for training or inference

For reference:

# Example: preprocessing, tokenization, and normalization in Python
import pandas as pd
from sklearn.preprocessing import MinMaxScaler
from transformers import AutoTokenizer

# Load raw data
data = pd.DataFrame({'text': ['AI is cool', 'ML is fun'], 'value': [100, 250]})

# Preprocessing: remove missing values
data = data.dropna()

# Tokenization: convert text to token IDs
tokenizer = AutoTokenizer.from_pretrained('bert-base-uncased')
tokens = tokenizer(data['text'].tolist(), padding=True, truncation=True)

# Normalization: scale numerical values to 0-1
scaler = MinMaxScaler()
data['value_normalized'] = scaler.fit_transform(data[['value']])

📤 Output: The data now contains cleaned text, token IDs, and normalized numerical values—ready for model training.

🕵️ Common Pitfalls for New Engineers¶

• Skipping preprocessing — leads to poor model performance or training failures
• Tokenizing inconsistently — using different tokenizers for training and inference breaks the model
• Normalizing after splitting data — always fit the scaler on training data only, then transform test data
• Forgetting to save tokenizers and scalers — you need them to process new data during deployment

✅ Summary¶

Mastering these three techniques gives you a solid foundation for building reliable AI pipelines—from raw data to production deployment.