9.1c Matrices: 2D arrays — a fully connected layer's weight matrix¶
🧭 Context Introduction¶
When you hear "fully connected layer" in a neural network, think of a giant spreadsheet where every input connects to every output. The weight matrix is the core of this connection — it's a 2D array (a matrix) that stores all the "strength" values (weights) between one layer and the next. For new engineers, understanding this matrix is like learning the blueprint of how information flows and transforms inside an AI model.
📊 What Is a Fully Connected Layer's Weight Matrix?¶
A fully connected (dense) layer takes an input vector and multiplies it by a weight matrix to produce an output vector. The weight matrix is:
- 2D array — rows and columns of numerical values
- Size = (input_size × output_size) — for example, 784 inputs to 128 outputs gives a 784×128 matrix
- Trainable — these values are adjusted during training to minimize error
⚙️ How the Weight Matrix Works¶
The weight matrix performs a linear transformation:
- Each column corresponds to one neuron in the current layer
- Each row corresponds to one input feature from the previous layer
- Each cell (row i, column j) represents the weight connecting input i to neuron j
The operation is: output = input × weight_matrix + bias
📊 Visual Representation: Matrix 2D Tensor Grid¶
This diagram displays a 2D tensor (matrix), arranging elements in rows and columns to represent grid-like datasets.
🛠️ Visualizing a Simple Weight Matrix¶
Imagine a tiny layer with 3 inputs and 2 outputs:
- Input vector: [x₁, x₂, x₃]
- Weight matrix (3 rows × 2 columns):
| Neuron 1 | Neuron 2 | |
|---|---|---|
| Input 1 | w₁₁ | w₁₂ |
| Input 2 | w₂₁ | w₂₂ |
| Input 3 | w₃₁ | w₃₂ |
- Output calculation:
- Output neuron 1 = (x₁ × w₁₁) + (x₂ × w₂₁) + (x₃ × w₃₁)
- Output neuron 2 = (x₁ × w₁₂) + (x₂ × w₂₂) + (x₃ × w₃₂)
🕵️ Why This Matters for AI Infrastructure¶
Understanding the weight matrix helps engineers optimize:
| Aspect | Why It Matters |
|---|---|
| Memory footprint | A large weight matrix (e.g., 4096×4096) consumes significant GPU memory |
| Data movement | Moving weights from memory to compute units is often the bottleneck |
| Parallelism | Matrix multiplication can be split across GPU cores for speed |
| Precision | Weights can be stored as float32, float16, or int8 to trade accuracy for speed |
🔍 Real-World Example: Image Classification¶
Consider a fully connected layer that processes a 28×28 grayscale image (784 pixels) to 10 output classes (digits 0–9):
- Weight matrix size: 784 rows × 10 columns = 7,840 weight values
- Each weight determines how strongly a pixel influences a digit prediction
- Training adjusts these 7,840 values so the network correctly identifies digits
🧪 Key Takeaway for New Engineers¶
The weight matrix is the "brain" of a fully connected layer — it stores the learned relationships between inputs and outputs. When you see a neural network diagram with arrows connecting circles, those arrows are all stored in a single 2D array called the weight matrix. Engineers optimize how this matrix is stored, moved, and multiplied to make AI training and inference faster.