Deep Learning & Neural Networks — The Mechanics Behind the Buzzword
"Deep Learning" gets thrown around constantly, but CCAT exams tend to test the mechanics — how a neural network actually trains — not just the buzzword. Here's the process broken down clearly.
Why "Deep"?
"Deep" simply refers to the number of hidden layers in a neural network:
- Shallow network: 1–2 layers
- Deep network: 3+ layers (often dozens or hundreds)
The Biological Inspiration
Artificial Neural Networks (ANNs) loosely mimic biological neurons:
| Biological | Artificial Equivalent |
|---|---|
| Dendrites (receive signals) | Input connections |
| Soma (processes signal) | Weighted sum + activation function |
| Axon (sends signal) | Output connection |
| Synapse (junction) | Weighted edge between nodes |
Layers of a Neural Network
| Layer | Role |
|---|---|
| Input Layer | Receives raw features |
| Hidden Layer(s) | Performs weighted sums + activation; learns abstract features |
| Output Layer | Produces final prediction |
The Training Process — Step by Step
This 4-step cycle is a near-guaranteed exam topic:
Step 1: Forward Pass Input flows through hidden layers to produce a prediction.
Step 2: Calculate Loss Compare the prediction to the correct answer using a loss function (MSE for regression, Cross-Entropy for classification).
Step 3: Backward Pass (Backpropagation)
Compute how much each weight contributed to the error, using the chain rule:
dL/dw = dL/dy × dy/dw
Step 4: Update Weights (Gradient Descent)
w = w − α × (dL/dw)
where α (alpha) is the learning rate.
This cycle repeats until the loss stops decreasing (convergence).
Key Hyperparameters — Memorize the Typical Ranges
| Hyperparameter | Description | Typical Range |
|---|---|---|
| Learning Rate (α) | Step size for weight updates | 0.001 – 0.1 |
| Batch Size | Samples per weight update | 32, 64, 128 |
| Epochs | Full passes through training data | 10 – 1000 |
| Dropout Rate | Fraction of neurons randomly disabled | 0.2 – 0.5 |
Activation Functions — A Frequently Tested Table
| Function | Formula | Range | Best Use |
|---|---|---|---|
| Sigmoid | 1/(1+e⁻ˣ) | (0, 1) | Binary output |
| Tanh | (eˣ−e⁻ˣ)/(eˣ+e⁻ˣ) | (−1, 1) | Hidden layers |
| ReLU | max(0, x) | [0, ∞) | Most hidden layers |
| Leaky ReLU | max(0.01x, x) | (−∞, ∞) | Fixes "dying ReLU" |
| Softmax | eˣⁱ / Σeˣʲ | (0,1), sums to 1 | Multi-class output |
Exam trap: Softmax is for the output layer in multi-class classification — not for hidden layers. Don't confuse it with ReLU's role.
The Perceptron — Where It All Started
A perceptron is the simplest artificial neuron, invented by Frank Rosenblatt in 1958:
z = w1x1 + w2x2 + w3x3 + b
y = 1 if z ≥ 0, else 0
Critical limitation: A single-layer perceptron can only solve linearly separable problems — it can implement AND/OR gates but cannot solve XOR. This exact limitation is why Multi-Layer Perceptrons (MLPs) with hidden layers were developed — a very commonly tested fact.
Quick Recap
- Deep = 3+ hidden layers
- Training = Forward Pass → Loss → Backpropagation → Gradient Descent
- Single-layer perceptron can't solve XOR; MLP can
Next post: CNNs and RNNs — the two architectures behind image recognition and language processing, and how to tell them apart on sight.