Activation Functions & Loss Functions
Activation and loss choices determine whether a network can learn the task at all and whether gradients vanish.
Learn it
Read the CS231n notes on neurons and activation functions to understand ReLU vs. saturating units and the dead-neuron problem. Pair this with 3Blue1Brown's cost-function chapter to connect loss surfaces to the gradient descent you already saw.
By the end you’ll be able to
- Choose appropriate activations (ReLU, sigmoid, softmax) per layer
- Match loss functions to task type (regression vs. classification)
Free resources
Practice
Take a small classifier and swap sigmoid for ReLU and MSE for cross-entropy; record how training loss and accuracy change.
Build it — your output
A comparison table of activation/loss combinations with the training curves you observed.
Prove it — check yourself
You’re ready to move on when you can honestly say yes to each:
- When would you use softmax vs. sigmoid on the output layer?
- Why is cross-entropy preferred over MSE for classification?
- What symptom indicates a saturating activation is killing gradients?
Do it with AI
Use a free AI tool as your tutor and pair-programmer. Copy a prompt below into Claude, ChatGPT, or Gemini (for building, try Cursor or Copilot).
The habit: Try it yourself first — a little struggle is where the learning actually happens. Then ask for a hint before the answer — never copy blindly.
- Understand itClaude / ChatGPT / Gemini
I'm a beginner learning "Activation Functions & Loss Functions". Explain it in simple terms with a real-world analogy, then ask me ONE question to check my understanding — don't reveal the answer.
- PracticeClaude / ChatGPT / Gemini
Give me 3 short practice problems on "Activation Functions & Loss Functions" (my goal: choose appropriate activations (relu, sigmoid, softmax) per layer), from easy to hard. Don't include the solutions yet — I'll try them first, then ask you to check.
- Get unstuckClaude / ChatGPT / Gemini
I'm stuck on this exercise about "Activation Functions & Loss Functions": [paste your attempt here]. Give me ONE small hint to move forward — do NOT give me the full solution.
- Build & reviewCursor / Copilot
Here is my work for a "Activation Functions & Loss Functions" task: [paste your code or answer]. Review it: point out what's wrong and WHY, and suggest improvements — but let me make the fixes myself.
- Prove itClaude / ChatGPT / Gemini
Quiz me on "Activation Functions & Loss Functions". Ask me 5 questions one at a time, wait for my answer each time, then tell me if I'm right and briefly why.