Drag points, step the learning rules and watch the losses fall. Each topic is one idea from the lectures, from fitting a line by gradient descent to the convolutions inside a CNN.
Fitting a line by gradient descent
Each weight vector \(\mathbf w=[w_1,w_2]\) is one line on the left and one point on the right. Gradient descent walks downhill on the right; the line on the left follows. Drag the data, or drag the point on the loss surface.
Lecture 2, slides 8–15. The lecture's data \((1,1),(2,3),(4,3)\) with \(\mathbf w=[1,0.57]\) gives TrainLoss \(=0.38\).
training example\(f_{\mathbf w}(x)\)residualloss contoursgradient-descent path−η∇ (next step)× least-squares minimum
Gradient descent
Push η past about 0.3 on the lecture data and the steps overshoot the valley and diverge.
Live
Score, margin and the hinge loss
The classifier is \(\text{sign}(\mathbf w\cdot\phi(x))\), so \(\mathbf w\) points into the +1 side and the boundary is perpendicular to it. Drag the tip of \(\mathbf w\). The right plot puts every example at its margin: the zero-one loss is flat there, so only hinge and logistic give gradient descent anything to follow.
Lecture 2, slides 17–27, with \(\phi(x)=[x_1,x_2]\) and the training set \(([0,2],+1),([-2,0],+1),([1,-1],-1)\).
y = +1y = −1w and boundary w·φ(x) = 0misclassifiedzero-onehingelogistican example at its margin
Train
Loss
Click adds
Live
Overfitting and the bias–variance tradeoff
Fit polynomials of growing degree to noisy samples of \(\sin 2\pi x\). The faint curves are fits to other training sets drawn from the same distribution: their spread is the variance, and the gap between their average and the truth is the bias.
Lecture 3, slides 3–9. \(h\) overfits when another \(h'\) has larger training error but smaller error over \(\mathcal D\).
training settrue f(x)fitted h(x)fits to other training setstraining errortesting errorbias²variance
Model and data
Live
k-fold cross-validation
The \(m\) examples are split into \(k\) folds. Each round trains on \(k-1\) folds and validates on the one left out, so every example is validated exactly once. Click a round to see its split; the averaged validation error picks the degree.
\(\displaystyle \text{CV}_k(d)=\frac1k\sum_{i=1}^{k}\text{error}_{\text{fold }i}\big(h_d^{(-i)}\big)\)\(\displaystyle \text{training runs}=k\times(\text{number of models})\)\(\displaystyle k=m\;\Rightarrow\;\text{leave-one-out}\)
Lecture 3, slides 10–13. Each example trains in \(k-1\) rounds, so a fraction \(1-\tfrac1k\) of the data is used for training each time.
training foldvalidation foldclick a round to inspect ittrained onheld outfit on this round's training foldsCV error (mean of folds)one foldtrue test errortraining error
Setup
Live
The perceptron learning rule
Examples are visited in turn. A correct prediction leaves the weights alone; a mistake moves them by \(\alpha I (T-O)\), which tilts and shifts the boundary toward the right answer. The default run reproduces the lecture's trace and stops at \(\mathbf w=(1,-3,4)\).
Lecture 4 (perceptron), slides 6–22. If the examples are linearly separable the rule converges in finitely many steps from any start.
T = 1T = 0boundary (O = 1 side shaded)before last updatew₀w₁w₂a mistake (weights change)
Learn
Data
Trace
AND, OR, NOT, and why XOR needs a hidden unit
A single perceptron draws one line, so it can only represent linearly separable functions. Set the weights by hand or let the rule learn them. XOR has no separating line; the lecture's two-layer network solves it by feeding a hidden AND unit into the output with weight −2.
target 1target 0w₁I₁ + w₂I₂ = θwrong outputhidden unit y₃ fires above its linenetwork outputs 1click a corner to trace it
Single perceptron
Target
Learning rate, batch GD and SGD
An Adaline is a linear unit, so its error surface is a bowl with one global minimum. A small η creeps down the valley; a large η zig-zags across it, and past \(2/\lambda_{\max}\) it diverges. SGD uses one example per update: cheap and noisy. Mini-batches sit in between.
batch GDSGDmini-batchclick to move the startbatch GDSGDmini-batchlog scale
Optimiser
Live
Activation functions and their gradients
Backprop needs a slope. The step function has none, the sigmoid's peaks at 0.25 and fades on both sides, and ReLU passes gradient 1 or 0. On the right, one hidden unit's response over the input plane: a sigmoid unit responds over a whole half-plane, an RBF unit only near its centre.
Lecture 4 (MLP), slides 5–9. With linear hidden units, \(W_2W_1\mathbf x=(W_2W_1)\mathbf x\): extra layers add nothing.
unit output (darker = closer to 1)drag to move the unit
Functions
One hidden unit
Unit
Backpropagation on a computation graph
Forward: values flow left to right in topological order. Backward: each node multiplies its local gradient by the gradient arriving from above and passes the product down. Step through it and watch the add, mul and max gates.
Lecture 4 (computation graphs), slides 1–40. Gradients add where a value branches.
value (forward)gradient ∂f/∂· (backward)current node
Graph
Inputs
This step
Momentum and local minima
A network's error surface can have several minima, and gradient descent stops in whichever basin it starts in. Momentum gives each weight inertia, which speeds it along valleys and can carry it over a shallow ridge. Click the surface to choose a start; the two paths share it.
A small multi-layer perceptron trained by backprop with momentum, in the spirit of the TensorFlow Playground demo. Watch the hidden units' lines combine into a curved boundary. One hidden unit can only draw a line; XOR needs at least two.
Lecture 4 (MLP and gradient descent). Sigmoid output, squared error \(\tfrac12(t-o)^2\), weights start at small random values.
class 1class 0test examplehidden-unit lines (first layer)positive weightnegative weightnode fill = mini decision maptraining losstest losslog scale
Data and network
Data
Hidden act.
Hidden layers
Live
Universal approximation
One hidden layer of sigmoids, each a soft step at its own position, can build any well-behaved function: the output layer scales the steps and adds them up. Add units and the staircase hugs the target more closely.
\(\displaystyle \hat f(x)=b+\sum_{j=1}^{H}v_j\,\sigma\big(k(x-c_j)\big)\)\(\displaystyle \max_x\lvert\hat f(x)-f(x)\rvert\to 0\ \text{ as } H\to\infty\)
Lecture 4 (MLP), slides 10–11. Here the hidden steps are spread evenly over the interval and the output weights \(v_j\) are fitted by least squares.
To get output \(m\), flip the kernel, slide it so it sits over position \(m\), multiply overlapping entries and add. Turn off the flip and you have cross-correlation, which is what CNN libraries actually compute.
signal f(n)kernel g(m − n) at shift moutput (f∗g)(m)drag across the plot to move m
Signals
Signal
Kernel
At this shift
Image filters
A 3×3 kernel slides over the image. Averaging kernels blur; difference kernels respond where intensity changes, which makes them edge detectors. Hover the output to see which pixels feed it. Draw on the input to try your own strokes.
Lecture 5 (image processing basics), slides 5–11. Edge pixels use zero padding, so the output is the same size as the input.
input I (draw on it)receptive field of the hovered outputpositivenegative(signed kernels) · grey for averaging kernels
Kernel
Edit any cell. Applying a blur repeatedly widens it.
Image
Hovered output
Padding and stride
Without padding every convolution shrinks the map by \(k-1\), which limits how deep the network can go. Zero-padding keeps the size; a stride of 2 skips every other position and halves it.
Once a feature is found, only its rough position matters. Max pooling keeps the strongest response in each window, so nudging the input by a pixel often leaves the pooled map unchanged. Shift the input and compare.
Lecture 5 (CNN), slides 15–17. Average pooling replaces the max with the mean.
feature map Fwindow maxpooled Psame as before the shiftchanged
Pooling
Type
Window
Shift input
Live
Receptive fields and parameter sharing
Each unit sees only a small window of the layer below, yet the window it covers in the input grows with every layer. And because one kernel is reused at every position, a convolution needs far fewer weights than a fully connected layer.