With features x₁ and x₂ the model becomes g(x) = σ(θᵀx + θ₀). Where θᵀx + θ₀ equals 0, the output is exactly 1/2. That is a line (a hyperplane in more dimensions) and it is the boundary.

g(x) = σ(θᵀx + θ₀)

Anywhere θᵀx + θ₀ is a constant, g has the same value, so the picture is a set of parallel lines. Move in the direction of θ and the output climbs toward 1. Move the other way and it falls toward 0.

To predict, take the positive label when θᵀx + θ₀ is greater than 0, and the other label when it is less than or equal to 0.

predict +1 if θᵀx + θ₀ > 0, otherwise −1

That is exactly the family of linear classifiers from earlier in the course. What is new is the confidence attached to each prediction, and a new way to choose θ and θ₀: pick the curve that fits the data best. The lecture shows a bad curve that gives a high probability of coat on days with no coat.

For each day, the curve gives a probability to what actually happened, whether a coat or no coat. The probability of the whole data set is the product of those probabilities, because the days are treated as independent events.

probability of data = product of the probability of each day

There is a catch: a product of many numbers between 0 and 1 gets very small, very fast. The lecture fixes this in the next episode with a logarithm.

The log is monotonic, so the weights that maximize the probability are exactly the weights that maximize its log, and sums are friendlier than products. Earlier in the course we worked with losses, where lower is better, so we take the negative. Averaged over the data, this is the negative log likelihood loss.

loss = −log( probability given to the true label )

One difference from before: the guess is now a probability between 0 and 1, not a plain yes or no.

Because the loss is smooth, it has a gradient, and the gradient points in the direction of steepest climb. Gradient descent starts somewhere, takes a step in the opposite direction with a chosen step size, and repeats.

θ ← θ − η · ∇loss(θ)

You can stop when the loss barely changes, when θ barely moves, or when the gradient is very small. The method is general: it minimizes any function, not only this loss.

A function is convex if, for any two points on its graph, the straight segment between them never dips below the graph. A simple quadratic is convex. A function with two dips is not, because a segment between the dips goes below it.

For a convex function every local optimum is a global one, so gradient descent run long enough, with sensible settings, gets to the best answer. The lecture notes that the logistic regression loss is convex, so descent works well for it.

Running gradient descent on the coat problem, the sigmoid changes at each step. It shifts, so the temperature where you start wearing a coat moves, and it gets steeper, because a steeper curve can raise the probability of what really happened.

The lecture ends this part with a worry: for some data the curve can keep getting steeper and still raise those probabilities. That leads straight to the next episode.

As θ gets larger, every probability gets pushed to an extreme, and that always lowers the loss. But we do not want that: a huge θ is too confident, and we would need a lot of data to justify it. A regularizer adds a penalty for large weights.

penalty = λ · size of θ,  with λ ≥ 0

The constant λ is a trade-off: the larger it is, the more the penalty counts against fitting the data. A common way to choose it is to try a few orders of magnitude and compare them with cross-validation.

The loss with the penalty added is what describes logistic regression. We can classify with it and run gradient descent on it, which means taking the gradient of that objective at each step.

Gradient descent is not the only way to minimize this objective. The lecture closes with a practical note: with millions of data points it feels wasteful to add up all of them before taking a single step, which is a hint of what comes next.