We mostly talk about two broad tasks in machine learning, which are Classification and Regression. Classification forms the basis of many tasks we perform, from recognizing images to being the backbone of large language model training. Yes! Even in LLM training, classification happens at multiple levels, typically optimized using the well-known classification loss: Cross Entropy.

Let’s take one example. How will I classify an image into a particular class? A CNN just gives me the feature vector of the image. What is my next step so that the classification actually works and does not suck?

Say ARI when it is ARI
Say ARI when it is ARI

Binary Cross Entropy

When we have only two classes of data, say images of Ari and Octopus, we can use Binary Cross Entropy Loss.

Octopus vs Ari
Octopus vs Ari

And the loss is,

To understand this loss, we need to understand -log(p) first!

We want our model to give high probability to the correct class. If the image is Ari, the probabilty should be high for the class of Ari.

Amazing Model1 and Awful Model2
Amazing Model1 and Awful Model2
Since it is binary classification, we can simply assign numbers 0 and 1 to the classes. So if the true label is Ari (Let’s say y = 1), we want p = probability(model says “Ari”) to be as close to 1 as possible.

In other words, the model should be trained in such a way. If the model says p=0.98, that’s amazing! If it says p=0.1, that’s terrible.

Let’s see what happens when we plug in different p values.

Higher the p, lower the log(p)
Higher the p, lower the log(p)
We take negative of log(p) just to make it positive. If the model is confident and correct (p ≈ 1), -log(p)≈ 0. If it’s wrong and gives a small probability to the true class (p ≈ 0), the loss shoots up.

Now, we know what -log(p) does. Let’s just fit this understanding into the Binary Cross Entropy Loss Function.

As we said, we have two classes (1 and 0).

If the true label is 1 (Ari), then only the first term matters: −log(p) and the second term becomes zero. So, the loss punishes low p values, the model must make p → 1.

If the true label is 0 (Octopus), then only the second term matters: −log(1−p) and the first term disappears. Now, the loss punishes high p values, the model must make p → 0.

We add both terms because we don’t know in advance whether the label will be 0 or 1. It is just a smart way to combine both cases into one single equation. Depending on whether y=1 or y=0, one term automatically becomes active, and the other goes to zero.

That’s BCE Loss!

Cross Entropy

Alright, Binary Cross Entropy worked great when we had only two classes: Ari and Octopus. But what if we now have Ari, Octopus, and StarFish?

Octopus vs Ari vs StarFish
Octopus vs Ari vs StarFish
Now it’s multi-class classification and here comes Cross Entropy Loss. When we have more than two classes, the model doesn’t just give us one probability. Instead, it gives a probability distribution over all classes.
Probability Distribution over the classes: Ari, Octopus, StarFish, seems like a good model
Probability Distribution over the classes: Ari, Octopus, StarFish, seems like a good model
The formula is nothing when you already know BCE Loss.

If we expand this, it’s just y₁log(p₁) + y₂log(p₂) + y₃log(p₃) + … + y꜀log(p꜀). Now, if the class is Ari, how are we going to zero out Octopus and StarFish?

Well! We have something called One-Hot Encoding. For each class, we have a vector filled with zeros except for a 1 at the index of the correct class. That means:

If it’s Ari, the vector is [1 0 0]ᵀ If it’s Octopus, it’s [0 1 0]ᵀ If it’s StarFish, it’s [0 0 1]ᵀ

When we multiply y₁, y₂, y₃ with their respective log(p) values, only the true class’s term survives, the rest all become zero.

For example, if the true class is Ari, the one-hot vector is [1 0 0]ᵀ. Substituting:

Loss = -(1·log(p₁) + 0·log(p₂) + 0·log(p₃)) Loss = -log(p₁)

Only the term corresponding to the true class survives, the rest become zero. That’s how Cross Entropy automatically focuses on the correct class without us manually handling the others.

After training with Cross Entropy Loss, the model learns to assign high probability to the correct class and very low probabilities to all other classes. In other words, for a given input, the output probability distribution becomes sharp and confident, with the true class close to 1 and the rest close to 0. This is the result of the model learning from the loss function over many examples.

Correct Probability Distribution: The Fruition of Cross Entropy
Correct Probability Distribution: The Fruition of Cross Entropy

Note: The model won’t give probability values itself; we use a softmax over the logits the model gives us.