<image>https://miro.medium.com/v2/resize:fit:1100/format:webp/1*apStXJ7fdXr2Ru3k3FFD9g.png</image>
We mostly talk about two broad tasks in machine learning, which are Classification and Regression. Classification forms the basis of many tasks we perform, from recognizing images to being the backbone of large language model training. Yes! Even in LLM training, classification happens at multiple levels, typically optimized using the well-known classification loss: Cross Entropy.

Let’s take one example. How will I classify an image into a particular class?
A CNN just gives me the feature vector of the image. What is my next step so that the classification actually works and does not suck?
<imageCaption>
<image>https://miro.medium.com/v2/resize:fit:1100/format:webp/1*qbJZisU90c8ReBWRVb4poQ.png</image>
<caption>Say ARI when it is ARI</caption>
</imageCaption>

## Binary Cross Entropy
When we have only two classes of data, say images of Ari and Octopus, we can use Binary Cross Entropy Loss.
<imageCaption>
<image>https://miro.medium.com/v2/resize:fit:720/format:webp/1*hwiEgdmmx4mGQwwfWKgUcw.png</image>
<caption>Octopus vs Ari</caption>
</imageCaption>

And the loss is,
<image>https://miro.medium.com/v2/resize:fit:1100/format:webp/1*Fkip9RqoAbAu76kvY3teDA.png</image>

To understand this loss, we need to understand -log(p) first!

We want our model to give high probability to the correct class. If the image is Ari, the probabilty should be high for the class of Ari.
<imageCaption>
<image>https://miro.medium.com/v2/resize:fit:1100/format:webp/1*9ZmJDxGCzrLpyQ6KzGMKng.png</image>
<caption>Amazing Model1 and Awful Model2</caption>
</imageCaption>
Since it is binary classification, we can simply assign numbers 0 and 1 to the classes. So if the true label is Ari (Let’s say y = 1), we want p = probability(model says “Ari”) to be as close to 1 as possible.

In other words, the model should be trained in such a way. If the model says p=0.98, that’s amazing! If it says p=0.1, that’s terrible.

Let’s see what happens when we plug in different p values.
<imageCaption>
<image>https://miro.medium.com/v2/resize:fit:1100/format:webp/1*Is287tDjh3ZeWDo2xJYyPg.png</image>
<caption>Higher the p, lower the log(p)</caption>
</imageCaption>
We take negative of log(p) just to make it positive.
If the model is confident and correct (p ≈ 1), -log(p)≈ 0.
If it’s wrong and gives a small probability to the true class (p ≈ 0), the loss shoots up.

Now, we know what -log(p) does. Let’s just fit this understanding into the Binary Cross Entropy Loss Function.
<img>https://miro.medium.com/v2/resize:fit:1100/format:webp/1*Fkip9RqoAbAu76kvY3teDA.png</img>
As we said, we have two classes (1 and 0).

If the true label is 1 (Ari), then only the first term matters: −log(p) and the second term becomes zero. So, the loss punishes low p values, the model must make p → 1.

If the true label is 0 (Octopus), then only the second term matters: −log(1−p) and the first term disappears. Now, the loss punishes high p values, the model must make p → 0.

We add both terms because we don’t know in advance whether the label will be 0 or 1. It is just a smart way to combine both cases into one single equation. Depending on whether y=1 or y=0, one term automatically becomes active, and the other goes to zero.

That’s BCE Loss!

## Cross Entropy
Alright, Binary Cross Entropy worked great when we had only two classes: Ari and Octopus. But what if we now have Ari, Octopus, and StarFish?
<imageCaption>
<image>https://miro.medium.com/v2/resize:fit:614/format:webp/1*g_UNBZ54LcqNTRX72e0I7Q.png</image>
<caption>Octopus vs Ari vs StarFish</caption>
</imageCaption>
Now it’s multi-class classification and here comes Cross Entropy Loss. When we have more than two classes, the model doesn’t just give us one probability. Instead, it gives a probability distribution over all classes.
<imageCaption>
<image>https://miro.medium.com/v2/resize:fit:1100/format:webp/1*J8-6v1bxSnBWmuA8D9dRCA.png</image>
<caption>Probability Distribution over the classes: Ari, Octopus, StarFish, seems like a good model</caption>
</imageCaption>
The formula is nothing when you already know BCE Loss.
<image>https://miro.medium.com/v2/resize:fit:1100/format:webp/1*T_eDSFkxaRzCPGWQ5RHs7Q.png</image>
If we expand this, it’s just y₁log(p₁) + y₂log(p₂) + y₃log(p₃) + … + y꜀log(p꜀). Now, if the class is Ari, how are we going to zero out Octopus and StarFish?

Well! We have something called One-Hot Encoding.
For each class, we have a vector filled with zeros except for a 1 at the index of the correct class. That means:

If it’s Ari, the vector is [1 0 0]ᵀ
If it’s Octopus, it’s [0 1 0]ᵀ
If it’s StarFish, it’s [0 0 1]ᵀ

When we multiply y₁, y₂, y₃ with their respective log(p) values, only the true class’s term survives, the rest all become zero.

For example, if the true class is Ari, the one-hot vector is [1 0 0]ᵀ. Substituting:

Loss = -(1·log(p₁) + 0·log(p₂) + 0·log(p₃))
Loss = -log(p₁)

Only the term corresponding to the true class survives, the rest become zero. That’s how Cross Entropy automatically focuses on the correct class without us manually handling the others.

After training with Cross Entropy Loss, the model learns to assign high probability to the correct class and very low probabilities to all other classes. In other words, for a given input, the output probability distribution becomes sharp and confident, with the true class close to 1 and the rest close to 0. This is the result of the model learning from the loss function over many examples.
<imageCaption>
<image>https://miro.medium.com/v2/resize:fit:1100/format:webp/1*rdyeE7mXRYgA-i-34HhHbQ.png</image>
<caption>Correct Probability Distribution: The Fruition of Cross Entropy</caption>
</imageCaption>

Note: The model won’t give probability values itself; we use a softmax over the logits the model gives us.

arizone/posts/classification-losses-bce-and-ce/content.md

Classification Losses: BCE and CE

Sound waves are analog signals, meaning they are continuous. To understand this, we can use a sine wave as a simple example, since it is also a continuous signal.
<imageWithCaption>
    <img>https://miro.medium.com/v2/resize:fit:1100/format:webp/1*I3aLKf99aY-SZsl8gyJylw.png</img>
    <caption>SINE WAVE | y = sin(2πx)</caption>
</imageWithCaption>
When we say a signal is continuous, it means there are infinitely many values between any two points on the wave. No matter how close two points are, there are always more (infinite) values in between. We can’t store all these values in a computer.
<imageWithCaption>
    <img>https://miro.medium.com/v2/resize:fit:1100/format:webp/1*n2oAHwG13pkzy-nquToNKw.gif</img>
    <caption>Infinite Values In Between Two Points in A Wave</caption>
</imageWithCaption>
So, we record only some values, which are called samples, and they are enough to represent the entire wave. In simple words, we use a few points to represent the entire signal.
<imageWithCaption>
    <img>https://miro.medium.com/v2/resize:fit:1100/format:webp/1*PVgyT7GBWW7EZ5TLtenF6w.png</img>
    <caption>Only Few Points</caption>
</imageWithCaption>
<imageWithCaption>
    <img>https://miro.medium.com/v2/resize:fit:1100/format:webp/1*qgfLJo3fpM5d1Kw6jsl-cw.png</img>
    <caption>Reconstructing Wave: Possibility 1: The Good Reconstruction</caption>
</imageWithCaption>
<imageWithCaption>
    <img>https://miro.medium.com/v2/resize:fit:1100/format:webp/1*GJjqYT7Cm-N-FIdwAuH1xQ.png</img>
    <caption>Reconstructing Wave: Possibility 2: The Bad Reconstruction</caption>
</imageWithCaption>

So, the next question would be: how many samples are required? 30 samples per second? 40 samples per second? Or what’s the number? Let it be whatever, but this number of samples recorded per second is called the sample rate.

Now, to answer this question correctly, understand that there are many frequency components in a sound wave. That means, per second, there could be an irregular number of complete cycles. One particular instance may have a frequency of 2 Hz, and another may have 4 Hz. That’s what a wave signal is like in the natural world.

We need to pick the highest frequency component of all, and our sampling rate should be at least twice the highest frequency value. This number helps in reconstructing the signal perfectly, and this is the so-called Nyquist–Shannon Sampling Theorem. If the highest frequency component in the wave is 3 Hz, the sampling rate should be at least 2*3 Hz = 6 Hz.
 <img>https://miro.medium.com/v2/resize:fit:1100/format:webp/1*_IfS8jxJN9IaQiwFTF0oyg.jpeg</img>
Before we answer how this number is enough to represent the complete signal, we have to discuss what happens if we don’t follow the proper sampling rate. Let’s say we have a sampling rate that is lower than the highest frequency component. In such case, If there are 10 complete cycles per second (10 Hz), our sampling rate would represent only 5 complete cycles, making the computer think that the wave is a lower-frequency one. This phenomenon is called Aliasing and it brings us some fake tones, distortion and beating effect in the audio.

Waves from the natural world contain many high-frequency components, often higher than what our sampling system can handle. To avoid problems, we use a low-pass filter to remove those high frequencies before sampling. This ensures that aliasing-causing frequencies are removed.

You might wonder: “Won’t we lose some audio by filtering these frequencies?” The answer is: yes, but that’s okay. Humans can only hear roughly from 20 Hz to 20 kHz. CD audio uses a sampling rate of 44.1 kHz, which means the Nyquist frequency is 22.05 kHz. Any frequency above ~22 kHz is removed, but we cannot hear those frequencies anyway. So there is no audible loss at all.

Once we have enough sampling points and we remove high frequencies, the wave is forced to be smooth. Given these points, there is only one smooth wave that can pass through them. Any other wave would require sharp changes, which are not allowed by the filter.

Sampling gives us dots, and filtering forces smoothness. Together, they force the original wave to come back.

arizone/posts/the-perfect-wave/content.md

AriZone

Classification Losses: BCE and CE

The Perfect Wave