nairadithya

← all notes

Negative Log Loss

concept

A loss function used to evaluate probability distributions. The idea is straightforward, Say you have a target
𝑦
𝑑
 that is the correct answer. You check the probablility output of your ground truth target
𝑦
𝑑
 in your model’s predicted distribution.

𝐿
=
βˆ’
log
(
𝑃
πœƒ
(
𝑦
𝑑
|
π‘₯
)
)

Reasoning about this, a probability distribution can only go from 0 to 1. a log of a value between 0 to 1 is negative. So negate it.

Another useful property of this is that very confident mistakes, where
𝑃
(
𝑦
𝑑
|
π‘₯
)
β†’
0
, the log approaches near infinity. So it penalizes very confident mistakes.

Another useful property of the logarithm is that joint likelihoods decompose into additive conditional operations, which helps with numerical stability.

βˆ’
log
(
𝑃
(
𝑦
𝑑
1
𝑦
𝑑
2
𝑦
𝑑
3
)
)
=
βˆ’
log
(
𝑃
(
𝑦
𝑑
1
)
)
βˆ’
log
(
𝑃
(
𝑦
𝑑
2
|
𝑦
𝑑
1
)
)
βˆ’
log
(
𝑃
(
𝑦
𝑑
3
|
𝑦
𝑑
2
,
𝑦
𝑑
1
)
)