tags:
- AI/ML/ActivationFunction
aliases:
- Rectified Linear Unit
- ReLU
This function introduces non-linearity by outputting input value if it's positive, and zero if it's negative, takes advantage of linear and non-linear functions. it’s range is (0, ∞)
Formula:
Where
The ReLU function is actually a function that takes the maximum value. Note that this is not fully interval-derivable, but we can take a sub-gradient, as shown in the figure above. Although ReLU is simple, it is an important achievement in recent years.
Leaky ReLU: It solves the dying ReLU problem, as it has a small positive slope in the negative area.
Formula:
Where:
ELU (Exponential Linear Units) function: similar to ReLU, it resolves some of it’s issues. however is computationally expensive. ELU, just like leaky ReLU also considers negative values by introducing a new alpha parameter and multiplying it will another equation.
$$
\begin{cases}
x &\text{for } x \ge 0 \\
\alpha(e^x-1) &\text{for } x < 0
\end{cases}
$$
![[activation-functions-ELU.png]]
- ELU is a strong alternative to ReLU. Different from the ReLU, ELU can produce negative outputs.
- Exponential operations are there in ELU, So it increases the computational time.
- No learning about the ‘a’ value takes place, and exploding gradient problem.