All Questions
1 question
2
votes
1
answer
86
views
ReLU Variance in Practice Disagrees with Theory
Let $X$ be a normally distributed random variable with $\mu=0$ and $\sigma=1$. Theory tells us that $\mathbb{V}[\text{ReLU}(X)] = \frac{1}{2}\mathbb{V}[X] = \frac{1}{2}$. Thus, we should have that $\...
- 123