How does floating point change anything? You still only have so many bits of precision, and the lower bits need to be rounded. In fact floating point has a lot less precision, since many of the bits are needed to store the exponent.
The range of the exponent. Stochastic rounding is intended to prevent bias towards zeroes which paralyze learning, but with floating point this is less of an issue since you only round the mantissa.
The issue is when you add a very small number to a larger number. This occurs in neural networks when accumulating gradients during learning. Many of the gradients are very small, and when adding them repeatedly they get rounded down. There was a paper studying the effect, and I think it was after about 14 bits of precision the accuracy is very diminished. But with stochastic rounding they could get down to very few bits.
All of this is fantastic if you are a deep learning Jedi Master. But there's a very small supply of those sorts.
I really worry about all these reduced floating-point representations when they are made use of by people who mostly understand deep learning through tinkering with existing tensorflow tutorials.
FP32 seems like a relative sweet spot with sufficient dynamic range to let most amateurs avoid getting trapped in the weeds. I could probably be persuaded to believe that FP24 is sufficient as well.
But I suspect that once you get down to 16 bits or so you have to do all sorts of stochastic rounding / dithering that is beyond the skill set of most data scientists. And that's because I suspect many of them don't really get dynamic range.
Which then leads me to believe that what we really need here is the equivalent of R for machine learning.
