github.com/jhegedus42/S... Is my attempt to break the transformer nonsense.
First of all, neurons fire in spikes, they have phase and frequency. Ergo, they don't lose information - due to softmax - coz unitarity keeps all information. So, that is suddenly a factor of 1000 in wasting energy....
github.com
GitHub - jhegedus42/Szima
Contribute to jhegedus42/Szima development by creating an account on GitHub.