The deepseek guys are producing very compute frugal models, at one point one has to think, how can US labs efficiently compete. The causal encoder decoder with 8b/16b per token active and 890 bytes per token of kv cache in total across all layers,Engram(associative memory)192b params, SWA, 1/2
