Continuous batching. Paged attention. Prefix caching. Speculative decoding. 12 inference ideas, a couple of minutes each, one pass. 🧵👇
youtu.be/MVo3P-D7eSc
youtu.be
YouTube Video
Continuous batching. Paged attention. Prefix caching. Speculative decoding. 12 inference ideas, a couple of minutes each, one pass. 🧵👇
https://youtu.be/MVo3P-D7eSc