Robert Nishihara @robertnishihara.bsky.social · 04/03/2025DeepSeek released smallpond, a big data processing framework built on top of Ray. - Smallpond targets high performance data processing. - It provides a high-level dataframe API - Targets petabyte-level scaling The challenges around training data prep only grow when you include multimodal data. 120
Robert Nishihara @robertnishihara.bsky.social · 22/11/2024Amazon published this only 4 months ago, but it feels like an eternity. It's one of the most impressive large-scale data processing migration efforts. Rare to see companies truly achieving order of magnitude cost improvements (while simultaneously increasing scale). aws.amazon.com/blogs/openso...aws.amazon.comAmazon’s Exabyte-Scale Migration from Apache Spark to Ray on Amazon EC2 | Amazon Web ServicesLarge-scale, distributed compute framework migrations are not for the faint of heart. There are backwards-compatibility constraints to maintain, performance expectations to meet, scalability limits to... 041
Robert Nishihara @robertnishihara.bsky.social · 22/11/2024Talked with John Schulman last year about the ChatGPT backstory and scaling laws 😍 John co-founded OpenAI and created ChatGPT. www.youtube.com/watch?v=6Ctv...youtube.comChatGPT Creator John Schulman on OpenAI | Ray Summit 2023YouTube video by Anyscale 000
Robert Nishihara @robertnishihara.bsky.social · 26/02/2024A good overview of the fundamentals of how to extend context windows for LLMs (if you care about RAG, you probably care about context lengths).anyscale.comFine-tuning LLMs for longer context and better RAG systemsBased on the popular “Needle In a Haystack” benchmark and RAG, we share our process of creating a problem-specific fine-tuning dataset to extend the context of models to build better RAG systems. 030
