iiinova — where your digital self keeps evolving

Ananth Packkildurai @ananthdurai.bsky.social · 26/01/2026

Data Engineering Weekly's 254th edition is out. Context Graph is the new talk of the town!!

000

Ananth Packkildurai @ananthdurai.bsky.social · 23/01/2026

The companies that build the most boring data stack often win the market!!! Prove me wrong.

010

Ananth Packkildurai @ananthdurai.bsky.social · 10/11/2025

The open source companies built their success on top of open-source platforms, benefited from community contributions and adoption, but now must abandon open-source principles to survive commercially.

010

Ananth Packkildurai @ananthdurai.bsky.social · 03/11/2025

Cricket has been India’s greatest force in overcoming centuries of colonial suppression. Today’s Women’s World Cup win echoes the spirit of 1983 — a triumph that will inspire generations to come. 🇮🇳🏆

000

Ananth Packkildurai @ananthdurai.bsky.social · 02/10/2025

Airbnb: Real-Time Key-Value Store Airbnb’s next-gen key-value store supports real-time ingestion and bulk uploads with sub-second latency, powering feature stores and fraud detection. Read the full story here: www.dataengineeringw...

010

Ananth Packkildurai @ananthdurai.bsky.social · 01/10/2025

Grab: Partner Gateway Metrics at Sub-Second Speed Real-time partner analytics at scale is tough. Grab uses Apache Pinot, Kafka–Flink ingestion, partitioning, and Star-tree indexing to cut query latency to <300 ms, enabling efficient API monitoring and fast issue resolution.

100

Ananth Packkildurai @ananthdurai.bsky.social · 30/09/2025

Netflix Muse: Scaling Analytics at Trillion-Row Scale Netflix evolved its Muse architecture to handle huge datasets efficiently: HyperLogLog sketches, Hollow in-memory feeds, and Druid optimizations cut query latency by ~50% and reduced concurrency load.

100

Ananth Packkildurai @ananthdurai.bsky.social · 29/09/2025

⚡ Latency Every Data Streaming Engineer Should Know “Real-time” has limits—disk, network, and replication delays add up. StreamNative explains latency tiers, common costs, and tuning levers like batching & async processing. 💡 Must-read for data streaming engineers!

100

Ananth Packkildurai @ananthdurai.bsky.social · 27/09/2025

MCP (Model Context Protocol) promises a new way for LLMs to use tools. Chris Riccomini argues it mostly reinvents OpenAPI, gRPC & CLIs. Resources = docs Tools = RPC Prompts = configs So… could MCP have just been a JSON file? 💡 More insights: www.dataengineeringw...

132

Ananth Packkildurai @ananthdurai.bsky.social · 26/09/2025

How Tables Got Smarter: Iceberg → DuckLake. From static snapshots to stream-native updates and catalog-first metadata, tables are evolving fast. Choose by intent, not hype. Subscribe → www.dataengineeringw... Full story → medium.com/fresha-da...

030

Ananth Packkildurai @ananthdurai.bsky.social · 25/09/2025

How Tables Grew a Brain: Iceberg → DuckLake Snapshots → incremental → stream-native → catalog-first. Metadata is the bottleneck. More insights → www.dataengineeringw... Full story → medium.com/fresha-da...

020

Ananth Packkildurai @ananthdurai.bsky.social · 24/09/2025

BlaBlaCar scales like a pro! dbt Core → Transform like a champ Airflow → Orchestrate effortlessly CI/CD → Deploy instantly Dev Containers → Standardized dev 📖 Full story →medium.com/blablacar... 💡 More insights → Subscribe to DEW #DataEngineering #dbt #Airflow #CICD #DevContainers

010

Ananth Packkildurai @ananthdurai.bsky.social · 23/09/2025

🚀 AI adoption is booming—but most data isn’t ready! AI-ready data is: Unified Real-time Human-verified Governed Without it, AI can confidently fail. With it? Reliable, scalable results. 📖 Read More 💡 More insights → Data Engineering Weekly #AI #AIReady #DataEngineering

000

Ananth Packkildurai @ananthdurai.bsky.social · 22/09/2025

The 238th edition of Data Engineering Weekly is available, featuring exciting Data & AI articles. Read more: www.dataengineeringw...

150

Ananth Packkildurai @ananthdurai.bsky.social · 18/09/2025

Apache Iceberg is now entering the classic paradox. Reference: www.dataengineeringw... www.warpstream.com/b...

081

Ananth Packkildurai @ananthdurai.bsky.social · 09/09/2025

Parquet paradox: supports pluggable indexing and bloom filters, but you must rewrite entire files to use them. Meanwhile, LanceDB rebuilds indexes independently. Is "self-contained" showing off its age to "composable" data architectures? 🤔

020

Ananth Packkildurai @ananthdurai.bsky.social · 09/09/2025

AI Hallucinations = confident answers that are flat-out wrong. Why they happen 👇 🎯 Training rewards sounding right, not being right 🎲 Guessing > “I don’t know” 📉 Missing data → confident fiction The fix? Retrieval grounding + truth-focused training.

100

Ananth Packkildurai @ananthdurai.bsky.social · 29/08/2025

Building a Search Engine at Scale 3 B embeddings. 2 months. From content parsing to vector indexing. Wilson Lin shares how—and why chunking is modeling. 📖 www.dataengineeringw... 💡 Subscribe → www.dataengineeringw...

020

Ananth Packkildurai @ananthdurai.bsky.social · 28/08/2025

Netflix is redefining data engineering. With LanceDB + Media ML, the Lakehouse now powers media intelligence, not just metrics. 📖 netflixtechblog.com 💡 Subscribe: dataengineeringweekl...

010

Ananth Packkildurai @ananthdurai.bsky.social · 27/08/2025

Reinforcement Learning Goes Enterprise RLaaS is powering adaptive AI agents in the real world. Dynamic AI that learns by doing = unstoppable. 🚀 📖 www.felicis.com/insi... 💡 More → www.dataengineeringw...

000

Ananth Packkildurai @ananthdurai.bsky.social · 26/08/2025

⚡ Dagster: Streaming Workflows Done Right From batch → event-first. Dagster+ shows how to design reliable, observable real-time pipelines with Kafka & Flink. Why wait for batch when your data can flow instantly? 📖 Full article → www.dataengineeringw...

100

Ananth Packkildurai @ananthdurai.bsky.social · 25/08/2025

The 234th edition is out with the latest trends and thoughts in Data Engineering.

030

Ananth Packkildurai @ananthdurai.bsky.social · 23/08/2025

🚀 LLMs: The Secret Superpower Behind Smarter Systems LLMs aren’t just AI models—they’re system superheroes! ⚡ Execute code 🗄️ Pull context from databases 🌐 Surf live knowledge online 🧩 Solve complex problems like pros

110

Ananth Packkildurai @ananthdurai.bsky.social · 22/08/2025

AI agents don’t just need training — they need the right context. ✨ From KV-cache optimization to external memory & error preservation, Manus shows how context engineering drives speed, recovery & scale. 📖 manus.im/blog/Contex... 💡 More deep dives → www.dataengineeringw...

000

Ananth Packkildurai @ananthdurai.bsky.social · 21/08/2025

Salesforce’s AIMS team shows how caching makes AI inference faster + more resilient 🚀 🔹 400ms → <1ms latency 🔹 27% faster requests 🔹 Survives DB outages Scaling AI = speed + reliability. 📖 engineering.salesfor... 💡 www.dataengineeringw...

000

Ananth Packkildurai @ananthdurai.bsky.social · 20/08/2025

Meta Reinvents Compliance Management Federation Platform → breaks down obligations into workstreams. Privacy Waves → batches tasks monthly for predictability + accountability. The goal: compliance that’s scalable and transparent. 📖 Full read: engineering.fb.com/2...

100

Ananth Packkildurai @ananthdurai.bsky.social · 18/08/2025

Shopify MCP UI → Breaking the Text Wall AI agents can now embed interactive UIs — product selectors, carts, image galleries — via MCP UI. 📌 Deep dive: shopify.engineering/... 💡 Also on DEW: www.dataengineeringw... #AI #ShopifyEngineering #TechInnovation

002

Ananth Packkildurai @ananthdurai.bsky.social · 18/08/2025

Data Engineering Weekly edition #233 is live!! Dive into the full edition here www.dataengineeringw...

010

Ananth Packkildurai @ananthdurai.bsky.social · 15/08/2025

Dimensional Modeling or data vault, which one is better? Property graph vs RDF, which one is best for metadata? All these techniques are defined for human-in-the-loop. The goal of a DW is to provide accurate metrics in a timely fashion. Do we still model the data for humans?

010

Ananth Packkildurai @ananthdurai.bsky.social · 14/08/2025

🎙️ New DEW pod: Bauplan CTO Jacopo Tagliabue on function-native pipelines, immutable versioning, Arrow-powered speed, and why vertical > horizontal for most 10–100 GB jobs. Apple: podcasts.apple.com/u... Spotify: open.spotify.com/epi... Substack: www.dataengineeringw...

010

Ananth Packkildurai @ananthdurai.bsky.social · 05/08/2025

Real-time contextual data is no longer optional — it’s foundational. LLMs demand live, relevant signals. We’re not building for dashboards anymore. We’re building for agents. From batch to streaming. From static to dynamic. From reports to APIs. #LLM #AI #DataOps

011

Ananth Packkildurai @ananthdurai.bsky.social · 28/07/2025

The 230th edition of Data Engineering Weekly is out!! Read more www.dataengineeringw...

040

Ananth Packkildurai @ananthdurai.bsky.social · 21/07/2025

Catch up on all the latest happenings in the data industry this week's DEW edition www.dataengineeringw...

010

Ananth Packkildurai @ananthdurai.bsky.social · 02/06/2025

The 222nd edition of Data Engineering Weekly is out!! www.dataengineeringweekly.com/p/data-engin...

060

Ananth Packkildurai @ananthdurai.bsky.social · 26/05/2025

The 221st edition of Data Engineering Weekly is out with a fresh set of articles. www.dataengineeringw...

011

Ananth Packkildurai @ananthdurai.bsky.social · 04/05/2025

The 219th edition of Data Engineering Weekly is out. Find the latest trends in AI & data by clicking here 👇🏼 www.dataengineeringw...

020

Ananth Packkildurai @ananthdurai.bsky.social · 29/04/2025

The 218th edition of Data Engineering Weekly is out, featuring the latest AI & Data stories. www.dataengineeringw...

010

Ananth Packkildurai @ananthdurai.bsky.social · 23/04/2025

This expert panel could have been replaced with ChatGPT, Claude & Gemini. You’re welcome.

000

Ananth Packkildurai @ananthdurai.bsky.social · 21/04/2025

The 217th edition of Data Engineering Weekly is out. Read more here www.dataengineeringw...

030

Ananth Packkildurai @ananthdurai.bsky.social · 14/04/2025

The 216th edition is fresh and out!! Read more here www.dataengineeringw...

020

Ananth Packkildurai @ananthdurai.bsky.social · 11/04/2025

Read more about composable data architecture in my recent blog. www.dataengineeringw...

120

Ananth Packkildurai @ananthdurai.bsky.social · 07/04/2025

The 215th edition of Data Engineering Weekly is out!! www.dataengineeringw...

010

Ananth Packkildurai @ananthdurai.bsky.social · 02/04/2025

On April 1st, Data Engineering Weekly officially acquired datashitposting.com, pinoner in data shit posting. As a bonus of the acquisition, please enjoy this shit post.

040

Ananth Packkildurai @ananthdurai.bsky.social · 30/03/2025

The 214th edition of Data Engineering Weekly is out!! www.dataengineeringw...

070

Ananth Packkildurai @ananthdurai.bsky.social · 24/03/2025

The 213th edition of Data Engineering Weekly is out!!. Read more www.dataengineeringw...

050

Ananth Packkildurai @ananthdurai.bsky.social · 24/02/2025

Perfecting my Briyani cooking skills. It's much easier than finding the null.

050