iiinova — where your digital self keeps evolving

Dipankar Mazumdar @dipankartnt.bsky.social · 18/03/2025

[NEW BLOG]: What is Apache Arrow Flight / Flight SQL / ADBC? 🎉 We need to ask, Why don’t ODBC & JDBC fit in today’s analytical world? These protocols were designed particularly for row-based workloads. What about columnar “Arrow” based data? dipankar-tnt.medium.com/what-is-apac...

dipankar-tnt.medium.com

What is Apache Arrow Flight, Flight SQL & ADBC?

ODBC (Open Database Connectivity) and JDBC (Java Database Connectivity) have long been the industry standard for connecting databases with…

050

Reposted by Dipankar Mazumdar

Apache Software Foundation (The ASF) @apache.org · 10/02/2025

📅 Save the Date 📅 Community Over Code North America 2025 has been announced! Where: Minneapolis, MN (USA) When: September 11-14, 2025 Read more about #CommunityOverCode --> buff.ly/4jQx36S

058

Dipankar Mazumdar @dipankartnt.bsky.social · 30/01/2025

Blogged: Concurrency control methods in a Lakehouse with Apache Hudi, Iceberg & Delta Lake. In this blog I go into the fundamentals of concurrency control, explore why it is essential for lakehouses with OCC, MVCC & Non blocking control. hudi.apache.org/blog/2025/01...

hudi.apache.org

Concurrency Control in Open Data Lakehouse | Apache Hudi

Introduction

010

Dipankar Mazumdar @dipankartnt.bsky.social · 24/01/2025

Blogged: Clustering Algorithms in Open Lakehouse formats such as Apache Hudi, Apache Iceberg & Delta Lake. Querying huge volumes of data from storage demands optimized query speed Your queries are fast today, but they might not be over time! Read: www.onehouse.ai/blog/what-is...

121

Dipankar Mazumdar @dipankartnt.bsky.social · 13/01/2025

“Bringing the database kernel to data lakes” - this is what Apache Hudi started with before the world heard of something called “Lakehouse”. Lakehouse means only one thing- data lakes needed the “transactional layer” on top of Parquet for running db-style workloads (both transactional & analytical)

100

Reposted by Dipankar Mazumdar

arynndoesdata @arynn.bsky.social · 07/01/2025

Now seems like a good time to remind people of my starter pack. go.bsky.app/T1SxhAe

107813

Reposted by Dipankar Mazumdar

Ananth Packkildurai @ananthdurai.bsky.social · 09/12/2024

The 200th edition of Data Engineering Weekly is out. Thank you all for your kind support www.dataengineeringw...

dataengineeringweekly.com

Data Engineering Weekly #200

The Weekly Data Engineering Newsletter

1111

Reposted by Dipankar Mazumdar

Dipankar Mazumdar @dipankartnt.bsky.social · 21/11/2024

Apache XTable at Scale in Production! This is a solid example to show the metadata translation capability for open table formats like Iceberg, Hudi & Delta. Fabric users can work with Iceberg tables written by Snowflake without any rewrites/stuff. Link: blog.fabric.microsoft.com/en-us/blog/s...

101

Dipankar Mazumdar @dipankartnt.bsky.social · 21/11/2024

101

Reposted by Dipankar Mazumdar

Alex Miller @alexmillerdb.bsky.social · 20/11/2024

New blog post on the fun new hardware advancements which databases can leverage for great gains, and why the cloud means it doesn't matter that they exist. 🫠 transactional.blog/b...

35418

Reposted by Dipankar Mazumdar

Jay 🦋 @jay.bsky.team · 13/11/2024

This might be the first time an open source app is at the top of the app store. Definitely the first open source social app.

623007398

Dipankar Mazumdar @dipankartnt.bsky.social · 09/11/2024

5 techniques to optimize performance for your Open Lakehouse. No matter how performant your compute engine is, your storage needs data to be optimally organized. Here’s a new blog I published! www.onehouse.ai/blog/how-to-...

onehouse.ai

How to Optimize Performance for Your Open Data Lakehouse

How can you speed up queries, and deal with new workloads and shifting query patterns? Learn about the tools you have to optimize your lakehouse for performance.

121

Dipankar Mazumdar @dipankartnt.bsky.social · 04/11/2024

Fast "Copy-On-Write" on Apache Parquet. Upserts are crucial for use cases like CDC. There are two ways for record-level updates in data lakes: Copy-On-Write (CoW) Merge-On-Read (MoR) A 🧵on Uber's new CoW optimization for Parquet.

110

Dipankar Mazumdar @dipankartnt.bsky.social · 01/11/2024

Announcing my new book - "Engineering Lakehouses with Open Table Formats" 🎉 TBH, I have been thinking about this for quite some time. Most of the times, in conversations with engineers exploring these formats, so many questions have come up.

3111

Reposted by Dipankar Mazumdar

Dipankar Mazumdar @dipankartnt.bsky.social · 29/10/2024

I started this new repo to serve as a central hub for content that I create & insights that I share from different sources about the Lakehouse architecture. Here’s what you will find in the repo: ⭐️ Blogs ⭐️ Research Papers summary ⭐️ Code ⭐️ Crisp Social posts Link: github.com/dipankarmazu...

111

Dipankar Mazumdar @dipankartnt.bsky.social · 29/10/2024

111

Dipankar Mazumdar @dipankartnt.bsky.social · 28/10/2024

Hey 👋 folks! It’s very nice to see a lot of data community people here. I am Dipankar & I have been involved in open source space for some time. Currently I focus on Data Infra with projects such as Apache Hudi, Iceberg, Arrow & XTable. In my prev gigs, I worked on different spectrum of Data.

140

Dipankar Mazumdar @dipankartnt.bsky.social · 28/10/2024

What is Incremental Processing & how orgs like Uber benefit from it in a Lakehouse? Incremental processing is a technique that processes data in small increments rather than in one large batch.

110

Reposted by Dipankar Mazumdar

Xiangpeng Hao @xiangpeng.systems · 25/10/2024

DataFusion implements one of the most advanced Parquet readers🚀, checkout how: blog.haoxp.xyz/posts/parque...

How DataFusion prune Parquet files based on its metadata can query semantics

1213

Reposted by Dipankar Mazumdar

Simon Späti 🏔️ @ssp.sh · 21/10/2024

A list of Data Engineers gathered from my Twitter following bsky.app/profile/ssp....

2102