ODBC (Open Database Connectivity) and JDBC (Java Database Connectivity) have long been the industry standard for connecting databases with…

Ensuring Atomicity, Consistency, Isolation, and Durability (ACID) in data systems is crucial for maintaining data integrity, especially in environments with concurrent operations. By the end of this b...

Introduction

Overview

The Weekly Data Engineering Newsletter

The open data lakehouse is much more than a data table format, such as Apache Hudi, Apache Iceberg, and Delta Lake. It works seamlessly with all three table formats and serves as a transactional datab...

Apache XTable™ (Incubating) is a cross-table interop of lakehouse table formats Apache Hudi, Apache Iceberg, and Delta Lake. Apache XTable™ is NOT a new or separate format, Apache XTable™ provides abs...

Background

How can you speed up queries, and deal with new workloads and shifting query patterns? Learn about the tools you have to optimize your lakehouse for performance.

With the evolution of storage table formats Apache Hudi®, Apache Iceberg®, and Delta Lake™, more and more companies are building up their lakehouse on top of these formats for many use cases, like incremental ingestion. But the speed of upserts sometimes is still a problem when the data volumes go up. In storage tables, Apache Parquet is used as the main file format. In this article, we will discuss how we built a row-level secondary index and the innovations we introduced in Apache Parquet to speed up the upsert data inside a Parquet file. We will also demonstrate benchmarking results that show 3x to 20x faster speeds than traditional copy-on-write in Delta Lake and Hudi.

Amazon.com: Engineering Lakehouses with Open Table Formats: Build scalable and efficient lakehouses with Apache Iceberg, Apache Hudi, and Delta Lake eBook : Mazumdar, Dipankar, Govindarajan, Vinoth: Kindle Store

A repository of blogs/videos that presents how Apache Iceberg is being used in Production by various orgs - dipankarmazumdar/iceberg-in-production