Quentin Lhoest 🤗 @lhoestq.hf.co · 16/05/2025CDC Parquet writer is out in PyArrow nightlies 🔥🔥 $ pip install \ -i pypi.anaconda.org/scientific-p... \ "pyarrow>=21.0.0.dev0" it's changing the way I view data versioning👇 110
Quentin Lhoest 🤗 @lhoestq.hf.co · 16/05/2025you can define it this way: >>> import pyarrow.parquet as pq >>> writer = pq.ParquetWriter( ... out, schema, ... use_content_defined_chunking=True, ... ) 100
Quentin Lhoest 🤗 @lhoestq.hf.co · 16/05/2025This writer outputs Parquet files that are robust to insertions/deletions/edits Which means versioned datasets cost only a fraction of their original storage ! 🔥🤯 e.g. if you store with Xet, which deduplicates files by chunk cc @julien.ledem.net FYI 100
