Quentin Lhoest 🤗 @lhoestq.hf.co · 12/05/20261M public datasets on @hf.co ! Congrats for pushing forward AI with open datasets ! 000
Quentin Lhoest 🤗 @lhoestq.hf.co · 16/05/2025It also speeds up files downloads and uploads, since now you only need to move the differentiating data around :) find more about Xet here: huggingface.co/blog/xet-on-... 000
Quentin Lhoest 🤗 @lhoestq.hf.co · 16/05/2025This writer outputs Parquet files that are robust to insertions/deletions/edits Which means versioned datasets cost only a fraction of their original storage ! 🔥🤯 e.g. if you store with Xet, which deduplicates files by chunk cc @julien.ledem.net FYI 100
Quentin Lhoest 🤗 @lhoestq.hf.co · 16/05/2025CDC Parquet writer is out in PyArrow nightlies 🔥🔥 $ pip install \ -i pypi.anaconda.org/scientific-p... \ "pyarrow>=21.0.0.dev0" it's changing the way I view data versioning👇 110
