iiinova — where your digital self keeps evolving

marcogorelli.bsky.social @marcogorelli.bsky.social · 16/09/2026

✨ The Pyrefly type-checker now tracks Polars DataFrame schema 🏷️ Declare a schema using Annotated 📚 Enjoy clearer self-documenting code and avoid some schema-related issues before hitting prod 😇 The annotated schema will be ignored by other type-checkers so you won't get conflicts between them

Example of using `Annotated` to declare Polars DataFrame schema

031

marcogorelli.bsky.social @marcogorelli.bsky.social · 09/04/2026

✨ New Narwhals expression: `nw.corr` 👥 Supported in both group-by and window contexts 🌐 Polars, DuckDB, PyArrow, PySpark, pandas...all supported 🗒️ PS. please don't draw conclusions about ice cream sales and drownings from this post

261

marcogorelli.bsky.social @marcogorelli.bsky.social · 10/03/2026

Bwlch Llanberis (Welsh folk tune) Named after a scenic road in Snowdonia, this is one of my favourite Welsh melodies

010

marcogorelli.bsky.social @marcogorelli.bsky.social · 22/02/2026

Tafliad Carreg (Welsh folk tune) This one was written by Mike Lease, and it means "A stone's throw" Here's my attempt at it. First time tuning my guitar to DADGAD, and my initial impression is that standard tuning might be a mistake

110

marcogorelli.bsky.social @marcogorelli.bsky.social · 13/02/2026

✨ @marimo.io now supports Pyrefly as an LSP, helping you to catch errors before you even execute your code! ⚙️ To enable it: - Install pyrefly - Settings -> User settings -> Enable pyrefly 🤸 This was a fun one to contribute to (🙏 Note that the ty type checker is also supported)

010

marcogorelli.bsky.social @marcogorelli.bsky.social · 30/01/2026

✨ New @pola.rs feature: if you misspell a time zone, it will suggest the correct one to you 🤸‍♂️This was a fun one to work on, Levenshtein and all!

Demo of Polars suggesting the correct time zone

020

marcogorelli.bsky.social @marcogorelli.bsky.social · 04/01/2026

✨ New Polars features: `min_by` and `max_by` 💡 Get the first (or last) value of a column ordered by another expression 🌊🦄 Narwhals will have this feature too, making it available to @arrow.apache.org , pandas, @duckdb.org , and more

061

marcogorelli.bsky.social @marcogorelli.bsky.social · 04/12/2025

😱🙀 The pandas 3.0 release-candidate is here! 💥 Will it break your code? 💡 Test it with `uv pip install -U --pre pandas` to find out! 🌊🦄 Narwhals users can relax, everything's taken care of for you, no need to do anything ☺️

032

marcogorelli.bsky.social @marcogorelli.bsky.social · 13/11/2025

✨ The next @pola.rs release will support lazy pivot! 💡 All you need is to specify the unique values to pivot on

010

marcogorelli.bsky.social @marcogorelli.bsky.social · 10/11/2025

✨ Narwhals now supports Daft! 🔌Just install the `narwhals-daft` plugin, and then `narwhals.from_native` will accept Daft DataFrame! 💡 No extra dependencies requires, computation will remain native (and distributed, and lazy!) until you call `collect`

150

marcogorelli.bsky.social @marcogorelli.bsky.social · 04/11/2025

😲 Wes McKinney mentioned Narwhals on a podcast

Screenshot of Wes mentioning Narwhals on a podcast

020

marcogorelli.bsky.social @marcogorelli.bsky.social · 23/10/2025

✨ New @pola.rs feature: `list.agg` 💡 Compute any Expr aggregation on a list column, just use `pl.element()` 🔜 Coming soon, in version 1.35!

060

marcogorelli.bsky.social @marcogorelli.bsky.social · 20/10/2025

🥧🧵 Narwhals now tests against Python 3.14 (pi-Python!) free-threading build 👉 For more on free-threaded Python, my colleague Nathan's blog post is an excellent read, link in comments

Demo of Narwhals running with free-threaded Python

100

marcogorelli.bsky.social @marcogorelli.bsky.social · 16/10/2025

✨ Narwhals now supports `format` across all backends 🧵 Create new string columns using placeholders and expressions 🌊🦄 Yet another piece of the @pola.rs API brought to pandas, @duckdb.org , PySpark, PyArrow, and more!

031

marcogorelli.bsky.social @marcogorelli.bsky.social · 13/10/2025

✨ NumPy's MaskedArray class is now type-complete 🏷️ Enjoy improved code completion and typing! 🙏 Thank you to Quansight Labs and Meta for having facilicated and funded the work (blog post in comments)

Before vs after of using MaskedArray methods in NumPy

140

marcogorelli.bsky.social @marcogorelli.bsky.social · 03/10/2025

🌝 The next @astropy.org release will support generic dataframe tables (like @pola.rs ) 🌊🦄 Powered by Narwhals

010

marcogorelli.bsky.social @marcogorelli.bsky.social · 12/09/2025

💡 Life-pro-tip: enable inline type hints 🏷️ Your editor will display inferred types for variables you haven't annotated 🕵️ Debugging those red squiggly lines will become much easier

151

marcogorelli.bsky.social @marcogorelli.bsky.social · 22/08/2025

✨🐼 Pandas 3.0 will have `pd.col` syntax 💡 You can use it for clean chaining in `assign` and `loc` 🚀 It's happening, the PR just got merged!

041

marcogorelli.bsky.social @marcogorelli.bsky.social · 12/08/2025

✨ New @pola.rs feature - you can now chain `name` expressions! 🌊🦄 As of Narwhals 2.0 (yes, we've released 2.0 🚀), this feature is available there as well! Little example below with our friend @duckdb.org

Example of `.name` chaining with Narwhals

030

marcogorelli.bsky.social @marcogorelli.bsky.social · 07/08/2025

💡 "There's nothing for permanent than a temporary fix" Case in point, this pandas comment which was added [checks git log] ... 12 years ago 🙈

temporary solution added to pandas 12 years ago

020

Reposted by @marcogorelli.bsky.social

Michael E. Mann @michaelemann.bsky.social · 07/08/2025

"If the U.S. can't get its greenhouse emissions under control, they need to face real consequences" said the rest of the world.

22483149

marcogorelli.bsky.social @marcogorelli.bsky.social · 29/07/2025

💡 Today in open source: a 17-year-old NumPy commit promising that a new feature will be coming soon Any day now..

Screenshot of 17-year-old commit message in NumPy

000

marcogorelli.bsky.social @marcogorelli.bsky.social · 26/07/2025

😍 @posit.co have done it again 🧙 gt-extras offers even more functionality for their table renditions of @pola.rs and pandas dataframes! 🌊🦄 Using Narwhals for some dataframe logic

091

marcogorelli.bsky.social @marcogorelli.bsky.social · 18/07/2025

💡 Solution: use Narwhals for universal dataframe support 🌊🦄

Meme of panicked Astronomer CEO caught on cam with the head of HR, with "When you find a new data science library" and "When you realise it only supports pandas" text overlaid

040

marcogorelli.bsky.social @marcogorelli.bsky.social · 15/07/2025

✨ New Formulaic release (1.2), featuring native support for @pola.rs ! 🌊🦄 Powered by Narwhals

Demo of Formulaic with native Polars support

1102

marcogorelli.bsky.social @marcogorelli.bsky.social · 09/07/2025

🎢 When the Q&A goes off-script 🤣 🇯🇵 Japanese math rock, 🅿️ porpoises and unfortunate aliases, and 🎤 Taylor Swift: 🥈All of these unexpectedly came up in this fun session, which is currently the 2nd most viewed of the conference 🙏 Thanks @pydatalondon.bsky.social for inviting me!

youtu.be

Marco Gorelli - How Narwhals brings Polars, DuckDB, PyArrow, & pandas together | PyData London 25

YouTube video by PyData

082

marcogorelli.bsky.social @marcogorelli.bsky.social · 01/07/2025

🥳 The next version of @mlflow.org will support @pola.rs ! 😩 Slightly disappointingly, they do so by converting to pandas 🌊🦄 Could Narwhals help make MLFlow gain native support for Polars? Link to the discussion below 👇

MLFlow maintainer announcing that they merged the Polars dataset PR

130

marcogorelli.bsky.social @marcogorelli.bsky.social · 19/06/2025

🙀 It's scary to merge a 2,000+ line PR! 😇 But, this one was so thorough that we just had to ship it 🏆 Thanks to an incredible effort by Ryan Whitten, Narwhals supports Ibis! 💡You can now use Narwhals with all of Ibis' backends, plus some extra, whilst keeping @pola.rs expressive API!

030

marcogorelli.bsky.social @marcogorelli.bsky.social · 06/06/2025

✨ @pola.rs now supports `list.filter` 🪄 The `list` namespace is now even more turbocharged ⚡ 👇 Full release notes for 1.30

160

marcogorelli.bsky.social @marcogorelli.bsky.social · 26/05/2025

✨ PySpark 4.0 is out 🌊🦄 And it's already supported by Narwhals, including PySpark connect (thanks Francesco Bruzzesi!)

011

marcogorelli.bsky.social @marcogorelli.bsky.social · 22/05/2025

✨ Narwhals now supports `fill_null` with forwards and backwards strategies 🎬 Here's a demo of a fill-forward function which supports PySpark, @pola.rs , @duckdb.org , pandas, cuDF, and more! 🎁 Bonus: curious about what this looks like as DuckDB SQL? We got you covered 😎

121

marcogorelli.bsky.social @marcogorelli.bsky.social · 09/05/2025

✨ Narwhals now supports PySpark 🎇 If you have a dataframe-agnostic function, you can pass in `pyspark.sql.DataFrame` 📈 Here's a demo, made with @marimo.io 🎁 BONUS feature: combine with SQLFrame, to generate SQL from @pola.rs syntax 🪄

Demo of Narwhals dataframe-agnostic function which supports PySpark

Plot of PySpark dataframe after converting it to PyArrow

011

marcogorelli.bsky.social @marcogorelli.bsky.social · 07/05/2025

🛠️ NVIDIA's cuDF library now runs the Narwhals test suite as part of its CI process 🙀

010

marcogorelli.bsky.social @marcogorelli.bsky.social · 25/04/2025

✨ New Polars feature: `is_business_day` 🌴 Use this to determine which dates fall on holidays, blazingly fast 🇩🇪 Using Germany as an example because I was just there to give a talk at @pyconde.bsky.social - thanks for having me and for putting on such a fun event!

011

marcogorelli.bsky.social @marcogorelli.bsky.social · 11/04/2025

🎈✨ Feeling pumped for this new @pola.rs feature! It's not a perf improvement, and it's not a new expression It's...working auto-complete in Jupyter / IPython / Marimo! 🙀😱 🙏 Credit to @krassowski.bsky.social for debugging and Alex Beedie for submitting a fix! 🧑‍🏫 This is gonna be great for teaching

Comparison of before vs after for Polars autocompletion - since 1.27.1, it gives correct suggestions!

040

marcogorelli.bsky.social @marcogorelli.bsky.social · 19/03/2025

📚 List normalisation: Polars vs DuckDB syntax 🐻‍❄️ Polars: divide the column by its list sum. Expressive as always 🎨 🦆 DuckDB: `a / list_sum(a)` would error...but you can write a list comprehension 😱🙀

010

marcogorelli.bsky.social @marcogorelli.bsky.social · 14/03/2025

✨✨ 2 big Narwhalified releases this week! 🎯 For time series enthusiasts, Darts now supports @pola.rs / PyArrow / cuDF / Modin via Narwhals 🎨 For visualisation enthusiasts, same story for @bokeh.org 🦄🌊 Underwater unicorn magic

041

Reposted by @marcogorelli.bsky.social

Bokeh @bokeh.org · 12/03/2025

We are excited to be here on @bsky.app and excited to announce the release of Bokeh version 3.7! Some great new features like windowed auto-ranging, support for more DataFame types, and new UI components—read all about it on our latest blog post discourse.bokeh.org/t/bokeh-3-7-...

discourse.bokeh.org

Bokeh 3.7 is released!

Check out the details in the blog post! For full details about this release, you can check out the release notes here: Releases — Bokeh 3.7.0 Documentation

055

marcogorelli.bsky.social @marcogorelli.bsky.social · 04/03/2025

🐻‍❄️ @pola.rs is my favourite dataframe library (surprise surprise) 🦆 @duckdb.org belongs alongside it in the modern data stack, learn both and you're in for a treat 📝 My latest blog post teaches the DuckDB equivalent of common dataframe operations, based on learnings from Narwhals development

labs.quansight.org

Mastering DuckDB when you're used to pandas or Polars

It's not as scary as you think

0113

marcogorelli.bsky.social @marcogorelli.bsky.social · 25/02/2025

✨ Streamlit merge PR to support Polars for cache hashing

010

marcogorelli.bsky.social @marcogorelli.bsky.social · 20/02/2025

🐻‍❄️ Under-appreciated @pola.rs feature: seamless expression composability 🎢 In Polars, `shift` and `rolling_mean` can be chained 💡 In SQL, you need a common table subexpression, because LAG isn't a scalar (i.e. elementwise) function

081

marcogorelli.bsky.social @marcogorelli.bsky.social · 18/02/2025

🎙️ Pleased to present at @pyconde.bsky.social this year! 🎯 This will be my first talk focused entirely on Narwhals and how it's bringing open source data tools together

030

marcogorelli.bsky.social @marcogorelli.bsky.social · 07/02/2025

🙀 Bokeh merged a pull request to use Narwhals for Polars / PyArrow support! 3️⃣.7️⃣ This will be available in their 3.7 release, probably later this month 🎢 Exciting times for Python dataframe visualisation!

Bokeh merge pull request to use Narwhals

040

marcogorelli.bsky.social @marcogorelli.bsky.social · 28/01/2025

✨New Narwhals release (1.24) 🏷️ Selectors and `nw.all` expressions are now supported across the API 💖🦆 Many improvements to the PySpark and @duckdb.org backends 😍 The more we work on Narwhals, the more we appreciate how expressive the @pola.rs API is

Demo of Narwhals.all with DuckDB backend

090

marcogorelli.bsky.social @marcogorelli.bsky.social · 20/01/2025

✨ New Polars release (1.20)! 🔎 Series.index_of and Expr.index_of added 🧵 New str.normalize method 🎢 Performance improvements for rolling functions (on my time series feature engineering benchmark it's a 2x improvement!) Full release notes: github.com/pola-rs/pola...

github.com

Release Python Polars 1.20.0 · pola-rs/polars

⚠️ Deprecations Make parameter of str.to_decimal keyword-only (#20570) 🚀 Performance improvements Extend functionality on BitmapBuilder and use in Growables (#20754) Specialize first/last agg fo...

0100

marcogorelli.bsky.social @marcogorelli.bsky.social · 09/01/2025

✨New Narwhals cover image 🧑‍🍳Can you guess what's cooking?

A panda, polar bear, narwhal, and duck in an office

270

marcogorelli.bsky.social @marcogorelli.bsky.social · 06/01/2025

✨ New Pyodide release (0.27)! 🤗 With support for 🐻‍❄️Polars and 🌊🦄Narwhals! 💡 So let's take it for a spin and write some dataframe-agnostic code!

Dataframe-agnostic aggregation which supports pandas, Polars, and more

121

marcogorelli.bsky.social @marcogorelli.bsky.social · 20/12/2024

✨ New temporal feature in the next Polars release! ⏲️ dt.replace lets you replace components of Date / Datetime columns ⚡🦀 It's an expressified vectorised rustified version of the Python standard library datetime.replace

091

marcogorelli.bsky.social @marcogorelli.bsky.social · 17/12/2024

🦀 10 minutes to live-coding in Rust from LunarVim 🥐 French pastries and Geoguessr meta 🔁 "Le Verlan", and my revenge on the French for having invented it 🎥 My @pydataparis.bsky.social video is now online!

youtu.be

Marco Gorelli - Polars Plugins: how you (yes, you!) can extend Polars Dataframes | PyData Paris 2024

YouTube video by PyData

041

marcogorelli.bsky.social @marcogorelli.bsky.social · 14/12/2024

🌊🦄 Narwhals featured in Tryolabs's top 10 new AI/ML/Data tools 2024 list! 🥳 Even more pleasing is that 2 of the other packages featured (Marimo and Rio) use Narwhals under-the-hood 🛠️ Given that Narwhals is a tool for tool-builders, this fills us with joy!

tryolabs.com

Top Python libraries of 2024

Dive into our 10th annual Python Libraries roundup for 2024, now featuring separate curated lists for General Use and AI / ML / Data tools. Discover this year's most innovative additions to the ecosys...

162