iiinova — where your digital self keeps evolving

Based on Timely 0.24, we released Differential Dataflow 0.17. Compatibility changes plus removing the Filter and Freeze wrappers because they weren't used and difficult to maintain. github.com/TimelyDatafl...

Release differential-dataflow-v0.17.0 · TimelyDataflow/differential-dataflow

Other Adjust Differential to recent Timely changes (#643) Remove Filter and Freeze wrappers (#644)

000

We also added a new way to chain multiple flat-map calls into a single Timely operator. Coming from Rust's iterator API, it's sometimes not obvious that Timely's API has a different cost structure (each transform renders an operator). Here, a builder can mitigate some of the cost.

Introduce builder for flatmap operators by frankmcsherry · Pull Request #704 · TimelyDataflow/timely-dataflow

We discussed this a year back, but I couldn't recall why we didn't move it forward. I copy/pasted the code, and dressed up the documentation and examples. Worth taking a peek!

000

I think these change are a great example of taking existing APIs and moving from row-first to batch-of-data. Computers generally seem better at handling data by column rather than row, and these changes allow Timely programs to represent data in their own form.

100

A Distributor trait takes the concepts of Exchange, but moves away from row-by-row processing, opening up new kinds of containers that aren't row-first. For example, what is a row in a trie-structured container? The new trait allows to partition by whatever the user wants, not just by a "row".

Distributor trait by antiguru · Pull Request #700 · TimelyDataflow/timely-dataflow

Introduce a distributor that knows how to partition containers across multiple pushers. This moves the partition logic from container builders into a bespoke trait and implementation instead of mix...

100

Previously, the Container trait described behavior that is now separate: Accountable reveals a container's record count for progress tracking; nothing else. For higher-level APIs, IterContainer and DrainContainer reveal their contents (non-)destructively, for container-agnostic row-by-row operators.

Rework container abstractions by antiguru · Pull Request #697 · TimelyDataflow/timely-dataflow

This is another attempt at reworking the container abstractions in Timely. At the moment, we have two types that a lot of the container abstractions hinge on, Container and ContainerBuilder. Both d...

100

We just released Timely Dataflow 0.24! Here are some exciting changes from @frankmcsherry.bsky.social and myself. The container abstractions got a complete rework, and we introduce a new pattern to distribute data. Details below. github.com/TimelyDatafl...

Release timely-v0.24.0 · TimelyDataflow/timely-dataflow

This version of Timely has some exciting new features. The Distributor trait offers a generalization of the Exchange type. It allows users to define custom distribution strategies for routing data...

151

Moritz Hoffmann @antiguru.bsky.social · 25/06/2025

A lot of performance improvements went into this one!

020

Reposted by Moritz Hoffmann

Materialize @materialize.com · 15/05/2025

The Materialize engineering team uncovered a rare concurrency bug 🪲in Rust’s 🦀 unbounded channels that could lead to double-free memory errors. After thorough debugging and working with the Rust and crossbeam communities, the fix is now part of @rust-lang.org 1.87.0. 🔗 bit.ly/3Fan1Om

021

Reposted by Moritz Hoffmann

Nate Stewart @nate-stewart.bsky.social · 11/04/2025

Agents generate more data and place more demand on systems than ever before—and standards like MCP will only accelerate this trend. Learn how Delphi is rethinking how they build data-intensive applications—from the db to the UI: bit.ly/42BZdf9

bit.ly

Scaling queries on agent-produced data: How Delphi transformed its data infrastructure

Join our webinar with Delphi to discover they evolved their data infrastructure to handle the rapid increase in AI-driven interactions with Materialize.

022

Moritz Hoffmann @antiguru.bsky.social · 11/03/2025

Jan '24: 1,338 kWh. That was a newly-built apartment in NYC. I don't miss this part!

010

Reposted by Moritz Hoffmann

Frank McSherry @frankmcsherry.bsky.social · 22/01/2025

For interested folks, the RustNYC presentation I gave about the `columnar` crate is now available in the repo: github.com/frankmcsherr... It's not as good without the talk track, but it's got the benchmark results, you can admire the stylistic watercolors (so therapeutic to make; strong recommend).

Moritz Hoffmann @antiguru.bsky.social · 15/01/2025

011

I wrote a bit about efficient maintenance of temporal filters in @materialize.com. By periodically restarting replicas, we can limit resource requirements (RAM and CPU), all without any other user-visible impact. materialize.com/blog/replica...

materialize.com

Replica expiration: Limiting temporal filters' resource requirements

Replica expiration is a new feature in Materialize that limits the resource requirements of temporal filters.

012

Moritz Hoffmann @antiguru.bsky.social · 29/12/2024

An ocelot with an odd shape at the Museum für Naturkunde in Berlin.

010

Reposted by Moritz Hoffmann

Frank McSherry @frankmcsherry.bsky.social · 23/12/2024

I was recently coveted a 10 year journal on jetpens, because that would be cool to do, and realized that I already had one! The blog turns 10 this month, and I thought it might be worth taking a tour through each of the years and seeing what I can jot down about each. github.com/frankmcsherr...

Moritz Hoffmann @antiguru.bsky.social · 28/11/2024

332

It'll certainly be an opt-in feature in its current form. Forcing a certain deployment pattern will be hard. Otoh, we could automate graceful replica restarts!

010

Sonne additional details by @frankmcsherry.bsky.social bsky.app/profile/fran...

000

So, we didn't solve the underlying problem, but mitigated it for many of our workloads.

001

Now, in Materialize time advanced about every second, and the result is high CPU utilization 😿 But: we restart most things every week, so updates beyond a week will never be surfaced! The mitigation we implemented is to drop such updates, and add some safeguards to never produce incorrect results. 🎉

100

Future updates are stashed at the input to indexes, and they'll only act on them once the time has advanced far enough. Because we use Differential Dataflows, the updates are only partially ordered. To extract ready updates, DD scans the data and splits it in ready and pending, ~ in linear time.

100

Improved performance for temporal filters

Sometimes it's easier to mitigate a problem than solve it: a feature I recently implemented reduces the memory and CPU @materialize.com requires to maintain future updates. Some details why solving the problem is hard and the mitigation easy-ish. materialize.com/changelog/20...

materialize.com

Optimization time! We've significantly reduced the cost of using temporal filters in Materialize.

252