iiinova — where your digital self keeps evolving

And it's only fitting that we'll be hosting this event a true lakehouse, the Lake Chalet, the best waterfront restaurant on Lake Merritt, steps from the Data Council main event. RSVP here while tickets last: www.rilldata.com/events/data-...

Real-time Roundtable Live from Data Council | Rill Data

010

With $21.8M in funding, Tobiko aims to build a modern data platform | TechCrunch

Similarly, Toby and his team at Tobiko Data have built an powerful yet elegant transformation platform -- combining SQLMesh and SQL dialect transpilation (SQLGlot) to allow portability of pipelines between databases, warehouses, and lakehouses. techcrunch.com/2024/06/05/w...

techcrunch.com

Tobiko aims to reimagine how teams work with data by offering a dbt-compatible data transformation platform.

The sub-second speed-at-scale of these real-time engines enable new kinds of applications: point-of-sale fraud detection, IoT monitoring, real-time context for AI agents -- these use cases just aren't supported by traditional data warehouses like Snowflake. www.rilldata.com/blog/scaling...

Rill | Scaling Beyond Postgres: How to Choose a Real-Time Analytical Database

This blog explores how real-time databases address critical analytical requirements. We highlight the differences between cloud data warehouses like Snowflake and BigQuery, legacy OLAP databases like ...

100

Why am I so excited to bring this crew together on stage? It's because real-time analytical databases like ClickHouse, Apache Pinot, and MotherDuck / DuckDB are reshaping data stacks the fastest-moving engineering teams on earth -- OpenAI, DoorDash, and @stackblitz.com.

This legendary panel of technical founders includes Yury Izrailevsky (co-founder of ClickHouse), Kishore Gopalakrishna (founder of StarTree, creator of Apache Pinot), @jrdntgn.bsky.social (co-founder of MotherDuck), and @captaintobs.bsky.social (founder of Tobiko, creators of SQLMesh and SQLGlot).

121

Yo SF Bay Area #databs crew, want to talk lakehouses at a real Lake House? :) Next week after Data Council, join the founders of @clickhouse.com, @motherduck.com, @startreedata.bsky.social, and @tobikodata.com to talk real-time databases and next-generation ETL. www.rilldata.com/events/data-...

1103

If you're interested in learning more about the "Shift Left" trend, please check out data engineering author @ssp.sh's blog post released today. www.rilldata.com/blog/what-sh...

By shifting left, data teams can create more maintainable, performant, and reliable data systems while reducing duplication and inconsistency throughout the data stack.

Rill | What

120

DuckCon 6 - A SQL-Based Metrics Layer Powered by DuckDB

At @rilldata.com, we've taken the step of shifting metric layers left from BI tools and pushing them into real-time analytical databases like ClickHouse and DuckDB -- to power insanely fast exploratory dashboards. (I'll be discussing at my Data Council in two weeks). docs.google.com/presentation...

docs.google.com

INTRODUCING A SQL-BASED METRICS LAYER POWERED BY DUCKDB Mike Driscoll Co-Founder, CEO at Rill Data

261

Now that SQL-on-data-lake frameworks are maturing (DuckDB SQL on Iceberg, Spark SQL on DeltaLake), and transpiling between SQL dialects is possible (thanks to SQLMesh and @tobikodata.com), it's possible to shift these SQL transformations left, out of the warehouse and onto object storage.

The advantage was that transformations could be written in SQL. The disadvantage is you pay the Snowflake tax for every compute cycle in their warehouse.

120

Transformation logic is another use case. "Shifting left" is in some ways a reaction to the "ELT" pattern (or anti-pattern, in my opinion) that big data warehouses like Snowflake were pushing -- whereby you extract, load, and only *then* transform data in the warehouse.

210

Data validation is a great example: an eCommerce platform might validate that order prices contain no negative numbers after its loaded into the database. "Shifting left" means moving that validation to the ingestion or even the collection step in the pipeline, before it hits the database.

120

Data pipelines can be visualized as flowing data left to right, starting with raw sources, ingested and modeled into database tables, and eventually served out through user-facing applications and dashboards. "Shifting left" means taking logic that lives on the right side and moving it leftward.

Mike Driscoll @medriscoll.com · 04/04/2025

"Shifting left" is the new trend among in data stacks -- but what does it mean and what does it matter?

251

Apache Pinot is one of the world’s fastest and most scalable real-time analytical databases, relied on by LinkedIn, Uber, and Stripe. It was awesome diving into the secrets behind its unique architecture with creator and @startreedata.bsky.social founder Kishore Gopalakrishna.

0101

Mike Driscoll @medriscoll.com · 15/03/2025

I wish I could say "yes almost certainly" but if the levels of competence we're witnessing in other areas I'm not placing any bets on DOGE's data security practices.

010

So what are DOGE's true priorities? As Maya Angelou wrote: "When someone shows you who they are, believe them the first time."

041

Cloud data centers have climate control, and more compute power than your MacBook Air! This set up could be done by competent data engineer in less time than it took to run her query. The DOGE tech wiz acknowledged this and wrote "it hasn't been a priority to get that done."

131

They should haved loaded this multi-terabytes contracts data set into a cloud database, or even better -- a database built for real-time analytics like @clickhouse.com, Pinot, or StarRocks (sorry @duckdb.org, this is more than you can handle).

180

But it doesn't absolve her, or her team, from ridicule. The DOGE tech wiz kids shouldn't be toting federal databases around on USB-attached external hard drives in "hot, humid hotel rooms" (her literal words): that's what database servers were invented for.

160

What actually overheated was an USB external hard drive, with several terabytes of contract data, that she was reading into her MacBook Air and then filtering to find contracts matching her criteria. (High-speed reads on NVMe drives can heat up to 175 °F before thermal throttling kicks in).

240

Simon Späti 🏔️ @ssp.sh · 11/03/2025

Like others, I jumped on the bandwagon to ridicule the DOGE analyst who "overheated her hard drive" by analyzing just 60k rows of data. I was wrong. The truth is even dumber. 🧵

382

Reposted by Mike Driscoll

Just published: Ever had to «Scale beyond Postgres»? You may have started with a simple ETL pipeline and crunched critical business logic into useful dashboards, but speed and scale didn't grow with data at some point, and it's the concurrent user. ✨ Below are some highlights from the article.

Rill | Scaling Beyond Postgres: How to Choose a Real-Time Analytical Database

143

Mike Driscoll @medriscoll.com · 04/03/2025

How mature is DuckDB WASM these days? I recall reading on HN about a similar app last year, called Pretzel, but I think they pivoted: news.ycombinator.com/item?id=3971...

news.ycombinator.com

Show HN: Open-source, browser-local data exploration using DuckDB-WASM and PRQL | Hacker News

330

Reposted by Mike Driscoll

Julian Hyde @julianhyde.bsky.social · 03/03/2025

The father of relational databases understood semantic data models.

This activity is sometimes called semantic data modeling. Actually, the task of capturing the meaning of data is a never-ending one. So the label “semantic” must not be interpreted in any absolute sense. – E.F. Codd, 1979

1151

Reposted by Mike Driscoll

rmoff 🏃‍♂️🫖🥓 @rmoff.net · 28/02/2025

Blogged: Exploring UK Environment Agency data with @duckdb.org and @rilldata.com rmoff.net/2025/02/28/e... #dataBS

0207

Mike Driscoll @medriscoll.com · 05/02/2025

Google couldn’t create a competitive product because they profit directly from that ad spam and indirectly from data selling. (Same reason Gmail doesn’t really want to clean up your Inbox, even though they could.)

010

Mike Driscoll @medriscoll.com · 31/01/2025

DuckCon #6 is now live from Amsterdam (link below!) In about 20 minutes I'll be sharing some work we've been doing at @rilldata.com on metrics-layer-powered dashboards. And speculating in my last slide, because why not, about what you would name an AI agent that runs on DuckDB... 🦆

172

Mike Driscoll @medriscoll.com · 29/01/2025

“Pivot tables are just a shorter, fatter GROUP BY”. ❤️ this by @gregat.es

161

Reposted by Mike Driscoll

DuckDB @duckdb.org · 24/01/2025

DuckCon #6 will start in 168 hours (January 31, 15:00)! If you plan to attend in-person, please register at duckdb.org/2025/01/31/d... The stream will be available without registration.

0227

Mike Driscoll @medriscoll.com · 24/01/2025

And compacts them periodically?

100

Mike Driscoll @medriscoll.com · 23/01/2025

#databs welcomes the creator of Pandas, founder of Datapad & Voltron, and all around nice guy Wes McKinney @wesmckinney.com to BlueSky.

0263

Mike Driscoll @medriscoll.com · 16/01/2025

Just sent you a note over there!

000

Mike Driscoll @medriscoll.com · 16/01/2025

I'm collaborating with @ssp.sh on a brief history of pivot tables. We'll be tracing their lineage and evolution across Visicalc, Lotus, Excel, PowerPivot, Qlik, and PowerBI. Any sites, videos, products (dead or alive) that you would recommend we should mention or dig into?

6221

Mike Driscoll @medriscoll.com · 16/01/2025

Real-time user experiences make applications magical. Google, WhatsApp, and ChatGPT would all fail if you added just 10 seconds to every search, message, or prompt interaction. We are surrounded by fast user experiences and yet most of us tolerate business dashboards that take minutes to load.

081

Mike Driscoll @medriscoll.com · 18/12/2024

The puritanical mob replaced “quaffing” with “singing”?? Shameful.

040

Reposted by Mike Driscoll

Burak @buremba.bsky.social · 15/12/2024

It took 5 minutes to create this dashboard with @rilldata.com's AI auto-generate features. Impressive @medriscoll.com!

0133

Mike Driscoll @medriscoll.com · 12/12/2024

@vasek.bsky.social Hit me up if you have any issues, you should be able to build a dashboard running on top of DuckDB locally in a few minutes. docs.rilldata.com

docs.rilldata.com

Let's get started! | Rill

Use the following to download Rill and start your first project, my-rill-project.