Sign in

Stephan Ewen

@stephanewen.bsky.social
1K followers 490 following 28 posts

Building distributed systems and data infra. Previously co-creator of Apache Flink (flink.apache.org), now building Restate (restate.dev) to make distributed apps more easily resilient and scalable.

PostsRepliesMedia
Reposted by Stephan Ewen
slinkydeveloper.bsky.social @slinkydeveloper.bsky.social · 07/04/2025
Coming up in @restate.dev 1.3: Implement scatter/gather scenarios with ease, while keeping your application resilient to failures! Those sub tasks can fail, the orchestrator itself can fail, and yet this code will run to completion thanks to Restate "run to completion" semantics.
Restate service handler written in Kotlin, showing how to implement a scatter/gather use case with few lines of code in a resilient way.
0103
Stephan Ewen @stephanewen.bsky.social · 07/04/2025
That is a good part of the motivation why we created Restate. We built Apache Flink before and even tried to use it for transactional use cases. Restate is the transactional event-processing twin to stream processing. With an API that looks like durable functions/execution/state/rpc.
180
Reposted by Stephan Ewen
Luke @lukehedger.dev · 26/03/2025
Exploring @restate.dev as a replacement for some long-running, complex, asynchronous orchestrators currently implemented in Step Functions. This 2023 post resonates hard. Lambda chain -> Step Functions is an increasingly common migration but comes up short vs RPC/ECS restate.dev/blog/we-repl...
192
Stephan Ewen @stephanewen.bsky.social · 07/04/2025
To visualize the state machine across services, you could export OTEL traces and use a tool like Jaeger to get the dependency diagram. Future versions of the UI will have a timeline view, showing when what step was executed. That is not a state machine, but still helpful. Is any of that helpful?
140
Stephan Ewen @stephanewen.bsky.social · 07/04/2025
To develop with a State Machine, you can use XState with Restate. See here for details: e.g., using it with XState. Is that restate.dev/blog/persist... 👇
restate.dev
Persistent serverless state machines with XState and Restate
Run state machines in the cloud, and suspend them when there's no work to do.
100
Stephan Ewen @stephanewen.bsky.social · 07/04/2025
True, not having an explicit state machine is a pity, but also somewhat a feature: It works with implicit state machines and dynamic control flow, e.g, as a backend for AI Agents. But there are ways to get state machines: 👇
110
Stephan Ewen @stephanewen.bsky.social · 25/03/2025
Wow, what a blast from the past. Thank you for bringing that back 🙂 This is from a talk "All roads lead to Rome" - hence the helmets.
060
Stephan Ewen @stephanewen.bsky.social · 13/03/2025
Is this where we currently are?
052
Stephan Ewen @stephanewen.bsky.social · 12/03/2025
Thank you so much for the kind words!
010
Reposted by Stephan Ewen
Kevin Webb @kevinwebb.io · 18/02/2025
Restate is one of the coolest pieces of software I've encountered in quite awhile. Not often a tiny, well-crafted application comes along that forces you to fundamentally rethink software design.
restate.dev
Restate 1.2: a distributed durable execution engine, built from first principles
Restate 1.2 adds highly-available, distributed deployments and a graphical UI.
193
Reposted by Stephan Ewen
Chris @chris.blue · 27/02/2025
This post from @stephanewen.bsky.social is *really* thought provoking. Their architecture seems to unify both database and stream processing architecture into one thing. I'm not totally sure what the implications of this are yet, but it seems important.
restate.dev
The Anatomy of a Durable Execution Stack from First Principles
The architecture of Restate, a Durable Execution engine built from the ground up.
1172
Stephan Ewen @stephanewen.bsky.social · 27/01/2025
It has been quite a journey from Stateful Functions to Restate - and while Restate can do a lot more, the Stateful Functions roots are quite visible, specifically in the Virtual Objects Api.
000
Stephan Ewen @stephanewen.bsky.social · 27/01/2025
Thank you so much for the kind words!
010
Stephan Ewen @stephanewen.bsky.social · 25/01/2025
But I would agree that BookKeeper is an underrated piece and a hidden gem of distributed systems design.
030
Stephan Ewen @stephanewen.bsky.social · 25/01/2025
One reason why Restate mentions Delos frequently is because our lead runtime architect worked with Delos before and adopted the vocabulary. Plus Restate internally has the heterogeneous loglet chain abstraction.
120
Stephan Ewen @stephanewen.bsky.social · 25/01/2025
Totally - I learned a ton from the BookKeeper design and your blog posts and there is a lot of similarity to Delos. We discussed this a lot internally.
100
Reposted by Stephan Ewen
Jack Vanlightly @vanlightly.bsky.social · 25/01/2025
Regarding Restate and its distributed log, many people talk about Delos but Apache BookKeeper is also highly relevant/similar, so I like to remind people that it also exists! I've written extensively about how BookKeeper works if you're interested: 1\ medium.com/splunk-maas/...
medium.com
Apache BookKeeper Insights Part 1 — External Consensus and Dynamic Membership
Series Introduction
2123
Stephan Ewen @stephanewen.bsky.social · 25/01/2025
That is exactly right - what would be a better term? There is some simplification of the coordination happening here, because it centralizes it - like updating the state under a lock.
110
Stephan Ewen @stephanewen.bsky.social · 25/01/2025
That's a fair comment - maybe the better way to phrase this is replacing different forms of coordination with multiple independent systems, with one type of coordination, which can be also pretty transparently abstracted in the application, so you keep coordination out of your code. Would that work?
120
Stephan Ewen @stephanewen.bsky.social · 25/01/2025
I guess the headline is brief - I would phrase it as every system is a log and and interpreter of events stored in the log (in the broadest sense).
000
Stephan Ewen @stephanewen.bsky.social · 24/01/2025
Maybe the biggest one is the per-key granularity in Restate, and what is conceptually conditional appends (only append if no newer retry subsumed event handler). This can be built on top of the log (in Restate its above the physical log partition layer), but more efficient if tightly integrated.
140
Stephan Ewen @stephanewen.bsky.social · 24/01/2025
I have this slide from a presentation at Kafka Summit / Current about that.
131
Reposted by Stephan Ewen
Jack Vanlightly @vanlightly.bsky.social · 24/01/2025
Great stuff. I'm watching the durable execution space closely and personally I'm quite bullish on it. I'll be writing my own thoughts on durable execution soon.
1233
Reposted by Stephan Ewen
Stephan Ewen @stephanewen.bsky.social · 24/01/2025
𝐄𝐯𝐞𝐫𝐲 𝐒𝐲𝐬𝐭𝐞𝐦 𝐢𝐬 𝐚 𝐋𝐨𝐠 Here is an idea on how to drastically 𝐫𝐞𝐝𝐮𝐜𝐞 𝐜𝐨𝐦𝐩𝐥𝐞𝐱𝐢𝐭𝐲 𝐚𝐧𝐝 𝐜𝐨𝐨𝐫𝐝𝐢𝐧𝐚𝐭𝐢𝐨𝐧 in distributed apps. A fun way to look at this is similar to the idea of Turning the Database Inside Out - like Turning the Microservice Inside Out. restate.dev/blog/every-s... 👇
restate.dev
Every System is a Log: Avoiding coordination in distributed applications
Distributed coordination makes application complex and brittle. Because all systems eventually build on logs, we can use a shared-log approach to eliminate most coordination. We discuss this conceptua...
45313
Stephan Ewen @stephanewen.bsky.social · 24/01/2025
But how is this practically usable? Our DBs and queues aren't built like this. How do we strike a sweet-spot between this model (great consistency) and maintaining healthy decoupling and separation of concerns? The blog post discusses that and how it inspired our work on @restatedev.bsky.social
010
Stephan Ewen @stephanewen.bsky.social · 24/01/2025
Applications coordinate with many systems (DBs, queues, locks, schedulers, ...) which are all logs underneath. Each maintains its own ground truth, without common order or conditions. By letting all those logs build on top of a common log, many dist. sys. problems in apps virtually disappear. 👇
110
Stephan Ewen @stephanewen.bsky.social · 24/01/2025
𝐄𝐯𝐞𝐫𝐲 𝐒𝐲𝐬𝐭𝐞𝐦 𝐢𝐬 𝐚 𝐋𝐨𝐠 Here is an idea on how to drastically 𝐫𝐞𝐝𝐮𝐜𝐞 𝐜𝐨𝐦𝐩𝐥𝐞𝐱𝐢𝐭𝐲 𝐚𝐧𝐝 𝐜𝐨𝐨𝐫𝐝𝐢𝐧𝐚𝐭𝐢𝐨𝐧 in distributed apps. A fun way to look at this is similar to the idea of Turning the Database Inside Out - like Turning the Microservice Inside Out. restate.dev/blog/every-s... 👇
restate.dev
Every System is a Log: Avoiding coordination in distributed applications
Distributed coordination makes application complex and brittle. Because all systems eventually build on logs, we can use a shared-log approach to eliminate most coordination. We discuss this conceptua...
45313
Stephan Ewen @stephanewen.bsky.social · 23/01/2025
This is why we are building software and open-source communities. To get such messages from users, the best possible reward. Kudos to the entire @restatedev.bsky.social team.
030
Stephan Ewen @stephanewen.bsky.social · 06/12/2024
My first post on @bsky.app - let's go!
0110
Stephan Ewen @stephanewen.bsky.social · 06/12/2024
- flexible latency-optimized log - a sql query engine over that log We'll unpack that for the next release in detail...
000
Stephan Ewen @stephanewen.bsky.social · 06/12/2024
Restate implements an elaborate stack🥞 in which all parts are carefully crafted to support each other: - multiplexing state, communication, orchestration, scheduling through the same log - deterministic event-based architecture - borrowed consensus 👇
100
Stephan Ewen @stephanewen.bsky.social · 06/12/2024
First peek at distributed @restatedev.bsky.social: a database for durable execution. Distributed setups scale, improve failover, and trade the dependency on persistent disk (eg EBS) with an S3 dependency. Restate replicates events internally to bridge S3's latency gap. 👇 restate.dev/blog/distrib...
restate.dev
Distributed Restate - A first look
A first look at distributed highly-available deployments of Restate and the architecture that supports those. We demonstate this through a geo-distributed active/active setup executing an order-proces...
1328
Reposted by Stephan Ewen
Christoph Molnar @christophmolnar.bsky.social · 27/11/2024
The unofficial GIF-based pandas library documentation. pandas.DataFrame.rolling
media.tenor.com
a panda bear is laying down in the grass .
Alt: a panda bear is rolling down in the grass. It's a side-ways roll, hlding some type of object. I give 10/10.
3546
Stephan Ewen @stephanewen.bsky.social · 27/11/2024
Finally started using this account for real, nice to be here!
081
Stephan Ewen @stephanewen.bsky.social · 27/11/2024
But in all seriousness, looks like a slick product at first glance. Kudos to the team!
010
Stephan Ewen @stephanewen.bsky.social · 27/11/2024
This made me laugh and feel humbled. Being in the same sentence as Currywurst and the Berghain. Life goal achieved!
120