iiinova — where your digital self keeps evolving

iiinova is part of the Atmosphere — an open, public network where you own your identity.

Jepsen

@jepsen.mastodon.jepsen.io.ap.brid.gy

2.1K followers 1 following 59 posts

Breaking distributed systems, one fault at a time. [bridged from mastodon.jepsen.io/@jepsen on the fediverse by fed.brid.gy ]

Posts Replies Media

Jepsen @jepsen.mastodon.jepsen.io.ap.brid.gy · 22/09/2026

Jepsen 0.3.14 is now available, including several new generators, a slew of bugfixes and performance improvements especially in generators, and some new features for the web interface. github.com/jepsen-io/jepsen/release…

Release v0.3.14 · jepsen-io/jepsen

This release focuses on generator performance and correctness, and introduces several new generators. It also improves the web interface and fixes some long-standing issues with time zones. Bugfixe...

Jepsen @jepsen.mastodon.jepsen.io.ap.brid.gy · 19/08/2026

Heads-up for Elle users--I believe there's another bug in the `rw` workload which causes `linearizable-keys?` and friends to incorrectly report cyclic version errors for versions which are not in fact cyclic. Working on that now.

Jepsen @jepsen.mastodon.jepsen.io.ap.brid.gy · 31/07/2026

A new #Jepsen release to support caching expensive generators and final reads for cycle-detection tests: github.com/jepsen-io/jepsen/release…

Release 0.3.13 · jepsen-io/jepsen

This is a small release. It makes expensive generators more efficient with a cache, allows rw and append tests a final chance to detect data loss at the end of tests, and lets file snapshot nemeses...

Jepsen @jepsen.mastodon.jepsen.io.ap.brid.gy · 20/07/2026

Delighted to announce the release of #Jepsen 0.3.12, now available on Clojars and GitHub, as usual. github.com/jepsen-io/jepsen/release…

Release 0.3.12 · jepsen-io/jepsen

A big release! This fixes several deadlocks and a safety error in Elle's inference of version orders for rw workloads which caused tests to incorrectly report that version orders were cyclic. It al...

Jepsen @jepsen.mastodon.jepsen.io.ap.brid.gy · 14/07/2026

Could someone please poke the AWS Marketplace support team for me? There's a bug in the add instances workflow, they no longer accept emails, and the web form they insist on using for filing tickets only accepts 1000 characters and seems to have an 80% chance of routing directly to /dev/null.

Jepsen @jepsen.mastodon.jepsen.io.ap.brid.gy · 29/06/2026

Percona also cites five issues in different programs where users complain about the new error, which suggests that those programs may have been losing data due to improper SI. Some users have responded by turning off the safety feature entirely: github.com/fisharebest/webtrees/iss…

WT2.2.1 SQLSTATE[HY000]: General error: 1020 Record has changed since last read in table / hitcounter · Issue #5177 · fisharebest/webtrees

I use PageSpeed Insights from time to time. it starts 2 requests in parallel (mobile / computer). The 2nd (computer) request fails regularly recently with error 500 SQLSTATE[HY000]: General error: ...

Jepsen @jepsen.mastodon.jepsen.io.ap.brid.gy · 29/06/2026

Missed this post on MariaDB's Snapshot Isolation implementation from Percona: www.percona.com/blog/mariadbs-snaps… In short, instead of allowing transactions to silently lose updates, MariaDB now (mostly) returns an error so the […]

mastodon.jepsen.io

Original post on mastodon.jepsen.io

Jepsen @jepsen.mastodon.jepsen.io.ap.brid.gy · 06/06/2026

AWS changed a few things about CloudFormation that broke the Jepsen cluster product. Version 0.3.11.2 should work now: github.com/jepsen-io/jepsen#aws

Jepsen @jepsen.mastodon.jepsen.io.ap.brid.gy · 26/03/2026

If you want to chat about checking predicate safety, come chat at BugBash 2026. April 22-24, DC: antithesis.com/bugbash/conference20…

A conference on extracting reliable software from the slop factory

Jepsen @jepsen.mastodon.jepsen.io.ap.brid.gy · 17/03/2026

A new #Jepsen report, on MariaDB Galera Cluster 12.1.2. Two cases of write loss, along with P4 (Lost Update) and Stale Read in healthy clusters. jepsen.io/analyses/mariadb-galera-c…

Jepsen: MariaDB Galera Cluster 12.1.2

Jepsen @jepsen.mastodon.jepsen.io.ap.brid.gy · 10/03/2026

Jepsen 0.3.11 is now available on Clojars. This is a fairly small release, but it does update Jepsen to work with DB nodes running Debian Trixie. github.com/jepsen-io/jepsen/release…

Release 0.3.11 · jepsen-io/jepsen

This is a fairly minor release. It updates Jepsen to work with Debian Trixie DB nodes, fixes a few minor bugs, and introduces some very small quality-of-life improvements, along with the usual slew...

Jepsen @jepsen.mastodon.jepsen.io.ap.brid.gy · 09/12/2025

In the vein of research.cs.wisc.edu/adsl/Publicati…, Lee Brotherston writes that NATS sometimes neglects to check whether calls to `fsync` actually succeeded: github.com/nats-io/nats-server/issu…

Filesystem sync errors unchecked · Issue #7629 · nats-io/nats-server

Observed behavior There are multiple places in server/filestore.go where the file .Sync() call is used: https://github.com/search?q=repo%3Anats-io%2Fnats-server%20%22.Sync()%22%20path%3Aserver%2Ffi...

Jepsen @jepsen.mastodon.jepsen.io.ap.brid.gy · 08/12/2025

A new #Jepsen report: we demonstrate data loss and persistent split-brain in the NATS streaming system, in response to simulated power failures/OS crashes. jepsen.io/analyses/nats-2.12.1

Jepsen: NATS 2.12.1

Jepsen @jepsen.mastodon.jepsen.io.ap.brid.gy · 02/12/2025

Jepsen and Antithesis worked together to write a glossary for anyone building, testing, and operating distributed systems. It covers the basics of concurrency, consistency models and phenomena, faults, and some testing approaches: antithesis.com/resources/reliabilit… […]

mastodon.jepsen.io

Original post on mastodon.jepsen.io

Jepsen @jepsen.mastodon.jepsen.io.ap.brid.gy · 02/12/2025

A new #Jepsen release, 0.3.10, brings improved support for controllable random value generation, and running tests inside Antithesis. Jepsen's composable generator system has also been extracted to a minimal library, making it easier to re-use in other systems […]

mastodon.jepsen.io

Original post on mastodon.jepsen.io

Jepsen @jepsen.mastodon.jepsen.io.ap.brid.gy · 18/08/2025

The latest Jepsen talk, from Systems Distributed in June, goes live in 15 minutes. We'll be doing a live chat during the premier, if you want to chat about databases and testing. :-) www.youtube.com/watch?v=dpTxWePmW5Y

Jepsen @jepsen.mastodon.jepsen.io.ap.brid.gy · 07/08/2025

A new #Jepsen report! We tested early builds of Capela, an unreleased distributed programming environment, and found twenty-two issues, including four language problems, fourteen crashes or non-fatal panics, performance degradation, and three safety issues including lost update […]

mastodon.jepsen.io

Original post on mastodon.jepsen.io

Jepsen @jepsen.mastodon.jepsen.io.ap.brid.gy · 06/08/2025

An interview with Kaivalya Apte, on The GeekNarrator podcast. We talk about mapping properties to tests, type I and II errors, performance, LLMs, and more. www.youtube.com/watch?v=IvE1VbOol88

Jepsen @jepsen.mastodon.jepsen.io.ap.brid.gy · 05/08/2025

The video of my BugBash talk, "Jepsen 17: ACID Jazz" is out now! www.youtube.com/watch?v=v8cG2hh10SQ

Jepsen @jepsen.mastodon.jepsen.io.ap.brid.gy · 15/07/2025

Antithesis and Jepsen are releasing a glossary of terms useful in distributed systems testing: antithesis.com/resources/reliabilit…

A distributed systems reliability glossary

A list of key concepts for building and testing reliable distributed systems, with basic definitions and deep references.

Jepsen @jepsen.mastodon.jepsen.io.ap.brid.gy · 07/06/2025

Systems Distributed. June 19-20, Amsterdam. systemsdistributed.com

A parody of John Waters' "Serial Mom", except it's "Serializable Mom". I'm holding scissors (to partition the network) and trying to channel my best homage to Kathleen Turner.

Jepsen @jepsen.mastodon.jepsen.io.ap.brid.gy · 06/06/2025

The companion blog post from TigerBeetle is great too--dives into detail on the bug we found in the index-intersection query code: tigerbeetle.com/blog/2025-06-06-fuz…

Two bugs sitting on cozy chairs, sewing. The Tiger Beetle bug is gesturing "No! Stop!" to the other bug, who is sewing a shirt all wrong. I think the other bug is Jepsen. :D

Jepsen @jepsen.mastodon.jepsen.io.ap.brid.gy · 06/06/2025

A new #Jepsen report! We worked with TigerBeetle to find seven crashes, elevated latencies during single-node failures, and requests which were retried forever in version 0.16.11. We found only two safety issues: missing results for queries with multiple predicates, and incorrect timestamps in a […]

mastodon.jepsen.io

Original post on mastodon.jepsen.io

Jepsen @jepsen.mastodon.jepsen.io.ap.brid.gy · 07/05/2025

Jepsen 0.3.9 is now available, including a module for restarting flaky databases, a few improvements to downloading logs, more capable generators, and friendlier error messages. github.com/jepsen-io/jepsen/release…

Release 0.3.9 · jepsen-io/jepsen

A medium-sized release, this version makes several quality-of-life improvements. We (finally!) download log files only once, rather than twice at the end of a run; this should make multi-GB log dow...

Jepsen @jepsen.mastodon.jepsen.io.ap.brid.gy · 06/05/2025

Justin Jaffray has written up a lovely companion to this piece, diving into Snapshot Isolation and how to understand transaction dependency structures. buttondown.com/jaffray/archive/how-…

How To Understand That Jepsen Report

There is a new Jepsen report which, as always, is an entertaining read and gives some insight into the way things might be broken in subtle ways. I learned...

Jepsen @jepsen.mastodon.jepsen.io.ap.brid.gy · 03/05/2025

Thanks to AWS's Sergey Melnik, as well as HN commenters matashii and Ants Aasma, we now know that Long Fork in PostgreSQL clusters is caused by a disagreement between primaries and secondaries on the order in which transactions become visible […]

mastodon.jepsen.io

Original post on mastodon.jepsen.io

Jepsen @jepsen.mastodon.jepsen.io.ap.brid.gy · 29/04/2025

A small issue in Amazon RDS for PostgreSQL: at the "Repeatable Read” isolation level, which in PostgreSQL normally means Snapshot Isolation, Amazon RDS for PostgreSQL clusters appear to exhibit Long Fork. We observed this behavior in healthy clusters, in versions ranging from 13.15 to 17.4 […]

mastodon.jepsen.io

Original post on mastodon.jepsen.io

Jepsen @jepsen.mastodon.jepsen.io.ap.brid.gy · 30/03/2025

Added four new phenomena to Jepsen's docs: P4 (Lost Update), A5A (Read Skew), A5B (Write Skew) and Process. jepsen.io/consistency/phenomena#sql

Jepsen @jepsen.mastodon.jepsen.io.ap.brid.gy · 10/03/2025

For the daytime crew: Jepsen's distributed systems class starts next week. The accompanying workshop, where we practice writing and debugging our own distributed systems, follows the week after […]

mastodon.jepsen.io

Original post on mastodon.jepsen.io

Jepsen @jepsen.mastodon.jepsen.io.ap.brid.gy · 06/03/2025

Jepsen 0.3.8 is now available. It includes a new nemesis for file corruption, and improvements to clock-skew tests. github.com/jepsen-io/jepsen/release…

Release 0.3.8 · jepsen-io/jepsen

The centerpiece of 0.3.8 is a new nemesis for corrupting files: jepsen.nemesis.file. This nemesis can be scoped to specific regions of a file, is aware of chunk structure (e.g. database pages) and ...

Jepsen @jepsen.mastodon.jepsen.io.ap.brid.gy · 05/03/2025

What IS Strong Serializability, really? Ever want to try writing your own gossip service. Two open sessions of Jepsen's training classes are coming up: the Distributed Systems Fundamentals class, and (for the first time!) its accompanying workshop […]

mastodon.jepsen.io

Original post on mastodon.jepsen.io

Jepsen @jepsen.mastodon.jepsen.io.ap.brid.gy · 05/03/2025

I'll be speaking on Jepsen at Bug Bash (DC, April 3-4), and Systems Distributed (Amsterdam, June 19-20). Come join! bugbash.antithesis.com systemsdistributed.com

bugbash.antithesis.com

BugBash 2025: software reliability conference

Join us at a tech conference organized by Antithesis to explore reliable software development. April 3-4, 2024 at Yours Truly DC hotel, Washington D.C.

Jepsen @jepsen.mastodon.jepsen.io.ap.brid.gy · 22/01/2025

Woke up to a bunch of excellent emails--y'all rock. Will try to write back to everyone in the next hour or so. ❤️

Jepsen @jepsen.mastodon.jepsen.io.ap.brid.gy · 21/01/2025

So, uh, the last time I had to call malloc() was a quarter century ago, and I am struggling to do basic tasks without corrupting the heap. I would love to hire one of you stripey-socked C witches for a very small contract to help finish […]

mastodon.jepsen.io

Original post on mastodon.jepsen.io

Jepsen @jepsen.mastodon.jepsen.io.ap.brid.gy · 22/12/2024

Added descriptions of the SQL isolation level anomalies P0, P1, P2, and P3 to the phenomena page: jepsen.io/consistency/phenomena#sql

Jepsen @jepsen.mastodon.jepsen.io.ap.brid.gy · 04/12/2024

Released version 0.2.4 of Maelstrom, Jepsen's workbench for writing toy distributed systems: github.com/jepsen-io/maelstrom

GitHub - jepsen-io/maelstrom: A workbench for writing toy implementations of distributed systems.

A workbench for writing toy implementations of distributed systems. - jepsen-io/maelstrom

Jepsen @jepsen.mastodon.jepsen.io.ap.brid.gy · 04/12/2024

There's a few tickets left for the distributed systems class coming up in just over a week. If you'd like to join, now's the time. :-) www.eventbrite.com/e/distributed-sy…

Distributed Systems Fundamentals

Learn the theory and practice behind distributed systems, from safety and liveness to deployment and monitoring.

Jepsen @jepsen.mastodon.jepsen.io.ap.brid.gy · 03/12/2024

Bin Wang put together a Jepsen test for Patroni, a PostgreSQL replication system. All sorts of good stuff in here, including that the cluster can't handle a series of single-node failures: www.binwang.me/2024-12-02-PostgreSQ…

Jepsen Test on Patroni: A PostgreSQL High Availability Solution | Bin Wang - My Personal Blog

Jepsen @jepsen.mastodon.jepsen.io.ap.brid.gy · 27/11/2024

Antithesis, Buf, and Jepsen are running a joint webinar on December 5th. We'll discuss a Kafka protocol safety issue, talk about the challenges of distributed systems testing, and show how Jepsen and Antithesis helped identify critical safety errors in Bufstream. Come watch Antithesis pause […]

mastodon.jepsen.io

Original post on mastodon.jepsen.io

Jepsen @jepsen.mastodon.jepsen.io.ap.brid.gy · 13/11/2024

Thanks to everyone who wrote in objecting to the report's description of data loss due to auto-commit. Some experiments this morning suggest that we got it wrong (at least for the official Java client). We've published an update to the report: jepsen.io/analyses/bufstream-0.1.0#…

Jepsen @jepsen.mastodon.jepsen.io.ap.brid.gy · 12/11/2024

"That's never been my understanding of auto-commit, that would be a crazy default wouldn't it?" I feel like there's a real need for database user research--building a quantitative and qualitative picture of how actual users understand the systems they use […]

mastodon.jepsen.io

Original post on mastodon.jepsen.io

Jepsen @jepsen.mastodon.jepsen.io.ap.brid.gy · 12/11/2024

We reproduced lost writes, aborted reads, and torn transactions in Bufstream and Kafka. We believe every Kafka-compatible system is likely vulnerable to these issues. Some clients may be able to mitigate them, but the official Java client presently does not.

Jepsen @jepsen.mastodon.jepsen.io.ap.brid.gy · 12/11/2024

One of the surprising things we found during this collaboration was that the #Kafka transaction protocol assumes messages are delivered in order, but allows those messages to be delivered to different nodes over different TCP connections. This means that a client's commit or abort message could […]

mastodon.jepsen.io

Original post on mastodon.jepsen.io

Jepsen @jepsen.mastodon.jepsen.io.ap.brid.gy · 12/11/2024

A new #Jepsen report! We worked with Buf to analyze the safety of Bufstream, a Kafka-compatible streaming system. We found three safety and two liveness issues in Bufstream 0.1.0, including the loss of acknowledged writes in healthy clusters. These problems were resolved by version 0.1.3 […]

mastodon.jepsen.io

Original post on mastodon.jepsen.io

Jepsen @jepsen.mastodon.jepsen.io.ap.brid.gy · 07/11/2024

As an aside, I'm cackling that MariaDB saw the Extremely Queer genders in the test suite and decided to put "male" and "female" in their blog post instead. Except they put "male" in the name field, so... I dunno, somehow this version is even *more* trans? […] [Original post on mastodon.jepsen.io]

The original test suite, whose genders are butch, dyke, femme, enby, leatherboy, and agender

The MariaDB blog post, which inserts into people the name "male" and the gender "female"

Jepsen @jepsen.mastodon.jepsen.io.ap.brid.gy · 07/11/2024

Last year we reported that MySQL and MariaDB's "Repeatable Read" was badly broken. The MariaDB team has been hard at work, and they've added a new flag, `--innodb-snapshot-isolation=true`, which causes "Repeatable Read" to prevent Lost Update, Non-repeatable Read, and violations of Monotonic […]

mastodon.jepsen.io

Original post on mastodon.jepsen.io

Jepsen @jepsen.mastodon.jepsen.io.ap.brid.gy · 26/10/2024

If you've been waiting for a chance to take Jepsen's distributed systems class, we've got great news. We're running a special session December 16-19 that anyone can register for. www.eventbrite.com/e/distributed-sy…

Jepsen @jepsen.mastodon.jepsen.io.ap.brid.gy · 24/10/2024

Just want to give a shout out to Irene Kannyo, who's edited every Jepsen analysis over the last four years. She brings her background as scientist and writer to our industry-focused reports, maintains an evolving style guide, streamlines prose, ensures diagrams match descriptions, and so much […]

mastodon.jepsen.io

Original post on mastodon.jepsen.io

Jepsen @jepsen.mastodon.jepsen.io.ap.brid.gy · 21/10/2024

Shoot! Caught a cold on Thursday and I'm still not quite over it. Won't be able to make it to GOTO after all. Sorry y'all. 😔

Jepsen @jepsen.mastodon.jepsen.io.ap.brid.gy · 18/10/2024

#Jepsen 0.3.6 is now available on GitHub and Clojars. This is a sizeable release. It includes a significant correctness bugfix for a rare bug that could make operations in the history print with the wrong data. It also adds a new namespace for composing databases, nemeses, and generators when […]

mastodon.jepsen.io

Original post on mastodon.jepsen.io