iiinova — where your digital self keeps evolving

fjallrs.bsky.social @fjallrs.bsky.social · 15/09/2026

"It's a property of the universe" is gonna be my future excuse for... everything

0100

fjallrs.bsky.social @fjallrs.bsky.social · 16/06/2026

I just recently revisited some In Flames after years, lmao what is this

010

Reposted by @fjallrs.bsky.social

Trifecta Tech Foundation @trifectatech.bsky.social · 01/06/2026

We're proud to announce the first release of our latest data compression project, Zstandard in Rust! Read why we did this, the current state of the project, and where we're hoping to take it: buff.ly/JoruN7e Thanks to Chainguard, Astral, NLnet Foundation, and @sovereign.tech. #rustlang #opensource

trifectatech.org

Announcing Zstandard in Rust - Trifecta Tech Foundation

Over the past year, we've been silently working on our third compression project. After zlib and bzip2 we're now taking on zstd with libzstd-rs-sys, and are proud to announce its first release.

14914

fjallrs.bsky.social @fjallrs.bsky.social · 04/04/2026

EINVAL (err 22) doesn't seem like corruption to me. But unfortunately POSIX errors are terribly intransparent, and it depends on what syscall even throws that...

030

fjallrs.bsky.social @fjallrs.bsky.social · 30/03/2026

fjall is at 1989 stars 😵 - The road to 2K is on!!

040

fjallrs.bsky.social @fjallrs.bsky.social · 07/03/2026

Oh yeah bluesky exists too. I just released 3.1 - which features compaction filter support, meaning you can custom logic during compaction (e.g. TTL rules, or soft deletions)! fjall-rs.github.io/post/fjall-3...

fjall-rs.github.io

Fjall 3.1: Compaction filters

Run custom logic during compactions

1151

fjallrs.bsky.social @fjallrs.bsky.social · 06/01/2026

Even found a second (minor) bug by coincidence

010

fjallrs.bsky.social @fjallrs.bsky.social · 02/01/2026

Changelog: github.com/fjall-rs/fja... V2->V3 migration: github.com/fjall-rs/mig...

github.com

050

fjallrs.bsky.social @fjallrs.bsky.social · 02/01/2026

A strange moment, but - I just released 3.0.0. fjall-rs.github.io/post/fjall-3/ What has started as a rewrite of the block format has become almost a full rewrite, interrupted by a bachelor's thesis, faulty hardware and a couple of weeks of sickness - but I think the end product is pretty good.

fjallrs.bsky.social @fjallrs.bsky.social · 24/12/2025

Love it when SQLite just stops working but allocates hundreds of GBs of disk space.

030

Reposted by @fjallrs.bsky.social

Carl Sverre @f0a.org · 09/12/2025

Check out my talk on physical replication and Graft that I delivered at the recent @syncconf.bsky.social in SF. youtu.be/QoKzDyH2MEA?...

youtu.be

Carl Sverre (SQLSync) - Why Physical Replication Still Matters

YouTube video by Local-First Conf

053

fjallrs.bsky.social @fjallrs.bsky.social · 07/12/2025

Restarting, the drive works again - strange!

020

fjallrs.bsky.social @fjallrs.bsky.social · 07/12/2025

So for the last 3 (!!!) days I was repro'ing a weird perf regression where read I/O would randomly spike. First I thought it was the block cache implementation, or compaction. But today I just got a "Input/output error". First I thought XFS died, but nope, even smartctl gets I/O error now, great.

130

fjallrs.bsky.social @fjallrs.bsky.social · 07/12/2025

Well... looks like my Kingston disk just died...?!

121

fjallrs.bsky.social @fjallrs.bsky.social · 25/11/2025

I have prepared a preliminary changelog, too. This all has been a bit of work, you could say. github.com/fjall-rs/fja...

github.com

000

fjallrs.bsky.social @fjallrs.bsky.social · 25/11/2025

RC 1 is out! I forgot to post about RC 0, but here we are. At this point API and disk format are unlikely to further change - now it's all about stabilization for the final release... crates.io/crates/fjall...

crates.io

crates.io: Rust Package Registry

1102

fjallrs.bsky.social @fjallrs.bsky.social · 13/11/2025

I rewrote the entire write buffer, and journal backpressure system to be a more robust queue + messaging system. Also, now no more threads are started and stopped in the background on the fly. There is a single thread worker pool now (adjustable, at least 1), and that's it.

050

fjallrs.bsky.social @fjallrs.bsky.social · 13/11/2025

v3 pre.6 is out - probably the penultimate prerelease before going into release candidate(s). At this point, all major architectural reworks are done, and just some APIs need some changes. Followed by a final cleanup and stabilization phase. crates.io/crates/fjall...

crates.io

crates.io: Rust Package Registry

1102

fjallrs.bsky.social @fjallrs.bsky.social · 08/11/2025

And because less data is written to disk, it frees up availabe IOPS and increases SSD endurance.

040

fjallrs.bsky.social @fjallrs.bsky.social · 08/11/2025

With key-value separation, blobs are written at least twice (journal + blob file). So if a blob is compressible (let's say by 40%), we will still incur a write amp of 1.6x. Journal compression uses more CPU, but because less data is transferred to OS and disk, it can actually be faster.

140

fjallrs.bsky.social @fjallrs.bsky.social · 08/11/2025

3.0.0 pre.5 is out and brings journal compression support. Here's 16K JSON blobs being written:

172

fjallrs.bsky.social @fjallrs.bsky.social · 28/10/2025

Yesterday or so I ran a benchmark over night, and v3 scales much better for extremely large (100+ GB) databases

020

fjallrs.bsky.social @fjallrs.bsky.social · 28/10/2025

v3 pre-release 2 is out! Still not file format -or API stable, but most importantly this release marks lsm-tree being *feature complete*. crates.io/crates/fjall...

crates.io

crates.io: Rust Package Registry

151

fjallrs.bsky.social @fjallrs.bsky.social · 26/10/2025

Well, it was a faulty RAM stick everybody

061

fjallrs.bsky.social @fjallrs.bsky.social · 25/10/2025

So I have now tried btrfs, xfs, ext4 on two diff. SSDs, updating to Ubuntu 24; and I'm still getting corruptions... at this point I don't really know what to do. This is not code specific (RocksDB is also affected), yet I don't understand how my OS and all apps just seem to continue to work fine...

152

fjallrs.bsky.social @fjallrs.bsky.social · 11/10/2025

Without having numbers (right now), LMDB will probably read absurdly fast, while not so much space- or write efficient. Most, if not all data, fits into RAM here. Would be interesting to benchmark 100s of GBs. But I found redb to be slower (sometimes much slower) than LMDB, pretty much always.

121

fjallrs.bsky.social @fjallrs.bsky.social · 11/10/2025

And here's another read-heavy bench with 4K values, 5% sync random updates and 95% Zipfian reads. ReDB uses 4x more disk space, and writes 6x slower. Interestingly, it's also ~5x slower in point reads - I think ReDB is not handling large values well; haven't read too much into its implementation.

120

fjallrs.bsky.social @fjallrs.bsky.social · 09/10/2025

Synchronous writes (fsync) I might add

010

fjallrs.bsky.social @fjallrs.bsky.social · 09/10/2025

95% Zipfian reads, 5% random updates on a Kingston PLP SSD with 16K JSON blobs - sled DNF! (OOM) fjall uses LZ4 compression

250

Reposted by @fjallrs.bsky.social

fig (aka:[phil]) @bad-example.com · 08/10/2025

constellation runs on a bare metal server with 4 dedicated CPU cores, 16GiB memory, 1TiB fast NVMe SSD attached, and 500mbit unmetered network connection.

2443

fjallrs.bsky.social @fjallrs.bsky.social · 05/10/2025

sled (bloodstone) is not in the bench too rn, because you can't compile it next to sled w/ compression; plus it's super unstable - in general I found it to be faster in writing, smaller on disk, but similar or slightly slower in writing than sled 0.x...

220

fjallrs.bsky.social @fjallrs.bsky.social · 05/10/2025

Blob GC working again (now fully automatic) - much less impact on foreground work (old GC was stop-the-world)

081

fjallrs.bsky.social @fjallrs.bsky.social · 02/10/2025

Hopefully I will have results in 6 hours 👀

130

fjallrs.bsky.social @fjallrs.bsky.social · 02/10/2025

Noo 🥲 I wanted to expand it to a 1-hour run but it crashed because of a bug in the workload (I think), in the middle of the night.

120

Reposted by @fjallrs.bsky.social

fjallrs.bsky.social @fjallrs.bsky.social · 26/09/2025

If you sponsor NOW (github.com/sponsors/fja...), you can get an exclusive preview of version 3!! jk I pushed the first pre-release to crates.io - it's unstable though! Breakages guaranteed, and some APIs are missing! bye

github.com

Sponsor @fjall-rs on GitHub Sponsors

Support fjall-rs's open source work

086

fjallrs.bsky.social @fjallrs.bsky.social · 26/09/2025

github.com

Sponsor @fjall-rs on GitHub Sponsors

Support fjall-rs's open source work

086

fjallrs.bsky.social @fjallrs.bsky.social · 26/09/2025

Looks like the default fillfactor is 100%, which should be good for disk space. So it's probably dead row overhead and lack of page compression (?) plus maybe secondary indexes.

110

fjallrs.bsky.social @fjallrs.bsky.social · 25/09/2025

But yeah, it feels like things are slotting into place. I have more than 50 git stashes and branches floating around... it's been a bit messy, but getting there.

010

fjallrs.bsky.social @fjallrs.bsky.social · 25/09/2025

I think I was lucky to fall into the pit of success because I think I made more progress the last two days than the last two months combined... V3 pre-release(s) very soon... 👀

120

fjallrs.bsky.social @fjallrs.bsky.social · 23/09/2025

Add to that, sweeping API changes for better ergonomics that easily touch over a hundred files - Rust's doctests are a godsend here, I would have gone insane already without them

120