iiinova — where your digital self keeps evolving

microcosm: atproto building blocks

@microcosm.blue

1.2K followers 4 following 271 posts

Open-source APIs to kick-start your next app on ATProto. Community-supported infrastructure ready for production. relay.fire.hose.cam constellation.microcosm.blue slingshot.microcosm.blue more: microcosm.blue tangled.org/@microcosm.blue

Posts Replies Media

microcosm: atproto building blocks @microcosm.blue · 15/09/2026

seeing some firehose weridness! significant drop in events received by both microcosm relays in the last 15mins. since these relays are hosted by different providers on different continents, this seems likely to be a real drop in network volume.

received event rate, commits received line charts. both show relay2 and relay3, almost exactly overlapping series. it's very stable over 400 evt/sec until the last 15mins of the hour-long window, where it drops and levels to just over 200 evt/sec (both charts are essentially the same)

1111

microcosm: atproto building blocks @microcosm.blue · 02/09/2026

availability warning. everything fine so far and most microcosm services have automatic failover to cloud providers, so any impact should be minimal if anything happens 🤞

EMERGENCY ALERT /
now
ALERTE D'URGENCE
Environment Canada: DESTRUCTIVE WINDS AND VERY LARGE HAIL in this mobile coverage area.
Take immediate cover in a basement or interior room if weather threat approaches. Shelter from flying debris if outside.
Severe Thunderstorm Warning, 5:08 p.m. EDT Wednesday 2 September

1172

microcosm: atproto building blocks @microcosm.blue · 28/03/2026

it's up, now backfilling. you can see the progress here: ufos.microcosm.blue/status (28h behind, ufos should be able to catch up relatively quick)

Database

storage: fjall storage v2
current size: 108.85GiB
Rollup cursor (-28h45m58s)

050

microcosm: atproto building blocks @microcosm.blue · 09/03/2026

🟢 UFOs-api has caught up on replay! back to normal. and a good thing too, we've got lexicons on fire this monday

1120

microcosm: atproto building blocks @microcosm.blue · 08/03/2026

doing the math on time remaining to restore completion left as an exercise to the reader

btop showing download at 151mibps (in a spike)

230

microcosm: atproto building blocks @microcosm.blue · 08/03/2026

whew okayyyyyyy some time to breathe now took some fiddling with bucket keys to get the public index mounted, but now it's just however long it takes to restore ~500GB to rocks

root@draco:~# ./microcosm-rs/target/release/rocks-restore-from-backup --from-backup-dir /mnt/remote_index --to-data-dir "$INDEX_DIR"
restoring latest rocksdb backup from "/mnt/remote_index" to "/mnt/volume_tor1_01/constellation-index"...

140

microcosm: atproto building blocks @microcosm.blue · 20/02/2026

not expecting any further issue from this! but just in case, we're making sure all devices and batteries are charged, and constellation is ready for alternate routing via cellular tethering if necessary.

a big black UPS with two power bricks and three normal plugs connected to its top, with small googly eyes looking downward on the front, rests on top of a white box.

3251

microcosm: atproto building blocks @microcosm.blue · 04/01/2026

Service note: Microcosm Jetstream in Montreal Canada at wss://jetstream.fire.hose.cam has started emitting events with periodic delays due to our hosting provider throttling its CPU We'll upsize the instance as soon as we can. In the mean time, the other microcosm jetstream instances are unaffected

line chart: cpu seconds (stacked)

a normal stack of mostly user and system, staying under 1 second total (there are two CPUs available, so this is under 50% utilization)

...until the end, when three huge spikes of "steal" occur, each lasting a couple hours

these spikes match up with the delays in the chart shared in the quoted post.

172

microcosm: atproto building blocks @microcosm.blue · 18/12/2025

Constellation will get some minor maintenance in a few moments: the primary host instance (cassiopeia, the raspberry pi 5) just started falling a bit behind. Failover to the secondary (mikrokosmos, higher capacity) is automatic, so any impact on public requests should only last a few moments.

line chart: jetstream event rate

two series, cassiopeia (green) and mikrokosmos (yellow) are very closely overlapping over the previous 24hrs until about 3 hours ago, when cassiopeia starts appearing noisier and peeling off from the smooth trend to a slightly lower average.

1261

microcosm: atproto building blocks @microcosm.blue · 03/09/2025

8 billion - bluesky likes, follows, blocks, replies, verifications, - tangled stars, comments, issues, - frontpage upvotes - leaflet subscriptions - smokesignal RSVPs - popsky list items - rocksky likes - fujocoded guestbook entries - pinksea avatars - recipes - flushes - anisota items - stream ch..

indexed 7,999,299,711 links

(cropped screenshot)

35811

microcosm: atproto building blocks @microcosm.blue · 12/08/2025

back up and confirmed the fix is working! proof: - left: jetstream.fire.hose.cam, no patch, errors `~` in rkey - right: jetstream2.fr.hose.cam, patched! fine with `~` in rkey

101

microcosm: atproto building blocks @microcosm.blue · 01/08/2025

it's not coming back up. rip baku 2025–2025. moving the demo to another host now. if you were using this demo, you'll need to log in again once it's back up.

closeup of a pi zero with a pink usb cord and purple hand-drawn googly eyes on its cpu

161