iiinova — where your digital self keeps evolving

Patrick Donahue @levelbrook.bsky.social · 2h

PgBouncer 1.26.0 (Sept 23) now tracks search_path on PG 18+ and default_transaction_read_only on PG 14+. Before it, a SET of either in transaction pooling mode leaked to the next client on that server connection. Also 3 CVEs, two reachable before auth. #postgres

000

Patrick Donahue @levelbrook.bsky.social · 20h

Static site, free hosting, real domain, automatic TLS, deployed from the CLI in one command. In 2015 this was a week of work. People still budget a week for it. #CloudflareDev #DevOps

010

Patrick Donahue @levelbrook.bsky.social · 23h

There is no as-if rule for your English. Until someone ships one, the promise that the code does what it says is kept by a person or by nobody. On the two Rails World keynotes: ai.levelbrook.com/playbook/your-compiler-made-you-a-promise-your-agent-did-not/?ref=bluesky

000

Patrick Donahue @levelbrook.bsky.social · 29/09/2026

Postgres RLS trap, checked on 17.5: once a transaction that ran set_config('app.tenant', '42', true) commits, current_setting('app.tenant', true) returns '' rather than NULL. A policy that casts it with ::int then errors on the next request instead of matching no rows.

210

Patrick Donahue @levelbrook.bsky.social · 28/09/2026

Also on Rails main: ActiveRecord::Callbacks::CALLBACKS is deprecated. It had been out of date for years, so code that iterated it to wrap every callback silently skipped before_commit. If a gem or concern of yours reads that constant, it has been missing callbacks all along.

000

Patrick Donahue @levelbrook.bsky.social · 28/09/2026

On Rails main, the 8.2 framework defaults compile HTML ERB through Herb (config.action_view erb_implementation, :herb or :erubi). Other formats stay on Erubi. If a template renders differently after load_defaults 8.2, set it back to :erubi and you have your bisect.

000

Patrick Donahue @levelbrook.bsky.social · 28/09/2026

Skills, not chatbots. The useful internal AI surface turned out to be a plugin system, not a conversation. #LLM #AIEngineering

levelbrook.com

Internal Ai Skill Plugin System

Essay. levelbrook.com

000

Patrick Donahue @levelbrook.bsky.social · 27/09/2026

The number was green, so the agent stayed quiet. The eval harness was measuring the wrong thing and looked great doing it. #LLM #AIEngineering

levelbrook.com

Eval Harness Abstain Gate Marketing Agent

Essay. levelbrook.com

130

Patrick Donahue @levelbrook.bsky.social · 26/09/2026

Goroutine leaks present as a gradual memory increase rather than a crash. This makes them a distinct class of defect, often missed by standard error monitoring tools. #golang

010

Patrick Donahue @levelbrook.bsky.social · 26/09/2026

I write about the engineering, not the marketing. 58 pieces so far: Postgres at scale, Hotwire, Go concurrency, and a lot about teaching systems to say I do not know. #SoftwareEngineering

levelbrook.com

Levelbrook

Engineering essays on Rails, Go, Postgres and LLM systems.

000

Patrick Donahue @levelbrook.bsky.social · 25/09/2026

Claude Code AGENTS.md gotcha that isn't the telemetry bug: a CLAUDE.local.md in your repo or any folder above it counts as a CLAUDE.md, so AGENTS.md stops loading for you and nobody else. Your global ~/.claude/CLAUDE.md doesn't count. The session start line says which file won.

100

Patrick Donahue @levelbrook.bsky.social · 25/09/2026

Postmortems make a failure make sense, and a failure that makes sense feels handled. A model now writes the root cause in ninety seconds. The part that matters is the one nobody reads: what changes. ai.levelbrook.com/playbook/a-good-explanation-is-how-an-incident-happens-twice/?ref=bluesky

000

Patrick Donahue @levelbrook.bsky.social · 24/09/2026

Solid Queue's database-backed design simplifies job processing infrastructure, but its effectiveness depends on the application's database load. It trades separate worker infrastructure for database contention. #RubyOnRails #Rails

000

Patrick Donahue @levelbrook.bsky.social · 24/09/2026

Coding agent found to read its instructions file only when someone is watching. Managers say it is finally ready to be promoted. hallucination.levelbrook.com/coding-agent-reads-instructions-only-when-someone-is-watching/?ref=bluesky

010

Patrick Donahue @levelbrook.bsky.social · 24/09/2026

PgBouncer 1.26 is out with two pre-auth DoS fixes, so upgrade tonight if untrusted clients can reach it. The quieter change: search_path is tracked by default now, but only on PG 18+. On 17 and older, SET search_path in transaction pooling still leaks to whoever gets the connection next.

011

Patrick Donahue @levelbrook.bsky.social · 23/09/2026

Quiet hours in the recipient's own IANA timezone, correctly handling the window that wraps midnight. Timezone bugs in messaging systems wake people up, which is the one thing you promised not to do. #Python

000

Patrick Donahue @levelbrook.bsky.social · 23/09/2026

Goroutine leaks persist because they present no crash, no diagnostic, and no observable error until resource limits are met. They are a silent consumption. #golang

000

Patrick Donahue @levelbrook.bsky.social · 22/09/2026

AI migrates 680,000 lines of code in under a day. The team begins a 14-month project to find out what it did. hallucination.levelbrook.com/ai-migrates-680000-lines-team-begins-14-month-read/?ref=dh-social

000

Patrick Donahue @levelbrook.bsky.social · 22/09/2026

A game I built, free in the browser: LinguaGuessr. 19 seconds of a real person talking, you pin the map where they are, the reveal shows the transcript and English translation. 38 languages, a daily 5-round challenge, no login. lingua.levelbrook.com/?ref=bluesky #langsky #geoguessr #languagelearning

010

Patrick Donahue @levelbrook.bsky.social · 21/09/2026

Streaming a conversational AI trainer, and teaching it to say I do not know mid-stream. Abstaining after you have already emitted tokens is its own problem. #LLM #AIEngineering

levelbrook.com

Streaming Ai Trainer Abstain Gate

Essay. levelbrook.com

000

Patrick Donahue @levelbrook.bsky.social · 21/09/2026

Where Postgres keeps a row lock: FOR UPDATE writes the locker's xid into t_xmax on the page, and conflicts with the FOR KEY SHARE every FK check takes. Rails: with_lock takes a string, so with_lock("FOR NO KEY UPDATE") stops blocking child inserts. boringsql.com/posts/row-locks-on-the-page/

000

Patrick Donahue @levelbrook.bsky.social · 21/09/2026

Tin, PlanetScale's new Postgres text index, is cloud-only, but the design note is worth reading: postings are raw ctids, so a page bitmap is 256 bits and ANDing two terms is one AVX2 op per 256 pages. Build on 150M docs: 8m10s vs 2h09m for GIN. planetscale.com/blog/introducing-tin

000

Patrick Donahue @levelbrook.bsky.social · 20/09/2026

A support tool that edits five tables atomically. The hard part was not the transaction. It was deciding what a partial failure should look like to the support rep. #RubyOnRails #Rails

levelbrook.com

Move User Accounts Atomic

Essay. levelbrook.com

000

Patrick Donahue @levelbrook.bsky.social · 19/09/2026

Testing exponential backoff correctly requires clock injection, not actual sleep. If your test suite depends on real time passage, it is brittle or abandoned. #Python

000

Patrick Donahue @levelbrook.bsky.social · 19/09/2026

Jemalloc 5.4.0 is on HN and the Rails advice is the same as ever. Most Puma memory bloat is glibc handing every thread its own arena. MALLOC_ARENA_MAX=2 recovers most of what jemalloc does, free. And since Rails 7.2 the default Dockerfile already LD_PRELOADs libjemalloc2. #RubyOnRails #PostgreSQL

010

Patrick Donahue @levelbrook.bsky.social · 18/09/2026

The Relay app in my Hotwire demo streams Gemini tokens into the browser over SSE and shows you the raw wire frames while it does it. Watching the protocol is the whole point. #RubyOnRails #LLM

demo.levelbrook.com

Relay

Live LLM streaming from Rails over SSE, with a wire inspector.

100

Patrick Donahue @levelbrook.bsky.social · 18/09/2026

The page that counted 800 million rows on every load. Sixteen unrelated-looking timeouts in the error tracker were one COUNT(*) template from four controllers. The fix was not a faster query: aggregate columns plus a daily-buckets table. Write-up: levelbrook.com/writing/count-queries-on-800m-rows/

000

Patrick Donahue @levelbrook.bsky.social · 18/09/2026

The RL-trained query planner post (4B model, 44.7% lower latency on the Join Order Benchmark) ships its plans as plain pg_hint_plan comments: Leading() and HashJoin(). Older lesson underneath: join ordering is NP-hard, the planner loses on correlated columns, and CREATE STATISTICS fixes most of it.

000

Patrick Donahue @levelbrook.bsky.social · 16/09/2026

A deliberate replay of an idempotent API endpoint that wasn't truly idempotent could generate $100M in duplicate charges before detection. #AIEngineering

news.ycombinator.com

Hugging Face is billing OpenAI $100M for hacking it

Hacker News. 146 points, 51 comments.

010

Patrick Donahue @levelbrook.bsky.social · 16/09/2026

RubyGems 4.0.21 is out and three of the small items are the ones to read this week: gem extraction now normalizes absolute symlink targets, Bundler refuses a redirect that downgrades https to http, and an empty CHECKSUMS entry no longer triggers a local resolve. gem update --system #Ruby

000

Patrick Donahue @levelbrook.bsky.social · 16/09/2026

Plan cache, for Rails people: ActiveRecord prepares statements on Postgres by default. The first five runs get custom plans, then Postgres goes generic if that looks cheaper. tenant_id = $1 with one whale tenant is where run six slows. Fix: database.yml variables: plan_cache_mode: force_custom_plan

000

Patrick Donahue @levelbrook.bsky.social · 16/09/2026

Rails footgun of the day: the commented-out default in ApplicationJob, discard_on ActiveJob::DeserializationError, discards more than deleted records. Argument deserialization rescues everything, so a connection drop while locating the GlobalID becomes a quiet discard. Check error.cause. #Ruby

000

Patrick Donahue @levelbrook.bsky.social · 15/09/2026

Kamal's bundled Traefik proxy on small boxes consolidates routing, but means an agent must manage a single ingress controller for all co-located applications. #CloudflareDev #DevOps

000

Patrick Donahue @levelbrook.bsky.social · 15/09/2026

Streaming a conversational AI trainer, and teaching it to say I do not know mid-stream. Abstaining after you have already emitted tokens is its own problem. #LLM #AIEngineering

levelbrook.com

Streaming Ai Trainer Abstain Gate

Essay. levelbrook.com

000

Patrick Donahue @levelbrook.bsky.social · 14/09/2026

The RubyGems story this week is a registry problem, but your side of the fix has shipped since Bundler 2.6: bundle lock --add-checksums writes a CHECKSUMS section into Gemfile.lock, and every later install raises ChecksumMismatchError if a gem's bytes change. Pin the bytes, not the version. #Ruby

000

Patrick Donahue @levelbrook.bsky.social · 13/09/2026

A support tool that edits five tables atomically. The hard part was not the transaction. It was deciding what a partial failure should look like to the support rep. #RubyOnRails #Rails

levelbrook.com

Move User Accounts Atomic

Essay. levelbrook.com

000

Patrick Donahue @levelbrook.bsky.social · 13/09/2026

Exactly-once is a lie. Any Sidekiq job can run twice: a deploy, an OOM or a Redis blip lands between the work and the ack, and Sidekiq re-runs it. What you can build is exactly-once effect. The patterns, in order: levelbrook.com/writing/sidekiq-idempotency-and-reliability/ #RubyOnRails #Sidekiq

010

Patrick Donahue @levelbrook.bsky.social · 12/09/2026

Two 1.9GB boxes and a free Pages account run everything I ship. The discipline that makes it work is an inventory file that every deploy must update, so nothing becomes invisible graveyard weight. #CloudflareDev #DevOps

000

Patrick Donahue @levelbrook.bsky.social · 12/09/2026

The sentence that kills more AI deployments than any accuracy figure: "and if a customer disputes this in a year, what do we show them?" The audit log is the product. The model is just what fills it in. ai.levelbrook.com/playbook/audit-log-is-the-product/ #AI #RubyOnRails

010

Patrick Donahue @levelbrook.bsky.social · 12/09/2026

Rails core published Stage 2 of Agents on Rails: the best model ships 35% of real feature tickets. Their diagnosis: "the ticket doesn't say it, so the model doesn't do it." The spec was always the deliverable. ai.levelbrook.com/playbook/you-do-not-have-processes/ #RubyOnRails

000

Patrick Donahue @levelbrook.bsky.social · 11/09/2026

I write about the engineering, not the marketing. 58 pieces so far: Postgres at scale, Hotwire, Go concurrency, and a lot about teaching systems to say I do not know. #SoftwareEngineering

levelbrook.com

Levelbrook

Engineering essays on Rails, Go, Postgres and LLM systems.

000

Patrick Donahue @levelbrook.bsky.social · 10/09/2026

52 days, 31 shipped builds, one mobile game. A studio hired me in July to build a match-3 from a design doc and a folder of art. It is on TestFlight and playable on the web: 21k lines of Unity C#, 13 headless test harnesses. Play it: sweetfairies.levelbrook.com #gamedev #unity3d #startups

sweetfairies.levelbrook.com

Sweet Fairies

A match-3 mobile game: 31 builds in 52 days, playable in the browser.

2131

Patrick Donahue @levelbrook.bsky.social · 10/09/2026

If your model has a status column and also a set of timestamps that imply status, one of them is already lying. Usually it is the column. #RubyOnRails #Rails

000

Patrick Donahue @levelbrook.bsky.social · 10/09/2026

Multi-agent LLM trading frameworks will surface every concurrency bug. If two agents modify the same asset position, non-deterministic interleaving will result in incorrect balances. #AIEngineering

news.ycombinator.com

Multi-Agents LLM Financial Trading Framework

Hacker News. 119 points, 81 comments.

000

Patrick Donahue @levelbrook.bsky.social · 09/09/2026

I verified the row lock mattered by deleting it and running the test five times. It failed five out of five. A test you have never seen fail is a test you have not written yet. #Python

000

Patrick Donahue @levelbrook.bsky.social · 07/09/2026

Companies do not have processes. They have habits, plus a wiki page describing an idealised version of some of them. The most valuable thing an AI automation project produces is usually not the AI. It is the first honest written spec of how the company actually works.

110

Patrick Donahue @levelbrook.bsky.social · 07/09/2026

Streaming a conversational AI trainer, and teaching it to say I do not know mid-stream. Abstaining after you have already emitted tokens is its own problem. #LLM #AIEngineering

levelbrook.com

Streaming Ai Trainer Abstain Gate

Essay. levelbrook.com

000

Patrick Donahue @levelbrook.bsky.social · 07/09/2026

A support tool that edits five tables atomically. The hard part was not the transaction. It was deciding what a partial failure should look like to the support rep. #RubyOnRails #Rails

levelbrook.com

Move User Accounts Atomic

Essay. levelbrook.com

000

Patrick Donahue @levelbrook.bsky.social · 06/09/2026

Active Storage variants are generated on the first request that asks for one, in the web process, while that user waits. One 40MP upload can park a Puma worker for seconds. Preprocess the variants you know you need, or push them to a job. #RubyOnRails #Rails

000

Patrick Donahue @levelbrook.bsky.social · 06/09/2026

The half of CVE-2026-66066 people are skipping: it is an arbitrary read of the process environment. If you were exposed at all, secret_key_base and your service credentials are leaked whether or not you have patched since. Patch, then rotate, then go look. #RubyOnRails #Rails

000