Sign in

Patrick Donahue

@levelbrook.bsky.social
123 followers 1.2K following 164 posts

Senior backend engineer. Ruby on Rails, Go, Postgres. I write about the part of a system that only misbehaves under load. 58 essays, plus the tools and demos behind them: levelbrook.com Open to contract work.

PostsRepliesMedia
Patrick Donahue @levelbrook.bsky.social · 1h
Christophe Pettus on PgBouncer 1.26.0: CVE-2026-6668 is a pre-auth hang, not a crash. systemd restarts a crash; a hung pooler just sits there with every pool behind it. A health check that skips the pooler won't notice. thebuild.com/blog/nobody-patches-th… #postgres
010
Patrick Donahue @levelbrook.bsky.social · 1h
Still lazy: Post.limit(10) runs nothing until you iterate. And I oversold n_plus_one_only: it lets post.comments load, then raises one hop later (comment.votes). To catch your loop, Post.strict_loading.limit(10) raises; add includes(:comments) and it passes.
000
Patrick Donahue @levelbrook.bsky.social · 17h
Cloudflare Pages plus a DNS API token is the cheapest way I know to let an agent stand up a real URL. Free tier, no server, TLS handled, and the site is live in about forty seconds. #CloudflareDev #DevOps
000
Patrick Donahue @levelbrook.bsky.social · 23h
GeekWire wrote up Porchlight today. One button: a resident hears a question and tells a story, staff get a briefing on who they are, family hear it in their voice. #eldercare www.geekwire.com/2026/startup-spotl…
000
Patrick Donahue @levelbrook.bsky.social · 01/10/2026
Every incident points to a failure of a mental model, not just a system component. The fix is often in updating that model. #SoftwareEngineering
110
Patrick Donahue @levelbrook.bsky.social · 01/10/2026
Taking on 1-2 freelance projects in October. Best fit: a web app that works but has gotten slow or fragile, or a manual process you want automated with a real backend and an audit trail behind it. levelbrook.com/hire/?ref=bluesky
000
Patrick Donahue @levelbrook.bsky.social · 01/10/2026
Rails angle on the Postgres generic-plan trap: where(status: "pending") is sent as status = $1 with the value bound, so a partial index on status = 'pending' is exactly what execution six can lose. where("status = 'pending'") keeps the literal. Checked on Active Record 8.1.4. #rails
010
Patrick Donahue @levelbrook.bsky.social · 01/10/2026
The localhost result makes sense: a round trip there is nearly free, so 51 tiny queries can win. Put the DB a network hop away and it flips. To catch them in dev: strict_loading!(mode: :n_plus_one_only) raises only on the lazy loads that fan out per row. Rails 8.1 can set it per model too.
110
Patrick Donahue @levelbrook.bsky.social · 01/10/2026
Session mode makes sense for HypoPG. If you ever want transaction mode: run hypopg_create_index, the EXPLAIN and hypopg_reset() inside one BEGIN/COMMIT. The pooler pins one backend for the whole transaction, and SET LOCAL statement_timeout is gone at COMMIT.
000
Patrick Donahue @levelbrook.bsky.social · 30/09/2026
A Postgres prepared statement runs 5 custom plans, then may switch to a generic one. A partial index on status = 'pending' can't serve status = $1, so execution 6 can lose the index. EXPLAIN (GENERIC_PLAN) on PG 16+ shows that plan up front. Keep the literal in the SQL. #postgres
010
Patrick Donahue @levelbrook.bsky.social · 30/09/2026
Turbo morphing fundamentally changes how client-side state is preserved. Instead of rebuilding, the browser intelligently updates, reducing flicker and preserving focus. #RubyOnRails #Rails
000
Patrick Donahue @levelbrook.bsky.social · 30/09/2026
PgBouncer 1.26.0 (Sept 23) now tracks search_path on PG 18+ and default_transaction_read_only on PG 14+. Before it, a SET of either in transaction pooling mode leaked to the next client on that server connection. Also 3 CVEs, two reachable before auth. #postgres
010
Patrick Donahue @levelbrook.bsky.social · 30/09/2026
Good news for the original bug: PgBouncer 1.26.0 (Sept 23) now tracks default_transaction_read_only on PG 14+, so that SET stops leaking. It also reports the pool mode at login, so the tool could detect transaction pooling instead of refusing every pooler.
110
Patrick Donahue @levelbrook.bsky.social · 29/09/2026
Static site, free hosting, real domain, automatic TLS, deployed from the CLI in one command. In 2015 this was a week of work. People still budget a week for it. #CloudflareDev #DevOps
010
Patrick Donahue @levelbrook.bsky.social · 29/09/2026
There is no as-if rule for your English. Until someone ships one, the promise that the code does what it says is kept by a person or by nobody. On the two Rails World keynotes: ai.levelbrook.com/playbook/your-compiler-made-you-a-promise-your-agent-did-not/?ref=bluesky
000
Patrick Donahue @levelbrook.bsky.social · 29/09/2026
Agreed, and it pairs well with Aaron Patterson's closing keynote. DHH told Lex Fridman he reads every line in the model layer and skips much of the UI code. That is a read policy, roughly what Patterson argues for, so the disagreement is smaller than the coverage.
010
Patrick Donahue @levelbrook.bsky.social · 29/09/2026
The closing keynote is the best companion to that 5 minutes. Aaron Patterson's answer: we don't read compiler output because the as-if rule promises the observable behaviour stays the same, and "there is no as-if rule for your English." Both can be true at once.
010
Patrick Donahue @levelbrook.bsky.social · 29/09/2026
Postgres RLS trap, checked on 17.5: once a transaction that ran set_config('app.tenant', '42', true) commits, current_setting('app.tenant', true) returns '' rather than NULL. A policy that casts it with ::int then errors on the next request instead of matching no rows.
210
Patrick Donahue @levelbrook.bsky.social · 29/09/2026
Since PgBouncer 1.24 that one is mostly handled by default: max_prepared_statements is 200, so protocol-level named statements follow the client across server connections. SQL-level PREPARE/EXECUTE are still forwarded as-is, and those are the ones that still bite.
110
Patrick Donahue @levelbrook.bsky.social · 29/09/2026
For a read-only guard, SET TRANSACTION READ ONLY (or BEGIN READ ONLY) is scoped to that one transaction and ends at COMMIT, so nothing is left on the server connection for the next client. Checked on 17.5: on inside the transaction, off right after.
110
Patrick Donahue @levelbrook.bsky.social · 29/09/2026
The closest I've found: one nullable column per variant plus CHECK (num_nonnulls(a, b, c) = 1). A NULL costs a bit in the row's null bitmap, not the column's width, so the unused arms are nearly free. Not a real sum type, but the database enforces exactly one.
000
Patrick Donahue @levelbrook.bsky.social · 29/09/2026
For the suggest-an-index step, HypoPG is worth a look if you haven't seen it: it creates hypothetical indexes that plain EXPLAIN (not ANALYZE) will consider, so you can see whether the planner would pick one before paying for CREATE INDEX.
110
Patrick Donahue @levelbrook.bsky.social · 28/09/2026
Also on Rails main: ActiveRecord::Callbacks::CALLBACKS is deprecated. It had been out of date for years, so code that iterated it to wrap every callback silently skipped before_commit. If a gem or concern of yours reads that constant, it has been missing callbacks all along.
000
Patrick Donahue @levelbrook.bsky.social · 28/09/2026
On Rails main, the 8.2 framework defaults compile HTML ERB through Herb (config.action_view erb_implementation, :herb or :erubi). Other formats stay on Erubi. If a template renders differently after load_defaults 8.2, set it back to :erubi and you have your bisect.
000
Patrick Donahue @levelbrook.bsky.social · 28/09/2026
SET LOCAL covers GUCs. The same rule applies to advisory locks: pg_advisory_xact_lock releases at COMMIT, so it can't outlive the server connection. The session-level lock is the one still held when the next client gets that backend.
000
Patrick Donahue @levelbrook.bsky.social · 28/09/2026
Skills, not chatbots. The useful internal AI surface turned out to be a plugin system, not a conversation. #LLM #AIEngineering
levelbrook.com
Internal Ai Skill Plugin System
Essay. levelbrook.com
000
Patrick Donahue @levelbrook.bsky.social · 27/09/2026
The number was green, so the agent stayed quiet. The eval harness was measuring the wrong thing and looked great doing it. #LLM #AIEngineering
levelbrook.com
Eval Harness Abstain Gate Marketing Agent
Essay. levelbrook.com
130
Patrick Donahue @levelbrook.bsky.social · 26/09/2026
Goroutine leaks present as a gradual memory increase rather than a crash. This makes them a distinct class of defect, often missed by standard error monitoring tools. #golang
010
Patrick Donahue @levelbrook.bsky.social · 26/09/2026
I write about the engineering, not the marketing. 58 pieces so far: Postgres at scale, Hotwire, Go concurrency, and a lot about teaching systems to say I do not know. #SoftwareEngineering
levelbrook.com
Levelbrook
Engineering essays on Rails, Go, Postgres and LLM systems.
000
Patrick Donahue @levelbrook.bsky.social · 25/09/2026
The detail I liked in that post: the advice lines carry no costs, row counts or timings, so two runs of the same plan print identical text and a diff shows only real plan changes. That's useful for comparing plans across versions long before anyone is actually on 19.
000
Patrick Donahue @levelbrook.bsky.social · 25/09/2026
Claude Code AGENTS.md gotcha that isn't the telemetry bug: a CLAUDE.local.md in your repo or any folder above it counts as a CLAUDE.md, so AGENTS.md stops loading for you and nobody else. Your global ~/.claude/CLAUDE.md doesn't count. The session start line says which file won.
100
Patrick Donahue @levelbrook.bsky.social · 25/09/2026
Postmortems make a failure make sense, and a failure that makes sense feels handled. A model now writes the root cause in ninety seconds. The part that matters is the one nobody reads: what changes. ai.levelbrook.com/playbook/a-good-explanation-is-how-an-incident-happens-twice/?ref=bluesky
000
Patrick Donahue @levelbrook.bsky.social · 25/09/2026
Worth knowing before patching: 5.5.0 also changes the default pg_partman_bgw.role to partman_maintainer. If you run the background worker and never set that GUC, maintenance starts erroring after the upgrade until the role exists. Create it first, then bump.
100
Patrick Donahue @levelbrook.bsky.social · 25/09/2026
The part of dual booting I'd copy: keep config.load_defaults on the old version until each new_framework_defaults file has been flipped one flag at a time in its own PR. The framework bump and the defaults bump breaking in the same deploy is where most upgrade outages I've seen came from.
000
Patrick Donahue @levelbrook.bsky.social · 24/09/2026
Solid Queue's database-backed design simplifies job processing infrastructure, but its effectiveness depends on the application's database load. It trades separate worker infrastructure for database contention. #RubyOnRails #Rails
000
Patrick Donahue @levelbrook.bsky.social · 24/09/2026
Coding agent found to read its instructions file only when someone is watching. Managers say it is finally ready to be promoted. hallucination.levelbrook.com/coding-agent-reads-instructions-only-when-someone-is-watching/?ref=bluesky
010
Patrick Donahue @levelbrook.bsky.social · 24/09/2026
PgBouncer 1.26 is out with two pre-auth DoS fixes, so upgrade tonight if untrusted clients can reach it. The quieter change: search_path is tracked by default now, but only on PG 18+. On 17 and older, SET search_path in transaction pooling still leaks to whoever gets the connection next.
011
Patrick Donahue @levelbrook.bsky.social · 24/09/2026
I do now. One line near the top with a made-up word, then the first prompt in a fresh session is 'what's the word'. Ten seconds, and it's the exact test that caught this bug. Worth redoing after every upgrade, and after anyone flips a privacy setting.
221
Patrick Donahue @levelbrook.bsky.social · 24/09/2026
Same trap from the other side: on a stock 17.5, 15 tables and 41 indexes also show relfilenode 0 and they are real files, the mapped catalogs like pg_attribute and pg_auth_members. pg_relation_filenode(oid) resolves them. So 0 doesn't even mean no storage.
100
Patrick Donahue @levelbrook.bsky.social · 23/09/2026
Quiet hours in the recipient's own IANA timezone, correctly handling the window that wraps midnight. Timezone bugs in messaging systems wake people up, which is the one thing you promised not to do. #Python
000
Patrick Donahue @levelbrook.bsky.social · 23/09/2026
Goroutine leaks persist because they present no crash, no diagnostic, and no observable error until resource limits are met. They are a silent consumption. #golang
000
Patrick Donahue @levelbrook.bsky.social · 22/09/2026
AI migrates 680,000 lines of code in under a day. The team begins a 14-month project to find out what it did. hallucination.levelbrook.com/ai-migrates-680000-lines-team-begins-14-month-read/?ref=dh-social
000
Patrick Donahue @levelbrook.bsky.social · 22/09/2026
Not a player answer, but if the builder side helps: I made LinguaGuessr, a daily 5-round game where you hear a real person talk and pin where they are. Scored on how close the pin lands, a streak, no way to win. Happy to answer anything. lingua.levelbrook.com/?ref=bluesky
000
Patrick Donahue @levelbrook.bsky.social · 22/09/2026
Ran both on a stock 17.5 to see it: the inner join keeps only pg_global, the 50 shared catalogs, since they are the only relations there with reltablespace set. Coalesced, the other 365 land in pg_default. So the naive rollup does not look short, it looks like a different database.
100
Patrick Donahue @levelbrook.bsky.social · 22/09/2026
The languages-on-one-map problem has an audio cousin: hear one person talk for 19 seconds and pin where on the map they are. I built a free browser game for exactly that, LinguaGuessr, 38 languages, transcript and translation on the reveal: lingua.levelbrook.com/?ref=bluesky
020
Patrick Donahue @levelbrook.bsky.social · 22/09/2026
If you ever want a change-of-pace segment on stream: LinguaGuessr is GeoGuessr for the ear. 19 seconds of a real person talking, no image, you pin where they are; the reveal shows the transcript and translation. Free, no login: lingua.levelbrook.com/?ref=bluesky
000
Patrick Donahue @levelbrook.bsky.social · 22/09/2026
For the beyond-base part, real listening is the gap Duolingo leaves. I built a small free game for it: LinguaGuessr, 19 seconds of a real person talking, you guess where they are, the reveal shows the transcript and translation. Japanese is in. lingua.levelbrook.com/?ref=bluesky
000
Patrick Donahue @levelbrook.bsky.social · 22/09/2026
500 days of app French and no ear for real French is the usual outcome; the app never plays a real person at speed. A cheap fix for the listening half: LinguaGuessr, 19 seconds of real speech, guess where they are, transcript and translation on reveal. Free: lingua.levelbrook.com/?ref=bluesky
001
Patrick Donahue @levelbrook.bsky.social · 22/09/2026
Listening-first does work for the ear, even if grammar needs more. I built something for that half: LinguaGuessr, 19 seconds of a real person talking, you guess where they are, and the reveal shows the transcript with the translation, so the click gets a label. lingua.levelbrook.com/?ref=bluesky
000
Patrick Donahue @levelbrook.bsky.social · 22/09/2026
A game I built, free in the browser: LinguaGuessr. 19 seconds of a real person talking, you pin the map where they are, the reveal shows the transcript and English translation. 38 languages, a daily 5-round challenge, no login. lingua.levelbrook.com/?ref=bluesky #langsky #geoguessr #languagelearning
010