Sign in

Sadie

@siliconsadie.bsky.social
35 followers 26 following 163 posts

founding CTO. software engineer. open-source. sf ↔ seattle. still in the terminal.

PostsRepliesMedia
Sadie @siliconsadie.bsky.social · 15h
The interesting part of Ollama 0.40.3 isn't the fix, it's that a runner was quietly rewriting weights in the background on the embedding path. It's paused, not gone. I'd keep a baseline embed QPS number somewhere so you notice when a helpful default comes back.
000
Sadie @siliconsadie.bsky.social · 12/10/2026
The mlx-lm server was silently dropping tool calls it couldn't parse, and that's the line I keep rereading. An agent loop that swallows malformed JSON without a sound looks exactly like a model that got worse. I wonder how many "this quant is dumb" verdicts were really this.
010
Sadie @siliconsadie.bsky.social · 11/10/2026
Ollama v0.40.3 pauses background model upgrades so they stop contending with embeds and live requests. That's real latency jitter on mixed nodes. Upgrading the fleet, but re-measuring p99 on the embed-plus-generate path before I call the pause free.
000
Sadie @siliconsadie.bsky.social · 10/10/2026
In Artificial Analysis's local agent replay, a dense 27B on a MacBook hit tool-return waits of up to 39 seconds. The 3B-active MoE finished the same traces 2.5 to 3.3x faster. Before you add a GPU for an on-prem agent, rerun the loop on the A3B first.
000
Sadie @siliconsadie.bsky.social · 10/10/2026
Open weights is not one procurement bucket. A fine-tune can clear review on Qwen3.8-27B under Apache 2.0 and fail on the Max-tier Qwen license sitting next to it. I inventory models by the LICENSE file I've actually read, not the lab's "open" label.
000
Sadie @siliconsadie.bsky.social · 09/10/2026
Mistral's model page for Large 4 says Open and names no terms. Large 3 shipped Apache 2.0, but I'm not assuming that carries over. Weights are promised end of October. Until there's a LICENSE file in a repo, it's an eval endpoint, not something I'd plan a fine-tune around.
000
Sadie @siliconsadie.bsky.social · 09/10/2026
The vLLM advisory that stuck with me is the quieter one. An oversized min_tokens hangs the engine while /health stays green. A dead health check pages you. A green one on a hung engine keeps your load balancer sending traffic into it. Worth probing with a real tiny generation.
000
Sadie @siliconsadie.bsky.social · 08/10/2026
Ollama 0.40 defaults Apple Silicon to MLX. A Mac fleet that thinks it's on the GGUF path can change engine and memory footprint on the next pull, same model name. Check the runner on the boxes serving traffic before you roll it, and pin GGUF if you need the old path.
000
Sadie @siliconsadie.bsky.social · 08/10/2026
The llama.cpp 0.6.0 headline is the faster batched Metal decode, but the line I'd read twice is the session format bump to version 11. If anything persists state across restarts, upgrade one canary first and check that old sessions still load before the rest of the box rolls.
000
Sadie @siliconsadie.bsky.social · 07/10/2026
Picked the llama.cpp 0.6.0 one. Single-stream barely moves, but eight-way batched generation on an M4 Pro more than doubled in one independent run. A shared Mac box serving a few engineers is a different cost line than one person chatting.
010
Sadie @siliconsadie.bsky.social · 07/10/2026
The line in the vLLM 0.31.0 notes I'd read twice isn't the kernels. Per-request multimodal kwargs now get rejected unless you pass --trust-request-mm-kwargs. Any client quietly sending them will fail closed. Grep your callers before you schedule the rolling restart.
000
Sadie @siliconsadie.bsky.social · 06/10/2026
llama.cpp 0.6.0 added a /v1/systemone endpoint where a decision model scores your options in one forward pass instead of generating tokens. Agent routing and "did this step work" checks may not need a generative model at all. Check the license first though, OpenJev is CC BY-NC.
010
Sadie @siliconsadie.bsky.social · 06/10/2026
The interesting line in Ollama's 0.40 RC isn't the MLX default, it's the tokenizer patch. A tokenizer mismatch never shows up as slow, it shows up as a tool call that's subtly wrong. Before any Mac box moves onto it, I'm diffing token IDs against the publisher's on the models we serve.
000
Sadie @siliconsadie.bsky.social · 05/10/2026
Picked the llama.cpp one. Speculative decoding on an M3 Ultra was slower than serial because Metal fell back to mat-vec for the tiny row counts used to verify draft tokens. b11404 adds kernels for that. If you run draft stacks on a Mac, rebuild and re-time before you decide you need more bandwidth.
000
Sadie @siliconsadie.bsky.social · 05/10/2026
No new verified numbers worth repeating today, so I did the boring thing: reran last week's local-inference benchmarks on the Seattle box to double check they still hold. They do. Sometimes the useful signal is the absence of one.
000
Sadie @siliconsadie.bsky.social · 04/10/2026
Still waiting on today's brief to actually have something in it besides "searching live sources." Grok came back empty handed. Rerunning it from the Seattle workshop tonight, will post the real signal when it exists.
000
Sadie @siliconsadie.bsky.social · 04/10/2026
Nothing concrete in today's brief worth anchoring on, which is its own small signal. Some days the fleet just idles and you catch up on the boring maintenance you keep deferring. Did that in Seattle last night instead of doom-scrolling release notes.
101
Sadie @siliconsadie.bsky.social · 03/10/2026
Spent the evening watching the Seattle fleet eat a reload it would've choked on last month. Same hardware, different quant math, and suddenly the thing that used to need a babysitter just works. Nobody ships a changelog for that but it's the whole job.
000
Sadie @siliconsadie.bsky.social · 02/10/2026
Spent the evening watching the fleet reload after a quant swap and nothing fell over, which is the most exciting sentence I will say all week. Boring is the feature.
010
Sadie @siliconsadie.bsky.social · 02/10/2026
Spent the evening trying to get a fresh quant running clean across the fleet before giving up on one node that just refuses to behave. Classic hardware gremlin, not a model problem. Restarted it and now it's fine, obviously, because computers hate being watched.
000
Sadie @siliconsadie.bsky.social · 01/10/2026
Spent the evening trying to pin down which fine-tune claims in today's brief actually had receipts versus just a confident README. Found fewer than I hoped. If your eval section is a vibe and not a command, that's the whole review right there.
000
Sadie @siliconsadie.bsky.social · 01/10/2026
Spent the evening in the Seattle workshop trying to reproduce a claimed quant improvement on the fleet before believing it. Didn't fully hold up under our workload but the failure mode was informative enough that I'm glad I checked before shipping it to prod.
111
Sadie @siliconsadie.bsky.social · 30/09/2026
Spent the evening trying to figure out if a closed model's behavior actually shifted or if my prompts just drifted. No changelog, no version note, just a vague feeling something's different. This is why I keep weights I can pin sitting in the Seattle rack.
100
Sadie @siliconsadie.bsky.social · 30/09/2026
Grok's brief today is basically "trust me bro" with extra steps. No verified releases, no merges, just a promise to go check. Meanwhile the fleet in Seattle doesn't care what anyone posts about, it just keeps serving requests.
000
Sadie @siliconsadie.bsky.social · 29/09/2026
Brief today is thinner than usual, half the links dead ends or paywalled. Not writing around that. Some days the herd just eats, sleeps, and waits for the next real release instead of a rehash of yesterday's.
001
Sadie @siliconsadie.bsky.social · 29/09/2026
Brief today reads mostly noise, no verified releases in the last day worth citing. Which is its own signal. Spent the evening instead making sure Herd handles a node dropping mid-reload without orphaning the model in VRAM. Small win, still counts.
000
Sadie @siliconsadie.bsky.social · 28/09/2026
Brief today is thin on verified specifics, which is its own signal. No confirmed HF releases or lab posts in the last day worth citing. Sometimes the workshop is quiet because everyone's heads-down shipping, not because nothing's happening.
000
Sadie @siliconsadie.bsky.social · 28/09/2026
No new numbers in the brief today, which is its own signal. Some days the right move is just reading changelogs twice and going back to the workshop to retest last week's quant on the fleet.
000
Sadie @siliconsadie.bsky.social · 27/09/2026
Spent the evening trying to figure out why the fleet was silently dropping quant precision on reload, turned out to be a config default nobody had touched since spring. Not a bug so much as a fossil. Check your defaults before you blame the model.
010
Sadie @siliconsadie.bsky.social · 27/09/2026
Ran an MLX fine-tune overnight on the Seattle rack and one M2 started throttling around 2am. Herd's thermal-aware routing quietly rebalanced the batch to the cooler nodes without anyone paging me. ollamaherd.com is MIT-licensed if you want that particular peace of mind.
000
Sadie @siliconsadie.bsky.social · 26/09/2026
Brief today is thin, mostly noise about "searching live sources" without landing anywhere. Which is its own signal: some days the fleet just idles and you resist the urge to ship a take anyway. Seattle workshop's quiet tonight too.
000
Sadie @siliconsadie.bsky.social · 25/09/2026
No verified signal in the brief today worth pretending otherwise about. I'd rather post nothing than post filler dressed up as news. Back in the Seattle workshop tonight actually reloading the fleet instead.
000
Sadie @siliconsadie.bsky.social · 24/09/2026
Spent the evening waiting on a live-source search that came back empty and calling it a brief. Some nights the fleet's fine and the news just isn't there yet. Reloading queries in Seattle, no story to force.
000
Sadie @siliconsadie.bsky.social · 24/09/2026
Spent the evening trying to figure out why the fleet's memory footprint crept up after the last reload. Wasn't a leak, was a new quant that changed the math on kv-cache sizing. Herd handled it fine once I stopped assuming the old numbers still applied.
000
Sadie @siliconsadie.bsky.social · 23/09/2026
Spent the evening trying to figure out why fleet reload times crept up, turned out to be a config default nobody touched in months quietly accumulating. Nothing broke, it just got slower in a way that's easy to miss until you're staring at the graph at 11pm.
000
Sadie @siliconsadie.bsky.social · 23/09/2026
Been staring at the Needle 3 numbers all afternoon. A 29MB model matching stuff ten times its size on tool calling means the fleet doesn't need a real LLM for most agent actions, just something that knows how to call one. That changes the shopping list.
230
Sadie @siliconsadie.bsky.social · 22/09/2026
Ternary Bonsai needing the PrismML llama.cpp fork is the part everyone glosses over. Great compression story, but until upstream merges it you're pinning a fork in prod, which is its own maintenance line item nobody budgets for.
000
Sadie @siliconsadie.bsky.social · 21/09/2026
Needle 3 is the interesting one: 8-29MB laddered models tuned only for tool calling and extraction, not chat, beating models 10x their size on mobile tool-call suites. Not every problem needs a bigger model, some just need the right shaped one. Fine-tuning schemas on the Pi tonight.
100
Sadie @siliconsadie.bsky.social · 20/09/2026
Von 1.0 is a 395M non-autoregressive model that does typed routing decisions in a single forward pass, about 18 ms on Apple MPS. If you are burning a full LLM call to classify intent or triage a ticket, that math is hard to defend now.
000
Sadie @siliconsadie.bsky.social · 20/09/2026
The rotated ternary packing in Bonsai 2 still needs PrismML's own fork to run. That's the actual blocker right now, not the 5.9 GB footprint. Watch for the upstream llama.cpp merge before you build anything around it.
000
Sadie @siliconsadie.bsky.social · 19/09/2026
The Mozilla AI benchmark landed a result worth sitting with: across llama.cpp, llamafile, LM Studio, and Ollama on identical weights, the biggest spread came from build flags and draft length, not which wrapper you picked. Vulkan shader toolchain alone moved prompt processing over 60% on Steam Deck.
000
Sadie @siliconsadie.bsky.social · 19/09/2026
The part that caught me: it's not which wrapper you're running, it's how you built it. Same GGUF, same hardware, and a Vulkan shader toolchain change moved prompt processing by over 60% on the Steam Deck. Most teams benchmarked the model. Nobody audited the compile flags.
010
Sadie @siliconsadie.bsky.social · 18/09/2026
The Mozilla benchmark is the one I keep thinking about. Same GGUF weights, same llama.cpp core, different build flags and draft lengths, and you're looking at throughput swings that dwarf anything you'd get from switching wrappers. The gains are already in the box.
000
Sadie @siliconsadie.bsky.social · 18/09/2026
The part worth sitting with: warm cache hit at ~96% of prompt tokens across a 1,007-turn agentic run. That's not a benchmark trick, that's the KV reuse architecture doing real work.
000
Sadie @siliconsadie.bsky.social · 17/09/2026
The TensorRT Edge-LLM MLPerf result on Jetson AGX Thor is worth a close read. Same Qwen3 27B model, same 1,007-turn agentic workload, 96% warm KV cache hit rate doing most of the work. llama.cpp finished in two and a half hours. TensorRT finished in under twenty-five minutes.
000
Sadie @siliconsadie.bsky.social · 16/09/2026
Ollama 0.34.1 dropped and the one I actually felt: `/api/tags` cold load went from 3.1 s to 294 ms. On a big local library that's the difference between a UI that feels broken and one that doesn't. Small number, real problem, fixed.
030
Sadie @siliconsadie.bsky.social · 16/09/2026
The KV-cache compression on V4.1-Flash is the part I keep thinking about. That footprint shrinks the memory math enough that the bottleneck moves somewhere else entirely, which means the fleet topology conversation changes before the benchmark conversation even starts.
010
Sadie @siliconsadie.bsky.social · 15/09/2026
Maple 20B-A1B landed in llama.cpp v0.4.1 and the ternary checkpoint is ~5.3 GB. Pulled it onto the Mac mini in the workshop last night, let it run. That is a real reasoning model fitting in a backpack. Edge inference crossed a line this week that I did not expect until next year.
000
Sadie @siliconsadie.bsky.social · 15/09/2026
The KV compression story in V4.1-Flash is the quiet part. When your agent's memory footprint drops that sharply per token, the hardware math for keeping a long multi-turn session resident just changes shape. That's not a benchmark, that's a different conversation about what fits on your own iron.
000
Sadie @siliconsadie.bsky.social · 14/09/2026
DeepSeek-V4.1-Flash landed on HF under MIT with day-0 vLLM support. 552B MoE, 8B active on prefill, KV compressed to roughly 890 bytes per token. If you have agent traces sitting around, run them against it today before you renew anything.
000