Damien Tournoud @damientournoud.bsky.social · 28/04/20263/ Place two: stripThinking in the gemma4 renderer walks past assistant content and removes `<|channel>...<channel|>` blocks before re-rendering. Comment says it mirrors Hugging Face's `strip_thinking` macro. 100
Damien Tournoud @damientournoud.bsky.social · 28/04/20262/ Place one: filterThinkTags in `server/routes.go` walks past assistant messages, runs each through a `<think>`/`</think>` parser, and rewrites `Content` to just the post-thinking remainder. Only fires for qwen3 family and deepseek-r1. 100
Damien Tournoud @damientournoud.bsky.social · 11/03/2026What could possibly be happening in that LLM to get this garbage as an output? 000
Damien Tournoud @damientournoud.bsky.social · 27/02/2026Yup. It can be very quick to prematurely declare Mission Accomplished. I have also seen it randomly decide not to do something and declare victory anyway. 000
Damien Tournoud @damientournoud.bsky.social · 27/02/20264/ Figuring out which years have messages without listing everything: scan backwards from the newest message. Record its year, jump to the start of that year, find the next oldest. Same for months and days. Empty periods cost zero API calls. 100
Damien Tournoud @damientournoud.bsky.social · 27/02/20263/ Every label becomes a folder, every email a .eml file, organized by year/month/day. Only populated directories are shown — no scrolling past empty months. 100
Damien Tournoud @damientournoud.bsky.social · 20/01/2026This results in the allocation of a map with 2.1 million entries, and the corresponding 2.1 million short key strings. In total, 416 MB of allocation. 100
Damien Tournoud @damientournoud.bsky.social · 20/01/2026The issue is a classic memory amplification issue: while Request.ParseForm refuses by default to parse request bodies bigger than 10 MB, even that can result in significant memory usage for specially crafted inputs. In this case, the input is a URL-encoded form data with small keys and no values. 100
Damien Tournoud @damientournoud.bsky.social · 06/01/20265/ That's not the only choice you can make. As noted in the Sync 1.1 release notes, if the repository was ordered in preorder depth-first search order, a reader implementation could both validate the cryptographic properties and iterate the keys in order. 100
Damien Tournoud @damientournoud.bsky.social · 05/01/20265/ Tap itself fetches the list of known repositories from the relay via the com.atproto.sync.listRepos method. A full fetch of the list takes a few hours, because the API scales pretty poorly. This is what the rate of discovery of new repositories looked like the first time I ran this: 110
Damien Tournoud @damientournoud.bsky.social · 31/12/202510/ The biggest collection excluding Bluesky (jp.5leaf.sync.mastodon) is only 280MB compressed in total. On the tail end, there is a large amount of very small collections, most of them appearing spamish. 242
Damien Tournoud @damientournoud.bsky.social · 31/12/20259/ At the collection level, the Bluesky-related collections (app.bsky.* and chat.bsky.*) form the overwhelming bulk of the data stored on the network (literally 99.9% in size). In decreasing order of size: likes (a majority of the data), then posts, then reposts and finally follows. 142
Damien Tournoud @damientournoud.bsky.social · 31/12/20258/ Speaking of compression, in my current storage (CARv1 format, zstd level 1), most repositories achieve a compression ratio above 2x. Still, 20% of the repositories compress very little, at 1.51x ratio. Surprising given the amount of redundancy in the data (i.e. structs are stored as maps). 241
Damien Tournoud @damientournoud.bsky.social · 31/12/20257/ On the other side of the size distribution, the biggest repository is 457MB compressed, and 1.18GB uncompressed. Definitely some interesting outliers here. 162
Damien Tournoud @damientournoud.bsky.social · 31/12/20256/ Not unexpectedly, most of the repositories are very small. 20% of the repositories are below 912B while 50% of the repositories are below 6.34kB. You have to wait until the 97th centile to reach 1.04MB. 1113
Damien Tournoud @damientournoud.bsky.social · 16/09/2025What you are looking at here is the 301 response of Nginx 1.22.1. 000
![model/renderers/gemma4.go:
// stripThinking removes <|channel>...<channel|> thinking blocks from content,
// matching the HF chat template's strip_thinking macro.
func stripThinking(text string) string {
var result strings.Builder
for {
start := strings.Index(text, "<|channel>")
if start == -1 {
result.WriteString(text)
break
}
result.WriteString(text[:start])
end := strings.Index(text[start:], "<channel|>")
if end == -1 {
break
}
text = text[start+end+len("<channel|>"):]
}
return strings.TrimSpace(result.String())
}](https://cdn.bsky.app/img/feed_thumbnail/plain/did:plc:6sge7lr6xli62ix3bgnvgbkk/bafkreibtwby4o475u7ievm2ctzz6cjlmujiyxnutukmbjwek2yeqfl47ue)
![server/routes.go:
func filterThinkTags(msgs []api.Message, m *Model) []api.Message {
if m.Config.ModelFamily == "qwen3" || model.ParseName(m.Name).Model == "deepseek-r1" {
finalUserIndex := -1
for i, msg := range msgs {
if msg.Role == "user" {
finalUserIndex = i
}
}
for i, msg := range msgs {
if msg.Role == "assistant" && i < finalUserIndex {
// TODO(drifkin): this is from before we added proper thinking support.
// However, even if thinking is not enabled (and therefore we shouldn't
// change the user output), we should probably perform this filtering
// for all thinking models (not just qwen3 & deepseek-r1) since it tends
// to save tokens and improve quality.
thinkingState := &thinking.Parser{
OpeningTag: "<think>",
ClosingTag: "</think>",
}
_, content := thinkingState.AddContent(msg.Content)
msgs[i].Content = content
}
}
}
return msgs
}](https://cdn.bsky.app/img/feed_thumbnail/plain/did:plc:6sge7lr6xli62ix3bgnvgbkk/bafkreifk7hdnsmfojzapoi7a2saq77oixza73si25x7pxjpbj67hzh4gha)
![gmail.go:
func (g *GmailClient) PopulatedYears(ctx context.Context, labelID string, cache *Cache) ([]int, error) {
var years []int
cursor := time.Now().AddDate(1, 0, 0).Unix()
for {
s, found, err := g.newestMessageInRange(ctx, labelID, 0, cursor, cache)
if err != nil {
return nil, err
}
if !found {
break
}
y := time.UnixMilli(s.InternalDate).In(time.Local).Year()
years = append(years, y)
cursor = time.Date(y, 1, 1, 0, 0, 0, 0, time.Local).Unix()
}
slices.Reverse(years)
return years, nil
}](https://cdn.bsky.app/img/feed_thumbnail/plain/did:plc:6sge7lr6xli62ix3bgnvgbkk/bafkreif33b465rsqp6ajzrum6ohhor3huw4ec4unuk3htm6ux5lpeg3f2e)
