iiinova — where your digital self keeps evolving

Matthew Carrigan

@carrigmat.bsky.social

255 followers 154 following 84 posts

Engineer @huggingface. I'm the reason your LLM frontend has a jinja2cpp dependency. Sometimes yells about housing and trans rights instead of working He/him

Posts Replies Media

Matthew Carrigan @carrigmat.bsky.social · 20/09/2026

The secret to being a great hardware builder for LLM inference is to hear sentences like "Inference is bottlenecked by memory bandwidth" then take them to their logical conclusion in the face of all decency and common sense

Four PCIE splitter cards containing 16 NVMe drives with a total read bandwidth of 200GB/s. This is significantly more than most systems' RAM bandwidth

020 0

Matthew Carrigan @carrigmat.bsky.social · 29/05/2026

a thousand usernames speak to me in the same voice, all repeating the same song: "I am real, I am unique, I am alive"

120 0

Matthew Carrigan @carrigmat.bsky.social · 23/03/2026

The code agents are requesting payment now

010 0

Matthew Carrigan @carrigmat.bsky.social · 18/03/2026

Deeply curious whether this was caused by the user's prompt, or whether this is Claude's well-documented support for animal rights shining through into its code agent work

000 0

Matthew Carrigan @carrigmat.bsky.social · 03/11/2025

PRs and issues on @hf.co have gotten a lot sloppier and weirder since the advent of code agents, but the weirdest ones still have an inexplicable human touch

010 0

Matthew Carrigan @carrigmat.bsky.social · 29/10/2025

In particular, this bit suggests that if you inject a concept too weakly the model doesn't notice, and too strongly it just talks about the concept rather than 'introspecting'. But maybe that just means a medium strength biases towards the concept without totally overriding the original question?

010 0

Matthew Carrigan @carrigmat.bsky.social · 15/06/2025

Yup, you can very clearly see a halving of stock value right after GPT-4 is released

0181 0

Matthew Carrigan @carrigmat.bsky.social · 27/04/2025

the betting markets are asking the real questions today

000 0

Matthew Carrigan @carrigmat.bsky.social · 28/01/2025

If all goes well, you should witness a short load period followed by the stream of consciousness as a state-of-the-art local LLM begins to ponder your question.

100 0

Matthew Carrigan @carrigmat.bsky.social · 17/01/2025

You'll also need heatsinks - 4U/tower heatsinks for Socket SP5 EPYC processors are hard to find, but some lunatic is making them in China and selling on Ebay/Aliexpress. Buy two.

100 0

Matthew Carrigan @carrigmat.bsky.social · 17/01/2025

Here's a sample setup you can use: Motherboard: Gigabyte MZ73-LM1 CPU: 2x AMD EPYC 9015 RAM: 24x 32GB DDR5 RDIMM PSU: Corsair HX1000i (You don't need all that power, you just need lots of CPU power cables for 2 sockets!) Case: PHANTEKS Enthoo Pro 2 Server

100 0

Matthew Carrigan @carrigmat.bsky.social · 17/01/2025

Server motherboards can have way more than 2 channels. The current leader is AMD EPYC, which has 12 channels of DDR5 *per socket*! 1 EPYC CPU = 12 RAM channels = 600GB/sec 2 EPYC CPUs = 24 RAM channels = 1200GB/sec! This should be enough to squeeze ~9tok/sec out of our model!

100 0

Matthew Carrigan @carrigmat.bsky.social · 17/01/2025

What about CPU RAM? Consumer RAM is quite cheap, but to fit 400+GB in a single motherboard, we're probably looking at a server board, which means we need RDIMM server memory. Right now, you can get 64GB of this for about $300-400, so 500GB should be $3000 or less.

100 0

Matthew Carrigan @carrigmat.bsky.social · 17/01/2025

The highest-memory datacenter GPUs (A100/H100) are 80GB. We would need ~6 of these. Although we can fit that in a single case, the cost at current prices will probably be >$60,000.

100 0

Matthew Carrigan @carrigmat.bsky.social · 17/01/2025

Let's work it out: The upcoming 5090 has 32GB of RAM. We would need about 16 of them for this. This would cost >$30,000, and you wouldn't even be able to fit them in a single case anyway.

100 0