iiinova — where your digital self keeps evolving

Andrew Gross @gross.systems · 4h

I'll believe we have AGI when the Artifical Analysis website renders properly on mobile.

010

Andrew Gross @gross.systems · 29/09/2026

Can't wait for the Open Weight models to finish distilling Opus 5.5 so they can produce legible descriptions again.

040

Andrew Gross @gross.systems · 28/09/2026

Agree that cost per completed task is the headline number, seeing the distribution per task across models could be interesting as well. And yeah, without something to de-correlate failures, 100 samples may be too much for most contained tasks. Unsure if that holds for a large attack surface

000

Andrew Gross @gross.systems · 28/09/2026

Yeah TBench is, but they arent doing the "cost for success" angle directly. Probably something I need to hack up myself.

110

Andrew Gross @gross.systems · 28/09/2026

This is really interesting, I wonder if they keep an up to date version somewhere, or perhaps I can just run it myself. Could be nice to cross reference it with the waffle view (www.tbench.ai?view=waffle) to understand the cutoff in terms of capability vs cost.

tbench.ai

TERMINAL-BENCH

A benchmark to measure and evolve with the frontier of agent work

100

Andrew Gross @gross.systems · 28/09/2026

Yeah, I do find the cost (and in particular the clarity around tokens used vs cost/speed) from AA to be helpful. I believe for their benchmarks they do run N trials, but im pretty sure its a static number.

100

Andrew Gross @gross.systems · 28/09/2026

It would be nice to see benchmarks that looked at things on a cost comparison basis for equal cost. So any things (especially security) often seem to scale based on how many tokens you throw at the problem. Is 1 run of Opus 5.5 better than 100 runs of GLM 5.3 Flash? etc

100

Andrew Gross @gross.systems · 24/09/2026

huggingface.co/datasets/tar... this is the deepswe one, is there similar for TBench? If its the same style, it would just be taking the k=5 (or 4 for deepswe) rollouts and scoring them and picking the best (essentially converting k=5 to k=1 for actual scoring) which is nifty

huggingface.co

000

Andrew Gross @gross.systems · 24/09/2026

For TBench 2.1, k=5 for the official run harness, I am curious if they used their model to pick the best of one of those, or if they picked the best of 4 or 5 runs of k=5, essentially getting k=20 (or k=25).

100

Andrew Gross @gross.systems · 24/09/2026

Also curious what it would look like to have this "in the loop" for a regular LLM, helping choose actions

100

Andrew Gross @gross.systems · 24/09/2026

Seems like the fine tune is likely on the embedding space so that you get better contrast? Hence the usage of some of the DeepSWE tasks for fine tuning. I wonder what perf is like without fine tuning.

100

Andrew Gross @gross.systems · 24/09/2026

Does seem like they used a similar approach to CLIP but for non-image inputs? Need to spend more time understanding the training data vs what is used for a fine tune and why etc.

000

Andrew Gross @gross.systems · 24/09/2026

They did seem to train/fine tune on the DeepSWE tasks, with a set held out for the actual testing. Not sure what the fine tuning gets here, "style" for the outputs? Shouldn't the action model be general w.r.t. these sorts of inputs and not need fine tuning?

100

Andrew Gross @gross.systems · 24/09/2026

Looks like Bo4 for the DeepSWE runs? huggingface.co/Contrastive-... In theory Fable/Opus Pass@4 should produce similar numbers right? I guess it depends if the model is choosing the final rollout results or being used mid stream to pick actions.

huggingface.co

Contrastive-LM/deepswe-clm-heads-8k · Hugging Face

We’re on a journey to advance and democratize artificial intelligence through open source and open science.

100

Andrew Gross @gross.systems · 24/09/2026

They have the data and code out there, so can dig in to see how they did it I guess. Downside is that the SOTA scores would come from running 10x (or more) tokens on frontier models to get candidates.

100

Andrew Gross @gross.systems · 24/09/2026

contrastive-lm.notion.site Interesting write up. No paper yet unfortunately. If some of the claims hold true it would be a pretty incredible leap. Their SOTA DeepSWE scores come from generating rollouts using Opus/Fable and then using their model to select the "right" action.

contrastive-lm.notion.site

Contrastive Language Models | Notion

A System One Model for Fast and Generalizable Decision-Making

200

Andrew Gross @gross.systems · 10/09/2026

disputing claims etc. A lot of this had a very real cost in human time in the past, so much so that being someone wealthy enough to have a lawyer to do it all for your was a huge privilege. Now many more people have the ability to send a properly bureacracy coded letter than before.

000

Andrew Gross @gross.systems · 10/09/2026

Im wondering if the next big "change" we see from AI is in our bureaucracy. Not so much in how it operates internally, but in the dramatic rise in inputs from people using AI. So many things were implicitly gated by the cost of time and attention that has mostly gone away. Filing complaints, forms..

100

Andrew Gross @gross.systems · 10/09/2026

With the way devices are going these days, Im going to need to have all of my private convos inside an MRI machine.

000

Andrew Gross @gross.systems · 23/08/2026

Interesting, I haven't explored much with that style of long term memory for a task like that. Curious how consistent it is.

000

Andrew Gross @gross.systems · 23/08/2026

Claude has reduced me to forcing it to summarize its own output without jargon every time it comes back to me. github.com/andrewgross/...

github.com

050

Andrew Gross @gross.systems · 23/08/2026

Ahh fair enough. The agents dont seam to really care that much about human readable naming do they. Curious how it handled the context length issue as the bundled code is a bit large even for 1mm context.

110

Andrew Gross @gross.systems · 23/08/2026

Curious how you generated this, just pointed the agent at the minified js?

100

Andrew Gross @gross.systems · 19/08/2026

Trying to find an old video on Zip codes. No luck with Youtube search, SEO'd to oblivion. Ask Gemini, gives crappy results, prompt it for a Youtube link, no luck. Tell the model its part of Google and it can search Youtube, responds telling me it can't, but also includes the exact link I needed.

Screenshot of a conversation with Google Gemini about trying to find a video. The model responds that it can't search Youtube indexes directly and for me to give it more info on what to look for. However, it also includes a link to the exact Youtube video I was searching for. Its not linked in the text, or a reference, just the last thing in the message, completely unrelated to the text.

101

Andrew Gross @gross.systems · 15/08/2026

Similar for things like the auto command classifier (if it uses an LLM at all), security review for Fable etc etc. I'd assume its no more than 10% of the cost of the actual token serving, so it probably doesn't change the numbers that much.

000

Andrew Gross @gross.systems · 15/08/2026

Im curious what the uncounted token overhead is for things like ChatGPT and Claude chat sessions. There are a lot of things like memory that involve some background LLM processing that don't seem to get reflected in usage limits, I wonder if it dramatically affects margin.

110

Andrew Gross @gross.systems · 07/08/2026

The latest crop of Claude models are too verbose and create incredible amounts of unnecessary jargon, requiring a re-prompt just to clean it up. Curious if its a side effect of RL rewarding lower response length, so they "create" vocabulary to compact more into less. Either way it sucks.

000

Andrew Gross @gross.systems · 07/08/2026

Ive tried wrangling it with things like Cyclomatic complexity checkers in `biome`, dead code via `knip` etc. They help, but I haven't figured out a good way to hint/check if code is "well organized". I suppose if I could I'd have solved some fundamental CS problems too

020

Andrew Gross @gross.systems · 07/08/2026

Finally, the LLMs are truly atrocious at structuring de novo system architectures. Endless amounts of duplicated code with subtle bugs, 1000+ line functions with if/else blocks, configs and defaults buried deep in execution paths etc. Much better to setup the structure yourself and let it modify

100

Andrew Gross @gross.systems · 07/08/2026

On a personal project (github.com/andrewgross/...) I have been trying out full "vibe" coding where I generally avoid looking at the code directly, and only interacting via the agent. A great learning experience, and some interesting results, but some really bad pitfalls. Happy to share session logs

github.com

humanify/experiments at main · andrewgross/humanify

Contribute to andrewgross/humanify development by creating an account on GitHub.

240

Andrew Gross @gross.systems · 07/08/2026

My general rule is that the better (ergonomic, consistent, fast) a development environment is for a human, the better an LLM will operate in it. Constrained properly they can be incredible for trying ideas, though final integration should play by the rules Ishan mentioned.

110

Andrew Gross @gross.systems · 07/08/2026

I have played around a lot with asking an LLM to try out an idea for how to fix some issue and to write up an experiment (hypothesis, data etc etc). Mixed results, and the most common issue is that it "wrote its own tests" when evaluating things, and those inevitably missed or skipped things.

110

Andrew Gross @gross.systems · 07/08/2026

Agree with @ishan.khatri.io on a lot of this advice. I do think there is room for letting the LLM explore ideas and experiment given the right setup. You should work closely with the LLM to make the evaluation environment and data, ideally making eval a single command.

210

Andrew Gross @gross.systems · 29/07/2026

Spaghettios Aglio e Olio

000

Andrew Gross @gross.systems · 27/07/2026

I'd read it. You got RSS?

010

Andrew Gross @gross.systems · 27/07/2026

Running out of signal I can use for some identifiers and right now they permute between files, creating noise in the final diff. Wondering if I need to find more signal or add more structure to reduce the variance.

000

Andrew Gross @gross.systems · 27/07/2026

Im curious if I have missed papers/projects out there for converting compiled/minified code to an AST/text code consistently. I know of humanify of course, and I found the BinDiff paper/project as well. Wondering what else is out there. My big problem right now is consistency between versions.

110

Andrew Gross @gross.systems · 24/07/2026

Whats the over/under on the flaw being HTTP request smuggling. I feel like that popped up all over the place recently.

000

Andrew Gross @gross.systems · 21/07/2026

Built in, and IIRC I liked the interface a bit more.

120

Andrew Gross @gross.systems · 20/07/2026

Difflib in place of fuzzywuzzy has been a nice surprise

100

Andrew Gross @gross.systems · 18/07/2026

How often do people get the prompt in CC asking if Anthropic can review their session to help improve Claude Code? I haven't gotten it on my work setup, but on my personal sub I have seen it prob 4-5 times. This is on a project that gets downgraded from Fable to Opus probably 50% of the time.

010

Andrew Gross @gross.systems · 16/07/2026

Fable seems to be really good at... deciding it needs to spin up 10 sub-agents, immediately running out of usage credits, being awoken and then deciding to do all the work itself while doing a bad job.

010

Andrew Gross @gross.systems · 14/07/2026

Something really weird is going on with the npm downloads counter for claude-code www.npmjs.com/package/@ant... Really weird spikes and total download counts in recent versions, while old versions have 1000s or even just 100s of downloads. No way every one is on the native installer.

npmjs.com

010

Andrew Gross @gross.systems · 13/07/2026

Im realizing my fork of humanify has become quite a different beast than the original repo, and I need a new name (while still giving credit to the original). Original: github.com/jehna/humanify Mine: github.com/andrewgross/...

github.com

GitHub - jehna/humanify: Deobfuscate Javascript code using ChatGPT

Deobfuscate Javascript code using ChatGPT. Contribute to jehna/humanify development by creating an account on GitHub.

000

Andrew Gross @gross.systems · 10/07/2026

The only thing I ever (rarely) use image generation for is making a customized meme for my friends. It seems that Google has decided that making a meme is a copyright violation and can't work with it at all.

010

Andrew Gross @gross.systems · 20/06/2026

TIL the etymology of cybernetics (and cyber). Thats... actually pretty cool. en.wikipedia.org/wiki/Cyberne...

en.wikipedia.org

Cybernetics - Wikipedia

130

Andrew Gross @gross.systems · 20/06/2026

I saw it, and new that I must spread the word of the future Claude models to come.

120

Andrew Gross @gross.systems · 20/06/2026

ianbarber.blog/2026/06/19/l... Sharing, if only for the reference to "Claude Telenovela"

Screenshot of blog entry: It’s tempting to assume that agents will Fix This: that you’ll hand your PyTorch or JAX definition to Claude Telenovela or whatever and have it generate optimally fused kernels2. To make that work you need a fixed, usable baseline to make sure that what is generated is… right.

101