Sign in

gavin.codes

@gavin.codes
11 followers 7 following 84 posts
PostsRepliesMedia
gavin.codes @gavin.codes · 15h
There are no measurements to make which identify the state. As such it is completely vacuous as a scientific category.
001
gavin.codes @gavin.codes · 15h
Consciousness is as methodologically interesting as revelations from God. Thomas Paine noted revelation can only have validity to the recipient. Beyond that is hearsay. The state of "consciousness" as an experience makes sense only to the one experiencing it.
110
Reposted by @gavin.codes
Scidonia @scidonia.bsky.social · 14/07/2026
The gap between what written code and human understanding is widening due to AI code generation. Merging software verification with design by contract offers a way out. Top-level contracts for humans, sub-contracts for the machine prover. Full breakdown: scidonia.ai/blog/contrac...
scidonia.ai
Understanding Software in the Large: Contracts and Compositionality | Scidonia
A top-level contract tells you what a system does. Sub-contracts tell the prover how it does it. You only need to read the first one. This is componentisation in real terms — and it changes how we thi...
011
gavin.codes @gavin.codes · 20/06/2026
The 0.06c is subsidised to what extent? If this is 60c it's still worth it. I think it's highly dubious it's a 10x subsidisation. I know it isn't because I've run these models myself on hardware. Regarding people, this is a generic problem with AI which requires AI governance on a world basis.
100
Reposted by @gavin.codes
Scidonia @scidonia.bsky.social · 18/06/2026
We used LLMs to prove PCF type preservation in Rocq from scratch. The cost? As low as $0.06. When a mechanised proof costs less than a bug report, vericoding (using AI to write code and a prover to guarantee it) becomes the new standard. scidonia.ai/blog/proof-a...
scidonia.ai
Proof as Commodity | Scidonia
We proved type preservation for PCF — a classic typed lambda calculus benchmark — using two frontier LLMs and rocq-piler. DeepSeek v4 completed it in 21 minutes for $0.06. Claude Opus 4.8 took 14 minu...
011
gavin.codes @gavin.codes · 18/06/2026
scidonia.ai/blog/proof-a...
scidonia.ai
Proof as Commodity | Scidonia
We proved type preservation for PCF — a classic typed lambda calculus benchmark — using two frontier LLMs and rocq-piler. DeepSeek v4 completed it in 21 minutes for $0.06. Claude Opus 4.8 took 14 minu...
000
gavin.codes @gavin.codes · 18/06/2026
$0.06 for a non-trivial proof about programming languages. This is going to change everything.
100
gavin.codes @gavin.codes · 18/06/2026
The costs of what though exactly? $0.06 for a significant proof. scidonia.ai/blog/proof-a...
scidonia.ai
Proof as Commodity | Scidonia
We proved type preservation for PCF — a classic typed lambda calculus benchmark — using two frontier LLMs and rocq-piler. DeepSeek v4 completed it in 21 minutes for $0.06. Claude Opus 4.8 took 14 minu...
100
gavin.codes @gavin.codes · 17/06/2026
The top level specification as a control surface. It will be designed to be as human readable as possible. It will not be simple so we need expert specification engineers and specification testing. The bulk of specification work which exists below and all of the implementation will be irrelevant.
110
gavin.codes @gavin.codes · 17/06/2026
It's possible to have composable specification! * Assume the specification of all called components are correct. * Check your own specification * Repeat. The same approach which works for syntactic types works for semantic types.
000
gavin.codes @gavin.codes · 17/06/2026
Type preservation for PCF+Ref now at 4c on DeepSeek + Opencode + Rocqpiler 4 cents ... proof is cheaper than bananas
000
gavin.codes @gavin.codes · 17/06/2026
The approach to Vericoding in axiomander in a nutshell
000
gavin.codes @gavin.codes · 17/06/2026
This is essentially a "compiler" for specifications — the dream of declarative programming. The change will be as big as shift as the shift away from assembly. But is this feasible? YES! We've already shown how it can be used for contracts on sequential python. This is the answer to AI slop.
000
gavin.codes @gavin.codes · 17/06/2026
1/2 Can we safely let AIs do almost all of the coding? What do we need to avoid getting mountains of slop? * Small control surface [Specification] * Deterministic proof of correctness [Proof assistant]
100
gavin.codes @gavin.codes · 15/06/2026
This is really a rough draft of the future. Specification as control surface, code as disposable and organisations of AIs with specific responsibilities.
011
gavin.codes @gavin.codes · 11/06/2026
Proving properties of software has historically been hard because proof is hard. But if we don't do the proofs, then the hard problem resolves to getting the right specification. This will remain hard, but it will probably also be the main activity in the near future. Time to take this seriously!
000
gavin.codes @gavin.codes · 11/06/2026
Only 60% more expensive than deepseek for my proofs, and twice as fast. The token efficiency almost makes up for the cost.
000
gavin.codes @gavin.codes · 10/06/2026
This is what I'm using: github.com/scidonia/roc...
github.com
GitHub - scidonia/rocq-piler: MCP server for Coq LSP
MCP server for Coq LSP. Contribute to scidonia/rocq-piler development by creating an account on GitHub.
210
gavin.codes @gavin.codes · 10/06/2026
Just tried to run my "benchmark" proof (PCF+ref type preservation) on Fable 5. It took 10 minutes and did not make ONE SINGLE ERROR. No tactics out of place, no iteration. Just full Rocq proofs the first time. This is mental.
110
gavin.codes @gavin.codes · 08/06/2026
Yes, but I'm not sure what the tool schema for Rocq MCP looks like.
210
gavin.codes @gavin.codes · 04/06/2026
There was no configuration difference. It was the same opencode, same model, same prompt, a new session, only difference was which MCPs were available.
110
gavin.codes @gavin.codes · 04/06/2026
3/3 Axiomander, our MCP for proving properties of python, shows how easy it is to augment python with this capability. Why isn't everyone already doing this?
000
gavin.codes @gavin.codes · 04/06/2026
2/3 Theorems which use dimensional analysis are absolutely trivial to prove and can be discharged 100% of the time as they fall into decidable theory: linear algebra. We can adorn python with units and get iron-clad assurance of correctness with no runtime cost and no additional code.
100
gavin.codes @gavin.codes · 04/06/2026
1/3 Proving very basic properties of software can have huge impacts. The NASA Mars Climate Orbiter failed due to a mismatch between Pound-seconds (Imperial) and Newton-seconds (Metric). This mistake cost $327 Million.
100
gavin.codes @gavin.codes · 03/06/2026
I will do this. My guess is that it is about the way that holes are managed and reporting happens about current proof context.
010
gavin.codes @gavin.codes · 03/06/2026
Very soon we will be neither reading, nor writing code. Software engineers will be engaged in specification management. The theorem prover is the key ingredient to make this possible. scidonia.ai/blog/opaque-...
scidonia.ai
Code Will Become Opaque — And That's Fine | Scidonia
Most software developers haven't twigged the real potential of automated theorem proving with LLMs. If provers are powerful enough, you no longer need to understand the code — only the contracts at th...
000
gavin.codes @gavin.codes · 03/06/2026
Based on the transcript, it seemed to perform quite well the entire time without obvious degradation. I think there are still opportunities to reduce this token burn significantly. I tried an alternative Rocq-MCP and it burned about 2x. I bet we can get another 2x from roqc-piler.
210
gavin.codes @gavin.codes · 30/05/2026
4/4 You can get Rocq-piler here: github.com/scidonia/roc...
github.com
GitHub - scidonia/rocq-piler: MCP server for Coq LSP
MCP server for Coq LSP. Contribute to scidonia/rocq-piler development by creating an account on GitHub.
000
gavin.codes @gavin.codes · 30/05/2026
3/4 Model: DeepSeek v4 Tokens to completion: 141k Cost: $3.43 Time: 20m (video is at 10x speed)
200
gavin.codes @gavin.codes · 30/05/2026
2/4 We (at Scidonia) have spent considerable effort to increase the theorem proving ergonomics for LLMs with our MCP and have gotten the following one-shot for a type preservation theorem about the programming language PCF with references (sort of a micro-ML language) on video.
100
gavin.codes @gavin.codes · 30/05/2026
1/4 Here is a video of neuro-symbolic theorem proving in action using the (opensource) Rocq-piler MCP (we renamed when updating to Rocq) with the Rocq theorem prover.
100
gavin.codes @gavin.codes · 24/05/2026
2/2 Done with the following prompt: "You are a proof assistant. You need to use only MCP tools to work your way through a coq proof. This proof is in mcp-coq-lsp/test_issues.v. You will need to add lemmata as you discover the need. Please fully close the main theorem."
001
gavin.codes @gavin.codes · 24/05/2026
1/2 One shotted type preservation for PCF+references, including obtaining all necessary lemmata for the proof using the latest version of our MCP for coq (github.com/scidonia/mcp...) using DeepSeek v4.
101
gavin.codes @gavin.codes · 21/05/2026
I'm making good progress on an automated theorem environment for python using only an MCP. But to make that work well, I needed interactive LLM theorem proving. For this I've developed an MCP for the Coq/Rocq theorem prover. github.com/scidonia/mcp...
github.com
GitHub - scidonia/mcp-coq-lsp: MCP server for Coq LSP
MCP server for Coq LSP. Contribute to scidonia/mcp-coq-lsp development by creating an account on GitHub.
000
gavin.codes @gavin.codes · 23/04/2026
But is it! Certainly at the minute product management in general still requires a lot of communication, but for SaaS I feel like AI could do pretty well here as well.
120
gavin.codes @gavin.codes · 23/04/2026
Software engineering is now mostly robot management. I'm not sure I like it, but that's where it has to go.
100
gavin.codes @gavin.codes · 16/04/2026
scidonia.ai/blog/extract...
scidonia.ai
Which Model Should Verify Your Extractions? A Cost-Quality Analysis of LLM Checkers | Scidonia
Every extraction pipeline needs a verification step. We tested eight models as quality scorers and found that for hallucination detection, a model costing 200× less than Claude performs identically. B...
000
gavin.codes @gavin.codes · 16/04/2026
Cross-checking generated content increases quality substantially, but what models should I use to *cross-check* generated content. It turns out for entity extraction you can use almost anything! Use the model which is 200x cheaper. See below 👇
100
gavin.codes @gavin.codes · 14/04/2026
scidonia.ai/blog/short-t...
scidonia.ai
Why a Structured Ontology Beats a Flat Notepad for LLM Short-Term Memory | Scidonia
Giving an LLM a typed, navigable knowledge structure instead of a flat scratchpad changes what it can remember, how it updates facts, and how much context it consumes doing so.
000
gavin.codes @gavin.codes · 14/04/2026
Riffing on an idea that is making the rounds of using a wiki knowledge base to drive your LLM, we tried this approach just for chat short term memory. The results were very sensitive to the ontology and organisation, we get better results than raw summary. It works amazingly well! See below 👇
120
gavin.codes @gavin.codes · 13/04/2026
scidonia.ai/blog/convers...
scidonia.ai
How Do You Measure an LLM's Memory? Precision and Recall for Conversation Facts | Scidonia
String matching cannot tell you whether an LLM remembers what was said in a conversation. We describe the QA-probing methodology we use to measure short-term memory recall and wiki precision, and what...
000
gavin.codes @gavin.codes · 13/04/2026
Ontologies can be helpful in structuring short term memory for interactive agents. We developed a methodology for short-term memory assessment and demonstrate a comparison between naive summarisation and ontology centric memory. Full discussion and methodology below 👇
110
gavin.codes @gavin.codes · 10/04/2026
scidonia.ai/blog/halluci...
scidonia.ai
Ghost Entities: Why LLM Hallucinations in Entity Extraction Are a Serious Downstream Risk | Scidonia
Hallucinated entities and relationships look identical to real ones inside a knowledge graph. We measured how often frontier models inject facts from parametric memory rather than from your documents ...
000
gavin.codes @gavin.codes · 10/04/2026
Entity extraction from documents can provide huge benefits to organisations. If your model is hallucinating information - it will pollute downstream assets. The key is appropriate provenance tracking and cross checking. Methodology link below 👇
100
Reposted by @gavin.codes
Scidonia @scidonia.bsky.social · 09/04/2026
We benchmarked Claude Sonnet 4.6, GPT-5.4, and Gemini 3 Pro on PERSON entity extraction across eight open-licence documents, spanning 18th-century literary prose, research papers, biomedical articles, & Wikipedia. Here is what we found - scidonia.ai/blog/llm-ent...
scidonia.ai
Which LLM Finds People Best? Benchmarking Claude, GPT-5.4 and Gemini 3 on PERSON Entity Extraction | Scidonia
We ran three frontier models on 8 open-licence documents and measured how accurately each one identifies named people — before and after cross-checking. The results reveal meaningful differences in hallucination rates and the value of verification.
011
gavin.codes @gavin.codes · 09/04/2026
Are you using the right models to analyze your documents? When performing large-context entity extraction, model matters and cross checking has an important impact on precision. It is easier for models to check than to generate, so checking is often worthwhile.
010
gavin.codes @gavin.codes · 22/12/2025
Here is concrete example of a Neurosymbolic programming framework using a proof assistant which is also a programming language (Rocq). All you have to do is drop this MCP in your opencode (or similar) coding framework and you can have LLMs write certifiably correct code. github.com/scidonia/mcp...
github.com
GitHub - scidonia/mcp-coq-lsp: MCP server for Coq LSP
MCP server for Coq LSP. Contribute to scidonia/mcp-coq-lsp development by creating an account on GitHub.
010
gavin.codes @gavin.codes · 11/12/2025
In the very near future code will be organised more around contract than implementation, just as currently we organise around high-level constructs and not machine code.
000
gavin.codes @gavin.codes · 11/12/2025
Generative AI is finally making software verification not just practical, but it a requirement (pun intended). AI slop is a serious problem, but it is also an opportunity.
110
gavin.codes @gavin.codes · 11/12/2025
At bookwyrm.ai we've been performing experiments in Neural-symbolic programming and we're getting amazing results. Generative AI driven by both specification and formal verification.
121