iiinova — where your digital self keeps evolving

Todd Morrill @toddmorrill.bsky.social · 17/08/2026

Implementing speculative execution on a GPU, writing custom VJP functions to override some of JAX's bad memory management behavior, etc. pushed me to the absolute limit of my coding abilities. I think it forced me to grow and learn a lot. N/N

020

Todd Morrill @toddmorrill.bsky.social · 17/08/2026

It turned out that the systems solutions we developed wound up being very valuable artifacts on their own. And another fun bit of history—this project started before Claude code was part of our workflow, which meant that all the JAX code you see is code that I largely wrote myself. 4/N

110

Todd Morrill @toddmorrill.bsky.social · 17/08/2026

I originally started working on decoders and synaptic delays, but I constantly found that simply running the event-based system was slow, resource intensive, and theoretically constrained which motivated the systems work—extreme parallelism and root solvers for flexibility of our design space. 3/N

120

Todd Morrill @toddmorrill.bsky.social · 17/08/2026

Here's a bit of backstory on this work for anyone that's interested. This work started in summer 2024 when I interned with @tonyzador.bsky.social and Christian Pehle at CSHL. I knew practically nothing about SNNs. I had to build the system from the ground up to develop my understanding. 2/N

110

Reposted by Todd Morrill

Columbia University's Zuckerman Institute @zuckermanbrain.bsky.social · 09/06/2026

Is forgetting useful? This was among the deep questions about how memories are formed and used explored at Local Circuits, a symposium that brought together leading experts in biological and artificial intelligence @columbiauniversity.bsky.social zuckermaninstitute.columbia.edu/symposium-me...

Christine Denny, PhD, presenting research on the molecular mechanisms underlying memory. Credit: Eileen Barroso.

093

Todd Morrill @toddmorrill.bsky.social · 28/03/2026

We hope to have an implementation out soon!

010

Todd Morrill @toddmorrill.bsky.social · 17/03/2026

This also means memory and compute are proportional to the number of events, not the number of discrete time steps, which is critical for modeling long sequences. Not to mention the benefits of precise spike times, which enable us to explore richer neural codes. 6/N

100

Todd Morrill @toddmorrill.bsky.social · 17/03/2026

Putting this all together was an immense engineering effort but I think it breaks ground on a new way to build SNNs, namely by operating strictly on events—only expending processing power on consuming or producing spikes. 5/N

100

Todd Morrill @toddmorrill.bsky.social · 17/03/2026

We also operate on machine precision spike times, which is a departure from how nearly all SNN implementations operate (they use a discrete time grid). We solve for precise spike times using differentiable root solvers—Newton-Raphson and Bisection. 4/N

100

Todd Morrill @toddmorrill.bsky.social · 17/03/2026

the non-linear neuron membrane reset breaks associativity. So we solve this via chunking (consume a chunk of input spikes in parallel), which essentially looks like speculative execution. 3/N

110

Todd Morrill @toddmorrill.bsky.social · 17/03/2026

This work started when Christian Pehle, @tonyzador.bsky.social, and I found that training sequentially (consuming/producing one spike at a time) in JAX was prohibitively slow. You certainly can consume multiple spikes in parallel—TLDR; just use an associative scan—but... 2/N

142

Todd Morrill @toddmorrill.bsky.social · 17/03/2026

Our new preprint on parallelizing training of temporally precise spiking neural networks is out! We show up to 44x speedups over a conventional sequential baseline. 1/N

Runtimes by batch size, hidden layer size, and parallel/serial computation

1124

Todd Morrill @toddmorrill.bsky.social · 13/02/2025

I think having universities pay PhD students would also give them more academic freedom. In CS labs, for instance, it’s common for the PI to get funding which is directed toward a particular project and PhD students are just getting paid to do these projects. That’s not always leading to new ideas

000

Todd Morrill @toddmorrill.bsky.social · 01/12/2024

To be sure, I believe in the community, I just think this is a big shift relative to just a couple of years ago when companies only had a compute moat. Now they seem to have both a compute and method moat. N/N

000

Todd Morrill @toddmorrill.bsky.social · 01/12/2024

I think another interesting point here is how far industry has gotten ahead of open source/academia on building these systems. This was a whole talk trying to figure out how to reinvent the wheel. 7/N

100

Todd Morrill @toddmorrill.bsky.social · 01/12/2024

Also, in case the bitter lesson wasn't enough, of course Rich Sutton had something to say about self-verifying AI in 2001! incompleteideas.net/IncIdeas/Key... 6/N

incompleteideas.net

Self-Verification, The Key to AI

100

Todd Morrill @toddmorrill.bsky.social · 01/12/2024

For example, should we ever expect an LLM to weigh in on whether P=NP? There are also more subjective domains requiring a notion of correctness, such as navigating interpersonal relations, where you can't write unit tests for every scenario you'll encounter. 5/N

100

Todd Morrill @toddmorrill.bsky.social · 01/12/2024

For instance, if we build verifiers that are capable of verifying the math and science we currently know, will it be able to push on and generate and verify new solutions to open problems? If not, then my question is, how do we build verification systems that are not bounded by human feedback? 4/N

100

Todd Morrill @toddmorrill.bsky.social · 01/12/2024

It seems like the system as a whole might still be bounded by the verification model's capabilities and how much they want to pay human annotators. 3/N

100

Todd Morrill @toddmorrill.bsky.social · 01/12/2024

I wonder if this 2-model system (primary model + verifier model) will be enough to reach escape velocity (i.e., primary models that can generate new ideas or reason about ideas they haven't been exposed to before). 2/N

100

Todd Morrill @toddmorrill.bsky.social · 01/12/2024

ICYMI, @srushnlp.bsky.social recently gave a nice talk speculating about the methods/data used to train OpenAI's o1 model. The key idea seems to be scaling up chain-of-thought (CoT) generation using auxiliary verifier models that can give feedback on the correctness of the generation. 1/N

1101

Todd Morrill @toddmorrill.bsky.social · 27/11/2024

What about private GitHub repos? Do you think info contained in those are already being used or will be used to train models?

000

Todd Morrill @toddmorrill.bsky.social · 27/11/2024

If you’re a PhD or masters student interested in working on NeuroAI topics, then consider applying by Dec. 10 to be a NeuroAI intern @cshlaboratory.bsky.social I interned with @tonyzador.bsky.social working on spiking neural networks. 10/10 experience. www.schooljobs.com/careers/cshl...

schooljobs.com

Career Opportunities | Cold Spring Harbor Laboratory Careers

040

Reposted by Todd Morrill

Tony Zador @tonyzador.bsky.social · 21/11/2024

Want to be part of the NeuroAI community CSHL ? Applications are open for outstanding graduate students in Artificial Intelligence to spend the summer at CSHL as NeuroAI Interns. Deadline: Dec 10th, 2024. Please spread the word! www.schooljobs.com/careers/cshl... @cshlaboratory.bsky.social

02317