iiinova — where your digital self keeps evolving

cogscikid.bsky.social @cogscikid.bsky.social · 28/06/2026

I think AI has tremendous potential to help us *help each other* This year I built an AI copilot to help support and scale mentorship. I plan to write a couple of blog posts on what I learned from this. Here's a link to the first one! infinitecare.substack.com/p/what-i-lea...

0152

cogscikid.bsky.social @cogscikid.bsky.social · 02/06/2026

Strikingly, higher diversity improves backward transfer to the very first distribution! So the agent keeps getting better on old tasks even as it stops improving on new ones. This suggests it's learning the shared structure, but losing the ability to specialize.

151

cogscikid.bsky.social @cogscikid.bsky.social · 02/06/2026

Task diversity inhibits continual reinforcement learning! At medium diversity (16 layouts × 16 topologies), performance climbs with each new distribution. But past that point, it monotonically gets worse.

120

cogscikid.bsky.social @cogscikid.bsky.social · 02/06/2026

We analyze backward transfer and find that varying the number of layouts or object assignments has no effect, but *varying the number of topologies has a big effect*! Again, this is true for both PPO and PQN.

130

cogscikid.bsky.social @cogscikid.bsky.social · 02/06/2026

Vary any single component and agents begin a new distribution at or above the same performance that they finished the previous one with! This is true for both PPO and PQN.

140

cogscikid.bsky.social @cogscikid.bsky.social · 02/06/2026

So we built Banyan: a procedurally-generated environment where you can systematically vary the components that define a task. A Banyan task is composed of a layout & a task tree, where each tree is defined by its topology and the objects assigned to its nodes.

250

cogscikid.bsky.social @cogscikid.bsky.social · 02/06/2026

Task diversity is supposedly key to generalization in RL. But what does it do to continual RL, where agents face one new task distribution after another? We find that past a point, more diversity actually inhibits continual reinforcement learning 🧵

45114

cogscikid.bsky.social @cogscikid.bsky.social · 07/05/2026

This same algorithm scales further. When learning many long and interdependent tasks like "make pickaxe" or "collect iron" from finite set of environments, it better generalizes to 10,000 unseen Craftax worlds. Dyna and model-free methods plateau long before that.

100

cogscikid.bsky.social @cogscikid.bsky.social · 07/05/2026

We study Multitask Preplay against other baselines - Dyna, Universal Value Functions, Successor Features, Hindsight Experience Replay - that all train to near-perfect success. At test, only humans and Multitask Preplay transfer. Every other method collapses.

100

cogscikid.bsky.social @cogscikid.bsky.social · 07/05/2026

We studied this with people where train and test conditions had distinct tasks. Despite only learning train tasks, people behave as if they've rehearsed goals they weren't told about. Multitask Preplay provides an explanation for why: they "preplayed" the new goal when they passed by it earlier on

100

cogscikid.bsky.social @cogscikid.bsky.social · 07/05/2026

The idea: during offline replay, you image pursuing goals that you encountered along the way but didn't actually pursue. Those counterfactual trajectories cache into a single value function that covers many goals. This generalizes Dyna, which only learns about the goal you were pursuing.

110

cogscikid.bsky.social @cogscikid.bsky.social · 07/05/2026

Do RL agents really need to replan from scratch for every new goal? Why do rodents "preplay" goals they haven't pursued? We think these questions share a candidate answer.

2185

cogscikid.bsky.social @cogscikid.bsky.social · 10/09/2025

Excited to share recent work on an AI model that doubles as a scalable cognitive theory. Basic idea: when learning one task, "preplay" other tasks to learn implicit goal-oriented maps that support fast performance later. This improves AI and predicts human behavior in naturalistic experiments!

1154

cogscikid.bsky.social @cogscikid.bsky.social · 12/07/2025

Main result 3: we speculate that this may have unexpected benefits for human generalization. We present AI simulations where Multitask Preplay improves generalization of complex, long-horizon behaviors to 10,000 unique new environments when they share subtask co-occurrence structure.

120

cogscikid.bsky.social @cogscikid.bsky.social · 12/07/2025

Main result 2: we generalize these predictions to Craftax, a partially observable, 2D minecraft domain. Here, we once again find evidence that people preplay completion of tasks that were accessible but unpursued, but now in a much larger world where generalization to new tasks is much harder.

110

cogscikid.bsky.social @cogscikid.bsky.social · 12/07/2025

Main result 1: Across 4 experiments, we find evidence that people preplay completion of tasks that were accessible but unpursued, even if they don't know those tasks will come up later on.

110

cogscikid.bsky.social @cogscikid.bsky.social · 12/07/2025

Key idea behind Multitask Preplay: when people engage in replay of one task, they might "preplay" completion of another task and leverage temporal-difference learning to cache the results into a neural network, enabling fast, automatic behavior for that task later on. arxiv.org/abs/2507.05561

130

cogscikid.bsky.social @cogscikid.bsky.social · 12/07/2025

Excited to share a new project spanning cognitive science and AI where we develop a novel deep reinforcement learning model---Multitask Preplay---that explains how people generalize to new tasks that were previously accessible but unpursued.

2449

cogscikid.bsky.social @cogscikid.bsky.social · 17/06/2025

NiceWebRL allows you to use the *exact same* environment for both developing computational models (AI or cognitive) and for conducting online human subject experiments

110

cogscikid.bsky.social @cogscikid.bsky.social · 16/06/2025

We discuss different avenues for garnering understanding: from looking at data as a source for explanation, to mechanistic interpretability, to developing reductive explanations that link task-performing models to the natural behaviors we observe.

120

cogscikid.bsky.social @cogscikid.bsky.social · 16/06/2025

We first describe how cognitive science can take lessons from the engineering practices of machine learning to better develop generalizable models. In particular, we emphasize frictionless reproducibility, generalizability-focused model development , and the utility of hypothesis-driven benchmarks.

110

cogscikid.bsky.social @cogscikid.bsky.social · 16/06/2025

Afterwards, we motivate learning-based models by highlighting how AI research has revealed that learning with naturalistic data can lead to qualitatively different patterns of behavior and generalization compared to simplified settings. This may have profound implications for our cognitive theories.

110

cogscikid.bsky.social @cogscikid.bsky.social · 16/06/2025

We first motivate the need for naturalistic experimental paradigms by showcasing examples from neuroscience, cognitive science, and AI where learning, behavior, and brain computation manifest differently in naturalistic experimental conditions.

120

cogscikid.bsky.social @cogscikid.bsky.social · 16/06/2025

For example, thanks to technologies like generative AI and augmented reality, we can now create *synthetic*, parametrically controllable naturalistic environments.

120

cogscikid.bsky.social @cogscikid.bsky.social · 16/06/2025

Naturalistic computational cognitive science is framework for theory-driven cognitive science that aims to develop theories that can predict and explain human behavior in increasingly naturalistic conditions.

120

cogscikid.bsky.social @cogscikid.bsky.social · 16/06/2025

In this paper, we specify a research strategy for building models and theories that aim to characterize natural behavior across the settings it manifests. To help readers, we provide an outline of our complete argument as our first figure. arxiv.org/abs/2502.20349

130

cogscikid.bsky.social @cogscikid.bsky.social · 16/06/2025

Excited to share this project specifying a research direction I think will be particularly fruitful for theory-driven cognitive science that aims to explain natural behavior! We're calling this direction "Naturalistic Computational Cognitive Science"

39922

cogscikid.bsky.social @cogscikid.bsky.social · 20/02/2025

Excited to return to my PhD institution, UMich, to give this talk next week! ece.engin.umich.edu/event/invite...

000

cogscikid.bsky.social @cogscikid.bsky.social · 10/02/2025

The second example we have allows two people to coordinate on "cooking" tasks in an environment called "overcooked". Importantly, this supports the popular JaxMARL (Multi-agent RL) codebase and all of its environments

000

cogscikid.bsky.social @cogscikid.bsky.social · 10/02/2025

Right now, I'm setting up an "examples" folder with (ideally) domains + algorithms researchers already care about. Are there any examples the community would like to see? The first example we have allows a human to control an agent in a 2D minecraft environment known as "crafter"

100

cogscikid.bsky.social @cogscikid.bsky.social · 10/02/2025

nicewebrl supports a growing set of environments (50+) developed by the research community

100

cogscikid.bsky.social @cogscikid.bsky.social · 10/02/2025

Are you interested in comparing modern AI/RL agents to Humans? To make it easier for researchers spanning cognitive science, multi-agent RL, and HCI to do this, I've made nicewebrl github.com/wcarvalho/ni...

131