iiinova — where your digital self keeps evolving

Reposted by Ken Shirakawa

Alessandro Gifford @alessandrogifford.bsky.social · 12/02/2026

NSD-synthetic, the out-of-distribution companion dataset of NSD consisting of 7T fMRI responses to 284 artificial images, is now published. #NeuroAI #CompNeuro #neuroscience #AI doi.org/10.1038/s414...

02714

Ken Shirakawa @kencan7749.bsky.social · 10/02/2026

This thread aligns closely with the core claim of our paper. While naturalistic stimuli are highly valuable, large-scale natural data can yield spurious successes due to unintended shortcuts in complex analysis pipelines. doi.org/10.1016/j.ne...

doi.org

Redirecting

010

Reposted by Ken Shirakawa

Ken Shirakawa @kencan7749.bsky.social · 13/06/2025

Our paper is now accepted at Neural Networks! This work builds on our previous threads in X, updated with deeper analyses. We revisit brain-to-image reconstruction using NSD + diffusion models—and ask: do they really reconstruct what we perceive? Paper: doi.org/10.1016/j.ne... 🧵1/12

doi.org

Redirecting

133

Ken Shirakawa @kencan7749.bsky.social · 13/06/2025

And here’s an experimental podcast-style of paper summary, generated via Notebook LM directed by me! Link: notebooklm.google.com/notebook/9c8...

notebooklm.google.com

000

Ken Shirakawa @kencan7749.bsky.social · 13/06/2025

This project wouldn’t have been done without the support of all our lab members. Huge thanks to co-authors, and especially to Prof. Kamitani ( @ykamit.bsky.social), for their invaluable support throughout this work!

010

Ken Shirakawa @kencan7749.bsky.social · 13/06/2025

Our paper goes further to formal analysis —including mathematical analysis, simulations, analysis of AI model representations, evaluation pitfalls, and meta-level insights into “realistic” reconstruction. If this thread sparked your interest, please take a look at our paper!

100

Ken Shirakawa @kencan7749.bsky.social · 13/06/2025

NSD’s image diversity is smaller than expected, but this doesn’t diminish its value. New datasets like NSD-synthetic (arxiv.org/abs/2503.06286) and NSD-imagery (www.arxiv.org/abs/2506.06898) will also be valuable. Yet, we should consider data splits that align with your research goals.

arxiv.org

A 7T fMRI dataset of synthetic images for out-of-distribution modeling of vision

Large-scale visual neural datasets such as the Natural Scenes Dataset (NSD) are boosting NeuroAI research by enabling computational models of the brain with performances beyond what was possible just ...

100

Ken Shirakawa @kencan7749.bsky.social · 13/06/2025

So, how should we interpret these reconstruction methods? We argue they’re better understood as visualizations of decoded content, not true reconstructions. Visualization itself also has value, but it’s crucial to recognize the huge gap between visualization and reconstruction.

100

Ken Shirakawa @kencan7749.bsky.social · 13/06/2025

Taken together, our results suggest recent diffusion-based reconstructions are a mix of classification into trained categories and hallucination by generative AIs. This deviates fundamentally from genuine visual reconstruction, which aims to recover arbitrary visual experiences.

100

Ken Shirakawa @kencan7749.bsky.social · 13/06/2025

What about the Generator (diffusion model)? We fed it true image features instead of predicted ones. The outputs were semantically similar—but perceptually quite different. It seems the Generator relies mainly on semantic features, with less focus on perceptual fidelity.

100

Ken Shirakawa @kencan7749.bsky.social · 13/06/2025

Given the overlap between training/test sets, can the Translator predict test stimuli effectively? Careful identification analyses revealed a fundamental limitation in generalizing beyond the training distribution. Translator, though a regressor, behaves more like a classifier.

100

Ken Shirakawa @kencan7749.bsky.social · 13/06/2025

We first check Latent features. UMAP visualization of NSD’s CLIP features revealed (A): - distinct clusters (~40) - substantial overlap between training and test sets NSD test images were also perceptually similar to training images (B), unlike in carefully curated Deeprecon (C).

100

Ken Shirakawa @kencan7749.bsky.social · 13/06/2025

To better understand what was happening, we decomposed these methods into a Translator–Generator pipeline. The Translator maps brain activity to the Latent features, and the Generator converts those features into images. We analyzed each component in detail.

100

Ken Shirakawa @kencan7749.bsky.social · 13/06/2025

We tested whether these methods generalize beyond NSD. They worked well on NSD (A), but performance severely dropped on Deeprecon (B). The latest MindEye2 even generated training-set categories unrelated to test stimuli. So what’s behind this generalization failure?

100

Ken Shirakawa @kencan7749.bsky.social · 13/06/2025

“Reconstruction” is often seen as recovering any instance from a space of interest. Prior works (e.g., Miyawaki+ 2008, Shen+ 2019) pursued this goal. Recent studies report realistic reconstructions from NSD using CLIP + diffusion models. But—do they truly achieve this goal?

100

Ken Shirakawa @kencan7749.bsky.social · 13/06/2025

doi.org

Redirecting

133

Reposted by Ken Shirakawa

arxiv cs.CV @arxiv-cs-cv.bsky.social · 14/05/2025

Yukiyasu Kamitani, Misato Tanaka, Ken Shirakawa Visual Image Reconstruction from Brain Activity via Latent Representation arxiv.org/abs/2505.08429

043

Reposted by Ken Shirakawa

Martin Hebart @martinhebart.bsky.social · 11/12/2024

One big issue with some of the previous claims are that NSD, the massive 7T fMRI dataset of 1000s of images, might not be the right dataset to test these hypotheses. The reason is that it is built on MSCoCo and has too high similarity between training and test. arxiv.org/abs/2405.10078 16/n

arxiv.org

161

Ken Shirakawa @kencan7749.bsky.social · 16/11/2024

I’m currently concerned about what the brain’s encoding model predicts. Given that the target brain state is collected under naturalistic condition and the inputs of encoding model derived from a deep neural network, I am not sure what the predictions are actually represent.

000