iiinova — where your digital self keeps evolving

This work was made possible by our amazing coauthors (inc. @iscienceluvr.bsky.social @ptoncompmemlab.bsky.social) and Stability AI 🙏

010

This project was developed using an open lab approach where we publicly worked with volunteers in the MedARC Discord. We are continuing to work on neuroAI projects in-the-open, check out our lab website to learn more: medarc.ai/fmri

This work shows it is now practical for patients to undergo a single MRI scanning session and produce enough data to perform high-quality image reconstructions. This could enable novel clinical diagnosis and assessment approaches, including locked-in patient communication and BCIs.

110

If we use the full 40 hours instead of 1, we get SOTA performance for reconstruction and retrieval. We found that the 1-hour setting offered a good balance between scan duration and reconstruction performance, with notable improvements from first pre-training on other subjects.

110

Other innovations: 1. Mapping fMRI to OpenCLIP & reconstructing via new fine-tuned Stable Diffusion XL unCLIP model. 2. Merging previously independent high- and low-level pipelines into one. 3. Predict text captions for conditional guidance during a final refinement step.

110

How do you do shared-subject modeling when brains are differently shaped with different functional topography? We first do subject-specific ridge regression to a shared latent space, followed by subject-agnostic non-linear mapping, and train this single model end-to-end.

Past work trained independent models per person, with each person needing dozens of hours of training data in the MRI machine for high-quality results. We show it’s now possible to get high-quality reconstructions from a single visit to the MRI facility.

130

🧠👁️ Our MindEye2 preprint is out! We reconstruct seen images from fMRI activity using only 1 hour of training data. This is possible by first pretraining a shared-subject model using other people's data, and then fine-tuning on a held-out subject with only 1 hr of data. arxiv.org/abs/2403.11207

2249

I am happy to say I'm not even leaving Princeton since I can work remotely for Stability; I'll become a visiting research scientist in the Norman lab to continue our existing projects while establishing new collaborations between MedARC and Princeton.

000

Also a massive thank you to my postdoc advisor Ken Norman for taking me into his lab, allowing me the freedom to pursue these crazy projects, and for being such a caring, hardworking, and insightful mentor throughout our time together.

100

I am extremely grateful for this opportunity. Huge thank you to Tanishq Abraham, PhD (CEO of MedARC) for his leadership in creating MedARC and Stability AI for being willing to invest in open neuroscience research.

100

This approach to doing research redefines the traditional research model, following similar footsteps as initiatives like EleutherAI, LAION, OpenBioML, and ML Collective.

100

Notion – The all-in-one workspace for your notes, tasks, wikis, and databases.

Happy to share that I am now Head of Neuroimaging at Stability AI! My role is to lead the MedARC Neuroimaging & AI Lab: medarc.ai/fmri The lab is remote, open-source, and open to the public to join. Tons of potential to rethink what a research lab is and benefit from crowd-sourced intelligence.

medarc.ai

A new tool that blends your everyday work apps into one. It's the all-in-one workspace for you and your team

Paul Scotti @humanscotti.bsky.social · 10/10/2023

arxiv link was pasted wrong 😓 heres correct link: arxiv.org/abs/2305.182...