iiinova — where your digital self keeps evolving

Yanchen Wang

@ppwang.bsky.social

CS PhD Student @ Stanford Computational Neuroscience | Neural Data Science | Computer Vision

Yanchen Wang @ppwang.bsky.social · 13/10/2026

Paper: arxiv.org/abs/2606.02937 Project: ppwangyc.github.io/projects/bea... Code: github.com/paninski-lab... Joint work with my co-authors at Columbia, Cold Spring Harbor Laboratory and Stanford, listed below. Thank you all!

100

Yanchen Wang @ppwang.bsky.social · 13/10/2026

We also predict neural activity from the 3D point cloud. Bits per spike, BEAST3D vs 3D keypoints: Mouse facial motor nucleus (8 neurons): 0.39 vs 0.29 (p=0.01) Chickadee hippocampus (52 neurons): 0.11 vs 0.07 (not significant) Our 2D BEAST features score the same, but aren't tied to body locations.

Neural encoding results. Title: Higher bits per spike than 3D keypoints. Subtitle: held-out data from the mouse facial motor nucleus and the chickadee hippocampus. Left: a raster of the recorded activity of 52 chickadee neurons over time, above a raster of the activity predicted from the 3D point cloud. Right: bars of bits per spike (higher is better), the mean across neurons. Mouse, 8 neurons: 3D keypoints 0.29, BEAST 0.39, Pose Splatter 0.29, BEAST3D 0.39. Chickadee, 52 neurons: 3D keypoints 0.07, BEAST 0.11, Pose Splatter 0.08, BEAST3D 0.11.

100

Yanchen Wang @ppwang.bsky.social · 13/10/2026

For pose, we train Lightning Pose on just 100 labeled frames. BEAST3D has the lowest error on 3 of 4 datasets (VGGT wins on rat, but it's ~6x bigger), and its keypoint traces are smoother than DINOv3's. Multi-view alone isn't enough: VGGT and E-RayZer don't reliably beat single-view models.

Pose estimation trained on 100 labeled frames, chickadee shown. Left: the tracked ankle position over time from BEAST3D (green) and a DINOv3 baseline (gray). The DINOv3 trace has sudden jumps that the BEAST3D trace lacks, and in the panel below, DINOv3's reprojection error has large spikes that BEAST3D's does not. Right: pixel error plotted against keypoint difficulty for DINOv3, BEAST, VGGT, E-RayZer and BEAST3D. The curves overlap for the easiest keypoints; as keypoints get harder, BEAST3D's error rises least and ends well below the others.

100

Yanchen Wang @ppwang.bsky.social · 13/10/2026

We render each hidden camera from the others and measure foreground PSNR. Animals seen in training, BEAST3D vs the best baseline: Mouse 27.0 vs 13.6 dB Rat 22.4 vs 12.5 dB Chickadee 18.6 vs 10.2 dB Human 22.4 vs 13.2 dB On animals it hasn't seen, it's still 3-8 dB ahead.

Real images of a mouse, rat, chickadee and human from hidden cameras, next to renderings of the same views predicted from the other cameras by E-RayZer, Pose Splatter and BEAST3D. A bar chart of foreground PSNR on animals seen in training shows BEAST3D at 27.0, 22.4, 18.6 and 22.4 dB versus 13.6, 12.5, 10.2 and 13.2 dB for the best baseline.

100

Yanchen Wang @ppwang.bsky.social · 13/10/2026

During training we hide one camera, predict 3D Gaussians from the other views (frozen DINOv3 features + known camera rays), render them from the hidden camera, and compare with the real frame. It also learns to segment the animal from SAM3 masks, so no mask model is needed at inference.

Diagram of how BEAST3D is trained, shown on a head-fixed mouse filmed by six cameras. One camera is hidden. The other five views, encoded as frozen DINOv3 features plus their known camera rays, go into a geometry transformer that predicts 3D Gaussians. These are rendered from the hidden camera, and the BEAST3D render is shown next to the real image; the two closely match. A label notes that no keypoint labels are used.

100

Yanchen Wang @ppwang.bsky.social · 13/10/2026

Off-the-shelf 3D models like VGGT and E-RayZer are built for lots of overlapping views from unknown cameras. On a lab rig, with 4-6 cameras far apart and close to the animal, they break down. Lab cameras are calibrated, though, so BEAST3D just takes the calibration as input.

A grid compares lab camera images of a chickadee and a mouse with the 3D point clouds predicted from them by VGGT, E-RayZer and BEAST3D. VGGT and E-RayZer produce fragmented or noisy clouds dominated by background, while BEAST3D produces a clean point cloud of the animal alone.

100

Yanchen Wang @ppwang.bsky.social · 13/10/2026

🎉 BEAST3D is a #NeurIPS2026 Oral! Can a moving animal's 3D shape predict what its neurons are doing, better than tracked keypoints? On average, yes. BEAST3D is a self-supervised vision transformer that reconstructs animals in 3D from unlabeled multi-camera lab video. 🧵

1102