iiinova — where your digital self keeps evolving

Jim RB @jbohnslav.bsky.social · 24/07/2025

ZEBRA-CoT Dataset for vision-language reasoning where the model *generates images during the CoT*. Example: for geometry problems, it's helpful to draw lines in image space. 182K CoT labels: math, visual search, robot planning, and more. Only downside: cc-by-nc license :(

140

Jim RB @jbohnslav.bsky.social · 23/07/2025

Cool technique: RASA, Removal of Absolute Spatial Attributes. They decode grid coords and find the plane in feature space that encodes position. They basically subtract this off, baking it into the last linear layer to leave the forward pass unchanged.

110

Jim RB @jbohnslav.bsky.social · 23/07/2025

Beats or is competitive to SigLIP/2, DinoV2 on linear eval, OOD detection, linear segmentation.

100

Jim RB @jbohnslav.bsky.social · 23/07/2025

Franca Fully open vision encoder. Masks image, encodes patches, then trains student to match teacher's clusters. Key advance: Matryoshka clustering. Each slice of the embedding gets its own projection head and clustering objective. Fewer features == fewer clusters to match.

140

Jim RB @jbohnslav.bsky.social · 15/07/2025

VRU-Accident New benchmark of 1K videos, 1K captions, and 6K MCQs from accidents involving VRUs. Example: "why did the accident happen?" "(B): pedestrian moves or stays on the road." Current VLMs get ~50-65% accuracy, much worse than humans (95%).

230

Jim RB @jbohnslav.bsky.social · 15/07/2025

Side note: I've always liked Pali/Gemma's Prefix-LM masking. Why have causal attention for image tokens?

100

Jim RB @jbohnslav.bsky.social · 15/07/2025

BlindSight AMD paper: they find attention heads often have stereotyped sparsity patterns (e.g. only attending within an image, not across). They generate sparse attention variants for each prompt. Theoretically saves ~35% FLOPs for 1-2% worse on benches.

110

Jim RB @jbohnslav.bsky.social · 11/07/2025

Long-RL Nvidia paper scaling RL to long videos. First trains with SFT on a synthetic long CoT dataset, then does GRPO with up to 512 video frames. Uses cached image embeddings + sequence parallelism, speeding up rollouts >2X. Bonus: code is already up!

120

Jim RB @jbohnslav.bsky.social · 09/07/2025

They identify entropy of "wait" or "alternatively" to be strongly correlated with MMMU. Neat!

120

Jim RB @jbohnslav.bsky.social · 09/07/2025

Fine-tuning the connector at the end gives a point or two of MMMU. I wonder how much of this is benchmaxxing--I haven't seen an additional SFT stage after RL before.

100

Jim RB @jbohnslav.bsky.social · 09/07/2025

They construct their warm-start SFT data with synthetic traces from Skywork-R1V2. GRPO is pretty standard, interesting that they just did math instead of math, grounding, other possible RLVR tasks. Qwen-2.5-Instruct 32B to judges the accuracy of the answer in addition to rule-based verification.

100

Jim RB @jbohnslav.bsky.social · 09/07/2025

Skywork-R1V3: new reasoning VLM with 76% MMMU. InternViT-6B stitched with QwQ-32B. SFT warmup, GRPO on math, then a small SFT fine-tune at the end. Good benches, actual ablations, and interesting discussion. Details: 🧵

110

Jim RB @jbohnslav.bsky.social · 09/07/2025

Training: use verl with vLLM for rollouts. Limit image resolution to 1280 visual tokens. Train on 32 H100s. Results: +18 points better on V* compared to Qwen2.5-VL, and +5 points better than GRPO alone.

110

Jim RB @jbohnslav.bsky.social · 09/07/2025

RL: GRPO. Reward: only correct answer, not valid grounding coordinates. Seems weird to not add that though. Data: training subset of MME-RealWorld. Evaluate on V*.

100

Jim RB @jbohnslav.bsky.social · 09/07/2025

MGPO: multi-turn grounding-based policy optimization. I've been waiting for a paper like this! Trains the LLM to iteratively crop regions of interest to answer a question, and the only reward is the final answer. Details in thread 👇

131

Jim RB @jbohnslav.bsky.social · 08/07/2025

Using automatically generated risk category labels and the front-facing view, they have GPT4o caption the scenarios. The metrics are based on caption similarity + classification metrics on riskiness-type.

100

Jim RB @jbohnslav.bsky.social · 08/07/2025

DriveMRP: interesting method to get a VLM to understand BEV maps + driving scenarios They synthesize high-risk scenes derived from NuPlan. They render it as both a bird's eye view image and a front camera view. 👇

100

Jim RB @jbohnslav.bsky.social · 08/07/2025

SeqGrowGraph Instead of segment + postprocess, generate lane graphs autoregressively. Node == vertex in BEV space, edge == control point for Bezier curves. At each step, a vertex is added and the adjacency matrix adds one row + column. They formulate this process as next token prediction. Neat!

110

Jim RB @jbohnslav.bsky.social · 02/07/2025

Excitingly, in one of their few shown results, multi-domain RL shows positive cross-task-transfer. Training on GUI agent data improves STEM answers, OCR, and grounding.

100

Jim RB @jbohnslav.bsky.social · 02/07/2025

They use a ton of specialized verifiers, judges, and reward models for RL.

100

Jim RB @jbohnslav.bsky.social · 02/07/2025

RL: both RLHF and RLVR with GRPO. They use humans and pass@k from prior checkpoints to judge difficulty. When training with many RL tasks, they found weakness in any one task leads to model collapse for all tasks: "effective RL demands finely tuned, hack-resistant verifiers in every domain"

100

Jim RB @jbohnslav.bsky.social · 02/07/2025

Pretraining recipe: stage 1, train all parameters for 120K steps at 8192 sequence length. Stage 2, interleaved + video data at 32K sequence length with tensor parallelism + context parallelism.

100

Jim RB @jbohnslav.bsky.social · 02/07/2025

Pretraining data 📚📷 * Captioning: 10B image-text pairs from the web * Interleaved data: websites, papers, and 100 million digitized books * OCR: 220 million images * Grounding: use GLIPv2 for images, playwright for GUIs * Video: "academic, web, and proprietary sources" * Instruction: 50M samples

100

Jim RB @jbohnslav.bsky.social · 02/07/2025

Model details: Uses AIMv2 as the vision encoder and GLM for the LLM, both unique choices. They add 3D convs to the vision encoder to downsample videos by 2X like Qwen 2-VL.

100

Jim RB @jbohnslav.bsky.social · 02/07/2025

GLM-4.1V-Thinking: new reasoning VLM with heavy emphasis on RL. Tons of hints but few ablations 😞 eg they upweight difficult-but-learnable samples every iteration, but don't show how it compares to baseline. 9B variant beats Qwen2.5-VL-7B on many standard benchmarks. Details in thread 👇

100

Jim RB @jbohnslav.bsky.social · 01/07/2025

Image-level caption details: stitch all the object captions into one global caption. You could just image + object captions in prompt -> unified caption, but there are too many objects for this to work. So, they crop the image into sub-images first then concat the outputs.

100

Jim RB @jbohnslav.bsky.social · 01/07/2025

Object-level caption details: RAM + APE + SAM for seg. Crop each object + caption with InternVL2.5-78B. Then caption the object in context of its original image. Qwen2.5-VL-72B to verify. Because the object captioning pipeline is so expensive, they train a 3B captioner using 600K examples.

100

Jim RB @jbohnslav.bsky.social · 01/07/2025

SAM4D: promptable 4D seg Data engine uses Grounding-Dino + SAM2 for segmentation + tracking in images. Lidar -> voxels -> ray casting to match to pixels. Clustering to stitch visual masklets into 3D instances. Model is like SAM2 but with a lidar encoder + motion-aware cross attention.

110

Jim RB @jbohnslav.bsky.social · 01/07/2025

MiCo: multi-image contrast Simple idea: Input is multiple augmented images, either from video or image edits. Prompt: "are these images the same or different?" Train with GRPO. Large bump in multi-image benchmarks, minor bump in general VQA / hallucination benches.

110

Jim RB @jbohnslav.bsky.social · 01/07/2025

SpatialReasoner-R1 Generates long CoT data with Multi-Model Monte Carlo Tree Search -- multiple candidate models for each step, evaluated by multiple LLM judges. DPO with separate losses on the "descriptive" caption and reasoning. Huge improvements on spatial datasets, good performance on VQA.

121

Jim RB @jbohnslav.bsky.social · 25/06/2025

Pretraining with ScaleCap improves benchmarks +1-3 points over ShareGPT4V.

100

Jim RB @jbohnslav.bsky.social · 25/06/2025

ScaleCap: synthetic image captioning pipeline. 2542 characters per cap. Given a caption, generates followup questions and answers with a VLM. Compute P(sentence | image,prompt) - P(sentence|prompt). Sentences with low scores are only using their text prior, so filter them out.

100

Jim RB @jbohnslav.bsky.social · 25/06/2025

UniVLA: VQA, world modeling, and robotic controls all-in-one transformer VQ-quantize images, DCT robotic actions, standard text tokens. Train the whole interleaved sequence with NTP. This type of approach would benefit from serious scaling. Anyone have a few thousand H100s laying around?

163

Jim RB @jbohnslav.bsky.social · 08/04/2025

They use MonST3R + Metric3Dv2 + grounding-SAM2 for autolabeling, very cool that you don't need ground-truth 3D labels.

100

Jim RB @jbohnslav.bsky.social · 08/04/2025

ST-Kit: new dataset + benchmark for kinematic understanding for VLMs. Examples: estimate the total distance covered by <object> in the video? In what direction does <object> move in BEV coordinates? Open source and closed source models do poorly but their fine-tune does well.

100

Jim RB @jbohnslav.bsky.social · 08/04/2025

Vision-R1: another RLVR for VLMs, this time for detection. Rewards: formatting, precision, recall. Adds +9mAP to Qwen-2.5-VL on COCO + ODINW 🤯. They change the threshold in a curriculum (easy -> hard) which adds ~2 points. Repo has training code based on R1-V and TRL 👍

100

Jim RB @jbohnslav.bsky.social · 08/04/2025

CAR-1000 Move over Stanford Cars, new dataset with 1000 hierarchical classes and 140,312 samples! Scraped from a Chinese car enthusiast forum, there's a wide variety of cars from around the world.

100

Jim RB @jbohnslav.bsky.social · 08/04/2025

NuPlanQA: uses nuPlan annotations + GPT4o to 1 million QA pairs for training and an 8K multiple-choice benchmark. BEV-LLM: baseline model. Multi-view video -> BEVFusion -> cross-attend with image features -> projector -> LLaMA3.2. Some helpful ablations on # views and frames.

100

Jim RB @jbohnslav.bsky.social · 03/04/2025

Hydra-NeXt: strong E2E closed-loop driving performance with only open-loop training Images -> encoder -> 4096 discrete trajectory vocab -> transformer -> bicycle model -> denoising diffusion refinement -> best trajectory selection. Much better closed-loop perf than UniAD, VAD

100

Jim RB @jbohnslav.bsky.social · 03/04/2025

Nice methods paper (/ advertisement) from Nvidia on training video models with NeMo. Nuts and bolts: use Ray + NVDEC for curation, S3 + WebDataset for dataloading, FSDP + TP + Context Parallelism + PP for video DiT training. 48.2% MFU

100

Jim RB @jbohnslav.bsky.social · 03/04/2025

ChatBEV: new dataset + benchmark for VQA on BEV maps for autonomous driving. Data from nuPlan. 116K train, 21K test. Render a BEV map + generate VQA with LLMs(?) Applications: VQA, use the model to condition a scene generator.

100

Jim RB @jbohnslav.bsky.social · 27/03/2025

TrajHF: diffusion-based planner fine-tuned with RLHF. Simple idea: humans have preferences for candidate trajectories. Collect human feedback on key frames from 4.5K clips mined for aggressive maneuvers. Improves these aggressive scenarios but worsens some open-loop metrics.

100

Jim RB @jbohnslav.bsky.social · 27/03/2025

DriveLMM-o1: new dataset + benchmark for autonomy VQA. Dataset is 18k QA pairs, each with step-by-step reasoning. Generated with GPT4o and human verified. Model is a fine-tuned InternVL2.5-8b. Nit: don't call your model o1 if you don't use RLVR!

100

Jim RB @jbohnslav.bsky.social · 20/03/2025

Action Dreaming: they find using only observed actions means the model ignores the instructions and just uses images. So they generate multiple instruction, action pairs for the same image. Make the other agents are open-loop and future ego position is a bicycle model.

100

Jim RB @jbohnslav.bsky.social · 20/03/2025

Generate a dataset of 3.1M samples in CARLA. Use DriveLM heuristics to turn CARLA data into captions.

100

Jim RB @jbohnslav.bsky.social · 20/03/2025

SimLingo: Wayve paper on VLAs for self driving Front camera + prompts -> InternVL2-0.5B. They add GPS targets with an MLP. Output language, and decode queries for waypoints + speed. Very strong CARLA performance, but CARLA-only training data.

110

Jim RB @jbohnslav.bsky.social · 20/03/2025

Examples look super promising.

100

Jim RB @jbohnslav.bsky.social · 20/03/2025

Key is a method to decompose captions to the smallest self-sufficient unit they call "primitive information units." Do the same for human captions, then compute F1-score. Note: DeCapBench is only 400 images!

100

Jim RB @jbohnslav.bsky.social · 20/03/2025

DeCapBench: new benchmark on super-detailed captioning. DCScore: new evaluation method for captioning, great correlation with human ratings. FeedQuill: VLM trained using PPO with a DScore-trained reward model. Great captioning specialist.

110

Jim RB @jbohnslav.bsky.social · 20/03/2025

Attention mechanisms are all vanilla. There are three types of interaction: Sensor-cross attention, Q=task tokens, KV=sensor tokens. Task self-attention: QKV=task tokens Temporal cross-attenion: Q=task tokens at t=T, KV=history tokens FIFO queue.

100