iiinova — where your digital self keeps evolving

- clean out the main repo and split off some parts to new repos under the new github organization (eg robot assets/python classes and some baseline algorithms) - better support for policy evals

Additionally there’s a new roadmap that is reflective what I’ve been planning to improve in the future. Some plans include - switching to Newton/Mujoco warp backend - a big redo of some apis inherited from ManiSkill2 based on feedback and my own pain points using maniskill

120

Community for developing the Maniskill robotics simulation framework - ManiSkill

The ManiSkill project now has a new home under a new mani-skill github organization, moving out of the old UCSD lab org. Long overdue and done as part of my plans for longer term maintainance as well as more organized community driven development github.com/mani-skill

github.com

ManiSkill

151

- clean out the main repo and split off some parts to new repos under the new github organization (eg robot assets/python classes and some baseline algorithms) - better support for policy evals

Stone Tao @stonet2000.bsky.social · 02/06/2026

and in the off chance you are a saber fencer, I brought my fencing gear (and will be at the Salle Auriol Seattle fencing club) 🤺

Stone Tao @stonet2000.bsky.social · 02/06/2026

in Seattle for 2 weeks finishing up my work at AI2 (at their extremely beautiful office). lmk if you want to meet!

Stone Tao @stonet2000.bsky.social · 13/05/2026

First blog post up on Robotics Simulation Infrastructure! I give a high-level overview, followed by an elementary example of better infrastructure for pose management. stoneztao.substack.com/p/robotics-s...

stoneztao.substack.com

Robotics Simulation Infrastructure Part 1

First part of a blog series diving into the world of robotics simulation and machine learning

0375

Stone Tao @stonet2000.bsky.social · 02/04/2026

Wasn't at GTC but a sneak peak of some of my internship work at NVIDIA was! (sim2real cable routing presented by my manager Yashraj) www.youtube.com/watch?v=d5wm...

youtube.com

AI Research Breakthroughs from NVIDIA Research (Hosted by Karoly of Two Minute Papers) | NVIDIA GTC

YouTube video by NVIDIA Developer

Stone Tao @stonet2000.bsky.social · 02/04/2026

some points of comparisons could include e.g. manager based configs (mjlab/isaac lab) vs direct style (maniskill/robosuite/also isaac lab), wrappers vs no gym wrappers, features for simplifying parallelization, visualization tooling and more

Stone Tao @stonet2000.bsky.social · 02/04/2026

would people be interested in a blog series on "simulation infrastructure" where I look at popular robotics sim tools (e.g. mjlab, isaac...) and explain the pros and cons of their design decisions. The series would cover what I believe to be key pillars of good, usable, sim code

2191

Stone Tao @stonet2000.bsky.social · 01/11/2025

I will be in SF from November 10-17 🌉 If you work on something interesting and want to meet up let me know! Or if there’s a fun event i’d love some invites 😃 I’m also looking to visit robotics companies (especially startups!), if you have time lmk if I can visit!

Stone Tao @stonet2000.bsky.social · 19/09/2025

1/1 neurips submission accepted! Will share more details later, covers an often overlooked aspect of on policy RL training when scaling to large scale parallelized environments

020

Reposted by Stone Tao

Nathan Lambert @natolambert.bsky.social · 19/09/2025

Rumors going around are that NeurIPS PCs had to reject ~400 papers because of venue constraints that were in the accept pile. Hard things like this happen but the PCs and conference should explicitly tell all the authors who had this happen to them. Or create something like accept w/o presentation.

1191

Stone Tao @stonet2000.bsky.social · 17/09/2025

what’s your next blog going to be? i’ve been also meaning to pick it up again but writers block lol

Stone Tao @stonet2000.bsky.social · 09/09/2025

It works best if the object meshes you use for diff. rendering have a corresponding segmentation mask. If there are occlusions then the default option of using SAM2 to generate segmentations will perform a little worse. That being said even if the mask has some gaps the optimization can work

Stone Tao @stonet2000.bsky.social · 09/09/2025

Big thanks to Linghao Chen (author of EasyHEC) for helping out. I simply made some examples and simplified some code for reproducibility/readability.

030

Stone Tao @stonet2000.bsky.social · 09/09/2025

Opensourcing a tool to calibrate camera extrinsics painlessly in a minute, no checkerboards! It's based on EasyHEC, using diff. rendering to optimize extrinsics given object meshes+poses. Crazy that even a piece of paper works too. Code: github.com/StoneT2000/s... (paper example in next post)

2247

Stone Tao @stonet2000.bsky.social · 19/07/2025

ManiSkill3 simple example, Mujoco Playground, and the DextrahRGB paper all do very similar things. I still think transfer techniques are useful, i’d think of these sim2real attempts as just explorations in the large scale visual DR space with parallel rendering. Both could be used together probably

Stone Tao @stonet2000.bsky.social · 19/07/2025

see section D of maniskill3 paper: arxiv.org/abs/2410.00425 tldr: Green screen background, segment out robot and cube in sim and only render that. Large scale visual domain randomization (robot, cube textures, camera parameters) very simple trick that is fairly reproducible for simple tasks

arxiv.org

ManiSkill3: GPU Parallelized Robotics Simulation and Rendering for Generalizable Embodied AI

Simulation has enabled unprecedented compute-scalable approaches to robot learning. However, many existing simulation frameworks typically support a narrow range of scenes/tasks and lack features crit...

Stone Tao @stonet2000.bsky.social · 19/07/2025

might be the first time I made it personally on the trending developers list on github! One of the contributing factors being the lerobot sim2real code

1170

Reposted by Stone Tao

Chris Paxton @cpaxton.bsky.social · 05/07/2025

Combining sim to real with cheap hardware like the so-100 really is making cutting edge robot learning so accessible and its wonderful

5487

Stone Tao @stonet2000.bsky.social · 05/07/2025

That doesn’t seem to make sense, you don’t need a RGBD camera, phone works fine. Happy to address it more if details can be shared on github if needed

Stone Tao @stonet2000.bsky.social · 05/07/2025

what bug did you encounter with OpenCVCameraConfig?

Stone Tao @stonet2000.bsky.social · 04/07/2025

For papers that aren’t ready / i think should be rejected due to many weaknesses, i often struggle to put anything in strengths and tend to just say some general thing. I only then elaborate more detailed (i hope) in the weaknesses section

Nathan Lambert @natolambert.bsky.social · 04/07/2025

Reposted by Stone Tao

My latest post: The American DeepSeek Project Build fully open models in the US in the next two years to enable a flourishing, global scientific AI ecosystem to balance China's surge in open-source and an alternative to building products ontop of leading closed models. buff.ly/kvJQE3I

1346

Stone Tao @stonet2000.bsky.social · 04/07/2025

Awesome to see people reproducing the accessible LeRobot zero-shot sim2real project! Trained for just 90 min in ManiSkill and deployed directly in real. Sim2real is not easy, but very rewarding when it works Original post by Jianwei Zhang on LinkedIn www.linkedin.com/posts/jianwe...

0296

Stone Tao @stonet2000.bsky.social · 28/06/2025

Excited to announce that I will be interning at @nvidia research this summer on robotics/embodied AI! I’ll be in seattle for the summer, let me know if you want to meet up and chat! 🦾

0100

June 25 EEB 248 @ 3pm - Towards Embodiment Scaling Laws in Robot Locomotion by @BoAi0110 June 25 OHE 122 @ 11am - ImVR: Immersive VR Teleoperation System for General Purpose by Yulin Liu

020

Rest of my lab has various presentations at RSS2025, please check out their awesome work! June 25 EEB 248 @ 10am - Hardware Optimization for In-Hand Rotation by K. Fay

~1 hour give or take maybe 20 mins (depends how close your camera is technically), and only 8gb of GPU memory! someone even reproduced our sim2real tutorial with lerobot during the lerobot hackathon with a google colab gpu and a macbook for robot deployment

140

Stone Tao @stonet2000.bsky.social · 20/06/2025

The sim2real demo had some mixed success, hampered primarily by the lighting conditions of the outdoors. At least it worked sometimes! Hindsight says that despite the weather, low-cost nature, only 1 hour of training, anything working is a miracle

192

I’ll be at #RSS2025 from June 21 to June 23! I’ll be giving a presentation on ManiSkill3 on June 21, 5:30 PM We will also have two live demo sessions, on June 21, 12:30-2:00PM and 6:30-8:00PM. Swing by to see live demos of zero shot RGB sim2real, cool sim demos, and VR teleop!

031

Reposted by Stone Tao

Phillip Isola @phillipisola.bsky.social · 15/06/2025

Our computer vision textbook is now available for free online here: visionbook.mit.edu We are working on adding some interactive components like search and (beta) integration with LLMs. Hope this is useful and feel free to submit Github issues to help us improve the text!

visionbook.mit.edu

Foundations of Computer Vision

The print version was published by

311633

Stone Tao @stonet2000.bsky.social · 15/06/2025

Email so far ahead is interesting. Is this just substack emails?

120

Stone Tao @stonet2000.bsky.social · 15/06/2025

the amount of cope on the other site is crazy

030

Reposted by Stone Tao

Tzu-Mao Li @tzumaoli.bsky.social · 14/06/2025

Rendering nerds! Check out our latest work "Vector-Valued Monte Carlo Integration Using Ratio Control Variates" that has just gotten the best paper award at SIGGRAPH 2025. This paper presents a method that reduces variance of a wide range of rendering and diff. rendering tasks with negligible cost.

79022

Stone Tao @stonet2000.bsky.social · 13/06/2025

Code/tutorial released for🤗LeRobot zero-shot visual sim2real Reinforcement Learning! Train your SO100 robot in ManiSkill's fast simulator+renderer💨 and deploy a cube picking model zero-shot. Just in time for the LeRobot Hackathon, happy hacking! github.com/StoneT2000/l...

github.com

GitHub - StoneT2000/lerobot-sim2real: lerobot sim2real code

lerobot sim2real code. Contribute to StoneT2000/lerobot-sim2real development by creating an account on GitHub.

090

Stone Tao @stonet2000.bsky.social · 12/06/2025

ah i didn’t know! I bought this one from WowRobo, they provided these clamps