iiinova — where your digital self keeps evolving

Reposted by Lerrel Pinto

Robot Talk @robottalkpod.bsky.social · 21/05/2025

How is AI helping robots to generalise their skills to unfamiliar environments? 🤖 🏠 In the latest episode, I chatted to Prof. Lerrel Pinto (@lerrelpinto.com) from New York University about #robot learning and decision making. Available wherever you get your podcasts: linktr.ee/robottalkpod

021

Lerrel Pinto @lerrelpinto.com · 18/04/2025

This project, which combines hardware design with learning-based controllers was a monumental effort led by @anyazorin.bsky.social and Irmak Guzey. More links and information about RUKA are below: Website: ruka-hand.github.io Assembly Instructions: ruka.gitbook.io/instructions

010

Lerrel Pinto @lerrelpinto.com · 18/04/2025

We just released RUKA, a $1300 humanoid hand that is 3D-printable, strong, precise, and fully open sourced! The key technical breakthrough here is that we can control joints and fingertips of the robot **without joint encoders**. All we need here is self-supervised data collection and learning.

1287

Lerrel Pinto @lerrelpinto.com · 29/03/2025

This would be funny! 😂

000

Lerrel Pinto @lerrelpinto.com · 28/03/2025

When life gives you lemons, you pick them up. (trained with robotutilitymodels.com)

1154

Reposted by Lerrel Pinto

Robot Talk @robottalkpod.bsky.social · 18/03/2025

What would you love to know about #robot learning and decision making? Later this season, I'll be chatting to Prof. Lerrel Pinto (@lerrelpinto.com) from NYU about using machine learning to train robots to adapt to new environments. Send me your questions for Lerrel: robottalk.org/ask-a-question/

0137

Lerrel Pinto @lerrelpinto.com · 02/03/2025

Is there a word for the feeling when you want to cheer for the other team?

150

Lerrel Pinto @lerrelpinto.com · 28/02/2025

The overall algorithm is simple: 1. Extract key points from human videos. 2. Train a transformer policy to predict future robot key points. 3. Convert predicted key points to robot actions.

110

Lerrel Pinto @lerrelpinto.com · 28/02/2025

Point Policy uses sparse key points to represent both human demonstrators and robots, bridging the morphology gap. The scene is hence encoded through semantically meaningful key points from minimal human annotations.

100

Lerrel Pinto @lerrelpinto.com · 28/02/2025

The robot behaviors shown below are trained without any teleop, sim2real, genai, or motion planning. Simply show the robot a few examples of doing the task yourself, and our new method, called Point Policy, spits out a robot-compatible policy!

1205

Reposted by Lerrel Pinto

Chris Paxton @cpaxton.bsky.social · 26/02/2025

This is important because the humble iPhone is one of the best accessories for embodied AI out there, if not actually the best. It's got a depth sensor, good camera, built-in internet, decent compute, and -- uniquely -- it has really good slam already built in.

3154

Lerrel Pinto @lerrelpinto.com · 26/02/2025

It should be accessible in EU now!

110

Lerrel Pinto @lerrelpinto.com · 26/02/2025

AnySense is built to empower researchers with better tools for robotics. Try it out below. Download on App store: apps.apple.com/us/app/anyse... Open-source code on GitHub: github.com/NYU-robot-le... Website: anysense.app AnySense is led by @raunaqb.bsky.social with several from NYU.

apps.apple.com

‎AnySense

‎AnySense is an open-source iPhone app that enables multi-sensory data collection by integrating the iPhone’s sensory suite with external sensors via Bluetooth and wired interfaces, enabling both offl...

030

Lerrel Pinto @lerrelpinto.com · 26/02/2025

With this 'wild' robot data, data collected by AnySense can then be used to train multimodal policies! In the video above, we use the Robot Utility Models framework to train Visuo-Tactile policies for a whiteboard erasing task. You can use it for so much more though!

130

Lerrel Pinto @lerrelpinto.com · 26/02/2025

We just released AnySense, an iPhone app for effortless data acquisition and streaming for robotics. We leverage Apple’s development frameworks to record and stream: 1. RGBD + Pose data 2. Audio from the mic or custom contact microphones 3. Seamless Bluetooth integration for external sensors

23510

Reposted by Lerrel Pinto

Eugene Vinitsky 🍒 @eugenevinitsky.bsky.social · 23/02/2025

A useful “productivity” trick is to remind yourself that research should be fun and inspiring and if it’s not that something should change.

2727

Lerrel Pinto @lerrelpinto.com · 20/02/2025

Just found a new winner for the most hype-baiting, unscientific plot I have seen. (From the recent Figure AI release)

1376

Reposted by Lerrel Pinto

Eugene Vinitsky 🍒 @eugenevinitsky.bsky.social · 20/02/2025

One reason to be intolerant of misleading hype in tech and science is that tolerating the small lies and deception is how you get tolerance of big lies

418427

Lerrel Pinto @lerrelpinto.com · 18/02/2025

Thanks Tucker! The timing of this is great given the uncertainty with other funding mechanisms.

000

Lerrel Pinto @lerrelpinto.com · 18/02/2025

Thank you to @sloanfoundation.bsky.social for this generous award to our lab. Hopefully this will bring us closer to building truly general-purpose robots!

3234

Lerrel Pinto @lerrelpinto.com · 13/02/2025

Yes, this is one of our inspirations!

110

Lerrel Pinto @lerrelpinto.com · 13/02/2025

A fun, clever idea from @upiter.bsky.social : treat code generation as a sequential editing problem -- this gives you loads of training data from synthetically editing existing code And it works! Higher performance on HumanEval, MBPP, and CodeContests across small LMs like Gemma-2, Phi-3, Llama 3.1

150

Lerrel Pinto @lerrelpinto.com · 07/02/2025

Thanks Eugene! Sounds exciting!

010

Lerrel Pinto @lerrelpinto.com · 07/02/2025

Hi Eugene, this sounds cool! Could you comment a bit on how well simulated driving agents translate to real world driving?

170

Lerrel Pinto @lerrelpinto.com · 31/01/2025

We have been working a bunch on offline world models. Pre-trained features from DINOv2 seem really powerful for modeling. I hope this opens up a whole set of applications for decision making and robotics! Check out the thread from @gaoyuezhou.bsky.social for more details.

040

Lerrel Pinto @lerrelpinto.com · 28/01/2025

nah they are friendly cat food by folks around NYU AD.

000

Reposted by Lerrel Pinto

Eugene Vinitsky 🍒 @eugenevinitsky.bsky.social · 09/01/2025

If you’re in grad school, finding a therapist can be really helpful. The thing you’re doing is hard and it’s harder if you don’t have help managing imposter syndrome, stress, self esteem, and a whole bunch of other things.

56513

Reposted by Lerrel Pinto

Laine Nooney @ 🔥🗑️🔥 @lainenooney.bsky.social · 30/12/2024

omg a student somehow accidentally wrote an email addressed to a faculty-wide NYU listserv and my inbox is now a master class on who understands the difference between a listserv and an email chain

2035382932

Reposted by Lerrel Pinto

Florian Ederer @florianederer.bsky.social · 25/12/2024

Humans vs Ants: Problem-solving Skills

911632

Lerrel Pinto @lerrelpinto.com · 18/12/2024

At NYU Abu Dhabi today and in love how cat friendly the campus is!

2111

Reposted by Lerrel Pinto

Daniel Rolph Author @danielrolphauthor.bsky.social · 15/12/2024

This holiday season, take a moment to visit your local bookstore. It’s about more than finding a great book—it’s about supporting the small businesses that keep our communities thriving.

572367312

Reposted by Lerrel Pinto

Remi Cadene @remicadene.bsky.social · 15/12/2024

HOT 🔥 fastest, most precise, and most capable hand control setup ever... Less than $450 and fully open-source 🤯 by @huggingface, @therobotstudio, @NepYope This tendon-driven technology will disrupt robotics! Retweet to accelerate its democratization 🚀 A thread 🧵

37327

Reposted by Lerrel Pinto

Alfredo Canziani @alfcnz.bsky.social · 13/12/2024

Outstanding presentation, finally! DynaMo: In-Domain Dynamics Pretraining for Visuo-Motor Control @jeffacce.bsky.social @lerrelpinto.com

0112

Lerrel Pinto @lerrelpinto.com · 10/12/2024

P3-PO uses a one time “point prescription” by a human to identify key points. After this it uses semantic correspondence to find the same points on different instances of the same object.

151

Lerrel Pinto @lerrelpinto.com · 10/12/2024

New paper! We show that by using keypoint-based image representation, robot policies become robust to different object types and background changes. We call this method Prescriptive Point Priors for robot Policies or P3-PO in short. Full project is here: point-priors.github.io

1377

Lerrel Pinto @lerrelpinto.com · 09/12/2024

More details are here: baku-robot.github.io BAKU was led by @haldarsiddhant.bsky.social who is will be presenting at NeurIPS this Thursday from 11 a.m. PST — 2 p.m. PST. So catch him if you around!

020

Lerrel Pinto @lerrelpinto.com · 09/12/2024

BAKU consists of three modules: 1. Sensor encoders for vision, language, and state 2. Observation trunk to fuse multimodal inputs 3. Action head for predicting actions. This allows BAKU to combine different action models like VQ-BeT and Diffusion Policy under one framework.

230

Lerrel Pinto @lerrelpinto.com · 09/12/2024

Modern policy architectures are unnecessarily complex. In our #NeurIPS2024 project called BAKU, we focus on what really matters for good policy learning. BAKU is modular, language-conditioned, compatible with multiple sensor streams & action multi-modality, and importantly fully open-source!

1309

Reposted by Lerrel Pinto

Raunaq Bhirangi @raunaqb.bsky.social · 09/12/2024

Robot utility models are not just among the first learned models that work zero-shot on a mobile manipulator, but also provide a nuanced discussion on what works and what doesn't in data-driven robot learning.

071

Lerrel Pinto @lerrelpinto.com · 08/12/2024

Maybe you have to force the baby to use only two fingers 😆

170

Lerrel Pinto @lerrelpinto.com · 08/12/2024

Yeah I agree we need to be more thoughtful. Even if one believe that data scaling is the answer, there is so much algorithmic work left to be done to make the process more efficient.

010

Lerrel Pinto @lerrelpinto.com · 08/12/2024

The policies here use only video as we want the trained policies to also work on webcam cameras. But in a previous project (dobb-e.com) we use the depth from it as well.

dobb-e.com

Dobb·E: On Bringing Robots Home

Dobb·E is an open-source framework for teaching robots new household tasks in 20 minutes via imitation learning.

040

Lerrel Pinto @lerrelpinto.com · 08/12/2024

Thank you! I think there is a nuanced conversation here. I think data is a crucial piece for robots, but pure data scaling for all possible things you would want your robot to do will need an ungodly amount of data.

110

Lerrel Pinto @lerrelpinto.com · 08/12/2024

Yes, that is a great catch. iPhones are an amazing vision sensor, much better than Realsense or other cameras we have tried.

120

Lerrel Pinto @lerrelpinto.com · 08/12/2024

Lol we thought quite a bit about the name 😆

010