Sign in

Lerrel Pinto

@lerrelpinto.com
3.1K followers 233 following 62 posts

Assistant Professor of CS @nyuniversity. I like robots!

PostsRepliesMedia
Lerrel Pinto @lerrelpinto.com · 18/04/2025
We just released RUKA, a $1300 humanoid hand that is 3D-printable, strong, precise, and fully open sourced! The key technical breakthrough here is that we can control joints and fingertips of the robot **without joint encoders**. All we need here is self-supervised data collection and learning.
1287
Lerrel Pinto @lerrelpinto.com · 28/03/2025
When life gives you lemons, you pick them up. (trained with robotutilitymodels.com)
1154
Lerrel Pinto @lerrelpinto.com · 28/02/2025
Point Policy uses sparse key points to represent both human demonstrators and robots, bridging the morphology gap. The scene is hence encoded through semantically meaningful key points from minimal human annotations.
100
Lerrel Pinto @lerrelpinto.com · 28/02/2025
The robot behaviors shown below are trained without any teleop, sim2real, genai, or motion planning. Simply show the robot a few examples of doing the task yourself, and our new method, called Point Policy, spits out a robot-compatible policy!
1205
Lerrel Pinto @lerrelpinto.com · 26/02/2025
We just released AnySense, an iPhone app for effortless data acquisition and streaming for robotics. We leverage Apple’s development frameworks to record and stream: 1. RGBD + Pose data 2. Audio from the mic or custom contact microphones 3. Seamless Bluetooth integration for external sensors
23510
Lerrel Pinto @lerrelpinto.com · 20/02/2025
Just found a new winner for the most hype-baiting, unscientific plot I have seen. (From the recent Figure AI release)
1376
Lerrel Pinto @lerrelpinto.com · 18/12/2024
At NYU Abu Dhabi today and in love how cat friendly the campus is!
2111
Lerrel Pinto @lerrelpinto.com · 10/12/2024
P3-PO uses a one time “point prescription” by a human to identify key points. After this it uses semantic correspondence to find the same points on different instances of the same object.
151
Lerrel Pinto @lerrelpinto.com · 10/12/2024
New paper! We show that by using keypoint-based image representation, robot policies become robust to different object types and background changes. We call this method Prescriptive Point Priors for robot Policies or P3-PO in short. Full project is here: point-priors.github.io
1377
Lerrel Pinto @lerrelpinto.com · 09/12/2024
BAKU consists of three modules: 1. Sensor encoders for vision, language, and state 2. Observation trunk to fuse multimodal inputs 3. Action head for predicting actions. This allows BAKU to combine different action models like VQ-BeT and Diffusion Policy under one framework.
230
Lerrel Pinto @lerrelpinto.com · 09/12/2024
Modern policy architectures are unnecessarily complex. In our #NeurIPS2024 project called BAKU, we focus on what really matters for good policy learning. BAKU is modular, language-conditioned, compatible with multiple sensor streams & action multi-modality, and importantly fully open-source!
1309
Lerrel Pinto @lerrelpinto.com · 08/12/2024
RUMs is the brainchild of @notmahi.bsky.social with several insightful experiments. The most important one being that data diversity >> data quantity. Another insight is that regardless of the algorithm there is a similar-ish scaling law across tasks. Check out the paper: arxiv.org/abs/2409.05865
140
Lerrel Pinto @lerrelpinto.com · 08/12/2024
Our awesome undergrad lead on this project @haritheja.bsky.social took RUMs to Munich for CoRL 2024 and showed it work zero-shot in opening doors and drawers bought from German IKEA.
130
Lerrel Pinto @lerrelpinto.com · 08/12/2024
There are three main components to build RUMs: diverse expert data + multi-modal behavior cloning + mLLM feedback. hardware, code & pretrained policies are fully opensourced: robotutilitymodels.com
130
Lerrel Pinto @lerrelpinto.com · 08/12/2024
Since we are nearing the end of the year, I'll revisit some of our work I'm most excited about from the last year and maybe a sneak peek of what we are up to next. To start of, Robot Utility Models, which enables zero-shot deployment. In the video below, the robot hasnt seen these doors before.
2368
Lerrel Pinto @lerrelpinto.com · 24/11/2024
We got stranded for a day in rural NY without electricity, running water, heat, internet, or cell service. It is crazy how difficult it is to live without these relatively modern inventions. I hope one day robots will join this list.
A snowy day in Windham, NY.
1233
Lerrel Pinto @lerrelpinto.com · 01/11/2024
In our latest project, we train robots to mimic a human video of the task by matching the object features using RL. We only need one video and under an hour of robot training. Project was led by Irmak Guzey w/ Yinlong Dai, Georgy Savva and Raunaq Bhirangi. More details: object-rewards.github.io
061
Lerrel Pinto @lerrelpinto.com · 25/10/2024
It is really hard to get robot policies that are both precise (small margins for error) and general (robust to variations). We just released ViSk, where skin sensing is used to train fine-grained policies with ~1 hour of data. I have attached a single-take video on this post. visuoskin.github.io
2101