RI Seminar: Lerrel Pinto : A Constructivist’s Guide to Robot Learning

Introduction

The speaker talks about his early days as an assistant professor and how he got into robotics.

Early Days in Robotics

• In 2014, the speaker wanted to switch to robotics and bought a Baxter robot with money from Marshall.

• He put the robot in Smith Hall but no one wanted to work with him because he had no experience in robotics.

• The speaker's first Master's student, Level, took a chance on him and became the first person to bring large-scale learning to robotics.

• Level continued to do great things during his PhD, including working on home robots.

The Gap Between Human and Robot Capabilities

The speaker discusses the gap between what humans can do with their hands and what robots are currently capable of doing.

Stop-Motion Animation Movie

• The speaker shows a video from the Pinocchio movie by Del Toro, which was created using stop-motion animation.

• He notes that humans can do amazing things with their hands in complex environments while robots struggle with basic tasks like picking up objects.

Machine Learning Advancements

• There is a revolution happening in language models and computer vision thanks to machine learning advancements.

• The speaker questions how we can get the same level of capabilities in robot models as we have achieved in language and vision.

Approaches for Closing the Gap

The speaker discusses various approaches to closing the gap between human and robot capabilities.

Different Approaches

• There are many researchers looking at different lines of approaches to improve robot capabilities.

• The speaker believes that we need to focus on developing better models for robots, rather than just improving hardware or software.

Model-Based Reinforcement Learning

• The speaker introduces model-based reinforcement learning as a promising approach for improving robot capabilities.

• He explains how this approach involves building a model of the environment and using it to plan actions that will achieve a desired goal.

Challenges with Model-Based Reinforcement Learning

• The speaker acknowledges that there are challenges with model-based reinforcement learning, such as the difficulty of accurately modeling complex environments.

• He suggests that combining model-based and model-free reinforcement learning could be a promising direction for future research.

Conclusion

The speaker concludes his talk by summarizing his main points and emphasizing the importance of developing better models for robots.

Key Takeaways

• Developing better models is crucial for improving robot capabilities.

• Model-based reinforcement learning is a promising approach but has its challenges.

• Combining different approaches could lead to significant advancements in robotics.

Learning from Scratch vs Interacting and Learning

In this section, the speaker discusses two schools of thought in robotics: learning from scratch and interacting with the world to learn. He explains that while learning from scratch has yielded some amazing successes, it requires an incredibly large amount of data. Instead, he proposes a constructivist approach where humans guide the learning and interaction of robots.

Two Schools of Thought in Robotics

• There are two schools of thought in robotics: learning from scratch and interacting with the world to learn.

• Learning from scratch requires an incredibly large amount of data, which limits the complexity of tasks that can be solved.

The Constructivist Approach

• The speaker proposes a constructivist approach where humans guide the learning and interaction of robots as they build their models and collect data.

• This approach is not new; it has been done since the 80s by Chris Atkinson who taught robots amazing skills by showing them how to solve them.

• Recent approaches have also embodied this idea of imitation and having humans in the loop.

Solving Intricate Hand Manipulation Tasks with VR

In this section, the speaker discusses how to solve intricate hand manipulation tasks using virtual reality (VR) technology.

Cyber Glove vs. Image-Based Approaches

• Cyber gloves are expensive and require calibration for each user.

• Image-based approaches are cheaper but suffer from inaccuracies and occlusions.

Using Oculus Headsets for Accurate Hand-Pose Estimation

• Oculus headsets can be used to accurately estimate hand poses in real-time.

• The headset allows users to operate the system remotely and use their hands as a high-dimensional joystick.

Collecting Data for Robot Training

• Collected data across six tasks of varying difficulty levels.

• Obtained around an hour of data per task, with more data collected for easier tasks.

Behavior Cloning vs. Self-Supervised Learning

• Behavior cloning fails due to weak signal and overfitting.

• Self-supervised learning is used to train models that can generalize beyond zero percent success rate.

Representation Learning Problem

In this section, the speaker explains what a representation learning problem is and how it works.

Self-Supervised Learning

• The speaker explains that self-supervised learning is used to train an encoder without knee labels.

• Once you have a good representation, you can use it to solve a downstream task such as classification or Q&A.

Hands-on Example

• The speaker demonstrates how self-supervised learning plus imitation can be used on hands.

• Once you have an encoder, you can find the nearest neighbor and apply the same action on the robot.

• For easy tasks, such as planar rotation and spinning tubular objects, reasonable performance is achieved. However, sliding tasks are much harder to solve.

Interpolation in Action Space

In this section, the speaker discusses interpolation in action space and retrieval augmented learning.

Comparison with Large Datasets

• The speaker wishes there was a benchmark for one-to-one comparison of different methods.

• Retrieval augmented learning allows for smarter nearest neighbors and interpolation of actions.

Self-Supervised Learning for Robotic Manipulation

In this section, the speaker discusses how self-supervised learning can be used to teach robots to manipulate objects. They explain how a nearest neighbor algorithm is used to reason with different types of objects and how visual features are used to capture the motion of the hand.

Nearest Neighbor Algorithm

• The nearest neighbor algorithm is used to figure out how to manipulate new objects.

• It can automatically start flipping new objects during testing time.

• The algorithm is interpretable, allowing users to look at the nearest neighbors and see what matches are like.

• The shape of the object in contact matters more than the shape of the object out of contact.

Training and Representations

• Orientation of the base of the hand relative to the main body does matter in some cases.

• Visual features are used for representation learning, which captures motion and helps understand that two scenes or configurations are similar.

Questions and Answers

• All code is public on their website for anyone who wants to try it out.

• These models have some amount of retraining as well, maybe from an initialization of imagenet free trained weights.

• In-domain self-supervised learning has pros and cons: coverage on task but small amounts of data. Out-of-domain learning has huge amounts of data but may not have coverage on robot task.

Introduction

In this section, the speaker introduces the topic of tactile data and how it can be used to solve tasks.

Tactile Data Representation

• The speaker discusses a representation of tactile data.

• They explain that SSL algorithms are applied to the tactile data itself.

• The human hand is used to show the robot hand how to play with objects.

Non-hand Tasks

• The speaker explains that the same idea can be applied to non-hand tasks.

• A Reacher grabber is used to collect data for these tasks.

• Policies can be easily transferred for opening drawers and cabinet doors using this setup.

Parametric vs Non-parametric Learning

• The speaker discusses whether parametric or non-parametric learning is better.

• They mention a recent study at CMU where models were evaluated on different tasks.

• Nearest neighbor models were found to work best in training locations but failed in test locations.

Limitations of Offline Imitation Learning

In this section, the speaker talks about limitations of offline imitation learning and how it fails when presented with new domains.

Flipping a Bagel Task

• The speaker demonstrates a task where a robot flips a bagel.

• An imitation learning algorithm is asked to transfer this task onto new domains.

Failure on New Domains

• The robot fails when presented with new Bagels as it has never seen them before.

• Offline imitation algorithms cannot transfer examples shown by humans onto new domains.

• Robots will always fail when presented with something unexpected during deployment.

Introduction

In this section, the speaker introduces the problem of robots failing and the need for them to learn from their failures. The speaker proposes a solution that involves using representations to do adaptation.

Using Representations for Adaptation

• Robots need to learn from their failures on the fly in deployment time.

• The same representations used previously can be used as a basis for adaptation.

• The robot needs to figure out how to take the failed trajectory and make it match the successful one.

• Alignment is necessary, where every point in the expert is matched with an optimal point on the agent using an optimal transport matching technique.

• The matching score obtained during alignment can be used as a reward for reinforcement learning (RL).

• RL can be used on-the-fly in deployment time to maximize this matching score.

Examples

In this section, examples are given of how this algorithm works. The speaker shows results of different tasks performed by robots after being trained with only one minute of human demonstration data.

Results of Tasks Performed by Robots

• Robots were able to perform household tasks like opening cabinet drawers and switching off lights.

• Robots were also able to perform more precise tasks like inserting keys into keyholes and flipping bagels.

• Across all these tasks, there was around 3x improvement compared to prior imitation learning algorithms.

• This algorithm allows some amount of generalization across objects as well. For example, if a robot is trained to pick up a dollar bill, it can also pick up a 50 bill or a card without additional training.

• However, if the strategy was not correct, these algorithms could fail.

Introduction to Behavior Transformer

In this section, the speaker introduces the concept of Behavior Transformer and explains how it can be used to learn multimodal behaviors in robotics.

Learning from Expert Demonstrations

• The robot learns from expert demonstrations provided by a human.

• The assumption is that the human knows what the right thing to do is and can imitate that.

Challenges with Multimodal Data

• Robotic data is often collected in a playful way or by observing humans, making it very multimodal.

• This makes it difficult to learn these behaviors when they were not intended for a specific task.

Using Transformers for Robotics Data

• Transformers are great at doing multimodal prediction, but their outputs are in discrete spaces.

• Robotic data has continuous outputs, so a behavior transformer is created to retrofit transformers to robotic data.

• A simple trick allows a discrete transformer-like model to operate on top of continuous actions.

Introduction

In this section, the speaker introduces the topic of constructivist robot learning and explains how they use multimodal data to train their robots.

Using Transformers for Sequential Data

• The speaker explains that they use Transformers instead of RNNs or MLPs because sequential data fits better with their robots.

• They mention that the Transformer has two outputs: a bin of K means cluster and a continuous correction of that.

Collecting Data

In this section, the speaker talks about collecting data for their robots and how they use it to train their models.

Using Multimodal Data

• The speaker mentions an undergrad in their lab who collected four and a half hours of data in their kitchen environment.

• They explain that having this long sequence of data allows them to view it as a large sequence and take snippets from it to train their models.

Training Models

• The speaker explains that after taking snippets from the data, they concatenate them and use the same Transformer model to output actions.

• They mention using a behavior Transformer which takes input observations along with future observations to predict actions.

• They also explain that this allows them to perform tasks without any reward information or online training.

Zero Shot Learning

In this section, the speaker talks about zero-shot learning and how they use it for robotics.

Performing Tasks

• The speaker explains that by showing an image of an open oven or pot being in the right place, they can get the robot to automatically figure out the sequence of actions needed.

• They mention being able to do more interesting things like opening both a microwave and oven at once using just images.

Arbitrary Actions

• The speaker notes that sometimes these Transformer models can do arbitrary things outside what was intended. They mention the robot turning a knob in between tasks.

• They also provide a link to their website where the code and Transformer models are available.

Summary

In this section, the speaker summarizes the main ideas discussed in the talk.

Main Ideas

• The speaker talks about their vision of constructivist robot learning with humans guiding the learning and interaction of robots.

• They mention three main ideas: interfacing, adaptation, and multimodal data.

• They explain that using tools like Oculus is promising for interfacing with robots and that adaptation is necessary for deploying models on real robots.

Robots Learning from Uncurated Data

In this section, the speaker discusses how robots can learn from large amounts of uncurated data using models like Transformers.

Extracting Useful Information from Uncurated Data

• Robots can learn from large amounts of uncurated data.

• Models like Transformers and Behavior Transformer can extract useful information and reason about stuff just from this uncreated data.

Acknowledgements

• The work was done by amazing students in our group.

Q&A Session

In this section, the speaker answers questions related to the previous presentation.

Scaling Robot Adaptation

• The speaker's research focuses on giving robots the least amount of data from humans and seeing how fast they can adapt to as many scenarios as possible.

• One way to scale is by showing an hour of demonstrations across lots of novel scenarios.

• Another way to scale is by showing one demonstration but then giving more time for adaptation to crazier environments.

Relevant Features Extraction

• The question is how to ensure that representations are capturing relevant features of the environment that you care about.

• Self-supervised algorithms should automatically figure out if the table, hand or object is important after being shown enough diverse data on that domain.

BYOL Algorithm Performance

• BYOL algorithm is simple with less hyperparameter tuning which makes it good enough for most projects.

• Even if a weak algorithm has enough data, it normally accounts for it.

Scaling Self-Supervised Approaches in Robotics

In this section, the speaker discusses the challenges of scaling self-supervised approaches in robotics and the need for higher quality data.

Challenges of Scaling Self-Supervised Approaches

• Collecting high-quality data is a challenge when scaling self-supervised approaches in robotics. It takes time to collect data, and academic labs may be slower than industry.

• The amount of diversity in the data is also important for generalization. Large-scale efforts are needed to collect diverse data from different environments.

• The role of simulation in terms of generalization is an interesting question. Simulation can be useful for cheaply generating large amounts of diverse data, but it has limitations as an algorithm.

RT1 vs QT Opt for Generalization

In this section, the speaker compares RT1 and QT Opt methods for generalization and discusses their limitations.

Comparison between RT1 and QT Opt Methods

• RT1 method involves collecting more demonstration data to improve generalization while QT Opt involves letting the robot collect more data by itself.

• However, RT1 has limitations as it still does not generalize well to different objects despite having a lot of data.

• The best way to deploy these methods is still unclear since there hasn't been enough research on this problem yet.

Role of Simulation in Generalization

In this section, the speaker discusses the role of simulation in terms of generalization and its limitations.

Limitations of Simulation as an Algorithm

• Simulation can be viewed as a nativist algorithm that requires human engineers to know all the rules of physics and structure the problem accordingly.

• While simulation can generate large amounts of diverse data, it has limitations in terms of generalization to the real world.

• The speaker's students have stopped using simulation as much and are exploring other methods for generalization.

Creating a Robot that can Solve New Problems on the Fly

In this section, the speaker discusses the challenges of creating a robot that can solve new problems without human intervention. They also discuss the role of third-person data in robotics research.

Challenges of Scaling Sim Tutorial Algorithm

• The speaker believes that asking a Sim tutorial algorithm to scale across many tasks without human intervention is difficult.

• They believe that creating a robot that can solve new problems on the fly is more feasible.

Role of Third-Person Data in Robotics Research

• The speaker acknowledges the role of third-person data in robotics research.

• They explain that they have not explored it much because other researchers are already working on it.

• However, they express interest in taking interfaces developed by other researchers and applying their algorithms to them.

• For example, they mention using videos of humans to develop robots with tactile sensors but note that it's challenging due to difficulties in determining force and contact points.