Accelerating Visual Policy Learning with Sampling-Based Model Predictive Control

Problem

This work addresses the high computational and GPU memory costs associated with learning visual policies for tasks such as locomotion and manipulation. The authors highlight the inefficiencies in existing methods, particularly in the context of real-time applications, and propose a solution to mitigate these challenges. The paper is a preprint and has not undergone peer review.

Method

The authors introduce Sampling-Guided Policy Search (SGPS) as their core method. SGPS incorporates several key components:

• Action-target refinement: This is integrated with sampling-based model predictive control to enhance the learning process.
• Policy initialization: The method employs behavior cloning from sampled actions to establish a robust starting point for policy learning.
• Training process: The training alternates between sampling-based refinement and short-horizon first-order policy gradient (FoPG) updates. The FoPG formulation is designed to be decoupled, excluding rendering from the computation graph, which allows the model to learn directly from depth observations without incurring the overhead of visual rendering.

Results

The proposed method is evaluated on tasks including locomotion, obstacle traversal, crate pushing, and bimanual carrying, using simulated Unitree Go2 and G1 robots. The results indicate that the refinement process significantly enhances policy learning beyond what is achieved through initialization and tracking alone. Notably, the distilled policy demonstrates successful zero-shot transfer to a real Go2 robot, enabling it to perform autonomous behaviors such as trotting, crawling, clearing hurdles, and switching between behaviors. The available text does not report quantitative results.

Limitations

The authors do not report any limitations in their work. However, the absence of quantitative results may limit the ability to fully assess the performance improvements over existing methods.

Why it matters

This research has significant implications for the field of robotics and reinforcement learning, particularly in enhancing the efficiency of visual policy learning. By reducing computational demands, the proposed method could facilitate the deployment of complex robotic behaviors in real-world scenarios, paving the way for more advanced autonomous systems.