Generalized Framework for Wheel Loader Automatic Shoveling Task with Expert Initialized Reinforcement Learning

This paper presents a generalized framework for fast retrofitting of wheel loaders to enable automatic bucket shoveling with human-level performance. The retrofitting is accomplished in three steps: parameter estimation, expert demonstration, reinforcement learning (RL), and can be accomplished on any wheel loader. First, the dynamics of the given wheel loader is identified from a simple parameter estimation procedure. Second, data of an expert demonstrating the task with the wheel loader is recorded; third, the recorded expert demonstrations are used in an expert initialized RL method called Circle of Learning (CoL). Unlike typical model-free RL methods, which take a long training time to learn such tasks with human-level performance, CoL can shorten the training phase by pre-training the initial behavior of the agent by imitating expert demonstrations. The proposed framework is validated on an industrial wheel loader. The results demonstrate that the retrofitted wheel loader can achieve a bucket fill rate above 80% for automatically shoveling wet soil and medium coarse gravel, the deployed policy trained by CoL only took 2 hours of training with 10 expert demonstration examples. In contrast, the policy trained using TD3 achieved less than half the bucket fill rate within the same training duration.