Difference: FRCVPublicProject (13 vs. 14)

A Monte Carlo Framework for Incremental Improvement of Simulation Fidelity

D.M. Lyons (Fordham) and J. Finocchiaro, M. Novitzky and C. Korpela (United States Military Academy, West Point NY)

Simulation tools are widely used in robot program development, whether the program/controller is built by hand or using machine learning. At the very least, simulation allows a robot programmer to eliminate obvious program flaws. The availability of physics engines has produced simulations that can more accurately model physical behavior, making it more attractive to use simulation in conjunction with machine learning techniques to develop robot programs. However, a robot program validated with simulation, when operating in a real, unstructured environment may come across phenomena that its designers just did not know to include in the simulation, even though the phenomenon could in fact be simulated if it were known a-priori to be relevant. Examples of this kind of simulation `reality gap' include inaccurate robot joint parameters, surface friction, object masses, sizes and locations.

This paper addresses closing the reality gap for simulations used to develop robot programs by combining sim-to-real and real-to-sim approaches. We will assume that the simulation is a black-box and we present a framework to coerce simulation behavior to more closely resemble real experience. The simulation has configuration parameters $\phi$. If deployment of the robot program ($\pi$) results in failure, as determined by a performance monitor, then our overall objective is for $\phi$ to be updated by the deployment experience and $\pi$ redeveloped to handle that experience. We expect this virtuous cycle of simulation, deployment and improvement to iterate.

The simulation is modeled as a transition function $T_{\phi}(s' | s , a)$ where $s$ is the current sensor data from the simulation, $a$ is the action to be carried out, $s'$ is the resulting sensor data after the action is taken, and $\phi$ is a setting of the configuration parameters for the simulation. Real world experience will also be modeled as a transition function $T_r(s' | s , a)$. Two important novel aspects of our work are 1) a domain independent proposal for $\phi$ (as opposed to the more domain specific examples of domain randomization) and 2) a Monte Carlo method to modify $\phi$ based on a direct comparison of estimated transitions functions (as opposed to the comparison of observations).

The approach collects simulation and real world observations and builds conditional probability functions for them. These are used to generate paired roll-outs and to look for points of divergence in behavior. The divergences are employed to generate state-space kernels coercing the simulation into behaving more like observed reality within a region of the state space. The method handles situations in which the environment behaves different than expected (motor divergence) and sensors behave differently than expected (sensor divergence).

The method was evaluated in field trials using ROS/Gazebo for simulation and a heavily modified Traaxas platform for outdoor deployment evaluated first in the environment in whichit was designed and secondly in an environment containing an eunexpected ramp that would prevent it from reaching its objectives. Achieving objectives resulted in the system receiving a reward (as in reinforcement learning) and an Average Total Reward (ATR) graph used to capture its performance. Our results support not just that the kernel approach can force the simulation to behave more like reality, but that the modification is such that a robot program with an improved control policy tested in the modified simulation also performs better in the real world.

This approach allowed us to sport immediately when the deployed robot did not behave as in the simulation, and to propose changes to the simulation to model the 'unexpected' phenomena see in deployment. Manual update or reinforcement learning (we used both) can be used to 'fix' program code and test it in the modified simulation. The redeployment to the world showed the fixes were effective.

A crucial point of difference with other similar work is that they address the reality gap problem with a principled domain randomization approach, providing a range of environment for policy development, and generating a policy that is robust along the right dimensions of variability. We address the same problem but from the perspective of making each simulation run more closely resemble reality. This is reflected in how the simulation is ``wrapped'' by each approach: our approach requires a more invasive configuration -- access to the Gazebo model information -- but we argue that our configuration is less application specific and more easily generalized. We are especially interested in generalization to higher-dimensional state-spaces.

The quality of the improvement that our method makes is proportional to the divergence information that it has to work with. An avenue of future study is development of control strategies that improve divergence information by exploring more of the state space during deployment.

For more information: https://research.library.fordham.edu/frcv_videos/2 https://research.library.fordham.edu/frcv_facultypubs/77 https://research.library.fordham.edu/frcv_facultypubs/67