Michal Nazarczuk, Krystian Mikolajczyk

Abstract

There is a striking similarity between a typical subroutine executed with a robotic manipulator and a symbolic program from neuro-symbolic approaches to visual reasoning tasks. Inspired by this observation, we present a neuro-symbolic action planning (NS-AP) framework that enables combined Visual and Manipulation Reasoning, as well as the use of symbolic programs for robotic manipulation. Our environment makes use of a physics engine and high-quality renderer to provide realistic visual observations that are also accurate to the physical state of the scene. We provide a modular pipeline that addresses various aspects of the reasoning-manipulation process. Additionally, the framework is easily extendable and allows to generate new, synthetic data. Finally, we propose a dataset generated with our NS-AP that is composed of 10 benchmarking scenarios focused on closely related, simultaneous visual and physical measurements.

Link

Arxiv link

Citation

@article{nazarczuk2022nsap,
  title={Neuro-Symbolic Action Planning: A Benchmark for Visual and Manipulation Reasoning},
  author={Nazarczuk, Michal and Mikolajczyk, Krystian},
  journal={#TODO},
  year={2022}
  }