Taskography
Taskography - Evaluating robot task planning over large 3D scene graphs

Taskography - Evaluating robot task planning over large 3D scene graphs

Conference on Robot Learning (CoRL) 2021

An example 3D scene graph with unary and binary attributes.

A 3D scene graph (3DSG) annotated with plannable attributes.

3D scene graphs (3DSGs) [1],[2] are an emerging description; unifying symbolic, topological, and metric scene representations. However, typical 3DSGs contain hundreds of objects and symbols even for small environments; rendering task planning on the full graph impractical. We construct Taskography, the first large-scale robotic task planning benchmark over 3DSGs. While most benchmarking efforts in this area focus on vision-based planning, we systematically study symbolic planning, to decouple planning performance from visual representation learning. We observe that, among existing methods, neither classical nor learning-based planners are capable of real-time planning over full 3DSGs. Enabling real-time planning demands progress on both (a) sparsifying 3DSGs for tractable planning and (b) designing planners that better exploit 3DSG hierarchies. Towards the former goal, we propose SCRUB, a task-conditioned 3DSG sparsification method; enabling classical planners to match (and surpass) state-of-the-art learning-based planners. Towards the latter goal, we propose SEEK, a procedure enabling learning-based planners to exploit 3DSG structure, reducing the number of replanning queries required by current best approaches by an order of magnitude. We will open-source all code and baselines to spur further research along the intersections of robot task planning, learning and 3DSGs.

Quick links

Click to open the taskography paper - opens as PDF
Click to open the taskography poster - opens as PNG
Click to open the taskography paper - opens as PDF
Paper
Poster
Code / Data

Tasks

Rearrangement(k) Courier(n, k) Lifted Rearrangement(k)/Courier(n, k)
Based on the recently proposed rearrangement challenge [71], this task requires a robot randomly spawned to rearrange a set of k objects of interest into k corre- sponding receptacles. The robot often needs to execute multiple other actions along the way, such as opening/closing doors, navigating to goals, planning the sequence of objects to visit, etc. A robot that couriers objects is equipped with a knapsack of maximum payload capacity of n units. The robot needs to locate and courier k objects (of varying weights w ∈ {1, 2, 3} units) to k distinct delivery points. The knapsack can be used to stow and retrieve items in random-access fashion; effectively embedding a combinatorial optimization problem into the task. Stow and retrieve actions increase branching, necessitating far deeper searches. We also provide lifted variants of these tasks. Here, goals are specified over desired object-receptacle class relations (e.g., “put a cup on a table”) as opposed to over object instances (e.g., “put this cup on the table”). These tasks introduce ambiguity in both the search of classical task-planners and learning-based techniques, which must now distinguish object instances of relevant classes.

Benchmark

The taskography benchmark evaluates multiple classical and learning based planners over a suite of 3DSG symbolic planning tasks.

TASKOGRAPHY benchmark results on select grounded and lifted Rearrangement (Rearr) and Courier (Cour) 3DSG domains. Planning times are reported in seconds and do not incorporate planner-specific domain translation times (factored into planning timeouts). A ‘-’ indicates planning timeouts or failures (10 minutes for optimal planners, 30 seconds for all others). Results are aggregated over 10 random seeds - see supplementary for standard deviations and results across all 40 domains. Optimal task planning is infeasible in larger prob- lem instances or for more complex domains, while most satisficing planners are unable to achieve real-time performance. PLOI, a recent learning-based planner consistently performs the best across all domains.

Citation

@inproceedings{agia2022taskography,
  title={Taskography: Evaluating robot task planning over large 3D scene graphs},
  author={Agia, Christopher and Jatavallabhula, {Krishna Murthy} and Khodeir, Mohamed and Miksik, Ondrej and Vineet, Vibhav and Mukadam, Mustafa and Paull, Liam and Shkurti, Florian},
  booktitle={Conference on Robot Learning},
  pages={46--58},
  year={2022},
  organization={PMLR}
}

Authors

Christopher Agia
University of Toronto
Vector Institute
Krishna Murthy Jatavallabhula
Universite de Montreal
Mila
Mohamed Khodeir
University of Toronto
Vector Institute
Ondrej Miksik
Microsoft Mixed Reality and AI Lab
Vibhav Vineet
Microsoft Research
Mustafa Mukadam
Facebook AI Research
Liam Paull
Universite de Montreal
Mila
Florian Shkurti
University of Toronto
Vector Institute

Subscribe

Evaluating robot task planning over large 3D scene graphs

Based on a Jekyll template from Krishna. Some icons made by Smashicons from www.flaticon.com.

Last updated: July 12, 2022.