3D Dynamic Scene Graphs: Actionable Spatial Perception with Places, Objects, and Humans

We propose 3D Dynamic Scene Graphs (DSGs) as a unified representation for actionable spatial perception. (a) A DSG is a layered and hierarchical representation that abstracts a dense 3D model (e.g., a metric-semantic mesh) into higherlevel spatial concepts (e.g., objects, agents, places, rooms) and models their spatio-temporal relations (e.g., "agent A is in room B at time t", traversability between places or rooms). We present a Spatial PerceptIon eNgine (SPIN) that reconstructs a DSG from visual-inertial data, and (a) segments places, structures (e.g., walls), and rooms, (b) is robust to extremely crowded environments, (c) tracks dense mesh models of human agents in real time, (d) estimates centroids and bounding boxes of objects of unknown shape, (e) estimates the 3D pose of objects for which a CAD model is given.