Machine-learning based 3D point cloud classification and multitemporal change analysis with simulated laser scanning data using open source scientific software

AIM: We will present how virtual laser scanning (VLS), i.e., simulation of realistic LiDAR campaigns, can be key for applying machine/deep learning (ML/DL) approaches to geographic point clouds. Recent results will be shown for semantic classification and change analysis in multitemporal point clouds using exclusively open source scientific software.MOTIVATION: Laser scanning is able to deliver precise 3D point clouds which have made huge progress in research in geosciences over the last decade. Capturing multitemporal (4D: 3D + time) point clouds enables to observe and quantify Earth surface process activities, their complex interactions and triggers. Due to the large size of 3D/4D datasets that can be captured by modern systems, automatic methods are required for point cloud analysis. Machine learning approaches applied to geographic point clouds, in particular DL, have shown very promising results for many different geoscientific applications 1,2.METHODS RESULTS: While new approaches for deep neural networks are rapidly developing 1, the bottleneck of sufficient and appropriate training data (typically annotated point clouds) remains the major obstacle for many applications in geosciences. Those data hungry learning methods depend on proper domain representation by training data, which is challenging for natural surfaces and dynamics, where there is high intra-class variability. Synthetic LiDAR point clouds generated by means of VLS, e.g., with the open-source simulator HELIOS++ 3, can be a possible solution to overcome the lack of training data for a given task. In a virtual 3D/4D scene representing the target surface classes, different LiDAR campaigns can be simulated, with all generated point clouds being automatically annotated. VLS software like HELIOS++ allows to simulate any LiDAR platform and settings for a given scene, which offers high potential for data augmentation and the creation of training samples tailored to specific applications. In recent experiments 1, purely synthetic training data could achieve similar performances to costly labeled training data from real-world acquisitions for semantic scene classification.Furthermore, surface changes can be introduced to create dynamic VLS scenes (e.g., erosion, accumulation, movement/transport). Combining LiDAR simulation with automatic change analysis, such as offered by the open-source scientific software py4dgeo 5, enables to perform ML for change analysis in multitemporal point clouds 6. Recent results show that rockfall activity mapping and classification for permanent laser scanning data can be successfully implemented by combining HELIOS++, py4dgeo and the open-source framework VL3D, which can be used for investigating various ML/DL approaches in parallel.CONCLUSION: Expert domain knowledge (i.e., definition of proper 3D/4D scenes) and the power of AI can be closely coupled in VLS-driven ML/DL approaches to analyze 3D/4D point clouds in the geosciences. Open-source scientific software already offers all required components (HELIOS++, VL3D, py4dgeo).REFERENCES:1 Esmors Pena, A. M., et al. (2024): Deep learning with simulated laser scanning data for 3D point cloud classification. ISPRS Journal of Photogrammetry and Remote Sensing. under revision.2 Winiwarter, L., et al. (2022): DOI: https://doi.org/10.1016/j.rse.2021.1127723 HELIOS++: https://github.com/3dgeo-heidelberg/helios4 VL3D framework: https://github.com/3dgeo-heidelberg/virtualearn3d5 py4dgeo: https://github.com/3dgeo-heidelberg/py4dgeo6 Zahs, V. et al. (2023): DOI: https://doi.org/10.1016/j.jag.2023.1034