Semantic segmentation of Point Clouds:
a deep learning framework for Cultural Heritage

CVPR 2021 Tutorial

Day: June 20th 2021 (half day)
Time: 10.00 am – 2.00 pm
(Eastern time)
Location: virtual

Slides, material and recorded videos will be provided on this page


The use of 3D point clouds for Cultural Heritage (CH) assets is becoming paramount, since they allow metric and morphological analyses impossible with the use of 2D data, better interpretation of phenomena, valorization and visualization, innovative management, development of conservation strategies, etc.
In the last few decades, the increasing adoption of geomatics techniques for data collection and processing, contributed to the massive 3D-metric documentation of the built heritage. However, the complexity of CH assets makes the exploitation of 3D point clouds very impervious. Moreover, the provision of massive 3D geometric information with only color attributes could hamper their full exploitation due to the lack of semantic information. Therefore the need of efficient, reliable and automated solutions for heritage point cloud classification is necessary in order to widespread the use of such kind of data among heritage conservators, restorators, managers and HBIM (Historical Building Information Modeling) experts. Towards this end, the development of Deep Learning frameworks for point clouds classification is filling this gap, proving to be a very promising, but complex, field of research. These frameworks are designed to semantically enrich point clouds based on some specific classes, quite often case-dependent. These frameworks may facilitate the recognition of historic architectural elements at an appropriate level of detail, thus speeding up the process of reconstruction of geometries in the HBIM environment, or they can automatically identify degraded areas to speed up restoration processes.

In this tutorial, we cover different Deep Learning methods for semantic segmentation tasks, providing the audience with a wide outlook upon the most recent Neural Networks. We also broaden the discussion over the more appropriate combination of hyperparameters, attempting to define a joint pipeline of work for the CH domain. Besides pre-processing and optimization techniques for CH datasets, we will also discuss the application of a specific layer designed to better capture local geometric features of point clouds, which is nowadays the more challenging task.
The goal is to explain the principles behind solving point cloud semantic segmentation and give practical means for engineers and researchers (whose main competences may lie elsewhere), to apply the most powerful methods that have been developed in the last years. It will be presented and practically demonstrated how to formulate and solve point cloud semantic segmentation in CH domain with freely available software that will be distributed to the participants of the tutorial. 


10:00 – 10:15

10:15 – 11:15






Opening remarks

Deep Point Cloud processing: challenges and progress + Q&A


As the closest form to a wide range of raw 3D sensory data, point cloud has become a prevalent data representation for 3D perception. However, leveraging deep learning methods to process massive 3D point clouds is not trivial. Different from 2D images, the irregular and unstructured nature of 3D point clouds introduce a range of learning challenges. In this talk, I will specifically focus on three challenges: the representation challenge which requires novel deep learning backbones capable of directly consuming set structures, the sampling challenge where non-uniform point samples introduce transferring gaps for learning modules, the multimodal fusion challenge which concerns the joint learning of 2D images and 3D point clouds. I will start by revisiting some popular 3D backbones, showing how they address the representation challenge and consume the irregular point set structure. Then I will present some recent progress in handling the sampling challenge and the multimodal fusion challenge, with the goal of stimulating future research in the field.

Designing Invariant and Equivariant Networks for Point Clouds + Q&A


Real-world problems usually have various symmetries inherently — for instance many image classification tasks are invariant to translations, point cloud classification are invariant to permutation, point cloud segmentations are equivariant permutation, molecular property predictions are invariant to rigid body transform. Models that take advantage of such invariance and equivariance structure have shown impressive empirical performance. In this talk, we will explore how to incorporate such symmetries in a neural network as an inductive bias. We begin by formalizing these symmetries as being groups and characterize a family of functions that are invariant/equivariant to the actions of these groups. In particular, we will consider translation, permutation, and rotation. This family of functions has a special structure which enables us to design efficient deep network architectures that are constrained to be equivariant or invariant. Next we will generalize to hierarchical composition of such symmetries, for example when each set element adheres to their own symmetries, e.g. sets of sequences or sets of sets. There are numerous applications in point cloud analysis itself, ranging from large-scale point-cloud segmentation to multi-view 3D shape recognition.


Digital Cultural Heritage: motivations, challenges and opportunities + Q&A

Deep learning framework and code implementation + Q&A

Semantic segmentation and Explainable AI for Cultural Heritage point clouds + Q&A

Marina Paolanti

Li Yi

Manzil Zaheer


Roberto Pierdicca and Francesca Matrone

Yue Wang

Francesca Matrone and Davide Manco


Marche Polytechnic University

Polytechnic of Turin

Davide MANCO
Marche Polytechnic University

Francesca MATRONE
Polytechnic of Turin

Marche Polytechnic University

Marche Polytechnic University

Massachusetts Institute of Technology


Eric li YI
Tsinghua University


Recorded videos


Eva Savina Malinverni – Department of Civil Construction Engineering and Architecture (DICEA), Marche Polytechnic University
Paolo Clini – Department of Civil Construction Engineering and Architecture (DICEA), Marche Polytechnic University
Primo Zingaretti – Department of Information Engineering (DII), Marche Polytechnic University
Adriano Mancini – Department of Information Engineering (DII), Marche Polytechnic University


Please contact Marina Paolanti if you have any questions or concerns.