With the advent of artificial intelligence and natural user interfaces, the need for multimedia material that can be semantically interpreted in real time becomes critical. In the field of 3D architectural survey, a significant amount of research has been conducted to allow domain experts represent semantic data while keeping spatial references. Such data becomes valuable for natural user interfaces designed to let non-expert users obtain information about architectural heritage. In this paper, we present the architectural data collection and annotation procedure adopted in the Cultural Heritage Orienting Multimodal Experiences (CHROME) project. This procedure aims at providing conversational agents with fast access to fine-detailed semantic data linked to the available 3D models. We will discuss how this will make it possible to support multimodal user interaction and generate cultural heritage presentations.
Segmentation protocols in the digital twins of monumental heritage: a methodological development
The paper shows an advancement of the research that the authors have been carrying out in recent years in semantic structuring of digital architectural representations field, with a focus on the issue of uncertainty of annotations. The studies carried out in this regard have shown how the domain experts specialization determines a vision and interpretation of the same architectural object that we could define “categorized”. The interest was, then, in verifying which categories of experts have a greater degree of agreement in classifying and segmenting architectural elements, to highlight which specializations contribute the most in enriching the semantic reasoning about such forms. Aiming to broaden this reasoning, the research was deepened with annotation sessions concerning architecture examples that didn’t correspond to the classical orders rule but included wider fields of historical heritage (from sacred to fortified architecture). The aim is to verify whether the uncertainty of annotation is actually ascribable to a specific segment of the historical heritage, for example, the classical world, or whether the question is broader and as such in needs deeper thinking.
Comparative Analyses Between Sensors and Digital Data for the Characterization of Historical Surfaces
In the “informational” potential included in 3D digital models obtained by laser scanner survey, several data can be processed in addition to geometric features in order to widen the application of digitization to conservation of Cultural Heritage. Among these data, the intensity value is a (potentially) powerful knowledge concerning the interpretation of surfaces. The visual-comparative analysis developed over years of experimentations demonstrate the need to target research towards comparative data, “sampling” the reflectance of different materials, measured with different sensors and in environments with different boundary conditions. If until recently the process already presented interesting research directions in terms of calibration or control of results on specific materials but difficult at a comparative level, today, thanks to new data segmentation processes and algorithmic procedures, advancements and further comparisons will be possible opening to new interpretative hypotheses. The paper explores ongoing experimentations aimed at comparing different radiometric and colorimetric data obtained by 3D surveying using different laser scanner technologies on historical surfaces, to support the identification of features directly on the 3D model. The goal is to test the link between the intensity value and materials, construction techniques, and decay pathologies, in order to use, in the future, also this parameter as a radiometric feature in machine learning segmentation and classification algorithms. The contribution develops and deepens at the application level the theoretical background and the first experiments carried out on two case studies.
Connecting geometry and semantics via artificial intelligence: from 3D classification of heritage data to H-BIM representations.
Cultural heritage information systems, such as H-BIM, are becoming more and more widespread today, thanks to their potential to bring together, around a 3D representation, the wealth of knowledge related to a given object of study. However, the reconstruction of such tools starting from 3D architectural surveying is still largely deemed as a lengthy and time-consuming process, with inherent complexities related to managing and interpreting unstructured and unorganized data derived, e.g., from laser scanning or photogrammetry. Tackling this issue and starting from reality-based surveying, the purpose of this paper is to semi-automatically reconstruct parametric representations for H-BIM-related uses, by means of the most recent 3D data classification techniques that exploit Artificial Intelligence (AI). The presented methodology consists of a first semantic segmentation phase, aiming at the automatic recognition through AI of architectural elements of historic buildings within points clouds; a Random Forest classifier is used for the classification task, evaluating each time the performance of the predictive model. At a second stage, visual programming techniques are applied to the reconstruction of a conceptual mock-up of each detected element and to the subsequent propagation of the 3D information to other objects with similar characteristics. The resulting parametric model can be used for heritage preservation and dissemination purposes, as common practices implemented in modern H-BIM documentation systems. The methodology is tailored to representative case studies related to the typology of the medieval cloister and scattered over the Tuscan territory.
Semantic segmentation through Artificial Intelligence from raw point clouds to H-BIM representation
This work describes a semi automatic workflow for the 3D reconstruction of Heritage Building Information Models from raw point clouds, based on Artificial Intelligence techniques. The BIM technology applied to historical architecture has made it possible to create a virtual repository of many heterogeneous pieces of information in order to make the process of storing and collecting data on the built heritage more effective. The modelling phase of an artefact is the most complex and problematic in terms of time, as the large architectural heritage of historical buildings does not allow the use of parametric models, so that manual modelling of components is required. Current scientific research focuses on automating this phase by means of segmentation and classification methods: these are based on associating different semantic information to the products of the three dimensional surveying as point clouds or polygonal meshes. To address these problems, the proposed approach relies on: (i) the application of machine learning algorithms with a multi level and multi resolution (MLMR) method to semantically classify 3D heritage data; (ii) the use of annotated data identified by relevant features to boost the scan to BIM process for 3D digital reconstruction. The procedure is tested and evaluated on the complex case of the Church of Santa Caterina d’Alessandria in Pisa, Italy. The classification results show the reliability and reproducibility of the developed method.
Machine Learning and Deep Learning for the Built Heritage Analysis: Laser Scanning and UAV-Based Surveying Applications on a Complex Spatial Grid Structure
The reconstruction of 3D geometries starting from reality-based data is challenging and timeconsuming due to the difficulties involved in modeling existing structures and the complex nature of built heritage. This paper presents a methodological approach for the automated segmentation and classification of surveying outputs to improve the interpretation and building information modeling from laser scanning and photogrammetric data. The research focused on the surveying of reticular, space grid structures of the late 19th–20th–21st centuries, as part of our architectural heritage, which might require monitoring maintenance activities, and relied on artificial intelligence (machine learning and deep learning) for: (i) the classification of 3D architectural components at multiple levels of detail and (ii) automated masking in standard photogrammetric processing. Focusing on the case study of the grid structure in steel named La Vela in Bologna, the work raises many critical issues in space grid structures in terms of data accuracy, geometric and spatial complexity, semantic classification, and component recognition.