2D Image-To-3D Model: Knowledge-Based 3D Building Reconstruction (3DBR) Using Single Aerial Images and Convolutional Neural Networks (CNNs)

SCE-Net: Self- and Cross-Enhancement Network for Single-View Height Estimation and Semantic Segmentation

Single-view height estimation and semantic segmentation have received increasing attention in recent years and play an important role in the photogrammetry and remote sensing communities. The height information and semantic information of images are correlated, and some recent works have shown that multi-task learning methods can achieve complementation of task-related features and improve the prediction results of the multiple tasks. Although much progress has been made in recent works, how to effectively extract and fuse height features and semantic features is still an open issue. In this paper, a self- and cross-enhancement network (SCE-Net) is proposed to jointly perform height estimation and semantic segmentation on single aerial images. A feature separation–fusion module is constructed to effectively separate and fuse height features and semantic features based on an attention mechanism for feature representation enhancement across tasks. In addition, a height-guided feature distance loss and a semantic-guided feature distance loss are designed based on deep metric learning to achieve task-aware feature representation enhancement. Extensive experiments are conducted on the Vaihingen dataset and the Potsdam dataset to verify the effectiveness of the proposed method. The experimental results demonstrate that the proposed SCE-Net could outperform the state-of-the-art methods and achieve better performance in both height estimation and semantic segmentation.

Rapid Reconstruction of 3D Structural Model Based on Interactive Graph Cuts

The image-based 3D reconstruction technique has been applied in many scenarios of civil engineering, such as earthquake prevention and disaster reduction, construction monitoring, and intelligent city construction. However, the traditional technique is time-consuming, and the modeling efficiency has become a bottleneck limiting its application in emergency scenarios. In this paper, a rapid reconstruction method is proposed which combines the traditional image-based 3D reconstruction technique and an interactive graph cuts algorithm. Firstly, a sequence of images is collected around the target structure. Then, the images are preprocessed using the interactive iterative graph cuts algorithm to extract the target from each image. Finally, the resulting sequence of images is used to perform the 3D reconstruction. During the preprocessing, only a few images require manual intervention while the rest can be processed automatically. To verify the modeling accuracy of the proposed method, a column that has been destroyed is selected as a target for 3D reconstruction. The results show that compared with the traditional method, the modeling efficiency of the fast reconstruction method is doubled. In addition, the modeling accuracy is 97.65%, which is comparable to the modeling accuracy of the traditional method (97.73%); as well, by comparing the point clouds, the alignment between the two models is tremendously close, with tiny difference. The proposed rapid reconstruction method can be applied in emergency scenarios, such as rapid assessment in post-disaster situations.

SFA-MDEN: Semantic-Feature-Aided Monocular Depth Estimation Network Using Dual Branches

Monocular depth estimation based on unsupervised learning has attracted great attention due to the rising demand for lightweight monocular vision sensors. Inspired by multi-task learning, semantic information has been used to improve the monocular depth estimation models. However, multi-task learning is still limited by multi-type annotations. As far as we know, there are scarcely any large public datasets that provide all the necessary information. Therefore, we propose a novel network architecture Semantic-Feature-Aided Monocular Depth Estimation Network (SFA-MDEN) to extract multi-resolution depth features and semantic features, which are merged and fed into the decoder, with the goal of predicting depth with the support of semantics. Instead of using loss functions to relate the semantics and depth, the fusion of feature maps for semantics and depth is employed to predict the monocular depth. Therefore, two accessible datasets with similar topics for depth estimation and semantic segmentation can meet the requirements of SFA-MDEN for training sets. We explored the performance of the proposed SFA-MDEN with experiments on different datasets, including KITTI, Make3D, and our own dataset BHDE-v1. The experimental results demonstrate that SFA-MDEN achieves competitive accuracy and generalization capacity compared to state-of-the-art methods.

Histogram Adjustment of Images for Improving Photogrammetric Reconstruction

The accuracy of photogrammetric reconstruction depends largely on the acquisition conditions and on the quality of input photographs. This paper proposes methods of improving raster images that increase photogrammetric reconstruction accuracy. These methods are based on modifying color image histograms. Special emphasis was placed on the selection of channels of the RGB and CIE L*a*b* color models for further improvement of the reconstruction process. A methodology was proposed for assessing the quality of reconstruction based on premade reference models using positional statistics. The analysis of the influence of image enhancement on reconstruction was carried out for various types of objects. The proposed methods can significantly improve the quality of reconstruction. The superiority of methods based on the luminance channel of the L*a*b* model was demonstrated. Our studies indicated high efficiency of the histogram equalization method (HE), although these results were not highly distinctive for all performed tests.

School of the Future: A Comprehensive Study on the Effectiveness of Augmented Reality as a Tool for Primary School Children’s Education

With the emerging technologies of augmented reality (AR) and virtual reality (VR), the learning process in today’s classroom is much more effective and motivational. Overlaying virtual content into the real world makes learning methods attractive and entertaining for students while performing activities. AR techniques make the learning process easy, and fun as compared to traditional methods. These methods lack focused learning and interactivity between the educational content. To make learning effective, we propose to use handheld marker-based AR technology for primary school students. We developed a set of four applications based on students’ academic course of primary school level for learning purposes of the English alphabet, decimal numbers, animals and birds, and an AR Globe for knowing about different countries around the world. These applications can be played wherever and whenever a user wants without Internet connectivity, subject to the availability of a tablet or mobile device and the required target images. These applications have performance evaluation quizzes (PEQs) for testing students’ learning progress. Our study investigates the effectiveness of AR-based learning materials in terms of learning performance, motivation, attitude, and behavior towards different methods of learning. Our activity results favor AR-based learning techniques where students’ learning motivation and performance are enhanced compared to the non-AR learning methods.

Digital Modelling and Accuracy Verification of a Complex Architectural Object Based on Photogrammetric Reconstruction

Data concerning heritage buildings are necessary for all kinds of building surveying and design. This paper presents a method for creating a precise model of a historical architectural and landscape object with complex geometry. Photogrammetric techniques were used, combining terrestrial imaging and photographs taken using UAVs. In large-scale objects, it is necessary to divide the reconstruction into smaller parts and adopt an iterative approach based on the gradual completion of missing fragments, especially those resulting from occlusions. The model developed via the reconstruction was compared with geometrically reliable data (LAS point clouds) available in the public domain. The degree of accuracy it achieved can be used in conservation, for example, in construction cost estimates. Despite extensive research on photogrammetric techniques and their applicability in reconstructing cultural heritage sites, the results obtained have not yet been compared by other researchers with LAS point clouds from the information system for land cover (ISOK).

Determining the Increase in a Building’s Appreciation Rate Due to a Reconstruction

The reconstruction of buildings generally prolongs their useful life, increases their utility value, and last but not least, leads to an increase in their value. These assumptions only apply if an independent third party reaches the same conclusion together with the owner. However, the undesirable effect of the reconstruction of a building may be a decrease in its value. The aim of this contribution is to determine the change in value of an older sample building assessed in the included case study as a result of its reconstruction. Valuation methods are applied, which, as it turns out, reveal the inaccuracy of the subjective view of the person who reconstructed the building. The resulting change in the value of the sample building is discussed from the point of view of the applied valuation methods and other value-creating aspects (subjective view of the owner on the value of the building, historical value of the building, etc.). The contribution concludes with recommendations for maximizing the increase in value of a property through its reconstruction so as to eliminate the risk of a decrease in its value.

IM2ELEVATION: Building Height Estimation from Single-View Aerial Imagery

Estimation of the Digital Surface Model (DSM) and building heights from single-view aerial imagery is a challenging inherently ill-posed problem that we address in this paper by resorting to machine learning. We propose an end-to-end trainable convolutional-deconvolutional deep neural network architecture that enables learning mapping from a single aerial imagery to a DSM for analysis of urban scenes. We perform multisensor fusion of aerial optical and aerial light detection and ranging (Lidar) data to prepare the training data for our pipeline. The dataset quality is key to successful estimation performance. Typically, a substantial amount of misregistration artifacts are present due to georeferencing/projection errors, sensor calibration inaccuracies, and scene changes between acquisitions. To overcome these issues, we propose a registration procedure to improve Lidar and optical data alignment that relies on Mutual Information, followed by Hough transform-based validation step to adjust misregistered image patches. We validate our building height estimation model on a high-resolution dataset captured over central Dublin, Ireland: Lidar point cloud of 2015 and optical aerial images from 2017. These data allow us to validate the proposed registration procedure and perform 3D model reconstruction from single-view aerial imagery. We also report state-of-the-art performance of our proposed architecture on several popular DSM estimation datasets.

Geo-Location Algorithm for Building Targets in Oblique Remote Sensing Images Based on Deep Learning and Height Estimation

To improve the accuracy of the geographic positioning of a single aerial remote sensing image, the height information of a building in the image must be considered. Oblique remote sensing images are essentially two-dimensional images and produce a large positioning error if a traditional positioning algorithm is used to locate the building directly. To address this problem, this study uses a convolutional neural network to automatically detect the location of buildings in remote sensing images. Moreover, it optimizes an automatic building recognition algorithm for oblique aerial remote sensing images based on You Only Look Once V4 (YOLO V4). This study also proposes a positioning algorithm for the building target, which uses the imaging angle to estimate the height of a building, and combines the spatial coordinate transformation matrix to calculate high-accuracy geo-location of target buildings. Simulation analysis shows that the traditional positioning algorithm inevitably leads to large errors in the positioning of building targets. When the target height is 50 m and the imaging angle is 70°, the positioning error is 114.89 m. Flight tests show that the algorithm established in this study can improve the positioning accuracy of building targets by approximately 20%–50% depending on the difference in target height.

Identification of Salt Deposits on Seismic Images Using Deep Learning Method for Semantic Segmentation

Several areas of Earth that are rich in oil and natural gas also have huge deposits of salt below the surface. Because of this connection, knowing precise locations of large salt deposits is extremely important to companies involved in oil and gas exploration. To locate salt bodies, professional seismic imaging is needed. These images are analyzed by human experts which leads to very subjective and highly variable renderings. To motivate automation and increase the accuracy of this process, TGS-NOPEC Geophysical Company (TGS) has sponsored a Kaggle competition that was held in the second half of 2018. The competition was very popular, gathering 3221 individuals and teams. Data for the competition included a training set of 4000 seismic image patches and corresponding segmentation masks. The test set contained 18,000 seismic image patches used for evaluation (all images are 101 × 101 pixels). Depth information of the sample location was also provided for every seismic image patch. The method presented in this paper is based on the author’s participation and it relies on training a deep convolutional neural network (CNN) for semantic segmentation. The architecture of the proposed network is inspired by the U-Net model in combination with ResNet and DenseNet architectures. To better comprehend the properties of the proposed architecture, a series of experiments were conducted applying standardized approaches using the same training framework. The results showed that the proposed architecture is comparable and, in most cases, better than these segmentation models.