Search

This paper proposes an effective infrared and visible image fusion method based on a texture conditional generative adversarial network (TC-GAN). The constructed TC-GAN generates a combined texture map for capturing gradient changes in image fusion. The generator in the TC-GAN is designed as a codec structure for extracting more details, and a squeeze-and-excitation module is applied to this codec structure to increase the weight of significant texture information in the combined texture map. The generator loss function is designed by combing the gradient loss and adversarial loss to retain the texture information of the source images. The discriminator brings the texture of the generated image closer to the visible image. To obtain significant texture information from the source images, a multiple decision map-based fusion strategy is proposed using a combined texture map and an adaptive guided filter. Extensive experiments on the public TNO and RoadScene datasets demonstrate that the proposed method is superior to other state-of-the-art algorithms in terms of a subjective evaluation and quantitative indicators.

How to truthfully and accurately reflects the spatial variation of complex geological bodies is a huge challenge. Most modeling software has made some progress in system functionality and diversity. However, there are still deficiencies in the external appearance and internal quality optimization. In order to improve the quality and effectiveness of geological model mesh, the parameterization method of digital geometric mesh is introduced into the process of geological modeling in this paper. The basic representation method of geological body mesh is studied, and different parameterization methods are designed for different types of meshes. The common geological surface, high curvature geological surface and tunnel surface were verified as examples. The results show that the optimization and quality improvement of three-dimensional (3D) geological models can be realized by using the method presented in this paper, the geometric features of the original mesh are maintained and the degree of distortion is reduced in the process of texture mapping, grid editing, surface reconstruction.

In minimally invasive surgery, the primary surgeon requires an assistant to hold an endoscope to obtain visual information from the body cavity. However, the two-dimensional images acquired by endoscopy lack depth information. Future automatic robotic surgeries need three-dimensional information of the target area. This paper presents a method to reconstruct a 3D model of soft tissues from image sequences acquired from a robotic camera holder. In this algorithm, a sparse reconstruction module based on the SIFT and SURF features is designed, and a multilevel feature matching strategy is proposed to improve the algorithm efficiency. To recover the realistic effect of the soft-tissue model, a complete 3D reconstruction algorithm is implemented, including densification, meshing of the point cloud and texture mapping reconstruction. During the texture reconstruction stage, a mathematical model is proposed to achieve the repair of texture seams. To verify the feasibility of the proposed method, we use a collaborative manipulator (AUBO i5) with a mounted camera to mimic an assistant surgeon holding an endoscope. To satisfy a pivotal constraint imposed by the remote center of motion (RCM), a kinematic algorithm of the manipulator is implemented, and the primary surgeon is provided with a voice control interface to control the directions of the camera with. We conducted an experiment to show a 3D reconstruction of soft tissue by the proposed method and the manipulator, which indicates that the manipulator works as a robotic assistant which can hold a camera to provide abundant information in the surgery.

Three-dimensional (3D) face reconstruction technology is a key issue in the field of computer vision and computer graphics. The image-based 3D face reconstruction method is the mainstream technology of 3D face reconstruction. This paper uses the 3D face reconstruction algorithm to analyze the 3D reconstruction of a static image face and the structure and realization of the computer standardized management system. In the process of high-resolution two-channel matching, combined with multilevel reconstruction technology, a better initial disparity map is obtained by comparison. Then, the improved disparity map optimization algorithm is used to optimize the initial value, so that the result of the cloud point is smoother on the model surface while maintaining good detail accuracy. The AdaBoost face detection algorithm based on embedded images is used to obtain the image representing the region, and grayscale conversion and normalization processing are performed. Finally, the Poisson surface reconstruction algorithm is used to reconstruct a 3D high-precision face model. The experimental results in this paper show that the AdaBoost face detection algorithm can effectively separate and express face features, reduce the feature dimensions to compactly represent the data and correctly classify expressions. The 3D face reconstruction technology based on static images can increase the recognition rate of a face by 24% and can also increase the texture mapping effect of the local area of the face by 13%.

We propose a deep videorealistic 3D human character model displaying highly realistic shape, motion, and dynamic appearance learned in a new weakly supervised way from multi-view imagery. In contrast to previous work, our controllable 3D character displays dynamics, e.g., the swing of the skirt, dependent on skeletal body motion in an efficient data-driven way, without requiring complex physics simulation. Our character model also features a learned dynamic texture model that accounts for photo-realistic motion-dependent appearance details, as well as view-dependent lighting effects. During training, we do not need to resort to difficult dynamic 3D capture of the human; instead we can train our model entirely from multi-view video in a weakly supervised manner. To this end, we propose a parametric and differentiable character representation which allows us to model coarse and fine dynamic deformations, e.g., garment wrinkles, as explicit spacetime coherent mesh geometry that is augmented with high-quality dynamic textures dependent on motion and view point. As input to the model, only an arbitrary 3D skeleton motion is required, making it directly compatible with the established 3D animation pipeline. We use a novel graph convolutional network architecture to enable motion-dependent deformation learning of body and clothing, including dynamics, and a neural generative dynamic texture model creates corresponding dynamic texture maps. We show that by merely providing new skeletal motions, our model creates motion-dependent surface deformations, physically plausible dynamic clothing deformations, as well as video-realistic surface textures at a much higher level of detail than previous state of the art approaches, and even in real-time.

Since neither a single high-precision 3D coordinate laser point cloud nor a oblique image of high-precision spectral information can make the computer understand the data as human, in order to accurately and completely understand the object in the real world, in this paper, the fusion of the point cloud and the oblique image is studied for the understanding and application of the real object buildings by establishing a BIM model. Firstly, point cloud data and image data are acquired by different data acquisition methods. Laser point cloud and oblique image are fused by installation parameters so that geometric elements are consistent and benchmark is unified. Processing level is not limited to element level only, so as to ensure the integrity of information, and the fused mutual information remains consistent in scale, texture and direction. Secondly, under the condition of space-time synchronization, the relationship between point cloud and image in geometric space is established, and the alternating "energy transfer" is used for shape prediction and visual filling by matching under texture mapping constraints to enhance the semantic information of the 3D scene. Finally, a 3D building information model is established. Using the experimental results to track the whole life cycle (design, construction, operation) in real time saves the cost of building rework, reduces the building cycle, and improves the building accuracy. Especially based on BIM during Novel Coronavirus Pneumonia: analyze hospital space resource management and elastic function; construct medical unit model of infectious disease hospital; simulate and optimize medical process in infectious disease hospital; rapidly construct prefabricated infectious disease emergency hospital; simulate hospital infection path and dynamics; the auxiliary design and construction of infectious disease treatment in existing hospitals; The monitoring analysis and automatic control of hospital infection safety operation.

This article focuses on improving the 3D reconstruction of a human model from a single pixel-aligned implicit function image presented by FaceBook Research. The drawbacks of the method are revealed, associated with limiting the quality of the original image, recommendations are presented to avoid its incorrect operation, and approaches to improve the original model are proposed, which increase the identity of the resulting model by 1.33 times. We also worked out the tactics of subsequent texture mapping and implementation of a set of animations.

Disaster risk management of movable and immovable cultural heritage is a highly significant research topic. In this work, we present a pipeline for 3D digitisation, segmentation and annotation of large scale urban areas in order to produce data that can be exploited in disaster management simulators (e.g fire spreading, crowd movement, firefighting training, evacuation planning, etc.). We have selected the old town of Xanthi (Greece) as a challenging case study. We developed a custom multispectral camera to be carried by a commercial drone. Using the structure from motion / multiview stereo (SFM/MVS) approach, we produced a 3D model of the urban area covering 0.5 k m 2 that is followed by a multilayer texture map which carries information from visible and near-infrared regions of the electromagnetic spectrum. We developed a set of machine learning approaches based on logistic regression, support vector machines and artificial neural networks that allow 3D model segmentation by exploiting not only morphological and structural features but also the multispectral behaviour of different material surfaces. We objectively evaluate the performance of the proposed segmentation approaches on six significant material-based classes (cobbled-roads granite kilns, building walls, ceramic roof-tiles, low-vegetation, high-vegetation and metal surfaces) that are used in simulating fire propagation and crowd movement. The experiments revealed that the segmentation accuracy can be enhanced by taking into consideration surface material multispectral properties as well as morphological features. A Web-based multi-user annotation tool complements our proposed pipeline by enabling further 3D model segmentation, fine tuning and semantics annotation (e.g. usage-based building classification and evacuation priorities, escape paths and gathering points).

Texture mapping is a key step in large-scale 3D scene reconstruction, which can greatly enhance visual reality of the reconstructed scenes. However, existing techniques are unable to accomplish this task efficiently due to the high computational complexity of reconstructing large-scale real-world 3D scenes. In this work, we propose a new efficient convex optimization-based approach, i.e. the mesh-based continuous max-flow method, which can be easily implemented and accelerated upon a modern parallel computing platform, e.g. GPU. Particularly, an Markov Random Fields (MRF) based model, i.e. Potts model, is introduced to mathematically formulate the key specific view selection problem; we show that the challenging combinatorial optimization problem can be efficiently solved by resolving its convex relaxation, which recovers textures from images with the proposed duality-based continuous max-flow approach. In addition, visual effects of sharpness and deformation are utilized to define a criterion of evaluating texture quality effectively, and a large 3D triangular mesh is partitioned into structural components so as to reduce memory consumption of the proposed algorithm. The proposed mesh-based continuous max-flow approach for large-scale texture mapping demonstrates its outperformance over state-of-the-art methods, over large-scale public datasets, in both numerical efficiency and visual quality; meanwhile, our GPU-accelerated algorithm can yield 3D textured models with high quality from complex large-scale scenes in minutes.

BACKGROUND: Virtual reality (VR) simulation is an established option for temporal bone surgical training. Most VR simulators are based on computed tomography imaging, whereas the Visible Ear Simulator (VES) is based on high-fidelity cryosections of a single temporal bone specimen. Recently published OpenEar datasets combine cone-beam computed tomography (CBCT) and micro-slicing to achieve similar model quality. This study explores integration of OpenEar datasets into VES to enable case variation in simulation with implications for patient-specific modeling based on CBCT. METHODS: The OpenEar dataset consists of segmented, coregistered, multimodal imaging sets of human temporal bones. We derived drillable bone segments from the dataset as well as triangulated surface models of critical structures such as facial nerve or dura. Realistic visualization was achieved using coloring from micro-slicing, custom tinting, and texture maps. Resulting models were validated by clinical experts. RESULTS: Six of the eight OpenEar datasets could be integrated in VES complete with instructional guides for various temporal bone surgical procedures. Resulting models were of high quality because of postprocessing steps taken to increase realism including colorization and imaging artifact removal. Bone artifacts were common in CBCT, resulting in dehiscences that most often could not be found in the ground truth micro-slicing data. CONCLUSION: New anatomy models are included in VES version 3.5 freeware and provide case variation for training which could help trainees to learn more quickly and transferably under variable practice conditions. The use of CBCT for VR simulation models without postprocessing results in bone artifacts, which should be considered when using clinical imaging for patient-specific simulation, surgical rehearsal, and planning.

Over the past few years closer cooperation can be observed in various aspects of digital techniques in such disciplines as dentistry and anthropology. And in most cases that consists in imaging and image processing which results in obtaining 3D reconstructions. And indeed, they can significantly improve research and practice. Thus, depending on imaging technique and application, they can support CAD/CAM technology or precisely reconstruct morphology of invisible structures. However the currently presented study refers to technical aspects of shade and texture mapping, which is more aimed to obtain more realistic 3D reconstructions of palaeoanthropological material. Colour or shade matching has become an integral part of dental practice. It can be carried in a traditional manner though matching the tooth with conventional shade-guides, or, which is in line with the subjects of our study, by means of spectrophotometry. And the main procedures of shade detection have been performed by SpectroShade (MHT). Necessary attention has been paid to conditions influencing shade detection process with respect to the studied material teeth taken from the Bronze Age findings. Reconstructive techniques have traditionally been a scientific and practical part of palaeoanthropological research which is directed at appearance reconstruction. Though the leading part of this branch has been always aimed at analysis of skull morphology. In our time of rapidly developing digital techniques reconstructions have become to a large extent a matter of improvements of imaging and image processing techniques. Even though this doesnt directly refer to soft tissue reconstruction, it undoubtedly applies to dental reconstructions. And the current study presents improvements in reconstruction through combining imaging with shade and texture mapping on 3D models of teeth.

With the development of AR/VR technologies, a reliable and straightforward way to digitize a three-dimensional human body is in high demand. Most existing methods use complex equipment and sophisticated algorithms, but this is impractical for everyday users. In this paper, we propose a pipeline that reconstructs a 3D human shape avatar from a single image. Our approach simultaneously reconstructs the three-dimensional human geometry and whole body texture map with only a single RGB image as input. We first segment the human body parts from the image and then obtain an initial body geometry by fitting the segment to a parametric model. Next, we warp the initial geometry to the final shape by utilizing a silhouette-based dense correspondence. Finally, to infer invisible back texture from a frontal image, we propose a network called InferGAN. Based on human semantic information, we also propose a method to handle partial occlusion by reconstructing the occluded body parts separately. Comprehensive experiments demonstrate that our solution is robust and effective on both public and our own datasets. Our human avatars can be easily rigged and animated using MoCap data. We have developed a mobile application that demonstrates this capability for AR applications.

Telemedicine has evolved significantly for detection and diagnosis of diseases remotely, which means more frequent transmission of medical images over the network. The security of any medical image can be characterized as its integrity, confidentiality and authentication. It is these security vulnerabilities that limit the development of mobile healthcare applications, which intend to improve the efficiency of medical image communication. To address the vulnerabilities associated with medical images, we propose a texture edge map and multilevel chaotic map driven encryption framework for medical images. This technique utilizes texture maps generated by utilizing a bank of gabor filters along with multiple chaotic maps viz.: Sine, Cubic and Logistic maps for enhancing the key space, robustness and security of medical images over an insecure channel. The security, speed and reliability of the proposed technique for medical images are illustrated via experiments for key sensitivity, statistical and performance analysis. The proposed technique offers a large key space, pixel diffusion at an acceptable speed. Security analysis shows a high sensitive dependence of the encryption and decryption techniques to any subtle change in the secret key, the plain medical image and the encrypted image. Also, the proposed technique has a large enough key space to see off brute force attacks. Therefore, the proposed technique is a potential candidate for addressing security vulnerabilities of medical images over the communication networks.

Building detection is a crucial task in the field of remote sensing, which can facilitate urban construction planning, disaster survey, and emergency landing. However, for large-size remote sensing images, the great majority of existing works have ignored the image tilt problem. This problem can result in partitioning buildings into separately oblique parts when the large-size images are partitioned. This is not beneficial to preserve semantic completeness of the building objects. Motivated by the above fact, we first propose a framework for detecting objects in a large-size image, particularly for building detection. The framework mainly consists of two phases. In the first phase, we particularly propose a tilt correction (TC) algorithm, which contains three steps: texture mapping, tilt angle assessment, and image rotation. In the second phase, building detection is performed with object detectors, especially deep-neural-network-based methods. Last but not least, the detection results will be inversely mapped to the original large-size image. Furthermore, a challenging dataset named Aerial Image Building Detection is contributed for the public research. To evaluate the TC method, we also define an evaluation metric to compute the cost of building partition. The experimental results demonstrate the effects of the proposed method for building detection.

As 2D and 3D data present different views of the same face, the features extracted from them can be both complementary and redundant. In this paper, we present a novel and efficient orthogonalization-guided feature fusion network, namely OGF $^2$ Net, to fuse the features extracted from 2D and 3D faces for facial expression recognition. While 2D texture maps are fed into a 2D feature extraction pipeline (FE2DNet), the attribute maps generated from 3D data are concatenated as input of the 3D feature extraction pipeline (FE3DNet). The two networks are separately trained at the first stage and frozen in the second stage for late feature fusion, which can well address the unavailability of a large number of 3D+2D face pairs. To reduce the redundancies among features extracted from 2D and 3D streams, we design an orthogonal loss-guided feature fusion network to orthogonalize the features before fusing them. Experimental results show that the proposed method significantly outperforms the state-of-the-art algorithms on both the BU-3DFE and Bosphorus databases. While accuracies as high as 89.05% (P1 protocol) and 89.07% (P2 protocol) are achieved on the BU-3DFE database, an accuracy of 89.28% is achieved on the Bosphorus database. The complexity analysis also suggests that our approach achieves a higher processing speed while simultaneously requiring lower memory costs.

Detecting and segmenting the lung regions in chest X-ray images is an important part in artificial intelligence-based computer-aided diagnosis/detection (AI-CAD) systems for chest radiography. However, if the chest X-ray images themselves are used as training data for the AI-CAD system, the system might learn the irrelevant image-based information resulting in the decrease of system’s performance. In this study, we propose a lung region segmentation method that can automatically remove the shoulder and scapula regions, mediastinum, and diaphragm regions in advance from various chest X-ray images to be used as learning data. The proposed method consists of three main steps. First, employ the simple linear iterative clustering algorithm, the lazy snapping technique and local entropy filter to generate an entropy map. Second, apply morphological operations to the entropy map to obtain a lung mask. Third, perform automated segmentation of the lung field using the obtained mask. A total of 30 images were used for the experiments. In order to verify the effectiveness of the proposed method, two other texture maps, namely, the maps created from the standard deviation filtering and the range filtering, were used for comparison. As a result, the proposed method using the entropy map was able to appropriately remove the unnecessary regions. In addition, this method was able to remove the markers present in the image, but the other two methods could not. The experimental results have revealed that our proposed method is a highly generalizable and useful algorithm. We believe that this method might act an important role to enhance the performance of AI-CAD systems for chest X-ray images.

International audience; Shape-from-Template (SfT) solves the registration and 3D reconstruction of a deformable 3D object, represented by the template, from a single image. Recently, methods based on deep learning have been able to solve SfT for the wide-baseline case in real-time, clearly surpassing classical methods. However, the main limitation of current methods is the need for fine tuning of the neural models to a specific geometry and appearance represented by the template texture map. We propose the first texture-generic deep learning SfT method which adapts to new texture maps at run-time, without the need for texture specific fine tuning. We achieve this by dividing the problem into a segmentation step and a registration and reconstruction step, both solved with deep learning. We include the template texture map as one of the neural inputs in both steps, training our models to adapt to different ones. We show that our method obtains comparable or better results to previous deep learning models, which are texture specific. It works in challenging imaging conditions, including complex deformations, occlusions, motion blur and poor textures. Our implementation runs in real-time, with a low-cost GPU and CPU.

During the annealing of Ni and Ni-Mo films, a {110} texture, usually considered to be both of high surface energy and high strain energy, was found to develop. Quantitative thermodynamic model calculations showed that the anomalous formation of this {110} texture originates fundamentally from the grain yielding anisotropy of Ni. Based on extensive grain yielding anisotropy model calculations, a “texture map” based on {111}, {100}, and {110} textures was constructed to predict the texture transition temperatures for different film thicknesses. A kinetic analysis of the texture evolution in the films is further presented, revealing that the texture evolution is controlled by the self-diffusion of atoms at grain boundaries. These findings pave the way for the achievement of unusual surface orientations through the quantitative texture design of thin films.

Recent technological advances in computer vision and the advent of commercial RGB-D sensors have certainly boosted 3D modeling applications. This work presents an integrated system that enables the digitization of big objects, like vehicles, with low-cost RGB-D sensors. The implemented system can be used for visualization and monitoring of vehicles in large fleets that currently require a time-consuming manual inspection process. The main objective is to achieve an efficient consolidation of multiple views of a vehicle inside a moving frame, to acquire color and depth data and generate its 3D representation. The proposed integrated system denoises the acquired depth maps, aligns the produced point clouds captured in different time instances, and builds the 3D-reconstructed mesh. Finally, we apply a texture mapping algorithm to acquire realistic texture details and remove any visible seams. We evaluate all modules of the implemented system by performing several experiments with scanned vehicles.

This paper proposes a novel in situ three-dimensional (3D) laser machining system that combines 3D projection algorithms with in situ measurement and 3D modeling. This system forms a complete “scanning-modeling-projection-machining” integrated processing system for rapid pattern machining on the free-form surfaces. In situ measurement was conducted by self-scanning of the 3D galvanometer scanner. A high-efficiency Delaunay triangulation algorithm was employed for the 3D reconstruction to generate a quality-controlled 3D model. The Least-Squares Conformal Mapping (LSCM) and As-Rigid-As-Possible (ARAP) algorithms were employed for model parameterization. Local parameterization and bitmap vectorization methods were proposed to improve the accuracy and speed of parameterization and texture mapping. In situ machining software was developed, and the algorithms were verified by in situ machining experiments. The LSCM algorithm achieves fast processing speed but suffers from a large distortion if the model is complex. The ARAP algorithm can further ensure the accuracy of the parameterization through iterative calculation. The developed model can better guarantee the model quality for parameterization. The 3D projection algorithm can transfer the two-dimensional (2D) pattern on a 3D surface, and the in situ method eliminates the necessity for assembly and clamping of parts. The local parameterization and bitmap vectorization methods improve both the accuracy and efficiency of 3D projections. Therefore, the proposed in situ machining system has practical application value for the rapid processing of patterns on curved surfaces.

Nowadays, image processing and 3D shape analysis are an integral part of clinical practice and have the potentiality to support clinicians with advanced analysis and visualization techniques. The main contribution of the paper is the integration of these two approaches in order to increase the amount of information available and, thus, allow a more accurate analysis of each patient. Given a segmented anatomical district, we propose a novel mapping of volumetric data onto the segmented surface. The grey levels of the image voxels are mapped through a volume-surface correspondence map, which defines a grey level texture on the segmented surface. The resulting texture mapping is coherent to the local morphology of the segmented anatomical structure and provides an enhanced visual representation of the anatomical district. The integration of volume-based and surface-based information in a unique 3D representation also supports the identification and characterization of morphological landmarks and pathology evaluations.

In this work, we present a methodology for precise 3D modelling and multi-source geospatial data blending for the purposes of Virtual Reality immersive and interactive experiences. We evaluate it on the volcanic island of Santorini due to its formidable geological terrain and the interest it poses for scientific and touristic purposes. The methodology developed here consists of three main steps. Initially, bathymetric and SRTM data are scaled down to match the smallest resolution of our dataset (LIDAR). Afterwards, the resulted elevations are combined based on the slope of the relief, while considering a buffer area to enforce a smoother terrain. As a final step, the orthophotos are combined with the estimated Digital Terrain Model, via applying a nearest neighbour matching schema leading to the final terrain background. In addition to this, both onshore and offshore points-of-interest were modelled via image-based 3D reconstruction and added to the virtual scene. The overall geospatial data that need to be visualized in applications demanding photo-textured hyper-realistic models pose a significant challenge. The 3D models are treated via a mesh optimization workflow, suitable for efficient and fast visualization in virtual reality engines, through mesh simplification, physically based rendering texture maps baking, and level-of-details.

In this paper, we present a new technique of tridimensional reconstruction from a sequence of uncalibrated stereo images taken with cameras having varying parameters. At first, our system allows to recover initial coordinates of a set of 3D points. In this context, we have used our method of self-calibration based on the use of unknown 3D scene with its image projections and genetic algorithms to estimate all intrinsic parameters. After that extrinsic parameters are estimated based on classical pose estimation algorithms. Matching points and estimated value of intrinsic and extrinsic parameters are used to estimate initial 3D model that helps us in the initialization step. In order to have a reliable and relevant 3D reconstruction the proposed method is based on good and new exploitation of bundle adjustment (without camera poses initialization) technique based on Levenberg-Marquardt optimization with the aim to estimate our optimal 3D model that has special features compared to the classical case because it masks the pose parameters estimation in the optimization process. Finally, 3D mesh of the 3D scene is constructed with Delaunay algorithm and the 2D image is projected on the 3D model to generate the texture mapping. Experiments is conducted on real data to achieve demonstrate the validity and the performance of the proposed approach in terms of convergence, simplicity, stability and reconstruction quality.

Purpose This paper aims to obtain a texture dental model with real images and improve the rendering effect of the dental model. Design/methodology/approach The paper proposes a semiautomatic method to construct a realistic dental model with real images based on two-dimensional/three-dimensional (2D/3D) registration. First, a 3D digital dental model and three intraoral images are obtained by a 3D scanner and digital single-lens reflex camera. Second, the camera projection poses for every intraoral images are calculated by using the single-objective optimization algorithm. Third, with camera poses, the preliminary projection texture mapping is performed; besides, the seam between two textures is marked. Finally, the marked regions are fused based on the image pyramid to eliminate obvious seams. Findings The paper provides a method to construct a realistic dental model. The method can map three intraoral images to the dental model. The experimental results show that the textured dental model without obvious distortion, dislocation and seams is constructed with simple interactions. Originality/value The proposed method can be applied to the digital smile design system to improve the communication efficiency between doctors, patients and technicians.

We present a method for example-based texturing of triangular 3D meshes. Our algorithm maps a small 2D texture sample onto objects of arbitrary size in a seamless fashion, with no visible repetitions and low overall distortion. It requires minimal user interaction and can be applied to complex, multi-layered input materials that are not required to be tileable. Our framework integrates a patch-based approach with per-pixel compositing. To minimize visual artifacts, we run a three-level optimization that starts with a rigid alignment of texture patches (macro scale), then continues with non-rigid adjustments (meso scale) and finally performs pixel-level texture blending (micro scale). We demonstrate that the relevance of the three levels depends on the texture content and type (stochastic, structured, or anisotropic textures).

Photorealistic multi-view face synthesis from a single image is an important but challenging problem. Existing methods mainly learn a texture mapping model from the source face to the target face. However, they fail to consider the internal deformation caused by the change of poses, leading to the unsatisfactory synthesized results for large pose variations. In this paper, we propose a Gated Deformable Face Synthesis Network to model the deformation of faces that aids the synthesis of the target face image. Specifically, we propose a dual network that consists of two modules. The first module estimates the deformation of two views in the form of convolution offsets according to the input and target poses. The second one, on the other hand, leverages the predicted deformation offsets to create the target face image. In this way, pose changes are explicitly modeled in the face generator to cope with geometric transformation, by adaptively focusing on pertinent regions of the source image. To compensate offset estimation errors, we introduce a soft-gating mechanism that enables adaptive fusion between deformable features and primitive features. Extensive experimental results on five widely-used benchmarks show that our approach performs favorably against the state-of-the-arts on multi-view face synthesis, especially for large pose changes.

There are some problems in the visualization of 3D model of buildings at present, such as redundant texture data and large memory occupation of loading texture data, which pose challenges to the smooth loading of 3D building visualization. In this paper, we propose a method which using fractal quadtrees to simplify texture data and compress management. Firstly, this method uses the fractal self-similarity and the feature that the same texture has the same fractal dimension to screen all the textures, when the fractal dimension of multiple textures is within the threshold range. Radon transform will be performed on these textures to calculate their standard deviation and further simplify the texture. Secondly, we use fractal texture compression to create a multi-resolution texture data structure with quadtree structure, so that the building 3D model can achieve dynamic visualization of different scales. Finally, the parallel texture mapping of all faces of the 3D model of the building is implemented according to the texture calling rules. Experimental results demonstrate that the texture is managed by the method proposed in this paper, which shortens the loading time of texture and reduces the memory occupation by about 48.92%. Therefore, the method of texture simplification and compression proposed in this paper has a significant effect in the process of building 3D model visualization.

Conformal surface parameterization is useful in graphics, imaging and visualization, with applications to texture mapping, atlas construction, registration, remeshing and so on. With the increasing capability in scanning and storing data, dense 3D surface meshes are common nowadays. While meshes with higher resolution better resemble smooth surfaces, they pose computational difficulties for the existing parameterization algorithms. In this work, we propose a novel parallelizable algorithm for computing the global conformal parameterization of simply-connected surfaces via partial welding maps. A given simply-connected surface is first partitioned into smaller subdomains. The local conformal parameterizations of all subdomains are then computed in parallel. The boundaries of the parameterized subdomains are subsequently integrated consistently using a novel technique called partial welding, which is developed based on conformal welding theory. Finally, by solving the Laplace equation for each subdomain using the updated boundary conditions, we obtain a global conformal parameterization of the given surface, with bijectivity guaranteed by quasi-conformal theory. By including additional shape constraints, our method can be easily extended to achieve disk conformal parameterization for simply-connected open surfaces and spherical conformal parameterization for genus-0 closed surfaces. Experimental results are presented to demonstrate the effectiveness of our proposed algorithm. When compared to the state-of-the-art conformal parameterization methods, our method achieves a significant improvement in both computational time and accuracy.

Surface parameterizations have been widely applied in computer-aided design for the geometric processing tasks of surface registration, remeshing, texture mapping, and so on. In this paper, we present an efficient balanced energy minimization algorithm for the computation of simply connected open surface parameterizations with balanced angle and area distortions. The existence of a nontrivial accumulation function of the proposed algorithm is guaranteed under some mild conditions, and the limiting function is shown to be one-to-one. Comparisons of the proposed algorithm with angle- and area-preserving parameterizations show that the angular distortion is close to that of an angle-preserving parameterization while the area distortion is significantly improved. An application of the proposed algorithm involving surface remeshing, registration, and morphing to the Chinese virtual broadcasting technique is demonstrated.

In the recent period, significant progress has been achieved towards reconstructing the 3D face model from face image. With the support of the render engines and sufficient data, the reconstruction results are fine in detail. Nevertheless, the research on the 3D face reconstruction with texture from a single unrestricted face image is imperfect. The rebuild process lacks essential structure and texture information in the profile and the craniofacial region. To address this problem, we present a method of creating a 3D full face-and-head model with photorealistic texture from a single “in-the-wild” face image in this paper. To this end, we introduce a pipeline to integrate the highly-detailed face model into the basic model. Specifically, the basic model was built by multilinear optimization, and the highly-detailed face model which represents the facial features generated by constrained illumination distribution. Additionally, to infer the invisible region texture information corresponding to the input face image, we design an effective architecture with the generative adversarial network (GAN) for panoramic UV texture generation. The final results after UV texture mapping were visualized in the experiment, which demonstrates that the model faithfully recovers the photorealistic details in arbitrary perspective. Furthermore, compared to the state-of-the-art facial modeling techniques and existing commercial solutions, our method takes less input and performs better in surface detail.

Acceleration techniques for Rendering in general, and Ray-Casting in particular, have been the subject of much research in Computer Graphics. Most efforts have been focused on new data structures for efficient ray/scene traversal and intersection. In this paper, we propose an acceleration technique that approximates rendering and that is built around a new feature-based clustering approach. The technique starts preprocessing the scene by grouping elements according to their features using a set of channels based on an information theory-based approach. Then, at run-time, a rendering strategy uses that clustering information to reconstruct the final image, by deciding which areas could take advantage of the coherence in the features and thus, could be interpolated; and which areas require more involved calculations. This process starts with a low-resolution render that is iteratively refined up to the desired resolution by reusing previously computed pixels. Our experimental results show a significant speedup of an order of magnitude, depending on the complexity of the per-pixel calculations, the screen size of the objects, and the number of clusters. Rendering quality and speed directly depend on the number of clusters and the number of steps performed during the reconstruction procedure, and both can easily be set by the user. Our findings show that feature-based clustering can significantly impact rendering speed if samples are chosen to enable interpolation of smooth regions. Our technique, thus, accelerates a range of popular and costly techniques, ranging from texture mapping up to complex ambient occlusion, soft and hard shadow calculations, and it can even be used in conjunction with more traditional acceleration methods his work was partially funded by the TIN2017-88515-C2-2-R grant from Ministerio de Ciencia, Innovación y Universidades, Spain Open Access funding provided thanks to the CRUE-CSIC agreement with Elsevier

Texture mapping can be defined as the colorization of a 3D mesh using one or multiple images. In the case of multiple images, this process often results in textured meshes with unappealing visual artifacts, known as texture seams, caused by the lack of color similarity between the images. The main goal of this work is to create textured meshes free of texture seams by color correcting all the images used. We propose a novel color-correction approach, called sequential pairwise color correction, capable of color correcting multiple images from the same scene, using a pairwise-based method. This approach consists of sequentially color correcting each image of the set with respect to a reference image, following color-correction paths computed from a weighted graph. The color-correction algorithm is integrated with a texture-mapping pipeline that receives uncorrected images, a 3D mesh, and point clouds as inputs, producing color-corrected images and a textured mesh as outputs. Results show that the proposed approach outperforms several state-of-the-art color-correction algorithms, both in qualitative and quantitative evaluations. The approach eliminates most texture seams, significantly increasing the visual quality of the textured meshes.

To realize three-dimensional (3-D) simulation of weft-knitted fabrics, this paper proposes a 3-D loop modeling method which includes yarn texture and rapid deformation of fabric. The deformation of the fabric is achieved using a simplified particle system. In this study, the shape of the loop is controlled by using 10 control points, and the coordinates of the loop type control points are calculated in combination with the actual loop shape and a quadratic Bezier curve equation. To realize the deformation of the loop, the relationship between the control points and the particle system is established. The 3-D surface model of the yarn is constructed based on a local frame of the loop path. In addition, the yarn simulation is achieved by texture mapping. The simulation of fancy weft-knitted fabric is realized by the joint programming of Visual Studio and OpenGL. The results show that the simplified particle model can accurately achieve the deformation of the fabric with high efficiency. The geometric model properly represents the loop shape and exhibits the yarn texture.

Structure from Motion (SfM) is a 3D reconstruction approach for estimating camera poses and 3D structure from calibrated images. The recovered 3D structure is a sparse 3D point cloud that not permit to well define the shape of the object/scene. We must therefore move to a dense 3D reconstruction that requires a step of the dense matching between pairs of consecutive images, which requires a long calculation time. To reduce computation time, we have proposed an algorithm for the good choice of image pairs that will be used by the Modified Match Propagation (MMP) to improve the sparse 3D reconstruction. These image pairs will be selected on the basis of the result already achieved by SfM. The MMP algorithm will be applied for each image pair to retrieve new matches and their 3D coordinates. The final 3D point cloud is achieved by fusion of results obtained from the image pairs selected. The realistic 3D model is recovered after applying the Poisson surface reconstruction method with texture mapping. The results of the experiments show the speed of the proposed approach without losing quality of 3D reconstructed models.

Exemplar based texture synthesis is an important technique for image processing and computer graph in texture mapping. So far, great achievements have been made in this field. However, both traditional and modern methods based on deep learning are making errors in synthesizing patterned texture due to the failure for catching the regularity of texture. To obtain a better synthesized result, a new framework for regular texture synthesis is proposed in this paper. Besides, we use recurrent neural network (RNN) of long-shot term memory (LSTM) to produce a regular texture based on exemplar. Our method can generate at any size texture without errors, which is an improvement for texture synthesis with deep learning techniques.Compared with traditional method as well as deep learning method, our method is obviously better in synthesizing regular texture.

Recently, the deep learning (DL)-based solution for single image super-resolution reconstruction has been extensively used. Though these methods have shown exceptional results, the computational requirement is very high and requires high-end graphic processors. Motivated to establish an efficient method without such requirement, we propose a modified Markov random field (MRF) model and two-dimensional (2D) phase congruency-based single-image super-resolution reconstruction method for remote sensing images. The 2D phase congruency-based feature extraction method is used to compute features such as edge and texture maps to achieve higher accuracy in finding similar example patches in feature space. To address an essential aspect of successful texture reconstruction, we have incorporated a texture prior for the computation of joint probability in our work. Image Euclidean distance (Ieuc) is integrated to achieve higher accuracy in finding the similarity between image patches in feature space and modeling compatibility functions. The experimental results demonstrate that the results of the proposed method are at par with the DL-based methods and outperform other state-of-the-art methods in perceptual quality as well as peak signal-to-noise ratio and structural similarity index parameters. Moreover, it shows significant improvement in the texture regions while reconstructing sharper edges for ×4 upscaling.

Applying textures to 3D models are means for creating realistic looking objects. This is especially important in the 3D manufacturing domain as manufactured models should ideally comprise a natural and realistic appearance. Nevertheless, natural material textures usually consist of dense patterns and fine details. Their embedding onto 3D models is typically cumbersome, requiring large processing time and resulting in large size meshes. This paper presents a novel approach for direct embedding of fine scale geometric textures onto 3D printed models by on-the-fly modification of the 3D printer’s head. Our idea is to embed 3D textures by revising the 3D printer’s G-code, i.e., incorporating texture details through modification of the printer’s path. Direct manipulation of the printer’s head movement allows for fine-scale texture mapping and editing on-the-fly in the 3D printing process. Thus, our method avoids the computationally expensive texture mapping, mesh processing and manufacturing preprocessing. This allows embedding detailed geometric textures of unlimited density which can model manual manufacturing artifacts and natural material properties. Results demonstrate that our direct G-code textured models are printed robustly and efficiently in both space and time compared to traditional methods.

Face forgery by deepfake is widely spread over the internet and this raises severe societal concerns. In this paper, we propose a novel video transformer with incremental learning for detecting deepfake videos. To better align the input face images, we use a 3D face reconstruction method to generate UV texture from a single input face image. The aligned face image can also provide pose, eyes blink and mouth movement information that cannot be perceived in the UV texture image, so we use both face images and their UV texture maps to extract the image features. We present an incremental learning strategy to fine-tune the proposed model on a smaller amount of data and achieve better deepfake detection performance. The comprehensive experiments on various public deepfake datasets demonstrate that the proposed video transformer model with incremental learning achieves state-of-the-art performance in the deepfake video detection task with enhanced feature learning from the sequenced data., Accepted at ACM International Conference on Multimedia, October 20 to 24, 2021, Virtual Event, China

This study investigates the use of textural features and texture maps in building a brain age prediction model using two publicly available datasets (IXI and OASIS), also the usage of texture maps to calculate the brain age delta and use it as a biomarker of dementia and investigate accelerated aging for demented subjects.

Per-pixel extrusion mapping consists of creating a virtual geometry stored in a texture over a polygon model without increasing its density. There are four types of extrusion mapping, namely, basic extrusion, outward extrusion, beveled extrusion, and chamfered extrusion. These different techniques produce satisfactory results in the case of plane surfaces, but when it is about the curved surfaces, the silhouette is not visible at the edges of the extruded forms on the 3D surface geometry because they not take into account the curvature of the 3D meshes. In this paper, we presented an improvement that consists of using a curved ray-tracing to correct the silhouette problem by combining the per-pixel extrusion mapping techniques and the quadratic approximation computed at each vertex of the 3D mesh.

In order to solve the problem that the shape change of flexible object is very complex and the projection on the flexible object seriously affects the viewing effect of the observer, a method of correcting the geometric distortion of flexible object based on texture mapping was proposed. By building the rules on the surface of the projection area of the largest calculating model, solved the camera and projector, projection object coordinate system to solve the problem of the conversion relationship between the three, implements the projection images aligned with the flexible projection shape of the object. The experimental results show that this method can reduce the limitation of projection device and simplify the operation flow of projection correction. To achieve a better projection mapping effect on the deformation of flexible objects.

Garment transfer aims to transfer the desired garment from a model image with the desired clothing to a target person, which has attracted a great deal of attention due to its wider potential applications. However, considering the model and target persons are often given at different views, body shapes and poses, realistic garment transfer is facing the following challenges that have not been well addressed: 1) deforming the garment; 2) inferring unobserved appearance; 3) preserving fine texture details. To tackle these challenges, we propose a novel SPatial-Aware Texture Transformer (SPATT) model. Different from existing models, SPATT establishes correspondence and infers unobserved clothing appearance by leveraging the spatial prior information of a UV-space. Specifically, the source image is transformed into a partial UV texture map guided by the extracted dense pose. To better infer the unseen appearance utilizing seen region, we first propose a novel coordinate-prior map that defines the spatial relationship between the coordinates in the UV texture map, and design an algorithm to compute it. Based on the proposed coordinate-prior map, we present a novel spatial-aware texture generation network to complete the partial UV texture. In the second stage, we first transform the completed UV texture to fit the target person. To polish the details and improve realism, we introduce a refinement generative network conditioned on the warped image and source input. Compared with existing frameworks as shown experimentally, the proposed framework can generate more realistic images with better-preserved texture details. Furthermore, difficult cases where two persons have large pose and view differences can also be well handled by SPATT.

In recent years, virtual reality equipment and technology have been rapidly developed. In order to protect and inherit traditional Chinese culture, this paper takes the chime-bells of Marquis Yi of Zeng and other contemporary musical instruments as research objects, and employs virtual reality technology to simulate and reproduce the cultural space and cultural activities at that time. Carrying out data acquisition via the use of photogrammetry and 3D scanning, data processing, 3D model reconstruction and model texture mapping, the creation of realistic models of chime-bells and other musical instruments are completed. On this basis, VR system based on HTC VIVE and Unity3D cross-platform development engine is selected to build an interactive platform to exhibit traditional culture. Compared with traditional cultural protection methods, this one is more acceptable and enjoyable. In addition, this method is better to enhance the experiencers' sense of on-site experience and attract experiencers' attention, which increases the breadth and depth of cultural communication.

We present a method to infer physically-based material properties for free-viewpoint video. Given a multi-camera image feed and reconstructed geometry, our method infers material properties, such as albedo, surface normal, metallic and roughness maps. We use a physically based, differentiable renderer to generate candidate images which are compared against the image feed. Our method searches for material textures which minimise an image-space loss metric between candidate renders and the ground truth image feed. Our method produces results that approximate state of the art reflectance capture, and produces texture maps that are compatible with common real-time and offline shading models.

Finger contains several discriminative biometric traits, including fingerprint, finger vein, finger knuckle, and finger shape, which are complementary in identity information. However, in most current researches and practical applications, only a single or several traits are utilized, which are prone to unsatisfactory recognition performance and easy forgery. Our work is the first attempt to collect and study all biometric traits on the finger. Firstly, a novel multi-view, multi-spectral 3D finger imaging system is designed. To the best of our knowledge, it is the first biometric imaging system that can capture almost all finger-based traits. With this 3D finger imaging system, we scanned numerous fingers, acquiring their external skin images and internal vein images from 6 different views. Then 3D finger models with skin and vein textures are reconstructed by space carving, mesh regularization, and texture mapping algorithms. Secondly, we establish a benchmark dataset, namely the Large- scale Finger Multi-Biometric database and benchmark for 3D Finger Biometrics (LFMB-3DFB). LFMB-3DFB contains 695 fingers, and each finger is captured 10 times. Then, 6 finger skin images and 6 finger vein images are obtained for each acquisition, and final 83,400 images and 6,950 3D finger models are obtained. Besides, we designed a more rigorous and comprehensive evaluation protocol for both identification and verification tasks. Finally, we designed corresponding baselines for 2D finger traits recognition, multi-view finger traits recognition, 3D finger traits recognition, and score-level fusion. Rigorous experiments have been conducted to verify the significance and usefulness of the proposed LFMB-3DFB.

3D reconstruction has been widely applied in medical images, industrial inspection, self-driving cars, and indoor modeling. The 3D model is built by the steps of data collection, point cloud registration, surface reconstruction, and texture mapping. In the process of data collection, due to the limited visibility of the scanning system, the scanner needs to scan multiple angles and then splice the data to obtain a complete point cloud model. The point clouds from different angles must be merged into a unified coordinate system, which is known as point cloud registration. The result of point cloud registration can directly affect the accuracy of the point cloud model; thus, point cloud registration is a key step in the construction of the point cloud model. The ICP (Iterative Closest Points) algorithm is the most known technique of the point cloud registration. The variational ICP problem can be solved not only by deterministic but also by stochastic methods. One of them is Grey Wolf Optimizer (GWO) algorithm. Recently, GWO has been applied to rough point clouds alignment. In the proposed paper, we apply the GWO approach to the realization of the point-to-point ICP algorithms. Computer simulation results are presented to illustrate the performance of the proposed algorithm.

Style transfer is a task that extracts the style from the style image to synthesize an output that has the content of the content image. The latest method with better effect transfers style image onto content image via a transformation matrix. These algorithms are based on a variety of assumptions to model the style. However, there are no accurate assumptions of how style and content are combined. They model the features covariance as style and produce copy and paste results in synthesized images which we called texture mapping effect. In this work, we define the coupling form as the multiplication of the style matrix and the content matrix. Then, we derive the form of transformation theoretically and present an arbitrary style transfer approach that decouples content and style from features based on our definition. By this assumption, the stylized image will not introduce a texture mapping effect. We demonstrate the effectiveness of our approach comparisons with the state-of-the-art methods in both style transfer and photorealistic image stylization. The results show that our method has no texture mapping effect and is more in line with the definition of stylization.

Polandball comics are hand-drawn satirical content that portray personified countries in a unique style. Although certain parts of these comics, such as ball outlines, are easy to draw, some country flags are complex and require time, effort, and skill to depict correctly. Convolutional Neural Networks have shown success in image synthesis tasks but lack the ability to rescale and rotate images for texture mapping. The domain of Virtual Try-On Networks has made great progress in networks that can handle spatially invariant transforms. We show that similar methods can be used in another domain dependent on texture mapping, namely generating valid, rule-abiding Poland-ball characters given an outline and a country flag. To evaluate our method we make use of the Frechet Inception Distance where we achieved a score of 34.9. Multiple configurations of the model were evaluated to show that all modules used in the model contribute to the achieved performance. The main contributions in this paper are: a model that can be used by Polandball artists to aid in comic creation and a dataset with over 40,000 labeled Polandball characters for computer vision tasks.

The 3D information of medical images can effectively assist surgeons in diagnosis and surgical planning. In this paper, the methods of monocular endoscopic reconstruction of the inner surface of the nasal cavity and nasal endoscopic surgery path planning are proposed, and a virtual nasal endoscopic navigation system with real surface texture information is realized, which provides an intuitive reference for surgeons' diagnosis and surgery. First, by extracting the ASIFT feature points of the nasal endoscopic images, the structure from motion and multi-view stereo reconstruction are performed to construct a point cloud model. The point cloud model is used to reconstruct the triangle surface and texture mapping to obtain a 3D model of the nasal cavity with real surface texture information. Secondly, in view of the lack of scale information of the obtained monocular nasal reconstruction model, a variable-scale registration method between the obtained monocular nasal reconstruction model and the CT image reconstruction model is proposed. The FPFH algorithm is used to achieve feature description and matching, and coarse registration is achieved; the variable scale iterative closest point algorithm is used to achieve fine registration. Then, considering the cylinder shape of the nasal endoscope, a collision-free 3D planning map is constructed, and the RRT algorithm is used to realize the collision-free path planning under the complex nasal cavity space constraints. Finally, a virtual nasal endoscopy navigation system with real surface texture information is constructed, and virtual nasal endoscopy experiments are carried out to verify the effectiveness of the navigation system.