Search

Projector deblurring is an important technology for dynamic projection mapping (PM), where the distance between a projector and a projection surface changes in time. However, conventional projector deblurring techniques do not support dynamic PM because they need to project calibration patterns to estimate the amount of defocus blur each time the surface moves. We present a deep neural network that can compensate for defocus blur in dynamic PM. The primary contribution of this paper is a unique network structure that consists of an extractor and a generator. The extractor explicitly estimates a defocus blur map and a luminance attenuation map. These maps are then injected into the middle layers of the generator network that computes the compensation image. We also propose a pseudo-projection technique for synthesizing physically plausible training data, considering the geometric misregistration that potentially happens in actual PM systems. We conducted simulation and actual PM experiments and confirmed that: (1) the proposed network structure is more suitable than a simple, more general structure for projector deblurring; (2) the network trained with the proposed pseudo-projection technique can compensate projection images for defocus blur artifacts in dynamic PM; and (3) the network supports the translation speed of the surface movement within a certain range that covers normal human motions. [ABSTRACT FROM AUTHOR]

Stereoscopic projection mapping (PM) allows a user to see a three-dimensional (3D) computer-generated (CG) object floating over physical surfaces of arbitrary shapes around us using projected imagery. However, the current stereoscopic PM technology only satisfies binocular cues and is not capable of providing correct focus cues, which causes a vergence-accommodation conflict (VAC). Therefore, we propose a multifocal approach to mitigate VAC in stereoscopic PM. Our primary technical contribution is to attach electrically focus-tunable lenses (ETLs) to active shutter glasses to control both vergence and accommodation. Specifically, we apply fast and periodical focal sweeps to the ETLs, which causes the “virtual image” (as an optical term) of a scene observed through the ETLs to move back and forth during each sweep period. A 3D CG object is projected from a synchronized high-speed projector only when the virtual image of the projected imagery is located at a desired distance. This provides an observer with the correct focus cues required. In this study, we solve three technical issues that are unique to stereoscopic PM: The 3D CG object is displayed on non-planar and even moving surfaces; the physical surfaces need to be shown without the focus modulation; the shutter glasses additionally need to be synchronized with the ETLs and the projector. We also develop a novel compensation technique to deal with the “lens breathing” artifact that varies the retinal size of the virtual image through focal length modulation. Further, using a proof-of-concept prototype, we demonstrate that our technique can present the virtual image of a target 3D CG object at the correct depth. Finally, we validate the advantage provided by our technique by comparing it with conventional stereoscopic PM using a user study on a depth-matching task. [ABSTRACT FROM AUTHOR]

Intrinsic projector calibration is essential in projection mapping (PM) applications, especially in dynamic PM. However, due to the shallow depth-of-field (DOF) of a projector, more work is needed to ensure accurate calibration. We aim to estimate the intrinsic parameters of a projector while avoiding the limitation of shallow DOF. As the core of our technique, we present a practical calibration device that requires a minimal working volume directly in front of the projector lens regardless of the projector's focusing distance and aperture size. The device consists of a flat-bed scanner and pinhole-array masks. For calibration, a projector projects a series of structured light patterns in the device. The pinholes directionally decompose the structured light, and only the projected rays that pass through the pinholes hit the scanner plane. For each pinhole, we extract a ray passing through the optical center of the projector. Consequently, we regard the projector as a pinhole projector that projects the extracted rays only, and we calibrate the projector by applying the standard camera calibration technique, which assumes a pinhole camera model. Using a proof-of-concept prototype, we demonstrate that our technique can calibrate projectors with different focusing distances and aperture sizes at the same accuracy as a conventional method. Finally, we confirm that our technique can provide intrinsic parameters accurate enough for a dynamic PM application, even when a projector is placed too far from a projection target for a conventional method to calibrate the projector using a fiducial object of reasonable size. [ABSTRACT FROM AUTHOR]

We present computational phase-modulated eyeglasses, a see-through optical system that modulates the view of the user using phase-only spatial light modulators (PSLM). A PSLM is a programmable reflective device that can selectively retardate, or delay, the incoming light rays. As a result, a PSLM works as a computational dynamic lens device. We demonstrate our computational phase-modulated eyeglasses with either a single PSLM or dual PSLMs and show that the concept can realize various optical operations including focus correction, bi-focus, image shift, and field of view manipulation, namely optical zoom. Compared to other programmable optics, computational phase-modulated eyeglasses have the advantage in terms of its versatility. In addition, we also presents some prototypical focus-loop applications where the lens is dynamically optimized based on distances of objects observed by a scene camera. We further discuss the implementation, applications but also discuss limitations of the current prototypes and remaining issues that need to be addressed in future research. [ABSTRACT FROM AUTHOR]

Aiming at realizing novel vision augmentation experiences, this paper proposes the IlluminatedFocus technique, which spatially defocuses real-world appearances regardless of the distance from the user's eyes to observed real objects. With the proposed technique, a part of a real object in an image appears blurred, while the fine details of the other part at the same distance remain visible. We apply Electrically Focus-Tunable Lenses (ETL) as eyeglasses and a synchronized high-speed projector as illumination for a real scene. We periodically modulate the focal lengths of the glasses (focal sweep) at more than 60 Hz so that a wearer cannot perceive the modulation. A part of the scene to appear focused is illuminated by the projector when it is in focus of the user's eyes, while another part to appear blurred is illuminated when it is out of the focus. As the basis of our spatial focus control, we build mathematical models to predict the range of distance from the ETL within which real objects become blurred on the retina of a user. Based on the blur range, we discuss a design guideline for effective illumination timing and focal sweep range. We also model the apparent size of a real scene altered by the focal length modulation. This leads to an undesirable visible seam between focused and blurred areas. We solve this unique problem by gradually blending the two areas. Finally, we demonstrate the feasibility of our proposal by implementing various vision augmentation applications. [ABSTRACT FROM AUTHOR]

This paper presents a novel active marker for dynamic projection mapping (PM) that emits a temporal blinking pattern of infrared (IR) light representing its ID. We used a multi-material three dimensional (3D) printer to fabricate a projection object with optical fibers that can guide IR light from LEDs attached on the bottom of the object. The aperture of an optical fiber is typically very small; thus, it is unnoticeable to human observers under projection and can be placed on a strongly curved part of a projection surface. In addition, the working range of our system can be larger than previous marker-based methods as the blinking patterns can theoretically be recognized by a camera placed at a wide range of distances from markers. We propose an automatic marker placement algorithm to spread multiple active markers over the surface of a projection object such that its pose can be robustly estimated using captured images from arbitrary directions. We also propose an optimization framework for determining the routes of the optical fibers in such a way that collisions of the fibers can be avoided while minimizing the loss of light intensity in the fibers. Through experiments conducted using three fabricated objects containing strongly curved surfaces, we confirmed that the proposed method can achieve accurate dynamic PMs in a significantly wide working range. [ABSTRACT FROM AUTHOR]

The appearance of fabric motion is suggested to affect the human perception of bending stiffness. This study presents a novel spatial augmented reality, or projection mapping, approach that can visually manipulate the perceived bending stiffness of a fabric. Particularly, we proposed a flow enhancement method that can change the apparent fabric motion by using a simple optical flow analysis technique rather than complex physical simulations for interactive applications. Through a psychophysical experiment, we investigated the relationship between the magnification factor of our flow enhancement and the perceived bending stiffness of a fabric. Furthermore, we constructed a prototype application system that allows users to control the stiffness of a fabric without changing the actual physical fabric. By evaluating the prototype, we confirmed that the proposed technique can manipulate the perceived stiffness of various materials (i.e., cotton, polyester, and mixed cotton and linen) at an average accuracy of 90.3 percent. [ABSTRACT FROM AUTHOR]

This paper presents an approach for non-contact haptic augmentation with spatial augmented reality (SAR). We construct a thermo-visual projection system by combining a standard RGB projector and a fabricated infrared (IR) projector. The primary contribution of this paper is that we conduct thorough psychophysical experiments to investigate a design guideline for spatiotemporal projection patterns for both RGB and IR projectors to render a warm object with high presence. We develop application systems to evaluate the validity of the proposed system and design guideline. The evaluation results demonstrate that the proposed system can render warm objects with significantly higher presence than a standard SAR system. [ABSTRACT FROM AUTHOR]

This paper introduces a novel photometric compensation technique for inter-projector luminance and chrominance variations. Although it sounds as a classical technical issue, to the best of our knowledge there is no existing solution to alleviate the spatial non-uniformity among strongly heterogeneous projectors at perceptually acceptable quality. Primary goal of our method is increasing the perceived seamlessness of the projection system by automatically generating an improved and consistent visual quality. It builds upon the existing research of multi-projection systems, but instead of working with perceptually non-uniform color spaces such as CIEXYZ, the overall computation is carried out using the RLab color appearance model which models the color processing in an adaptive, perceptual manner. Besides, we propose an adaptive color gamut acquisition, spatially varying gamut mapping, and optimization framework for edge blending. The paper describes the overall workflow and detailed algorithm of each component, followed by an evaluation validating the proposed method. The experimental results both qualitatively and quantitatively show the proposed method significant improved the visual quality of projected results of a multi-projection display with projectors with severely heterogeneous color processing. [ABSTRACT FROM AUTHOR]

Spatial augmented reality (SAR) pursues realism in rendering materials and objects. To advance this goal, we propose a hybrid SAR (HySAR) that combines a projector with optical see-through head-mounted displays (OST-HMD). In an ordinary SAR scenario with co-located viewers, the viewers perceive the same virtual material on physical surfaces. In general, the material consists of two components: a view-independent (VI) component such as diffuse reflection, and a view-dependent (VD) component such as specular reflection. The VI component is static over viewpoints, whereas the VD should change for each viewpoint even if a projector can simulate only one viewpoint at one time. In HySAR, a projector only renders the static VI components. In addition, the OST-HMD renders the dynamic VD components according to the viewer's current viewpoint. Unlike conventional SAR, the HySAR concept theoretically allows an unlimited number of co-located viewers to see the correct material over different viewpoints. Furthermore, the combination enhances the total dynamic range, the maximum intensity, and the resolution of perceived materials. With proof-of-concept systems, we demonstrate HySAR both qualitatively and quantitatively with real objects. First, we demonstrate HySAR by rendering synthetic material properties on a real object from different viewpoints. Our quantitative evaluation shows that our system increases the dynamic range by 2.24 times and the maximum intensity by 2.12 times compared to an ordinary SAR system. Second, we replicate the material properties of a real object by SAR and HySAR, and show that HySAR outperforms SAR in rendering VD specular components. [ABSTRACT FROM PUBLISHER]

We propose a visual marker embedding method for the pose estimation of a projection surface to correctly map projected images onto the surface. Assuming that the surface is fabricated by a full-color or multi-material three-dimensional (3D) printer, we propose to automatically embed visual markers on the surface with mechanical accuracy. The appearance of the marker is designed such that the marker is detected by infrared cameras even when printed on a non-planar surface while its appearance can be diminished by the projection to be as imperceptible as possible to human observers. The marker placement is optimized using a genetic algorithm to maximize the number of valid viewpoints from which the pose of the object can be estimated correctly using a stereo camera system. We also propose a radiometric compensation technique to quickly diminish the marker appearance. Experimental results confirm that the pose of projection objects are correctly estimated while the appearance of the markers was diminished to an imperceptible level. At the same time, we confirmed the limitations of the current method; only one object can be handled, and pose estimation is not performed at interactive frame rates. Finally, we demonstrate the proposed technique to show that it works successfully for various surface shapes and target textures. [ABSTRACT FROM AUTHOR]

This paper presents a novel projected pixel localization principle for online geometric registration in dynamic projection mapping applications. We propose applying a time measurement of a laser projector raster-scanning beam using a photosensor to estimate its position while the projector displays meaningful visual information to human observers. Based on this principle, we develop two types of position estimation techniques. One estimates the position of a projected beam when it directly illuminates a photosensor. The other localizes a beam by measuring the reflection from a retro-reflective marker with the photosensor placed in the optical path of the projector. We conduct system evaluations using prototypes to validate this method as well as to confirm the applicability of our principle. In addition, we discuss the technical limitations of the prototypes based on the evaluation results. Finally, we build several dynamic projection mapping applications to demonstrate the feasibility of our principle. [ABSTRACT FROM AUTHOR]

This paper presents the extended work on LazyNav, a head-free, eyes-free and hands-free mid-air ground navigation control model presented at the IEEE 3D User Interfaces (3DUI) 2015, in particular with a new application to the head-mounted display (HMD). Our mid-air interaction metaphor makes use of only a single pair of the remaining tracked body elements to tailor the navigation. Therefore, the user can navigate in the scene while still being able to perform other interactions with her hands and head, e.g., carrying a bag, grasping a cup of coffee, or observing the content by moving her eyes and locally rotating her head. We design several body motions for navigation by considering the use of non-critical body parts and develop assumptions about ground navigation techniques. Through the user studies, we investigate the motions that are easy to discover, easy to control, socially acceptable, accurate and not tiring. Finally, we evaluate the desired ground navigation features with a prototype application in both a large display (LD) and a HMD navigation scenarios. We highlight several recommendations for designing a particular mid-air ground navigation technique for a LD and a HMD. [ABSTRACT FROM AUTHOR]

We propose a method to calibrate viewpoint-dependent, channel-wise image blur of near-eye displays, especially of Optical See-Through Head-Mounted Displays (OST-HMDs). Imperfections in HMD optics cause channel-wise image shift and blur that degrade the image quality of the display at a user's viewpoint. If we can estimate such characteristics perfectly, we could mitigate the effect by applying correction techniques from the computational photography in computer vision as analogous to cameras. Unfortunately, directly applying existing calibration techniques of cameras to OST-HMDs is not a straightforward task. Unlike ordinary imaging systems, image blur in OST-HMDs is viewpoint-dependent, i.e., the optical characteristic of a display dynamically changes depending on the current viewpoint of the user. This constraint makes the problem challenging since we must measure image blur of an HMD, ideally, over the entire 3D eyebox in which a user can see an image. To overcome this problem, we model the viewpoint-dependent blur as a Gaussian Light Field (GLF) that stores spatial information of the display screen as a (4D) light field with depth information and the blur as point-spread functions in the form of Gaussian kernels, respectively. We first describe both our GLF model and a calibration procedure to learn a GLF for a given OST-HMD. We then apply our calibration method to two HMDs that use different optics: a cubic prism or holographic gratings. The results show that our method achieves significantly better accuracy in Point-Spread Function (PSF) estimations with an accuracy about 2 to 7 dB in Peak SNR. [ABSTRACT FROM PUBLISHER]

We propose a novel inter-reflection compensation technique for immersive projection displays wherein we spatially modulate the reflectance pattern on the screen to improve the compensation performance of conventional methods. As the luminance of light reflected on a projection surface is mathematically represented as the multiplication of the illuminance of incident light and the surface reflectance, we can reduce undesirable intensity elevation because of inter-reflections by decreasing surface reflectance. Based on this principle, we improve conventional inter-reflection compensation techniques by applying reflectance pattern modulation. We realize spatial reflectance modulation of a projection screen by painting it with a photochromic compound, which changes its color (i.e., the reflectance of the screen) when ultraviolet (UV) light is applied and by controlling UV irradiation with a UV LED array placed behind the screen. The main contribution of this paper is a computational model to optimize a reflectance pattern for the accurate reproduction of a target appearance by decreasing the intensity elevation caused by inter-reflection while maintaining the maximum intensity of the target appearance. Through simulation and physical experiments, we demonstrate the feasibility of the proposed model and confirm its advantage over conventional methods. [ABSTRACT FROM AUTHOR]