Search

Sketches, with their expressive potential, allow humans to convey the essence of an object through even a rough contour. For the first time, we harness this expressive potential to improve segmentation performance in challenging tasks like camouflaged object detection (COD). Our approach introduces an innovative sketch-guided interactive segmentation framework, allowing users to intuitively annotate objects with freehand sketches (drawing a rough contour of the object) instead of the traditional bounding boxes or points used in classic interactive segmentation models like SAM. We demonstrate that sketch input can significantly improve performance in existing iterative segmentation methods, outperforming text or bounding box annotations. Additionally, we introduce key modifications to network architectures and a novel sketch augmentation technique to fully harness the power of sketch input and further boost segmentation accuracy. Remarkably, our model' s output can be directly used to train other neural networks, achieving results comparable to pixel-by-pixel annotations--while reducing annotation time by up to 120 times, which shows great potential in democratizing the annotation process and enabling model training with less reliance on resource-intensive, laborious pixel-level annotations. We also present KOSCamo+, the first freehand sketch dataset for camouflaged object detection. The dataset, code, and the labeling tool will be open sourced.

Fully supervised training of semantic segmentation models is costly and challenging because each pixel within an image needs to be labeled. Therefore, the sparse pixel-level annotation methods have been introduced to train models with a subset of pixels within each image. We introduce a Bayesian active learning framework based on sparse pixel-level annotation that utilizes a pixel-level Bayesian uncertainty measure based on Balanced Entropy (BalEnt) [84]. BalEnt captures the information between the models' predicted marginalized probability distribution and the pixel labels. BalEnt has linear scalability with a closed analytical form and can be calculated independently per pixel without relational computations with other pixels. We train our proposed active learning framework for Cityscapes, Camvid, ADE20K and VOC2012 benchmark datasets and show that it reaches supervised levels of mIoU using only a fraction of labeled pixels while outperforming the previous state-of-the-art active learning models with a large margin.

Assessing response quality to instructions in language models is vital but challenging due to the complexity of human language across different contexts. This complexity often results in ambiguous or inconsistent interpretations, making accurate assessment difficult. To address this issue, we propose a novel Uncertainty-aware Reward Model (URM) that introduces a robust uncertainty estimation for the quality of paired responses based on Bayesian approximation. Trained with preference datasets, our uncertainty-enabled proxy not only scores rewards for responses but also evaluates their inherent uncertainty. Empirical results demonstrate significant benefits of incorporating the proposed proxy into language model training. Our method boosts the instruction following capability of language models by refining data curation for training and improving policy optimization objectives, thereby surpassing existing methods by a large margin on benchmarks such as Vicuna and MT-bench. These findings highlight that our proposed approach substantially advances language model training and paves a new way of harnessing uncertainty within language models., Comment: Accepted to ICML 2024

Referring expression segmentation (RES), a task that involves localizing specific instance-level objects based on free-form linguistic descriptions, has emerged as a crucial frontier in human-AI interaction. It demands an intricate understanding of both visual and textual contexts and often requires extensive training data. This paper introduces RESMatch, the first semi-supervised learning (SSL) approach for RES, aimed at reducing reliance on exhaustive data annotation. Extensive validation on multiple RES datasets demonstrates that RESMatch significantly outperforms baseline approaches, establishing a new state-of-the-art. Although existing SSL techniques are effective in image segmentation, we find that they fall short in RES. Facing the challenges including the comprehension of free-form linguistic descriptions and the variability in object attributes, RESMatch introduces a trifecta of adaptations: revised strong perturbation, text augmentation, and adjustments for pseudo-label quality and strong-weak supervision. This pioneering work lays the groundwork for future research in semi-supervised learning for referring expression segmentation.

As 3D models become critical in today's manufacturing and product design, conventional 3D modeling approaches based on Computer-Aided Design (CAD) are labor-intensive, time-consuming, and have high demands on the creators. This work aims to introduce an alternative approach to 3D modeling by utilizing free-hand sketches to obtain desired 3D models. We introduce Deep3DSketch+, which is a deep-learning algorithm that takes the input of a single free-hand sketch and produces a complete and high-fidelity model that matches the sketch input. The neural network has view- and structural-awareness enabled by a Shape Discriminator (SD) and a Stroke Enhancement Module (SEM), which overcomes the limitations of sparsity and ambiguity of the sketches. The network design also brings high robustness to partial sketch input in industrial applications.Our approach has undergone extensive experiments, demonstrating its state-of-the-art (SOTA) performance on both synthetic and real-world datasets. These results validate the effectiveness and superiority of our method compared to existing techniques. We have demonstrated the conversion of free-hand sketches into physical 3D objects using additive manufacturing. We believe that our approach has the potential to accelerate product design and democratize customized manufacturing.

The rise of AR/VR has led to an increased demand for 3D content. However, the traditional method of creating 3D content using Computer-Aided Design (CAD) is a labor-intensive and skill-demanding process, making it difficult to use for novice users. Sketch-based 3D modeling provides a promising solution by leveraging the intuitive nature of human-computer interaction. However, generating high-quality content that accurately reflects the creator's ideas can be challenging due to the sparsity and ambiguity of sketches. Furthermore, novice users often find it challenging to create accurate drawings from multiple perspectives or follow step-by-step instructions in existing methods. To address this, we introduce a groundbreaking end-to-end approach in our work, enabling 3D modeling from a single free-hand sketch, Deep3DSketch+$\backslash$+. The issue of sparsity and ambiguity using single sketch is resolved in our approach by leveraging the symmetry prior and structural-aware shape discriminator. We conducted comprehensive experiments on diverse datasets, including both synthetic and real data, to validate the efficacy of our approach and demonstrate its state-of-the-art (SOTA) performance. Users are also more satisfied with results generated by our approach according to our user study. We believe our approach has the potential to revolutionize the process of 3D modeling by offering an intuitive and easy-to-use solution for novice users., Comment: Accepted at IEEE SMC 2023

Emerging wireless IoT applications increasingly venture beyond over-the-air communication, such as deep-tissue networking for implantable sensors, air-water communication for ocean monitoring, and soil sensing. These applications face the fundamental challenge of significant power loss due to reflection at media interfaces. We present RF-Mediator, a programmable metasurface system placed at media interfaces to virtually mask the presence of the physical boundary. It is designed as a single-layer metasurface comprising arrays of varactor-based elements. By tuning the bias voltage element-wise, the surface mediates between media on both sides dynamically and beamforms towards the endpoint to boost transmission through the interface, as if no media interface existed. The control algorithm determines the surface configuration by probing the search space efficiently. We fabricate the surface on a thin, flexible substrate, and experiment with several cross-media setups. Extensive evaluation shows that RF-Mediator provides a median power gain of 8 dB for air-to-tissue links and up to 30 dB for cross-media backscatter links.

With new frequency bands opening up, emerging wireless IoT devices are capitalizing on an increasingly divergent range of frequencies. However, existing coverage provisioning practice is often tied to specific standards and frequencies. There is little shareable wireless infrastructure for concurrent links on different frequencies, across networks and standards. This paper presents Scrolls, a frequency-tunable soft smart surface system to enhance wideband, multi-network coverage. Scrolls' hardware comprises many rows of rollable thin plastic film, each attached with flexible copper strips. When rolled to different lengths, the copper strips act as wire antennas reflecting signals on the corresponding frequencies. The surface control algorithm determines the unrolled strip lengths for link enhancement by probing the search space efficiently. We build a set of distributed, composable Scrolls prototypes and deploy them in an office. Extensive evaluation shows that Scrolls can adapt the antenna lengths effectively to provide link enhancement across diverse standards on sub-6 GHz bands. For concurrent links on 900 MHz (LoRa), 2.4 GHz (Wi-Fi), 3.7 GHz, and 5 GHz, Scrolls can provide received signal strength gains to all links simultaneously, by a median of 4 dB and up to 10 dB

Deep neural networks have been shown as a potentially powerful ansatz in variational Monte Carlo for solving quantum many-body problems. We propose two improvements in this direction. The first is graph neural ansatz (GNA), which is a variational wavefunction universal to arbitrary geometry. GNA results in accurate ground-state energies on 2D Kagome lattices, triangular lattices, and randomly connected graphs. Secondly, we design a distributed workflow on multiple accelerators to scale up the computation. We compute Kagome lattices with sizes up to 432 sites on 128 TPU cores. The parameter sharing nature of the GNA also leads to transferability across different system sizes and geometries., Comment: 9 pages, 3 figures, NeurIPS 2020 workshop on Machine Learning and the Physical Sciences

The recent discovery of superconductivity in Sr-doped NdNiO$_2$ films grown on SrTiO$_3$ started a novel field within unconventional superconductivity. To understand the similarities and differences between nickelate and cuprate layers on the same SrTiO$_3$ substrate, here based on the density functional theory we have systematically investigated the structural, electronic, and magnetic properties of NdNiO$_2$/SrTiO$_3$ and CaCuO$_2$/SrTiO$_3$ systems. Our results revealed a strong lattice reconstruction in the case of NdNiO$_2$/SrTiO$_3$, resulting in a polar film, with the surface and interfacial NiO$_2$ layers presenting opposite displacements. However, for CaCuO$_2$/SrTiO$_3$, the distortions of those same two CuO$_2$ layers were in the same direction. In addition, we found this distortion to be approximately independent of the studied range of film thickness for both the nickelate and cuprates films. Furthermore, we also observed a two-dimensional electron gas at the interface between NdNiO$_2$ and SrTiO$_3$, caused by the polar discontinuity, in agreement with recent literature. For NdNiO$_2$/SrTiO$_3$ the two-dimensional electron gas extends over several layers, while for CaCuO$_2$/SrTiO$_3$ this electronic rearrangement is very localized at the interface between CaCuO$_2$ and SrTiO$_3$. The electronic reconstruction found at the interface involves a strong occupation of the Ti $3d_{xy}$ state. In both cases, there is a significant electronic charge transfer from the surface Ni or Cu layers to the Ti interface layer. The interfacial Ni and Cu layer is hole and electron doped, respectively. By introducing magnetism and electronic correlation, we observed that the $d_{3z^2-r^2}$ orbital of Ni becomes itinerant while the same orbital for Cu remains doubly occupied, establishing a clear two- vs one-orbital active framework for the description of these systems., Comment: 12 pages, 9 figures

Small, low-cost IoT devices are typically equipped with only a single, low-quality antenna, significantly limiting communication range and link quality. In particular, these antennas are typically linearly polarized and therefore susceptible to polarization mismatch, which can easily cause 10-15 dBm of link loss on communication to and from such devices. In this work, we highlight this under-appreciated issue and propose the augmentation of IoT deployment environments with programmable, RF-sensitive surfaces made of metamaterials. Our smart meta-surface mitigates polarization mismatch by rotating the polarization of signals that pass through or reflect off the surface. We integrate our metasurface into an IoT network as LAMA, a Low-power Lattice of Actuated Metasurface Antennas, designed for the pervasively used 2.4 GHz ISM band. We optimize LAMA's metasurface design for both low transmission loss and low cost, to facilitate deployment at scale. We then build an end-to-end system that actuates the metasurface structure to optimize for link performance in real time. Our experimental prototype-based evaluation demonstrates gains in link power of up to 15 dBm, and wireless capacity improvements of 100 and 180 Kbit/s/Hz in through-surface and surface-reflective scenarios, respectively, attributable to the polarization rotation properties of LAMA'S metasurface.