Search

Teaching to improve student models (e.g., knowledge distillation) is an extensively studied methodology in LLMs. However, for humans, teaching not only improves students but also improves teachers. We ask: Can LLMs also learn by teaching (LbT)? If yes, we can potentially unlock the possibility of continuously advancing the models without solely relying on human-produced data or stronger models. In this paper, we provide a preliminary exploration of this ambitious agenda. We show that LbT ideas can be incorporated into existing LLM training/prompting pipelines and provide noticeable improvements. Specifically, we design three methods, each mimicking one of the three levels of LbT in humans: observing students' feedback, learning from the feedback, and learning iteratively, with the goals of improving answer accuracy without training and improving models' inherent capability with fine-tuning. The findings are encouraging. For example, similar to LbT in human, we see that: (1) LbT can induce weak-to-strong generalization: strong models can improve themselves by teaching other weak models; (2) Diversity in students might help: teaching multiple students could be better than teaching one student or the teacher itself. We hope that this early promise can inspire future research on LbT and more broadly adopting the advanced techniques in education to improve LLMs. The code is available at https://github.com/imagination-research/lbt., Comment: Under review

Semantic segmentation datasets often exhibit two types of imbalance: \textit{class imbalance}, where some classes appear more frequently than others and \textit{size imbalance}, where some objects occupy more pixels than others. This causes traditional evaluation metrics to be biased towards \textit{majority classes} (e.g. overall pixel-wise accuracy) and \textit{large objects} (e.g. mean pixel-wise accuracy and per-dataset mean intersection over union). To address these shortcomings, we propose the use of fine-grained mIoUs along with corresponding worst-case metrics, thereby offering a more holistic evaluation of segmentation techniques. These fine-grained metrics offer less bias towards large objects, richer statistical information, and valuable insights into model and dataset auditing. Furthermore, we undertake an extensive benchmark study, where we train and evaluate 15 modern neural networks with the proposed metrics on 12 diverse natural and aerial segmentation datasets. Our benchmark study highlights the necessity of not basing evaluations on a single metric and confirms that fine-grained mIoUs reduce the bias towards large objects. Moreover, we identify the crucial role played by architecture designs and loss functions, which lead to best practices in optimizing fine-grained metrics. The code is available at \href{https://github.com/zifuwanggg/JDTLosses}{https://github.com/zifuwanggg/JDTLosses}., Comment: NeurIPS 2023

This work aims at decreasing the end-to-end generation latency of large language models (LLMs). One of the major causes of the high generation latency is the sequential decoding approach adopted by almost all state-of-the-art LLMs. In this work, motivated by the thinking and writing process of humans, we propose Skeleton-of-Thought (SoT), which first guides LLMs to generate the skeleton of the answer, and then conducts parallel API calls or batched decoding to complete the contents of each skeleton point in parallel. Not only does SoT provide considerable speed-ups across 12 LLMs, but it can also potentially improve the answer quality on several question categories. SoT is an initial attempt at data-centric optimization for inference efficiency, and showcases the potential of eliciting high-quality answers by explicitly planning the answer structure in language., Comment: In ICLR'24

The soft Dice loss (SDL) has taken a pivotal role in numerous automated segmentation pipelines in the medical imaging community. Over the last years, some reasons behind its superior functioning have been uncovered and further optimizations have been explored. However, there is currently no implementation that supports its direct utilization in scenarios involving soft labels. Hence, a synergy between the use of SDL and research leveraging the use of soft labels, also in the context of model calibration, is still missing. In this work, we introduce Dice semimetric losses (DMLs), which (i) are by design identical to SDL in a standard setting with hard labels, but (ii) can be employed in settings with soft labels. Our experiments on the public QUBIQ, LiTS and KiTS benchmarks confirm the potential synergy of DMLs with soft labels (e.g. averaging, label smoothing, and knowledge distillation) over hard labels (e.g. majority voting and random selection). As a result, we obtain superior Dice scores and model calibration, which supports the wider adoption of DMLs in practice. The code is available at https://github.com/zifuwanggg/JDTLosses, Comment: MICCAI 2023

Intersection over Union (IoU) losses are surrogates that directly optimize the Jaccard index. Leveraging IoU losses as part of the loss function have demonstrated superior performance in semantic segmentation tasks compared to optimizing pixel-wise losses such as the cross-entropy loss alone. However, we identify a lack of flexibility in these losses to support vital training techniques like label smoothing, knowledge distillation, and semi-supervised learning, mainly due to their inability to process soft labels. To address this, we introduce Jaccard Metric Losses (JMLs), which are identical to the soft Jaccard loss in standard settings with hard labels but are fully compatible with soft labels. We apply JMLs to three prominent use cases of soft labels: label smoothing, knowledge distillation and semi-supervised learning, and demonstrate their potential to enhance model accuracy and calibration. Our experiments show consistent improvements over the cross-entropy loss across 4 semantic segmentation datasets (Cityscapes, PASCAL VOC, ADE20K, DeepGlobe Land) and 13 architectures, including classic CNNs and recent vision transformers. Remarkably, our straightforward approach significantly outperforms state-of-the-art knowledge distillation and semi-supervised learning methods. The code is available at \href{https://github.com/zifuwanggg/JDTLosses}{https://github.com/zifuwanggg/JDTLosses}., Comment: NeurIPS 2023

Broadband Kerr combs with a flat comb spectral profile are expected in a number of applications, such as high-capacity optical communication. Here, we propose novel concentric dual-ring microresonators (DRMs) for advanced dispersion engineering to tailor the comb spectral profile. The dispersion can be flexibly engineered not only by the cross-section of the DRMs, but also by the gap between concentric dual-ring microresonators, which provides a new path to geometrically control the spectral profile of the soliton Kerr combs. An octave-spanning dissipative Kerr soliton comb with superior spectral flatness has been achieved numerically, covering from the telecommunication band to the mid-infrared (MIR) band region with a -40 dB bandwidth of 1265 nm (99.82 THz). Our results are promising to fully understand the nonlinear dynamics in hybrid modes in DRMs, which helps control broadband comb formation., Comment: 12 pages, 4 figures

UNet [27] is widely used in semantic segmentation due to its simplicity and effectiveness. However, its manually-designed architecture is applied to a large number of problem settings, either with no architecture optimizations, or with manual tuning, which is time consuming and can be sub-optimal. In this work, firstly, we propose Markov Random Field Neural Architecture Search (MRF-NAS) that extends and improves the recent Adaptive and Optimal Network Width Search (AOWS) method [4] with (i) a more general MRF framework (ii) diverse M-best loopy inference (iii) differentiable parameter learning. This provides the necessary NAS framework to efficiently explore network architectures that induce loopy inference graphs, including loops that arise from skip connections. With UNet as the backbone, we find an architecture, MRF-UNet, that shows several interesting characteristics. Secondly, through the lens of these characteristics, we identify the sub-optimality of the original UNet architecture and further improve our results with MRF-UNetV2. Experiments show that our MRF-UNets significantly outperform several benchmarks on three aerial image datasets and two medical image datasets while maintaining low computational costs. The code is available at: https://github.com/zifuwanggg/MRF-UNets., Comment: ECML-PKDD 2022

Photonic integrated microcombs have enabled advanced applications in optical communication, microwave synthesis, and optical metrology, which in nature unveil an optical dissipative soliton pattern under cavity-enhanced nonlinear processes. The most decisive factor of microcombs lies in the photonic material platforms, where materials with high nonlinearity and in capacity of high-quality chip integration are highly demanded. In this work, we present a home-developed chalcogenide glasses-Ge25Sb10S65 (GeSbS) for the nonlinear photonic integration and for the dissipative soliton microcomb generation. Compared with the current integrated nonlinear platforms, the GeSbS features wider transparency from the visible to 11 um region, stronger nonlinearity, and lower thermo-refractive coefficient, and is CMOS compatible in fabrication. In this platform, we achieve chip-integrated optical microresonators with a quality (Q) factor above 2 x 10^6, and carry out lithographically controlled dispersion engineering. In particular, we demonstrate that both a bright soliton-based microcomb and a dark-pulsed comb are generated in a single microresonator, in its separated fundamental polarized mode families under different dispersion regimes. The overall pumping power is on the ten-milliwatt level, determined by both the high Q-factor and the high material nonlinearity of the microresonator. Our results may contribute to the field of nonlinear photonics with an alternative material platform for highly compact and high-intensity nonlinear interactions, while on the application aspect, contribute to the development of soliton microcombs at low operation power, which is potentially required for monolithically integrated optical frequency combs., Comment: 22 pages, 5 figures

The sudden outbreak of the Coronavirus disease (COVID-19) swept across the world in early 2020, triggering the lockdowns of several billion people across many countries, including China, Spain, India, the U.K., Italy, France, Germany, and most states of the U.S. The transmission of the virus accelerated rapidly with the most confirmed cases in the U.S., and New York City became an epicenter of the pandemic by the end of March. In response to this national and global emergency, the NSF Spatiotemporal Innovation Center brought together a taskforce of international researchers and assembled implemented strategies to rapidly respond to this crisis, for supporting research, saving lives, and protecting the health of global citizens. This perspective paper presents our collective view on the global health emergency and our effort in collecting, analyzing, and sharing relevant data on global policy and government responses, geospatial indicators of the outbreak and evolving forecasts; in developing research capabilities and mitigation measures with global scientists, promoting collaborative research on outbreak dynamics, and reflecting on the dynamic responses from human societies., Comment: 27 pages, 18 figures. International Journal of Digital Earth (2020)

In order to project electromagnetic fields between different meshes with respect to the conservation of energetic values, Galerkin projection formulations based on the energetic norm are developed in this communication. The proposed formulations are applied to an academic example., Comment: published at Compumag 2013

Spatial Galerkin projection transfers fields between different meshes. In the area of finite element analysis of electromagnetic fields, it provides great convenience for remeshing, multi-physics, domain decomposition methods, etc. In this paper, a space-time Galerkin projection is developed in order to transfer fields between different spatial and temporal discretization bases., Comment: Published at Compumag 2015