Search

Multimodal large language models (MLLMs) have shown impressive success across modalities such as image, video, and audio in a variety of understanding and generation tasks. However, current MLLMs are surprisingly poor at understanding webpage screenshots and generating their corresponding HTML code. To address this problem, we propose Web2Code, a benchmark consisting of a new large-scale webpage-to-code dataset for instruction tuning and an evaluation framework for the webpage understanding and HTML code translation abilities of MLLMs. For dataset construction, we leverage pretrained LLMs to enhance existing webpage-to-code datasets as well as generate a diverse pool of new webpages rendered into images. Specifically, the inputs are webpage images and instructions, while the responses are the webpage's HTML code. We further include diverse natural language QA pairs about the webpage content in the responses to enable a more comprehensive understanding of the web content. To evaluate model performance in these tasks, we develop an evaluation framework for testing MLLMs' abilities in webpage understanding and web-to-code generation. Extensive experiments show that our proposed dataset is beneficial not only to our proposed tasks but also in the general visual domain, while previous datasets result in worse performance. We hope our work will contribute to the development of general MLLMs suitable for web-based content generation and task automation. Our data and code will be available at https://github.com/MBZUAI-LLM/web2code., Comment: Website at https://mbzuai-llm.github.io/webpage2code/

Multi-rater annotations commonly occur when medical images are independently annotated by multiple experts (raters). In this paper, we tackle two challenges arisen in multi-rater annotations for medical image segmentation (called ambiguous medical image segmentation): (1) How to train a deep learning model when a group of raters produces a set of diverse but plausible annotations, and (2) how to fine-tune the model efficiently when computation resources are not available for re-training the entire model on a different dataset domain. We propose a multi-rater prompt-based approach to address these two challenges altogether. Specifically, we introduce a series of rater-aware prompts that can be plugged into the U-Net model for uncertainty estimation to handle multi-annotation cases. During the prompt-based fine-tuning process, only 0.3% of learnable parameters are required to be updated comparing to training the entire model. Further, in order to integrate expert consensus and disagreement, we explore different multi-rater incorporation strategies and design a mix-training strategy for comprehensive insight learning. Extensive experiments verify the effectiveness of our new approach for ambiguous medical image segmentation on two public datasets while alleviating the heavy burden of model re-training., Comment: Accepted by BIBM 2024

Automatic ophthalmic disease diagnosis on fundus images is important in clinical practice. However, due to complex fundus textures and limited annotated data, developing an effective automatic method for this problem is still challenging. In this paper, we present a self-supervised method via polar transformation based progressive contrastive learning, called PoCo, for ophthalmic disease diagnosis. Specifically, we novelly inject the polar transformation into contrastive learning to 1) promote contrastive learning pre-training to be faster and more stable and 2) naturally capture task-free and rotation-related textures, which provides insights into disease recognition on fundus images. Beneficially, simple normal translation-invariant convolution on transformed images can equivalently replace the complex rotation-invariant and sector convolution on raw images. After that, we develop a progressive contrastive learning method to efficiently utilize large unannotated images and a novel progressive hard negative sampling scheme to gradually reduce the negative sample number for efficient training and performance enhancement. Extensive experiments on three public ophthalmic disease datasets show that our PoCo achieves state-of-the-art performance with good generalization ability, validating that our method can reduce annotation efforts and provide reliable diagnosis. Codes are available at \url{https://github.com/wjh892521292/PoCo}.

In clinical practice, medical image segmentation provides useful information on the contours and dimensions of target organs or tissues, facilitating improved diagnosis, analysis, and treatment. In the past few years, convolutional neural networks (CNNs) and Transformers have dominated this area, but they still suffer from either limited receptive fields or costly long-range modeling. Mamba, a State Space Sequence Model (SSM), recently emerged as a promising paradigm for long-range dependency modeling with linear complexity. In this paper, we introduce a Large Kernel Vision Mamba U-shape Network, or LKM-UNet, for medical image segmentation. A distinguishing feature of our LKM-UNet is its utilization of large Mamba kernels, excelling in locally spatial modeling compared to small kernel-based CNNs and Transformers, while maintaining superior efficiency in global modeling compared to self-attention with quadratic complexity. Additionally, we design a novel hierarchical and bidirectional Mamba block to further enhance Mamba's global and neighborhood spatial modeling capability for vision inputs. Comprehensive experiments demonstrate the feasibility and the effectiveness of using large-size Mamba kernels to achieve large receptive fields. Codes are available at https://github.com/wjh892521292/LKM-UNet., Comment: Accepted by MICCAI 2024

Uveitis demands the precise diagnosis of anterior chamber inflammation (ACI) for optimal treatment. However, current diagnostic methods only rely on a limited single-modal disease perspective, which leads to poor performance. In this paper, we investigate a promising yet challenging way to fuse multimodal data for ACI diagnosis. Notably, existing fusion paradigms focus on empowering implicit modality interactions (i.e., self-attention and its variants), but neglect to inject explicit modality interactions, especially from clinical knowledge and imaging property. To this end, we propose a jointly Explicit and implicit Cross-Modal Interaction Network (EiCI-Net) for Anterior Chamber Inflammation Diagnosis that uses anterior segment optical coherence tomography (AS-OCT) images, slit-lamp images, and clinical data jointly. Specifically, we first develop CNN-Based Encoders and Tabular Processing Module (TPM) to extract efficient feature representations in different modalities. Then, we devise an Explicit Cross-Modal Interaction Module (ECIM) to generate attention maps as a kind of explicit clinical knowledge based on the tabular feature maps, then integrated them into the slit-lamp feature maps, allowing the CNN-Based Encoder to focus on more effective informativeness of the slit-lamp images. After that, the Implicit Cross-Modal Interaction Module (ICIM), a transformer-based network, further implicitly enhances modality interactions. Finally, we construct a considerable real-world dataset from our collaborative hospital and conduct sufficient experiments to demonstrate the superior performance of our proposed EiCI-Net compared with the state-of-the-art classification methods in various metrics.

Ordinal regression refers to classifying object instances into ordinal categories. It has been widely studied in many scenarios, such as medical disease grading, movie rating, etc. Known methods focused only on learning inter-class ordinal relationships, but still incur limitations in distinguishing adjacent categories thus far. In this paper, we propose a simple sequence prediction framework for ordinal regression called Ord2Seq, which, for the first time, transforms each ordinal category label into a special label sequence and thus regards an ordinal regression task as a sequence prediction process. In this way, we decompose an ordinal regression task into a series of recursive binary classification steps, so as to subtly distinguish adjacent categories. Comprehensive experiments show the effectiveness of distinguishing adjacent categories for performance improvement and our new approach exceeds state-of-the-art performances in four different scenarios. Codes are available at https://github.com/wjh892521292/Ord2Seq., Comment: Accepted by ICCV2023

Image ordinal regression has been mainly studied along the line of exploiting the order of categories. However, the issues of class imbalance and category overlap that are very common in ordinal regression were largely overlooked. As a result, the performance on minority categories is often unsatisfactory. In this paper, we propose a novel framework called CIG based on controllable image generation to directly tackle these two issues. Our main idea is to generate extra training samples with specific labels near category boundaries, and the sample generation is biased toward the less-represented categories. To achieve controllable image generation, we seek to separate structural and categorical information of images based on structural similarity, categorical similarity, and reconstruction constraints. We evaluate the effectiveness of our new CIG approach in three different image ordinal regression scenarios. The results demonstrate that CIG can be flexibly integrated with off-the-shelf image encoders or ordinal regression models to achieve improvement, and further, the improvement is more significant for minority categories., Comment: 8 pages, 12 figures, to be published in IJCAI2023

Understanding forest health is of great importance for the conservation of the integrity of forest ecosystems. In this regard, evaluating the amount and quality of dead wood is of utmost interest as they are favorable indicators of biodiversity. Apparently, remote sensing-based machine learning techniques have proven to be more efficient and sustainable with unprecedented accuracy in forest inventory. This study, for the first time, automatically categorizing individual coniferous trees (Norway spruce) into five decay stages (live, declining, dead, loose bark, and clean) from combined airborne laser scanning (ALS) point clouds and color infrared (CIR) images using three different Machine Learning methods - 3D point cloud-based deep learning (KPConv), Convolutional Neural Network (CNN), and Random Forest (RF). First, CIR colorized point clouds are created by fusing the ALS point clouds and color infrared images. Then, individual tree segmentation is conducted, after which the results are further projected onto four orthogonal planes. Finally, the classification is conducted on the two datasets (3D multispectral point clouds and 2D projected images) based on the three Machine Learning algorithms. All models achieved promising results, reaching overall accuracy (OA) of up to 88.8%, 88.4% and 85.9% for KPConv, CNN and RF, respectively. The experimental results reveal that color information, 3D coordinates, and intensity of point clouds have significant impact on the promising classification performance. The performance of our models, therefore, shows the significance of machine/deep learning for individual tree decay stages classification and landscape-wide assessment of the dead wood amount and quality by using modern airborne remote sensing techniques. The proposed method can contribute as an important and reliable tool for monitoring biodiversity in forest ecosystems.

Surgery is the only viable treatment for cataract patients with visual acuity (VA) impairment. Clinically, to assess the necessity of cataract surgery, accurately predicting postoperative VA before surgery by analyzing multi-view optical coherence tomography (OCT) images is crucially needed. Unfortunately, due to complicated fundus conditions, determining postoperative VA remains difficult for medical experts. Deep learning methods for this problem were developed in recent years. Although effective, these methods still face several issues, such as not efficiently exploring potential relations between multi-view OCT images, neglecting the key role of clinical prior knowledge (e.g., preoperative VA value), and using only regression-based metrics which are lacking reference. In this paper, we propose a novel Cross-token Transformer Network (CTT-Net) for postoperative VA prediction by analyzing both the multi-view OCT images and preoperative VA. To effectively fuse multi-view features of OCT images, we develop cross-token attention that could restrict redundant/unnecessary attention flow. Further, we utilize the preoperative VA value to provide more information for postoperative VA prediction and facilitate fusion between views. Moreover, we design an auxiliary classification loss to improve model performance and assess VA recovery more sufficiently, avoiding the limitation by only using the regression metrics. To evaluate CTT-Net, we build a multi-view OCT image dataset collected from our collaborative hospital. A set of extensive experiments validate the effectiveness of our model compared to existing methods in various metrics. Code is available at: https://github.com/wjh892521292/Cataract OCT., Comment: 5 pages, 3 figures, accepted for publication in BIBM

Computer-aided medical image segmentation has been applied widely in diagnosis and treatment to obtain clinically useful information of shapes and volumes of target organs and tissues. In the past several years, convolutional neural network (CNN) based methods (e.g., U-Net) have dominated this area, but still suffered from inadequate long-range information capturing. Hence, recent work presented computer vision Transformer variants for medical image segmentation tasks and obtained promising performances. Such Transformers model long-range dependency by computing pair-wise patch relations. However, they incur prohibitive computational costs, especially on 3D medical images (e.g., CT and MRI). In this paper, we propose a new method called Dilated Transformer, which conducts self-attention for pair-wise patch relations captured alternately in local and global scopes. Inspired by dilated convolution kernels, we conduct the global self-attention in a dilated manner, enlarging receptive fields without increasing the patches involved and thus reducing computational costs. Based on this design of Dilated Transformer, we construct a U-shaped encoder-decoder hierarchical architecture called D-Former for 3D medical image segmentation. Experiments on the Synapse and ACDC datasets show that our D-Former model, trained from scratch, outperforms various competitive CNN-based or Transformer-based segmentation models at a low computational cost without time-consuming per-training process.

Understanding human driving behaviors quantitatively is critical even in the era when connected and autonomous vehicles and smart infrastructure are becoming ever more prevalent. This is particularly so as that mixed traffic settings, where autonomous vehicles and human driven vehicles co-exist, are expected to persist for quite some time. Towards this end it is necessary that we have a comprehensive modeling framework for decision-making within which human driving preferences can be inferred statistically from observed driving behaviors in realistic and naturalistic traffic settings. Leveraging a recently proposed computational framework for smart vehicles in a smart world using multi-agent based simulation and optimization, we first recapitulate how the forward problem of driving decision-making is modeled as a state space model. We then show how the model can be inverted to estimate driver preferences from naturalistic traffic data using the standard Kalman filter technique. We explicitly illustrate our approach using the vehicle trajectory data from Sugiyama experiment that was originally meant to demonstrate how stop-and-go shockwave can arise spontaneously without bottlenecks. Not only the estimated state filter can fit the observed data well for each individual vehicle, the inferred utility functions can also re-produce quantitatively similar pattern of the observed collective behaviors. One distinct advantage of our approach is the drastically reduced computational burden. This is possible because our forward model treats driving decision process, which is intrinsically dynamic with multi-agent interactions, as a sequence of independent static optimization problems contingent on the state with a finite look ahead anticipation. Consequently we can practically sidestep solving an interacting dynamic inversion problem that would have been much more computationally demanding.

We present the second prototype of a time-to-digital (TDC) ASIC for the upgrade of the ATLAS Monitored Drift Tube (MDT) detector for High-Luminosity LHC operations. Compared to the first prototype, triple modular redundancy has been implemented for the configuration and flow control logic. The total power consumption is increased by less than 10 mW while achieving the same time resolution and channel uniformity. A mini-DAQ system has been built to verify the front-end electronics chain with the new prototype together with other ASICs and boards in triggered mode. Cosmic ray tests with a small-diameter MDT chamber indicate that the configuration and data transmission of the readout electronics perform well. It is expected that this prototype design will be used in the final production., Comment: 11 pages, 7 figures

The Trigger Data Serializer (TDS) is a custom ASIC designed for the upgrade of the innermost station of the endcap ATLAS Muon Spectrometer. It is a mixed-signal chip with two working modes that can handle up to 128 detector channels. A total of 6,000 TDS ASICs have been produced for detector operation. This paper discusses a custom automatic test platform we developed to provide quality control of the TDS ASICs. We introduce the design, test procedures, and results obtained from this TDS testing platform., Comment: 6 pages, 7 figures

We present a multi-layer single-event upset mitigation strategy implemented in a low-cost Xilinx Artix-7 FPGA. The implementation is targeted for a trigger data router for the ATLAS muon spectrometer upgrade. The mitigation strategy employs three layers of protection to improve overall FPGA design robustness: use of triple-modular redundancy for FPGA fabric logic and embedded soft-error mitigation in the first layer; further enhancement with multi-boot FPGA reconfiguration across multiple copies of configuration memory in the second layer; and FPGA power cycling and configuration memory re-initialization in the third layer. The effectiveness of this scheme has been evaluated at two different neutron facilities, LANSCE and NCSR Demokritos, with 800 MeV and 25 MeV beam energies, respectively. Testing was performed with a similar configuration to that planned for final operation. We discuss the testing strategy and summarize the test results to estimate the expected data loss over 10 years of operation in the ATLAS experiment., Comment: submitted to Nuclear Instruments and Methods in Physics research A

We present the prototype of a time-to-digital (TDC) ASIC for the upgrade of the ATLAS Monitored Drift Tube (MDT) detector for high-luminosity LHC operation. This ASIC is based on a previously submitted demonstrator ASIC designed for timing performance evaluation, and includes all features necessary for the various operation modes, as well as the migration to the TSMC 130 nm CMOS technology. We present the TDC design with the emphasis on added features and performance optimization. Tests of the timing performance demonstrate that this ASIC meets the design specifications. The TDC has a bin size of about 780 ps, and a timing bin variations within 40 ps for all 24 channels with leading and trailing edge digitization, while the power consumption has been limited to 250 mW, corresponding to a consumption of about 5.2 mW per edge measurement., Comment: 9 pages, 7 figures

The upgrade of the ATLAS muon spectrometer for high-luminosity LHC requires new trigger and readout electronics for the various elements of the detector. We present the design of a time-to-digital converter (TDC) ASIC prototype for the ATLAS Monitored Drift Tube (MDT) detector. The chip was fabricated in a GlobalFoundries 130 nm CMOS technology. Studies indicate that its timing and power consumption characteristics meet the design specifications, with a timing bin variation of 40 ps for all 48 channels with a power consumption of about 6.5 mW per channel., Comment: 9 pages, 12 figures

The small-strip Thin Gap Chamber (sTGC) will be used for both triggering and precision tracking purposes in the upgrade of the ATLAS forward muon spectrometer. Both sTGC pad and strip detectors are readout by a Trigger Data Serializer (TDS) ASIC in the trigger path. This ASIC has two operation modes to prepare trigger data from pad and strip detectors respectively. The pad mode (pad-TDS) collects the firing status for up to 104 pads from one detector layer and transmits the data at 4.8 Gbps to the pad trigger extractor every 25 ns. The pad trigger extractor collects pad-TDS data from eight detector layers and defines a region of interest along the path of a muon candidate. In the strip mode (strip-TDS), the deposited charges from up to 128 strips are buffered, time-stamped, and a trigger matching procedure is performed to read out strips underneath the region of interest. The strip-TDS output is also transmitted at 4.8 Gbps to the following FPGA processing circuits. Details about the ASIC design and test results are presented in this paper., Comment: 10 pages, 12 figures

We present a programmable time alignment scheme used in an ASIC for the ATLAS forward muon trigger development. The scheme utilizes regenerated clocks with programmable phases to compensate for the timing offsets introduced by different detector trace lengths. Each ASIC used in the design has 104 input channels with delay compensation circuitry providing steps of ~ 3 ns and a full range of 25 ns for each channel. Detailed implementation of the scheme including majority logic to suppress single-event effects is presented. The scheme is flexible and fully synthesizable. The approach is adaptable to other applications with similar phase shifting requirements. In addition, the design is resource efficient and is suitable for cost-effective digital implementation with a large number of channels., Comment: 5 pages, 8 figures