Search

Internet video delivery has undergone a tremendous explosion of growth over the past few years. However, the quality of video delivery system greatly depends on the Internet bandwidth. Deep Neural Networks (DNNs) are utilized to improve the quality of video delivery recently. These methods divide a video into chunks, and stream LR video chunks and corresponding content-aware models to the client. The client runs the inference of models to super-resolve the LR chunks. Consequently, a large number of models are streamed in order to deliver a video. In this paper, we first carefully study the relation between models of different chunks, then we tactfully design a joint training framework along with the Content-aware Feature Modulation (CaFM) layer to compress these models for neural video delivery. {\bf With our method, each video chunk only requires less than $1\% $ of original parameters to be streamed, achieving even better SR performance.} We conduct extensive experiments across various SR backbones, video time length, and scaling factors to demonstrate the advantages of our method. Besides, our method can be also viewed as a new approach of video coding. Our primary experiments achieve better video quality compared with the commercial H.264 and H.265 standard under the same storage cost, showing the great potential of the proposed method. Code is available at:\url{https://github.com/Neural-video-delivery/CaFM-Pytorch-ICCV2021}, Comment: Accepted by ICCV 2021

We propose a compact and effective framework to fuse multimodal features at multiple layers in a single network. The framework consists of two innovative fusion schemes. Firstly, unlike existing multimodal methods that necessitate individual encoders for different modalities, we verify that multimodal features can be learnt within a shared single network by merely maintaining modality-specific batch normalization layers in the encoder, which also enables implicit fusion via joint feature representation learning. Secondly, we propose a bidirectional multi-layer fusion scheme, where multimodal features can be exploited progressively. To take advantage of such scheme, we introduce two asymmetric fusion operations including channel shuffle and pixel shift, which learn different fused features with respect to different fusion directions. These two operations are parameter-free and strengthen the multimodal feature interactions across channels as well as enhance the spatial feature discrimination within channels. We conduct extensive experiments on semantic segmentation and image translation tasks, based on three publicly available datasets covering diverse modalities. Results indicate that our proposed framework is general, compact and is superior to state-of-the-art fusion frameworks., Comment: ACMMM 2020 (2020.3)

Recent years have witnessed rapid advances in learnt video coding. Most algorithms have solely relied on the vector-based motion representation and resampling (e.g., optical flow based bilinear sampling) for exploiting the inter frame redundancy. In spite of the great success of adaptive kernel-based resampling (e.g., adaptive convolutions and deformable convolutions) in video prediction for uncompressed videos, integrating such approaches with rate-distortion optimization for inter frame coding has been less successful. Recognizing that each resampling solution offers unique advantages in regions with different motion and texture characteristics, we propose a hybrid motion compensation (HMC) method that adaptively combines the predictions generated by these two approaches. Specifically, we generate a compound spatiotemporal representation (CSTR) through a recurrent information aggregation (RIA) module using information from the current and multiple past frames. We further design a one-to-many decoder pipeline to generate multiple predictions from the CSTR, including vector-based resampling, adaptive kernel-based resampling, compensation mode selection maps and texture enhancements, and combines them adaptively to achieve more accurate inter prediction. Experiments show that our proposed inter coding system can provide better motion-compensated prediction and is more robust to occlusions and complex motions. Together with jointly trained intra coder and residual coder, the overall learnt hybrid coder yields the state-of-the-art coding efficiency in low-delay scenario, compared to the traditional H.264/AVC and H.265/HEVC, as well as recently published learning-based methods, in terms of both PSNR and MS-SSIM metrics.

The rapidly emerging field of deep learning-based computational pathology has demonstrated promise in developing objective prognostic models from histology whole slide images. However, most prognostic models are either based on histology or genomics alone and do not address how histology and genomics can be integrated to develop joint image-omic prognostic models. Additionally identifying explainable morphological and molecular descriptors from these models that govern such prognosis is of interest. We used multimodal deep learning to integrate gigapixel whole slide pathology images, RNA-seq abundance, copy number variation, and mutation data from 5,720 patients across 14 major cancer types. Our interpretable, weakly-supervised, multimodal deep learning algorithm is able to fuse these heterogeneous modalities for predicting outcomes and discover prognostic features from these modalities that corroborate with poor and favorable outcomes via multimodal interpretability. We compared our model with unimodal deep learning models trained on histology slides and molecular profiles alone, and demonstrate performance increase in risk stratification on 9 out of 14 cancers. In addition, we analyze morphologic and molecular markers responsible for prognostic predictions across all cancer types. All analyzed data, including morphological and molecular correlates of patient prognosis across the 14 cancer types at a disease and patient level are presented in an interactive open-access database (http://pancancer.mahmoodlab.org) to allow for further exploration and prognostic biomarker discovery. To validate that these model explanations are prognostic, we further analyzed high attention morphological regions in WSIs, which indicates that tumor-infiltrating lymphocyte presence corroborates with favorable cancer prognosis on 9 out of 14 cancer types studied., Comment: Demo: http://pancancer.mahmoodlab.org

The expanding adoption of digital pathology has enabled the curation of large repositories of histology whole slide images (WSIs), which contain a wealth of information. Similar pathology image search offers the opportunity to comb through large historical repositories of gigapixel WSIs to identify cases with similar morphological features and can be particularly useful for diagnosing rare diseases, identifying similar cases for predicting prognosis, treatment outcomes, and potential clinical trial success. A critical challenge in developing a WSI search and retrieval system is scalability, which is uniquely challenging given the need to search a growing number of slides that each can consist of billions of pixels and are several gigabytes in size. Such systems are typically slow and retrieval speed often scales with the size of the repository they search through, making their clinical adoption tedious and are not feasible for repositories that are constantly growing. Here we present Fast Image Search for Histopathology (FISH), a histology image search pipeline that is infinitely scalable and achieves constant search speed that is independent of the image database size while being interpretable and without requiring detailed annotations. FISH uses self-supervised deep learning to encode meaningful representations from WSIs and a Van Emde Boas tree for fast search, followed by an uncertainty-based ranking algorithm to retrieve similar WSIs. We evaluated FISH on multiple tasks and datasets with over 22,000 patient cases spanning 56 disease subtypes. We additionally demonstrate that FISH can be used to assist with the diagnosis of rare cancer types where sufficient cases may not be available to train traditional supervised deep models. FISH is available as an easy-to-use, open-source software package (https://github.com/mahmoodlab/FISH).

Cancer prognostication is a challenging task in computational pathology that requires context-aware representations of histology features to adequately infer patient survival. Despite the advancements made in weakly-supervised deep learning, many approaches are not context-aware and are unable to model important morphological feature interactions between cell identities and tissue types that are prognostic for patient survival. In this work, we present Patch-GCN, a context-aware, spatially-resolved patch-based graph convolutional network that hierarchically aggregates instance-level histology features to model local- and global-level topological structures in the tumor microenvironment. We validate Patch-GCN with 4,370 gigapixel WSIs across five different cancer types from the Cancer Genome Atlas (TCGA), and demonstrate that Patch-GCN outperforms all prior weakly-supervised approaches by 3.58-9.46%. Our code and corresponding models are publicly available at https://github.com/mahmoodlab/Patch-GCN., Comment: MICCAI 2021

Frozen sectioning (FS) is the preparation method of choice for microscopic evaluation of tissues during surgical operations. The high speed of the procedure allows pathologists to rapidly assess the key microscopic features, such as tumour margins and malignant status to guide surgical decision-making and minimise disruptions to the course of the operation. However, FS is prone to introducing many misleading artificial structures (histological artefacts), such as nuclear ice crystals, compression, and cutting artefacts, hindering timely and accurate diagnostic judgement of the pathologist. Additional training and prolonged experience is often required to make highly effective and time-critical diagnosis on frozen sections. On the other hand, the gold standard tissue preparation technique of formalin-fixation and paraffin-embedding (FFPE) provides significantly superior image quality, but is a very time-consuming process (12-48 hours), making it unsuitable for intra-operative use. In this paper, we propose an artificial intelligence (AI) method that improves FS image quality by computationally transforming frozen-sectioned whole-slide images (FS-WSIs) into whole-slide FFPE-style images in minutes. AI-FFPE rectifies FS artefacts with the guidance of an attention mechanism that puts a particular emphasis on artefacts while utilising a self-regularization mechanism established between FS input image and synthesized FFPE-style image that preserves clinically relevant features. As a result, AI-FFPE method successfully generates FFPE-style images without significantly extending tissue processing time and consequently improves diagnostic accuracy. We demonstrate the efficacy of AI-FFPE on lung and brain frozen sections using a variety of different qualitative and quantitative metrics including visual Turing tests from 20 board certified pathologists.

We establish automorphisms with closed formulas on quasi-split $\imath$quantum groups of symmetric Kac-Moody type associated to restricted Weyl groups. The proofs are carried out in the framework of $\imath$Hall algebras and reflection functors, thanks to the $\imath$Hall algebra realization of $\imath$quantum groups in our previous work. Several quantum binomial identities arising along the way are established., Comment: 53 pages, minor edits, to appear in Selecta Math

We use semi-derived Ringel-Hall algebras of quivers with loops to realize the whole quantum Borcherds-Bozec algebras and quantum generalized Kac-Moody algebras., Comment: v2, 12 pages, minor edits, accepted by Proc. AMS

let $\widetilde{\bf U}^\imath$ be a quasi-split universal $\imath$quantum group associated to a quantum symmetric pair $(\widetilde{\bf U}, \widetilde{\bf U}^\imath)$ of Kac-Moody type with a diagram involution $\tau$. We establish the Serre-Lusztig relations for $\widetilde{\bf U}^\imath$ associated to a simple root $i$ such that $i \neq \tau i$, complementary to the Serre-Lusztig relations associated to $i=\tau i$ which we obtained earlier. A conjecture on braid group symmetries on $\widetilde{\bf U}^\imath$ associated to $i$ disjoint from $\tau i$ is formulated., Comment: 13 pages, accepted for publication

We propose a novel algorithm for the computation of canonical polyadic decomposition (CPD) of large-scale tensors. The proposed algorithm generalizes the random projection (RAP) technique, which is often used to compute large-scale decompositions, from one single projection to multiple but coupled random projections (CoRAP). The proposed CoRAP technique yields a set of tensors that together admits a coupled CPD (C-CPD) and a C-CPD algorithm is then used to jointly decompose these tensors. The results of C-CPD are finally fused to obtain factor matrices of the original large-scale data tensor. As more data samples are jointly exploited via C-CPD, the proposed CoRAP based CPD is more accurate than RAP based CPD. Experiments are provided to illustrate the performance of the proposed approach.

We show that the $\imath$Hall algebra of the Jordan quiver is a polynomial ring in infinitely many generators and obtain transition relations among several generating sets. We establish a ring isomorphism from this $\imath$Hall algebra to the ring of symmetric functions in two parameters $t, \theta$, which maps the $\imath$Hall basis to a class of (modified) inhomogeneous Hall-Littlewood ($\imath$HL) functions. The (modified) $\imath$HL functions admit a formulation via raising and lowering operators. We formulate and prove Pieri rules for (modified) $\imath$HL functions. The modified $\imath$HL functions specialize at $\theta=0$ to the modified HL functions; they specialize at $\theta=1$ to the deformed universal characters of type C, which further specialize at $(t=0, \theta =1)$ to the universal characters of type C., Comment: v2,41 pages,references added

This paper reviews the first NTIRE challenge on quality enhancement of compressed video, with a focus on the proposed methods and results. In this challenge, the new Large-scale Diverse Video (LDV) dataset is employed. The challenge has three tracks. Tracks 1 and 2 aim at enhancing the videos compressed by HEVC at a fixed QP, while Track 3 is designed for enhancing the videos compressed by x265 at a fixed bit-rate. Besides, the quality enhancement of Tracks 1 and 3 targets at improving the fidelity (PSNR), and Track 2 targets at enhancing the perceptual quality. The three tracks totally attract 482 registrations. In the test phase, 12 teams, 8 teams and 11 teams submitted the final results of Tracks 1, 2 and 3, respectively. The proposed methods and solutions gauge the state-of-the-art of video quality enhancement. The homepage of the challenge: https://github.com/RenYang-home/NTIRE21_VEnh, Comment: Corrected the MOS values in Table 2, and corrected some minor typos

We study the electronic structure of a double-twisted few layer graphite (DTFLG), which consists of three few layer graphite (FLG), i.e. ABA-stacked graphene multilayer, stacked with two twist angles. We consider two categories of DTFLG, alternately twisted case and chirally twisted one, according to the rotation direction of the two twist angles. We show that, once the middle FLG of DTFLG is not thinner than trilayer, both kinds of DTFLG can remarkably host two pairs of degenerate moir\'{e} flat bands (MFBs) at $E_f$, twice that of the magic angle twisted bilayer graphene (TBG). The doubled MFBs of DTFLG lead to doubled DOS at $E_f$, which implies much stronger correlation effects than the TBG. The degeneracy of MFBs can be lifted by a perpendicular electric field, and the isolated MFBs have nonzero valley Chern number. We also reveal the peculiar wave function patterns of the MFBs in the DTFLG. Our results establish a new family of moir\'{e} systems that have the much larger DOS at $E_f$, and thus possible much stronger correlation effects., Comment: 6 pages, 4 figures

Hermitian theories play a major role in understanding the physics of most phenomena. It has been found only in the past decade that non-Hermiticity enables unprecedented effects such as exceptional points, spectral singularities and bulk Fermi arcs. Recent studies further show that non-Hermiticity can fundamentally change the topological band theory, leading to the non-Hermitian band topology and non-Hermitian skin effect, as confirmed in one-dimensional (1D) systems. However, in higher dimensions, these non-Hermitian effects remain unexplored in experiments. Here, we demonstrate the spin-polarized, higher-order non-Hermitian skin effect in two-dimensional (2D) acoustic metamaterials. Using a lattice of coupled whisper-gallery acoustic resonators, we realize a spinful 2D higher-order topological insulator (HOTI) where the spin-up and spin-down states are emulated by the anti-clockwise and clockwise modes, respectively. We find that the non-Hermiticity drives wave localizations toward opposite edge boundaries depending on the spin polarizations. More interestingly, for finite systems with both edge and corner boundaries, the higher-order non-Hermitian skin effect leads to wave localizations toward two corner boundaries for the bulk, edge and corner states in a spin-dependent manner. We further show that such a non-Hermitian skin effect enables rich wave manipulation through the loss configuration in each unit-cell. The reported spin-dependent, higher-order non-Hermitian skin effect reveals the interplay between higher-order topology and non-Hermiticity, which is further enriched by the spin degrees of freedom. This unveils a new horizon in the study of non-Hermitian physics and the design of non-Hermitian metamaterials.