Search

Patient prognosis in hypopharyngeal carcinoma remains difficult to predict, necessitating new, readily available biomarkers.Platelet-lymphocyte ratio (PLR)'s effects on recurrence-free survival (RFS) and overall survival (OS) were evaluated in individuals undergoing radical resection for advanced hypopharyngeal squamous cell carcinoma (HSCC).A total of 89 patients were retrospectively assessed. PLR, and derived neutrophil-lymphocyte (dNLR) and neutrophil-lymphocyte (NLR) ratios were determined based on complete blood count. Then, the prognostic values of PLR, dNLR and NLR were assessed by univariate and multivariate Cox regression analyses adjusted for disease-specific prognostic factors. Endpoints of interest were RFS and OS.The optimal cutoff of PLR was 98.815, based on which individuals were categorized into the high- (PLR ≥98.815) and low- (PLR98.815) PLR groups. High PLR (Elevated PLR has associations with increased risk of recurrence and reduced survival in advanced HSCC cases undergoing radical resection. High presurgical PLR may independently predict RFS. Therefore, further multi-institutional prospective studies are needed to better characterize the role of pre-operative blood PLR as prognostic factors in HSCC.

Detecting segments of interest from videos is a common problem for many applications. And yet it is a challenging problem as it often requires not only knowledge of individual target segments, but also contextual understanding of the entire video and the relationships between the target segments. To address this problem, we propose the Sequence-to-Segments Network (S2N), a novel and general end-to-end sequential encoder-decoder architecture. S2N first encodes the input video into a sequence of hidden states that capture information progressively, as it appears in the video. It then employs the Segment Detection Unit (SDU), a novel decoding architecture, that sequentially detects segments. At each decoding step, the SDU integrates the decoder state and encoder hidden states to detect a target segment. During training, we address the problem of finding the best assignment of predicted segments to ground truth using the Hungarian Matching Algorithm with Lexicographic Cost. Additionally we propose to use the squared Earth Mover's Distance to optimize the localization errors of the segments. We show the state-of-the-art performance of S2N across numerous tasks, including video highlighting, video summarization, and human action proposal generation.

Di-methylated lysine 9 of histone H3 (H3K9me2), regulated by histone methyltransferases, is involved in the epigenetic regulation of tumor-associated genes. The present study aimed to evaluate whether the H3K9me2 methylation level is associated with the expression level of euchromatic histone lysine methyltransferase 2 (EHMT2) in the prognosis of gastric cancer (GC). H3K9me2 methylation level and EHMT2 expression level were detected by immunohistochemistry in 118 GC samples. The clinicopathological significance of H3K9me2 and EHMT2 in patients with GC was assessed using a paired Student's t-test, χ2 test, Kaplan-Meier analysis with a log-rank test and Cox's proportional hazard analysis. Strong positive immunostaining of H3K9me2 and EHMT2 was observed in cancerous tissues compared with adjacent non-cancerous tissues. Positive immunostaining of EHMT2 and H3K9me2 was associated with lymph node metastasis, pathological grade and tumor-node-metastasis stage. H3K9me2 expression level was increased in tumor tissue and associated with worse specific-disease and disease-free survival time. In addition, EHMT2 protein expression levels were associated with the expression levels of H3K9me2. Low expression levels of H3K9me2 and EHMT2 predicted a better prognosis of patients with GC. The survival time of patients with a high expression of H3K9me2 and/or EHMT2 was significantly shorter compared with that of the patients with a low expression of H3K9me2 and/or EHMT2. In conclusion, an overexpression pattern of H3K9me2 and/or EHMT2 may be associated with clinicopathological features of GC and may be predictor markers of progression and prognosis in patients with GC, in addition to putative therapeutic targets.

Power demand side management (DSM) regulations stipulate that electric power grid enterprises of cities and provinces must complete their target of power saving through efficient power plant (EPP) programs. This investigation studies power DSM and establishes a supervision and evaluation system for the EPP program. The proposed system can be used to evaluate the performance of an EPP program such as electricity saving and investment benefit. A model of EPP is established. A cuckoo search algorithm is applied for the model optimization and its result gives an optimal decision scheme. Finally, a fuzzy comprehensive evaluation method is used to assess an EPP program. The results show that the established supervision and evaluation system and the model of an EPP program are effective and applicable for power grid enterprises.

Aim: To investigate hypoxia-inducible factor-1α (HIF-1α) and histone methylation markers as potential indicators of prognosis in esophageal squamous cell carcinoma (ESCC). Patients & methods: The prognostic value of HIF-1α and histone methylation markers levels was analyzed using Kaplan–Meier survival analysis. Results: HIF-1α protein expression was higher in ESCC tumors than in paracancerous tissues. Histone H3 Lys9 trimethylation (H3K9me3), histone H3 Lys27 trimethylation (H3K27me3), histone H3 Lys4 trimethylation (H3K4me3) and histone H4 Lys20 trimethylation (H4K20me3) were significantly upregulated in ESCC tissues. HIF-1α was only positively correlated with H3K9me3 and H3K4me3 expression. Kaplan–Meier analysis indicated that H3K9me3, H3K27me3, H4K20me3 and histone H3 Lys36 trimethylation (H3K36me3) were independent indicators of prognosis for ESCC. Conclusion: This study identified a pattern of epigenetic methylation markers and HIF-1α expression in ESCC, and their combined evaluation might improve survival prediction for ESCC patients.

Background: Oxaliplatin resistance is a major challenge for treatment of metastatic colorectal cancer (mCRC). Many molecular targeted drugs for refractory CRC have been developed to solve colorectal cancer drug resistance, but their effectiveness and roles in the progression of CRC and oxaliplatin- resistance still not clear.Methods: PDOs derived from CRC patients were constructed to conduct the sensitivity assays of oxaliplatin in vitro. Oxaliplatin resistant PDOs were selected and treated under the combined treatment of ML264(a KLF5 inhibitor) and oxaliplatin to determine the effects of KLF5 inhibition on apoptosis. Using CRC cell lines to investigate downstream mechanisms and xenograft models to confirm whether ML264 can restore oxaliplatin sensitivity of CRC cells in vivo.Results: We successfully constructed CRC PDOs and conducted the sensitivity test of oxaliplatin in PDOs from different patients. We found that ML264 restores oxaliplatin sensitivity in CRC PDOs by restoring the apoptotic response, and this effect was achieved by inhibiting the KLF5/Bcl-2/caspase3 signal pathway. Chromatin immunoprecipitation (ChIP) and luciferase reporter assay verified that KLF5 promoted the transcription of Bcl-2 in CRC cells. KLF5 inhibition also overcomed oxaliplatin resistance in xenograft tumors.Conclusions: Our study demonstrated that ML264 can restores oxaliplatin sensitivity in CRC PDOs by restoring the apoptotic response. KLF5 might be a potential therapeutic target for CRC resistant to oxaliplatin. PDOs have strong potential in evaluating inhibitors and drug combinations therapy in a preclinical environment.

Objectives: Misuse of antibiotics, especially in low-and-middle-income countries is a major contributor to antimicrobial resistance. Medical students are antibiotic consumers and prescribers in the...

Recently, significant progress has been made in single-view depth estimation thanks to increasingly large and diverse depth datasets. However, these datasets are largely limited to specific application domains (e.g. indoor, autonomous driving) or static in-the-wild scenes due to hardware constraints or technical limitations of 3D reconstruction. In this paper, we introduce the first depth dataset DynOcc consisting of dynamic in-the-wild scenes. Our approach leverages the occlusion cues in these dynamic scenes to infer depth relationships between points of selected video frames. To achieve accurate occlusion detection and depth order estimation, we employ a novel occlusion boundary detection, filtering and thinning scheme followed by a robust foreground/background classification method. In total our DynOcc dataset contains 22M depth pairs out of 91K frames from a diverse set of videos. Using our dataset we achieved state-of-the-art results measured in weighted human disagreement rate (WHDR). We also show that the inferred depth maps trained with DynOcc can preserve sharper depth boundaries., 3DV 2020

Interactive fashion image manipulation, which enables users to edit images with sketches and color strokes, is an interesting research problem with great application value. Existing works often treat it as a general inpainting task and do not fully leverage the semantic structural information in fashion images. Moreover, they directly utilize conventional convolution and normalization layers to restore the incomplete image, which tends to wash away the sketch and color information. In this paper, we propose a novel Fashion Editing Generative Adversarial Network (FE-GAN), which is capable of manipulating fashion images by free-form sketches and sparse color strokes. FE-GAN consists of two modules: 1) a free-form parsing network that learns to control the human parsing generation by manipulating sketch and color; 2) a parsing-aware inpainting network that renders detailed textures with semantic guidance from the human parsing map. A new attention normalization layer is further applied at multiple scales in the decoder of the inpainting network to enhance the quality of the synthesized image. Extensive experiments on high-resolution fashion image datasets demonstrate that the proposed FE-GAN significantly outperforms the state-of-the-art methods on fashion image manipulation.

People usually take short videos to record meaningful moments in their lives. However, selecting the most representative frame, which not only has high image visual quality but also captures video content, from a short video to share or keep is a time-consuming process for one may need to manually go through all the frames in a video to make a decision. In this paper, we introduce the problem of the best frame selection in a short video and aim to solve it automatically. Towards this end, we collect and will release a diverse large-scale short video dataset that includes 11, 000 videos shoot in our daily life. All videos are assumed to be short (e.g., a few seconds) and each video has human-annotated of the best frame. Then we introduce a deep convolutional neural network (CNN) based approach with ranking objective to automatically pick the best frame from frame sequences extracted via short videos. Additionally, we propose new evaluation metrics, especially for the best frame selection. In experiments, we show our approach outperforms various other methods significantly.

Purpose To evaluate the diagnostic value of 99mTc-HYNIC-TOC SPECT/CT in the detection of primary small cell neuroendocrine carcinomas of the nasal cavity and paranasal sinuses. Methods From September 2005 to November 2015, 19 patients (mean age 53.9 years, range 37–77; male 6, female 13) with sinonasal small cell neuroendocrine carcinoma (SNEC) underwent whole-body 99mTc-HYNIC-TOC SPECT/CT scan. Results All 19 primary lesions were detected by 99mTc-HYNIC-TOC SPECT/CT. A total of 212 metastatic lesions were identified by organ-based analysis, and the sensitivity and specificity of 99mTc-HYNIC-TOC SPECT/CT were 90.5% and 92.3%, respectively. The application of 99mTc-HYNIC-TOC SPECT/CT resulted in the detection of a higher number of lesions in 11 patients (57.9%) and in the successful alteration of the treatment in 7 patients (36.8%) compared with the morphological imaging methods. Conclusion 99mTc-HYNIC-TOC scintigraphy is an effective method for the detection of the sinonasal SNEC.

We aim to model the top-down attention of a convolutional neural network (CNN) classifier for generating task-specific attention maps. Inspired by a top-down human visual attention model, we propose a new backpropagation scheme, called Excitation Backprop, to pass along top-down signals downwards in the network hierarchy via a probabilistic Winner-Take-All process. Furthermore, we introduce the concept of contrastive attention to make the top-down attention maps more discriminative. We show a theoretic connection between the proposed contrastive attention formulation and the Class Activation Map computation. Efficient implementation of Excitation Backprop for common neural network layers is also presented. In experiments, we visualize the evidence of a model’s classification decision by computing the proposed top-down attention maps. For quantitative evaluation, we report the accuracy of our method in weakly supervised localization tasks on the MS COCO, PASCAL VOC07 and ImageNet datasets. The usefulness of our method is further validated in the text-to-region association task. On the Flickr30k Entities dataset, we achieve promising performance in phrase localization by leveraging the top-down attention of a CNN model that has been trained on weakly labeled web images. Finally, we demonstrate applications of our method in model interpretation and data annotation assistance for facial expression analysis and medical imaging tasks.

We study the problem of salient object subitizing, i.e. predicting the existence and the number of salient objects in an image using holistic cues. This task is inspired by the ability of people to quickly and accurately identify the number of items within the subitizing range (1---4). To this end, we present a salient object subitizing image dataset of about 14 K everyday images which are annotated using an online crowdsourcing marketplace. We show that using an end-to-end trained convolutional neural network (CNN) model, we achieve prediction accuracy comparable to human performance in identifying images with zero or one salient object. For images with multiple salient objects, our model also provides significantly better than chance performance without requiring any localization process. Moreover, we propose a method to improve the training of the CNN subitizing model by leveraging synthetic images. In experiments, we demonstrate the accuracy and generalizability of our CNN subitizing model and its applications in salient object detection and image retrieval.

In this paper, we tackle the problem of human motion transfer, where we synthesize novel motion video for a target person that imitates the movement from a reference video. It is a video-to-video translation task in which the estimated poses are used to bridge two domains. Despite substantial progress on the topic, there exist several problems with the previous methods. First, there is a domain gap between training and testing pose sequences–the model is tested on poses it has not seen during training, such as difficult dancing moves. Furthermore, pose detection errors are inevitable, making the job of the generator harder. Finally, generating realistic pixels from sparse poses is challenging in a single step. To address these challenges, we introduce a novel pose-to-video translation framework for generating high-quality videos that are temporally coherent even for in-the-wild pose sequences unseen during training. We propose a pose augmentation method to minimize the training-test gap, a unified paired and unpaired learning strategy to improve the robustness to detection errors, and two-stage network architecture to achieve superior texture quality. To further boost research on the topic, we build two human motion datasets. Finally, we show the superiority of our approach over the state-of-the-art studies through extensive experiments and evaluations on different datasets.

Despite the great progress in human motion prediction, it remains a challenging task due to the complicated structural dynamics of human behaviors. In this paper, we address this problem in three aspects. First, to capture the long-range spatial correlations and temporal dependencies, we apply a transformer-based architecture with the global attention mechanism. Speci cally, we feed the network with the sequential joints encoded with the temporal information for spatial and temporal explorations. Second, to further exploit the inherent kinematic chains for better 3D structures, we apply a progressive-decoding strategy, which performs in a central-to-peripheral extension according to the structural connectivity. Last, in order to incorporate a general motion space for high-quality prediction, we build a memory-based dictionary, which aims to preserve the global motion patterns in training data to guide the predictions.We evaluate the proposed method on two challenging benchmark datasets (Human3.6M and CMU-Mocap). Experimental results show our superior performance compared with the state-of-the-art approaches. National Research Foundation (NRF) Accepted version This research / project is supported by the National Research Foundation, Singapore under its International Research Centres in Singapore Funding Initiative. Any opinions, fndings and conclusions or recommendations expressed in this material are those of the author(s) and do not reflect the views of National Research Foundation, Singapore. This research is partially supported by the Monash FIT Start-up Grant, start-up funds from University at Buffalo and SUTD project PIE-SGP-Al-2020-02.

While single-image object detectors can be naively applied to videos in a frame-by-frame fashion, the prediction is often temporally inconsistent. Moreover, the computation can be redundant since neighboring frames are inherently similar to each other. In this work we propose to improve video object detection via temporal aggregation. Specifically, a detection model is applied on sparse keyframes to handle new objects, occlusions, and rapid motions. We then use real-time trackers to exploit temporal cues and track the detected objects in the remaining frames, which enhances efficiency and temporal coherence. Object status at the bounding-box level is propagated across frames and updated by our aggregation modules. For keyframe scheduling, we propose adaptive policies using reinforcement learning and simple heuristics. The proposed framework achieves the state-of-the-art performance on the Imagenet VID 2015 dataset while running real-time on CPU. Extensive experiments are done to show the effectiveness of our training strategies and justify the model designs.

Deep CNNs have been pushing the frontier of visual recognition over past years. Besides recognition accuracy, strong demands in understanding deep CNNs in the research community motivate developments of tools to dissect pre-trained models to visualize how they make predictions. Recent works further push the interpretability in the network learning stage to learn more meaningful representations. In this work, focusing on a specific area of visual recognition, we report our efforts towards interpretable face recognition. We propose a spatial activation diversity loss to learn more structured face representations. By leveraging the structure, we further design a feature activation diversity loss to push the interpretable representations to be discriminative and robust to occlusions. We demonstrate on three face recognition benchmarks that our proposed method is able to improve face recognition accuracy with easily interpretable face representations., 10 pages, 9 figures, 6 tables, To appear in ICCV 2019 as an oral paper

We present a generative image inpainting system to complete images with free-form mask and guidance. The system is based on gated convolutions learned from millions of images without additional labelling efforts. The proposed gated convolution solves the issue of vanilla convolution that treats all input pixels as valid ones, generalizes partial convolution by providing a learnable dynamic feature selection mechanism for each channel at each spatial location across all layers. Moreover, as free-form masks may appear anywhere in images with any shape, global and local GANs designed for a single rectangular mask are not applicable. Thus, we also present a patch-based GAN loss, named SN-PatchGAN, by applying spectral-normalized discriminator on dense image patches. SN-PatchGAN is simple in formulation, fast and stable in training. Results on automatic image inpainting and user-guided extension demonstrate that our system generates higher-quality and more flexible results than previous methods. Our system helps user quickly remove distracting objects, modify image layouts, clear watermarks and edit faces. Code, demo and models are available at: https://github.com/JiahuiYu/generative_inpainting, Accepted in ICCV 2019 Oral; open sourced; interactive demo available: http://jiahuiyu.com/deepfill/

Beyond current image-based virtual try-on systems that have attracted increasing attention, we move a step forward to developing a video virtual try-on system that precisely transfers clothes onto the person and generates visually realistic videos conditioned on arbitrary poses. Besides the challenges in image-based virtual try-on (e.g., clothes fidelity, image synthesis), video virtual try-on further requires spatiotemporal consistency. Directly adopting existing image-based approaches often fails to generate coherent video with natural and realistic textures. In this work, we propose Flow-navigated Warping Generative Adversarial Network (FW-GAN), a novel framework that learns to synthesize the video of virtual try-on based on a person image, the desired clothes image, and a series of target poses. FW-GAN aims to synthesize the coherent and natural video while manipulating the pose and clothes. It consists of: (i) a flow-guided fusion module that warps the past frames to assist synthesis, which is also adopted in the discriminator to help enhance the coherence and quality of the synthesized video; (ii) a warping net that is designed to warp clothes image for the refinement of clothes textures; (iii) a parsing constraint loss that alleviates the problem caused by the misalignment of segmentation maps from images with different poses and various clothes. Experiments on our newly collected dataset show that FW-GAN can synthesize high-quality video of virtual try-on and significantly outperforms other methods both qualitatively and quantitatively.

Virtual try-on systems under arbitrary human poses have significant application potential, yet also raise extensive challenges, such as self-occlusions, heavy misalignment among different poses, and complex clothes textures. Existing virtual try-on methods can only transfer clothes given a fixed human pose, and still show unsatisfactory performances, often failing to preserve person identity or texture details, and with limited pose diversity. This paper makes the first attempt towards a multi-pose guided virtual try-on system, which enables clothes to transfer onto a person with diverse poses. Given an input person image, a desired clothes image, and a desired pose, the proposed Multi-pose Guided Virtual Try-On Network (MG-VTON) generates a new person image after fitting the desired clothes into the person and manipulating the pose. MG-VTON is constructed with three stages: 1) a conditional human parsing network is proposed that matches both the desired pose and the desired clothes shape; 2) a deep Warping Generative Adversarial Network (Warp-GAN) that warps the desired clothes appearance into the synthesized human parsing map and alleviates the misalignment problem between the input human pose and the desired one; 3) a refinement render network recovers the texture details of clothes and removes artifacts, based on multi-pose composition masks. Extensive experiments on commonly-used datasets and our newly-collected largest virtual try-on benchmark demonstrate that our MG-VTON significantly outperforms all state-of-the-art methods both qualitatively and quantitatively, showing promising virtual try-on performances.

Resistance to molecular targeted therapies is a significant challenge for advanced colorectal cancer (CRC). Understanding the underlying mechanisms and developing effective strategies against regorafenib resistance are highly desired in the clinic. Here, we screened the expression of chemokine receptors and identified CC chemokine receptor 2 (CCR2) as a top upregulated gene in regorafenib-resistant cells. CCR2 silencing alleviated drug tolerance in regorafenib-resistant cells, while overexpression of CCR2 enhanced CRC cells resistance to regorafenib. Moreover, CCR2-mediated regorafenib tolerance was demonstrated to be associated with AKT/GSK3β-regulated β-catenin stabilization. In turn, β-catenin modulation is sufficient to trigger the transcriptional activation of CCR2 expression. Clinically, high-CCR2 expression was correlated to shorter overall survival and disease-free survival of patients. A positive correlation between CCR2 and nuclear β-catenin expression was observed in a cohort of CRC tissues. Altogether, these findings suggest β-catenin and CCR2 are part of a positive-feedback loop, which sustains a high CCR2 expression level, conferring CRC cells resistance to regorafenib. Thus, targeting CCR2 may be a useful therapeutic strategy to alleviate regorafenib tolerance to increase the efficacy of CRC treatments.

Deep learning-based methods have achieved remarkable success in image restoration and enhancement, but are they still competitive when there is a lack of paired training data? As one such example, this paper explores the low-light image enhancement problem, where in practice it is extremely challenging to simultaneously take a low-light and a normal-light photo of the same visual scene. We propose a highly effective unsupervised generative adversarial network, dubbed EnlightenGAN , that can be trained without low/normal-light image pairs, yet proves to generalize very well on various real-world test images. Instead of supervising the learning using ground truth data, we propose to regularize the unpaired training using the information extracted from the input itself, and benchmark a series of innovations for the low-light image enhancement problem, including a global-local discriminator structure, a self-regularized perceptual loss fusion, and the attention mechanism. Through extensive experiments, our proposed approach outperforms recent methods under a variety of metrics in terms of visual quality and subjective user study. Thanks to the great flexibility brought by unpaired training, EnlightenGAN is demonstrated to be easily adaptable to enhancing real-world images from various domains. Our codes and pre-trained models are available at: https://github.com/VITA-Group/EnlightenGAN .

Deep neural network-based methods were proposed for face attribute manipulation. There still exist, however, two major issues, i.e., insufficient visual quality (or resolution) of the results and lack of user control. They limit the applicability of existing methods since users may have different editing preference on facial attributes. In this paper, we address these issues by proposing a semantic component model. The model decomposes a facial attribute into multiple semantic components, each corresponds to a specific face region. This not only allows for user control of edit strength on different parts based on their preference, but also makes it effective to remove unwanted edit effect. Further, each semantic component is composed of two fundamental elements, which determine the edit effect and region respectively. This property provides fine interactive control. As shown in experiments, our model not only produces high-quality results, but also allows effective user interaction.

Prior highly-tuned human parsing models tend to fit towards each dataset in a specific domain or with discrepant label granularity, and can hardly be adapted to other human parsing tasks without extensive re-training. In this paper, we aim to learn a single universal human parsing model that can tackle all kinds of human parsing needs by unifying label annotations from different domains or at various levels of granularity. This poses many fundamental learning challenges, e.g. discovering underlying semantic structures among different label granularity, performing proper transfer learning across different image domains, and identifying and utilizing label redundancies across related tasks. To address these challenges, we propose a new universal human parsing agent, named "Graphonomy", which incorporates hierarchical graph transfer learning upon the conventional parsing network to encode the underlying label semantic structures and propagate relevant semantic information. In particular, Graphonomy first learns and propagates compact high-level graph representation among the labels within one dataset via Intra-Graph Reasoning, and then transfers semantic information across multiple datasets via Inter-Graph Transfer. Various graph transfer dependencies (\eg, similarity, linguistic knowledge) between different datasets are analyzed and encoded to enhance graph transfer capability. By distilling universal semantic graph representation to each specific task, Graphonomy is able to predict all levels of parsing labels in one system without piling up the complexity. Experimental results show Graphonomy effectively achieves the state-of-the-art results on three human parsing benchmarks as well as advantageous universal human parsing performance., Accepted to CVPR 2019. The Code is available at https://github.com/Gaoyiminggithub/Graphonomy

Aims To examine the efficacy of angiography derived endothelial shear stress (ESS) in predicting atherosclerotic disease progression. Methods and results Thirty-five patients admitted with ST-elevation myocardial infarction that had three-vessel intravascular ultrasound (IVUS) immediately after revascularization and at 13 months follow-up were included. Three dimensional (3D) reconstruction of the non-culprit vessels were performed using (i) quantitative coronary angiography (QCA) and (ii) methodology involving fusion of IVUS and biplane angiography. In both models, blood flow simulation was performed and the minimum predominant ESS was estimated in 3 mm segments. Baseline plaque characteristics and ESS were used to identify predictors of atherosclerotic disease progression defied as plaque area increase and lumen reduction at follow-up. Fifty-four vessels were included in the final analysis. A moderate correlation was noted between ESS estimated in the 3D QCA and the IVUS-derived models (r = 0.588, P

In this work, we address the human parsing task with a novel Contextualized Convolutional Neural Network (Co-CNN) architecture, which well integrates the cross-layer context, global image-level context, semantic edge context, within-super-pixel context and cross-super-pixel neighborhood context into a unified network. Given an input human image, Co-CNN produces the pixel-wise categorization in an end-to-end way. First, the cross-layer context is captured by our basic local-to-global-to-local structure, which hierarchically combines the global semantic information and the local fine details across different convolutional layers. Second, the global image-level label prediction is used as an auxiliary objective in the intermediate layer of the Co-CNN, and its outputs are further used for guiding the feature learning in subsequent convolutional layers to leverage the global image-level context. Third, semantic edge context is further incorporated into Co-CNN, where the high-level semantic boundaries are leveraged to guide pixel-wise labeling. Finally, to further utilize the local super-pixel contexts, the within-super-pixel smoothing and cross-super-pixel neighbourhood voting are formulated as natural sub-components of the Co-CNN to achieve the local label consistency in both training and testing process. Comprehensive evaluations on two public datasets well demonstrate the significant superiority of our Co-CNN over other state-of-the-arts for human parsing. In particular, the F-1 score on the large dataset [1] reaches $81.72\,\text{percent}$ by Co-CNN, significantly higher than $62.81\,\text{percent}$ and $64.38\,\text{percent}$ by the state-of-the-art algorithms, M-CNN [2] and ATR [1] , respectively. By utilizing our newly collected large dataset for training, our Co-CNN can achieve $85.36\,\text{percent}$ in F-1 score.

// Hao Feng 1 , 2 , Tobias S. Schiergens 2 , Zhi-hai Mao 1 , Jingkun Zhao 1 , 2 , Xiaohui Shen 1 , Ai-Guo Lu 1 and Wolfgang E. Thasler 3 1 Department of Digestive Surgery, Ruijin Hospital, Shanghai Jiao-tong University School of Medicine, Shanghai, China 2 Department of General, Visceral, Transplant, and Vascular Surgery, University Hospital of Ludwig Maximilian University of Munich, Munich, Germany 3 Department of General and Visceral Surgery, Red Cross Hospital of Munich, Munich, Germany Correspondence to: Ai-guo Lu, email: // Hao Feng, email: // Keywords : laparoscopy; rectal cancer; operative risk; Cr-POSSUM system; complications Received : July 20, 2016 Accepted : October 17, 2016 Published : December 09, 2016 Abstract BACKGROUND: It is still controversial about the treatment strategy for rectal cancer patients with elevated operative risk and elder rectal cancer patients. METHODS: This study presented a retrospective single center experience in rectal cancer proctectomy for high operative risk patients. High operative risk patient was defined as Cr-POSSUM > 5% combined with associated risk factors. 220 in 1477 consecutive patients met the inclusion criteria. RESULTS: 132 patients were selected (66:66) after propensity score matching. The total complication rate between conventional open rectal resection (71 %) and laparoscopic surgery (41%) was significantly different ( p = 0.0005). There is a significantly positive correlation between open surgery and advanced Dindo Classification ( p = 0.02). Cr-POSSUM is positively correlated with Dindo Classification ( p = 0.01). There was no significant difference in survival rate among stage I~II, different age groups or different Cr-POSSUM score sub-groups. However, stage III-IV tumor patients in laparoscopic group experienced improved overall survival rate. ( p < 0.0001). For patients with preoperative pulmonary or renal disease, patients in laparoscopic group also had better long term prognosis ( p = 0.03, p = 0.049). CONCLUSIONS: The results demonstrate the potential advantages of laparoscopic rectal cancer resection for high operative risk patients, especially for the patients with preoperative respiratory or renal disease and stage III cancer.

Skies are common backgrounds in photos but are often less interesting due to the time of photographing. Professional photographers correct this by using sophisticated tools with painstaking efforts that are beyond the command of ordinary users. In this work, we propose an automatic background replacement algorithm that can generate realistic, artifact-free images with a diverse styles of skies. The key idea of our algorithm is to utilize visual semantics to guide the entire process including sky segmentation, search and replacement. First we train a deep convolutional neural network for semantic scene parsing, which is used as visual prior to segment sky regions in a coarse-to-fine manner. Second, in order to find proper skies for replacement, we propose a data-driven sky search scheme based on semantic layout of the input image. Finally, to re-compose the stylized sky with the original foreground naturally, an appearance transfer method is developed to match statistics locally and semantically. We show that the proposed algorithm can automatically generate a set of visually pleasing results. In addition, we demonstrate the effectiveness of the proposed algorithm with extensive user studies.

Chemokines and chemokine receptors are causally involved in the metastasis of human malignancies. As a crucial chemokine receptor for mediating immune homeostasis, however, the role of CCR4 in colorectal cancer (CRC) remains unknown. In this study, we found that high expression of CCR4 in CRC tissues was correlated with shorter overall survival and disease free survival. In vitro and in vivo experiments revealed that silencing CCR4 attenuated the invasion and metastasis of CRC cells, whereas ectopic overexpression of CCR4 contributed to the forced metastasis of these cells. We further demonstrated that matrix metalloproteinase 13 (MMP13) played an important role in CCR4-mediated cancer cell invasion, which is up-regulated by ERK/NF-κB signaling. Positive correlation between CCR4 and MMP13 expression was also observed in CRC tissues. Moreover, our investigations showed that the level of CCR4 could be induced by TNF-α dependent of NF-κB activation in CRC cells. CCR4 might be implicated in TNF-α-regulated cancer cells metastasis. Combination of CCR4 and TNF-α is a more powerful prognostic marker for CRC patients. These findings suggest that CCR4 facilitates metastasis through ERK/NF-κB/MMP13 signaling and acts as a downstream target of TNF-α. CCR4 inhibition may be a promising therapeutic option for suppressing CRC metastasis.

In order to satisfy the high accuracy and high security requirements of time synchronization in smart grid, this paper has conducted a research on high-precision time synchronization control algorithm based on IEEE 1588 protocol. A slave clock model is built considering both phase offset and frequency drift. A novel approach using proportional–integral–derivative (PID) controller in cooperation with the wolf colony algorithm (WCA) is proposed to eliminate the offset of the slave clock with respect to the master clock. As a new swarm-intelligence algorithm, WCA is introduced for self-tuning the optimal parameters of the PID controller, which simulates the intelligent predatory behaviors and prey distribution rule of the wolf colony. The WCA-based PID control approach is compared with the method based on standard Particle Swarm Optimization (PSO) and Back-Propagation (BP) neural network. The experimental results show that the proposed WCA-based PID control approach in time synchronization for smart grid has higher precision, can adaptively adjust compensation and keep the master–slave clock synchronization effectively.

In this paper, we propose a new method for detecting primary objects in unconstrained videos in a completely automatic setting. Here, we define the primary object in a video as the object that presents saliently in most of the frames. Unlike previous works considering only local saliency detection or common pattern discovery, the proposed method integrates the local visual/motion saliency extracted from each frame, global appearance consistency throughout the video, and spatiotemporal smoothness constraint on object trajectories. We first identify a temporal coherent salient region throughout the whole video, and then explicitly learn a global appearance model to distinguish the primary object against the background. In order to obtain high-quality saliency estimations from both appearance and motion cues, we propose a novel self-adaptive saliency map fusion method by learning the reliability of saliency maps from labeled data. As a whole, our method can robustly localize and track primary objects in diverse video content, and handle the challenges such as fast object and camera motion, large scale and appearance variation, background clutter, and pose deformation. Moreover, compared with some existing approaches that assume the object is present in all the frames, our approach can naturally handle the case where the object is present only in part of the frames, e.g., the object enters the scene in the middle of the video or leaves the scene before the video ends. We also propose a new video data set containing 51 videos for primary object detection with per-frame ground-truth labeling. Quantitative experiments on several challenging video data sets demonstrate the superiority of our method compared with the recent state of the arts.

Instance-level object segmentation is an important yet under-explored task. The few existing studies are almost all based on region proposal methods to extract candidate segments and then utilize object classification to produce final results. Nonetheless, generating accurate region proposals itself is quite challenging. In this work, we propose a Proposal-Free Network (PFN ) to address the instance-level object segmentation problem, which outputs the instance numbers of different categories and the pixel-level information on 1) the coordinates of the instance bounding box each pixel belongs to, and 2) the confidences of different categories for each pixel, based on pixel-to-pixel deep convolutional neural network. All the outputs together, by using any off-the-shelf clustering method for simple post-processing, can naturally generate the ultimate instance-level object segmentation results. The whole PFN can be easily trained in an end-to-end way without the requirement of a proposal generation stage. Extensive evaluations on the challenging PASCAL VOC 2012 semantic segmentation benchmark demonstrate that the proposed PFN solution well beats the state-of-the-arts for instance-level object segmentation. In particular, the $AP^r$ over 20 classes at 0.5 IoU reaches 58.7% by PFN, significantly higher than 43.8% and 46.3% by the state-of-the-art algorithms, SDS [9] and [16], respectively., 14 pages

Finding views with good photo composition is a challenging task for machine learning methods. A key difficulty is the lack of well annotated large scale datasets. Most existing datasets only provide a limited number of annotations for good views, while ignoring the comparative nature of view selection. In this work, we present the first large scale Comparative Photo Composition dataset, which contains over one million comparative view pairs annotated using a cost-effective crowdsourcing workflow. We show that these comparative view annotations are essential for training a robust neural network model for composition. In addition, we propose a novel knowledge transfer framework to train a fast view proposal network, which runs at 75+ FPS and achieves state-of-the-art performance in image cropping and thumbnail generation tasks on three benchmark datasets. The superiority of our method is also demonstrated in a user study on a challenging experiment, where our method significantly outperforms the baseline methods in producing diversified well-composed views.

In this paper, we address referring expression comprehension: localizing an image region described by a natural language expression. While most recent work treats expressions as a single unit, we propose to decompose them into three modular components related to subject appearance, location, and relationship to other objects. This allows us to flexibly adapt to expressions containing different types of information in an end-to-end framework. In our model, which we call the Modular Attention Network (MAttNet), two types of attention are utilized: language-based attention that learns the module weights as well as the word/phrase attention that each module should focus on; and visual attention that allows the subject and relationship modules to focus on relevant image components. Module weights combine scores from all three modules dynamically to output an overall score. Experiments show that MAttNet outperforms previous state-of-art methods by a large margin on both bounding-box-level and pixel-level comprehension tasks. Demo and code are provided., Equation of word attention fixed; MAttNet+Grabcut results added

While many approaches have been proposed to estimate and remove blur in a photo, few efforts were made to have an algorithm automatically understand the blur desirability: whether the blur is desired or not, and how it affects the quality of the photo. Such a task not only relies on low-level visual features to identify blurry regions, but also requires high-level understanding of the image content as well as user intent during photo capture. In this paper, we propose a unified framework to estimate a spatially-varying blur map and understand its desirability in terms of image quality at the same time. In particular, we use a dilated fully convolutional neural network with pyramid pooling and boundary refinement layers to generate high-quality blur response maps. If blur exists, we classify its desirability to three levels ranging from good to bad, by distilling high-level semantics and learning an attention map to adaptively localize the important content in the image. The whole framework is end-to-end jointly trained with both supervisions of pixel-wise blur responses and image-wise blur desirability levels. Considering the limitations of existing image blur datasets, we collected a new large-scale dataset with both annotations to facilitate training. The proposed methods are extensively evaluated on two datasets and demonstrate state-of-the-art performance on both tasks.

In this paper, we introduce the problem of simultaneously detecting multiple photographic defects. We aim at detecting the existence, severity, and potential locations of common photographic defects related to color, noise, blur and composition. The automatic detection of such defects could be used to provide users with suggestions for how to improve photos without the need to laboriously try various correction methods. Defect detection could also help users select photos of higher quality while filtering out those with severe defects in photo curation and summarization. To investigate this problem, we collected a large-scale dataset of user annotations on seven common photographic defects, which allows us to evaluate algorithms by measuring their consistency with human judgments. Our new dataset enables us to formulate the problem as a multi-task learning problem and train a multi-column deep convolutional neural network (CNN) to simultaneously predict the severity of all the defects. Unlike some existing single-defect estimation methods that rely on low-level statistics and may fail in many cases on natural photographs, our model is able to understand image contents and quality at a higher level. As a result, in our experiments, we show that our model has predictions with much higher consistency with human judgments than low-level methods as well as several baseline CNN models. Our model also performs better than an average human from our user study., Comment: Accepted to WACV'18

In this paper, we propose a novel adaptive metric learning algorithm (AML) for visual saliency detection. A key observation is that the saliency of a superpixel can be estimated by the distance from the most certain foreground and background seeds. Instead of measuring distance on the Euclidean space, we present a learning method based on two complementary Mahalanobis distance metrics: 1) generic metric learning (GML) and 2) specific metric learning (SML). GML aims at the global distribution of the whole training set, while SML considers the specific structure of a single image. Considering that multiple similarity measures from different views may enhance the relevant information and alleviate the irrelevant one, we try to fuse the GML and SML together and experimentally find the combining result does work well. Different from the most existing methods which are directly based on low-level features, we devise a superpixelwise Fisher vector coding approach to better distinguish salient objects from the background. We also propose an accurate seeds selection mechanism and exploit contextual and multiscale information when constructing the final saliency map. Experimental results on various image sets show that the proposed AML performs favorably against the state-of-the-arts.

Recent deep learning based approaches have shown promising results for the challenging task of inpainting large missing regions in an image. These methods can generate visually plausible image structures and textures, but often create distorted structures or blurry textures inconsistent with surrounding areas. This is mainly due to ineffectiveness of convolutional neural networks in explicitly borrowing or copying information from distant spatial locations. On the other hand, traditional texture and patch synthesis approaches are particularly suitable when it needs to borrow textures from the surrounding regions. Motivated by these observations, we propose a new deep generative model-based approach which can not only synthesize novel image structures but also explicitly utilize surrounding image features as references during network training to make better predictions. The model is a feed-forward, fully convolutional neural network which can process images with multiple holes at arbitrary locations and with variable sizes during the test time. Experiments on multiple datasets including faces (CelebA, CelebA-HQ), textures (DTD) and natural images (ImageNet, Places2) demonstrate that our proposed approach generates higher-quality inpainting results than existing ones. Code, demo and models are available at: https://github.com/JiahuiYu/generative_inpainting., Accepted in CVPR 2018; add CelebA-HQ results; open sourced; interactive demo available: http://jhyu.me/demo

We aim to generate high resolution shallow depth-of-field (DoF) images from a single all-in-focus image with controllable focal distance and aperture size. To achieve this, we propose a novel neural network model comprised of a depth prediction module, a lens blur module, and a guided upsampling module. All modules are differentiable and are learned from data. To train our depth prediction module, we collect a dataset of 2462 RGB-D images captured by mobile phones with a dual-lens camera, and use existing segmentation datasets to improve border prediction. We further leverage a synthetic dataset with known depth to supervise the lens blur and guided upsampling modules. The effectiveness of our system and training strategies are verified in the experiments. Our method can generate high-quality shallow DoF images at high resolution, and produces significantly fewer artifacts than the baselines and existing solutions for single image shallow DoF synthesis. Compared with the iPhone portrait mode, which is a state-of-the-art shallow DoF solution based on a dual-lens depth camera, our method generates comparable results, while allowing for greater flexibility to choose focal points and aperture size, and is not limited to one capture setup., Comment: 11 pages, 15 figures, accepted by SIGGRAPH Asia 2018, low-resolution version

Existing works on semantic segmentation typically consider a small number of labels, ranging from tens to a few hundreds. With a large number of labels, training and evaluation of such task become extremely challenging due to correlation between labels and lack of datasets with complete annotations. We formulate semantic segmentation as a problem of image segmentation given a semantic concept, and propose a novel system which can potentially handle an unlimited number of concepts, including objects, parts, stuff, and attributes. We achieve this using a weakly and semi-supervised framework leveraging multiple datasets with different levels of supervision. We first train a deep neural network on a 6M stock image dataset with only image-level labels to learn visual-semantic embedding on 18K concepts. Then, we refine and extend the embedding network to predict an attention map, using a curated dataset with bounding box annotations on 750 concepts. Finally, we train an attention-driven class agnostic segmentation network using an 80-category fully annotated dataset. We perform extensive experiments to validate that the proposed system performs competitively to the state of the art on fully supervised concepts, and is capable of producing accurate segmentations for weakly learned and unseen concepts.

Photo blending is a common technique to create aesthetically pleasing artworks by combining multiple photos. However, the process of photo blending is usually time-consuming, and care must be taken in the process of blending, filtering, positioning, and masking each of the source photos. To make photo blending accessible to general public, we propose an efficient approach for automatic photo blending via deep learning. Specifically, given a foreground image and a background image, our proposed method automatically generates a set of blending photos with scores that indicate the aesthetics quality with the proposed quality network and policy network. Experimental results show that the proposed approach can effectively generate high quality blending photos with efficiency.

Multi-task learning has been widely adopted in many computer vision tasks to improve overall computation efficiency or boost the performance of individual tasks, under the assumption that those tasks are correlated and complementary to each other. However, the relationships between the tasks are complicated in practice, especially when the number of involved tasks scales up. When two tasks are of weak relevance, they may compete or even distract each other during joint training of shared parameters, and as a consequence undermine the learning of all the tasks. This will raise destructive interference which decreases learning efficiency of shared parameters and lead to low quality loss local optimum w.r.t. shared parameters. To address the this problem, we propose a general modulation module, which can be inserted into any convolutional neural network architecture, to encourage the coupling and feature sharing of relevant tasks while disentangling the learning of irrelevant tasks with minor parameters addition. Equipped with this module, gradient directions from different tasks can be enforced to be consistent for those shared parameters, which benefits multi-task joint training. The module is end-to-end learnable without ad-hoc design for specific tasks, and can naturally handle many tasks at the same time. We apply our approach on two retrieval tasks, face retrieval on the CelebA dataset [12] and product retrieval on the UT-Zappos50K dataset [34, 35], and demonstrate its advantage over other multi-task learning methods in both accuracy and storage efficiency.

We present a new image search technique that, given a background image, returns compatible foreground objects for image compositing tasks. The compatibility of a foreground object and a background scene depends on various aspects such as semantics, surrounding context, geometry, style and color. However, existing image search techniques measure the similarities on only a few aspects, and may return many results that are not suitable for compositing. Moreover, the importance of each factor may vary for different object categories and image content, making it difficult to manually define the matching criteria. In this paper, we propose to learn feature representations for foreground objects and background scenes respectively, where image content and object category information are jointly encoded during training. As a result, the learned features can adaptively encode the most important compatibility factors. We project the features to a common embedding space, so that the compatibility scores can be easily measured using the cosine similarity, enabling very efficient search. We collect an evaluation set consisting of eight object categories commonly used in compositing tasks, on which we demonstrate that our approach significantly outperforms other search techniques.

Human parsing and pose estimation have recently received considerable interest due to their substantial application potentials. However, the existing datasets have limited numbers of images and annotations and lack a variety of human appearances and coverage of challenging cases in unconstrained environments. In this paper, we introduce a new benchmark named "Look into Person (LIP)" that provides a significant advancement in terms of scalability, diversity, and difficulty, which are crucial for future developments in human-centric analysis. This comprehensive dataset contains over 50,000 elaborately annotated images with 19 semantic part labels and 16 body joints, which are captured from a broad range of viewpoints, occlusions, and background complexities. Using these rich annotations, we perform detailed analyses of the leading human parsing and pose estimation approaches, thereby obtaining insights into the successes and failures of these methods. To further explore and take advantage of the semantic correlation of these two tasks, we propose a novel joint human parsing and pose estimation network to explore efficient context modeling, which can simultaneously predict parsing and pose with extremely high quality. Furthermore, we simplify the network to solve human parsing by exploring a novel self-supervised structure-sensitive learning approach, which imposes human pose structures into the parsing results without resorting to extra supervision. The dataset, code and models are available at http://www.sysu-hcp.net/lip/., Comment: We proposed the most comprehensive dataset around the world for human-centric analysis! (Accepted By T-PAMI 2018) The dataset, code and models are available at http://www.sysu-hcp.net/lip/ . arXiv admin note: substantial text overlap with arXiv:1703.05446

To investigate the metal flow during the railway wheel forming process, experiments and finite element method (FEM) simulation were carried out. An axisymmetric modeling for the wheel rolling process was proposed to predict the metal flow in radial direction, by which the whole multi-stage forming process could be simulated in axisymmetric and integral way. The result shows that the axisymmetric simulation method was an effective method to explore the metal flow in radial direction and to analyze the relationships of tools motion during the wheel rolling. The detail information about metal flow in railway wheel forming process was obtained. The metal in the wheel web was from the area near the half radius of the original billet; the chill zone of the billet became an envelope of the rim and part of the web with a maximum thickness of about 6 mm below the tread. At the wheel rolling stage, the metal in the rim flowed towards the web; the metal near the surfaces of the conjunction region between the web and rim suffered severe shear deformation.

We present a scene parsing method that utilizes global context information based on both the parametric and nonparametric models. Compared to previous methods that only exploit the local relationship between objects, we train a context network based on scene similarities to generate feature representations for global contexts. In addition, these learned features are utilized to generate global and spatial priors for explicit classes inference. We then design modules to embed the feature representations and the priors into the segmentation network as additional global context cues. We show that the proposed method can eliminate false positives that are not compatible with the global context representations. Experiments on both the MIT ADE20K and PASCAL Context datasets show that the proposed method performs favorably against existing methods.

In this work, we address the challenging video scene parsing problem by developing effective representation learning methods given limited parsing annotations. In particular, we contribute two novel methods that constitute a unified parsing framework. (1) \textbf{Predictive feature learning}} from nearly unlimited unlabeled video data. Different from existing methods learning features from single frame parsing, we learn spatiotemporal discriminative features by enforcing a parsing network to predict future frames and their parsing maps (if available) given only historical frames. In this way, the network can effectively learn to capture video dynamics and temporal context, which are critical clues for video scene parsing, without requiring extra manual annotations. (2) \textbf{Prediction steering parsing}} architecture that effectively adapts the learned spatiotemporal features to scene parsing tasks and provides strong guidance for any off-the-shelf parsing model to achieve better video scene parsing performance. Extensive experiments over two challenging datasets, Cityscapes and Camvid, have demonstrated the effectiveness of our methods by showing significant improvement over well-established baselines., Comment: 15 pages, 7 figures, 5 tables, currently v2

Parsing urban scene images benefits many applications, especially self-driving. Most of the current solutions employ generic image parsing models that treat all scales and locations in the images equally and do not consider the geometry property of car-captured urban scene images. Thus, they suffer from heterogeneous object scales caused by perspective projection of cameras on actual scenes and inevitably encounter parsing failures on distant objects as well as other boundary and recognition errors. In this work, we propose a new FoveaNet model to fully exploit the perspective geometry of scene images and address the common failures of generic parsing models. FoveaNet estimates the perspective geometry of a scene image through a convolutional network which integrates supportive evidence from contextual objects within the image. Based on the perspective geometry information, FoveaNet “undoes” the camera perspective projection — analyzing regions in the space of the actual scene, and thus provides much more reliable parsing results. Furthermore, to effectively address the recognition errors, FoveaNet introduces a new dense CRFs model that takes the perspective geometry as a prior potential. We evaluate FoveaNet on two urban scene parsing datasets, Cityspaces and CamVid, which demonstrates that FoveaNet can outperform all the well-established baselines and provide new state-of-the-art performance.

Human parsing has recently attracted a lot of research interests due to its huge application potentials. However existing datasets have limited number of images and annotations, and lack the variety of human appearances and the coverage of challenging cases in unconstrained environment. In this paper, we introduce a new benchmark "Look into Person (LIP)" that makes a significant advance in terms of scalability, diversity and difficulty, a contribution that we feel is crucial for future developments in human-centric analysis. This comprehensive dataset contains over 50,000 elaborately annotated images with 19 semantic part labels, which are captured from a wider range of viewpoints, occlusions and background complexity. Given these rich annotations we perform detailed analyses of the leading human parsing approaches, gaining insights into the success and failures of these methods. Furthermore, in contrast to the existing efforts on improving the feature discriminative capability, we solve human parsing by exploring a novel self-supervised structure-sensitive learning approach, which imposes human pose structures into parsing results without resorting to extra supervision (i.e., no need for specifically labeling human joints in model training). Our self-supervised learning framework can be injected into any advanced neural networks to help incorporate rich high-level knowledge regarding human joints from a global perspective and improve the parsing results. Extensive evaluations on our LIP and the public PASCAL-Person-Part dataset demonstrate the superiority of our method., Comment: Accepted to appear in CVPR 2017