Search

Image content safety has become a significant challenge with the rise of visual media on online platforms. Meanwhile, in the age of AI-generated content (AIGC), many image generation models are capable of producing harmful content, such as images containing sexual or violent material. Thus, it becomes crucial to identify such unsafe images based on established safety rules. Pre-trained Multimodal Large Language Models (MLLMs) offer potential in this regard, given their strong pattern recognition abilities. Existing approaches typically fine-tune MLLMs with human-labeled datasets, which however brings a series of drawbacks. First, relying on human annotators to label data following intricate and detailed guidelines is both expensive and labor-intensive. Furthermore, users of safety judgment systems may need to frequently update safety rules, making fine-tuning on human-based annotation more challenging. This raises the research question: Can we detect unsafe images by querying MLLMs in a zero-shot setting using a predefined safety constitution (a set of safety rules)? Our research showed that simply querying pre-trained MLLMs does not yield satisfactory results. This lack of effectiveness stems from factors such as the subjectivity of safety rules, the complexity of lengthy constitutions, and the inherent biases in the models. To address these challenges, we propose a MLLM-based method includes objectifying safety rules, assessing the relevance between rules and images, making quick judgments based on debiased token probabilities with logically complete yet simplified precondition chains for safety rules, and conducting more in-depth reasoning with cascaded chain-of-thought processes if necessary. Experiment results demonstrate that our method is highly effective for zero-shot image safety judgment tasks.

The increasing reliance on smartphones for communication, financial transactions, and personal data management has made them prime targets for cyberattacks, particularly smishing, a sophisticated variant of phishing conducted via SMS. Despite the growing threat, traditional detection methods often struggle with the informal and evolving nature of SMS language, which includes abbreviations, slang, and short forms. This paper presents an enhanced content-based smishing detection framework that leverages advanced text normalization techniques to improve detection accuracy. By converting nonstandard text into its standardized form, the proposed model enhances the efficacy of machine learning classifiers, particularly the Naive Bayesian classifier, in distinguishing smishing messages from legitimate ones. Our experimental results, validated on a publicly available dataset, demonstrate a detection accuracy of 96.2%, with a low False Positive Rate of 3.87% and False Negative Rate of 2.85%. This approach significantly outperforms existing methodologies, providing a robust solution to the increasingly sophisticated threat of smishing in the mobile environment.

Robust content moderation classifiers are essential for the safety of Generative AI systems. In this task, differences between safe and unsafe inputs are often extremely subtle, making it difficult for classifiers (and indeed, even humans) to properly distinguish violating vs. benign samples without context or explanation. Scaling risk discovery and mitigation through continuous model fine-tuning is also slow, challenging and costly, preventing developers from being able to respond quickly and effectively to emergent harms. We propose a Classification approach employing Retrieval-Augmented Generation (Class-RAG). Class-RAG extends the capability of its base LLM through access to a retrieval library which can be dynamically updated to enable semantic hotfixing for immediate, flexible risk mitigation. Compared to model fine-tuning, Class-RAG demonstrates flexibility and transparency in decision-making, outperforms on classification and is more robust against adversarial attack, as evidenced by empirical studies. Our findings also suggest that Class-RAG performance scales with retrieval library size, indicating that increasing the library size is a viable and low-cost approach to improve content moderation., Comment: 11 pages, submit to ACL

Diffusion models have demonstrated remarkable efficacy across various image-to-image tasks. In this research, we introduce Imagine yourself, a state-of-the-art model designed for personalized image generation. Unlike conventional tuning-based personalization techniques, Imagine yourself operates as a tuning-free model, enabling all users to leverage a shared framework without individualized adjustments. Moreover, previous work met challenges balancing identity preservation, following complex prompts and preserving good visual quality, resulting in models having strong copy-paste effect of the reference images. Thus, they can hardly generate images following prompts that require significant changes to the reference image, \eg, changing facial expression, head and body poses, and the diversity of the generated images is low. To address these limitations, our proposed method introduces 1) a new synthetic paired data generation mechanism to encourage image diversity, 2) a fully parallel attention architecture with three text encoders and a fully trainable vision encoder to improve the text faithfulness, and 3) a novel coarse-to-fine multi-stage finetuning methodology that gradually pushes the boundary of visual quality. Our study demonstrates that Imagine yourself surpasses the state-of-the-art personalization model, exhibiting superior capabilities in identity preservation, visual quality, and text alignment. This model establishes a robust foundation for various personalization applications. Human evaluation results validate the model's SOTA superiority across all aspects (identity preservation, text faithfulness, and visual appeal) compared to the previous personalization models.

The calculation of material phonon thermal conductivity from density functional theory calculations requires computationally expensive evaluation of anharmonic interatomic force constants and has remained a computational bottleneck in the high-throughput discovery of materials. In this work, we present a machine learning-assisted approach for the extraction of anharmonic force constants through local learning of the potential energy surface. We demonstrate our approach on a diverse collection of 220 ternary materials for which the total computational time for anharmonic force constants evaluation is reduced by more than an order of magnitude from 480,000 cpu-hours to less than 12,000 cpu-hours while preserving the thermal conductivity prediction accuracy to within 10%. Our approach removes a major hurdle in computational thermal conductivity evaluation and will pave the way forward for the high-throughput discovery of materials.

Generative AI image models may inadvertently generate problematic representations of people. Past research has noted that millions of users engage daily across the world with these models and that the models, including through problematic representations of people, have the potential to compound and accelerate real-world discrimination and other harms (Bianchi et al, 2023). In this paper, we focus on addressing the generation of problematic associations between demographic groups and semantic concepts that may reflect and reinforce negative narratives embedded in social data. Building on sociological literature (Blumer, 1958) and mapping representations to model behaviors, we have developed a taxonomy to study problematic associations in image generation models. We explore the effectiveness of fine tuning at the model level as a method to address these associations, identifying a potential reduction in visual quality as a limitation of traditional fine tuning. We also propose a new methodology with twice-human-in-the-loop (T-HITL) that promises improvements in both reducing problematic associations and also maintaining visual quality. We demonstrate the effectiveness of T-HITL by providing evidence of three problematic associations addressed by T-HITL at the model level. Our contributions to scholarship are two-fold. By defining problematic associations in the context of machine learning models and generative AI, we introduce a conceptual and technical taxonomy for addressing some of these associations. Finally, we provide a method, T-HITL, that addresses these associations and simultaneously maintains visual quality of image model generations. This mitigation need not be a tradeoff, but rather an enhancement., Comment: 11 pages, 8 figures

Microorganisms can create engineered materials with exquisite structures and living functionalities. Although synthetic biology tools to genetically manipulate microorganisms continue to expand, the bottom-up rational design of engineered living materials still relies on prior knowledge of genotype-phenotype links for the function of interest. Here, we utilize a high-throughput directed evolution platform to enhance the fitness of whole microorganisms under selection pressure and identify novel genetic pathways to program the functionalities of engineered living materials. Using Komagataeibacter sucrofermentans as a model cellulose-producing microorganism, we show that our droplet-based microfluidic platform enables the directed evolution of these bacteria towards a small number of cellulose overproducers from an initial pool of 40'000 random mutants. Sequencing of the evolved strains reveals an unexpected link between the cellulose-forming ability of the bacteria and a gene encoding a protease complex responsible for protein turnover in the cell. The ability to enhance the fitness of microorganisms towards specific phenotypes and to discover new genotype-phenotype links makes this high-throughput directed evolution platform a promising tool for the development of the next generation of engineered living materials.

The cross-plane (across-layers) phonon thermal transport of five diverse, layered semiconductors is investigated by accounting for higher-order four-phonon scattering, phonon renormalization, and multi-channel thermal transport. For materials having relatively large cross-plane thermal conductivity (AlB6, MoS2, and MoSi2N4), phonons contributing to cross-plane conductivity have an order of magnitude larger mean free path than that for the basal-plane thermal transport, whereas the opposite effect is observed for materials with low thermal conductivity (MoO3 and KCuSe). The contribution from the wave-like coherent transport channel is less than 5% in all considered materials. Our work unravels the contrasting role of nano-structuring on the basal- vs. cross-plane thermal conductivity of low and high thermal conductivity layered materials.

We investigated the accelerated prediction of the thermal conductivity of materials through end- to-end structure-based approaches employing machine learning methods. Due to the non-availability of high-quality thermal conductivity data, we first performed high-throughput calculations based on first principles and the Boltzmann transport equation for 225 materials, effectively more than doubling the size of the existing dataset. We assessed the performance of state-of-the-art machine learning models for thermal conductivity prediction on this expanded dataset and observed that all these models suffered from overfitting. To address this issue, we introduced a novel graph-based neural network model, which demonstrated more consistent and regularized performance across all evaluated datasets. Nevertheless, the best mean absolute percentage error achieved on the test dataset remained in the range of 50-60%. This suggests that while these models are valuable for expediting material screening, their current accuracy is still limited.

The phonon thermal transport properties of twisted bi-layer graphene are investigated using lattice dynamics and the Boltzmann transport equation. The thermal conductivities of 13.2 and 21.8 twisted configurations are 56% and 36% lower than the untwisted configuration which has a room temperature thermal conductivity of 2260 W/m-K. The reason for this lower thermal conductivity is unraveled from phonon mode-level analysis made possible through the untwisting of layers. Due to a large commensurate unitcell of twisted configuration, the Brillouin zone is folded but this folding has no bearing on the phonon scattering phase space. The major impact of twisting is felt by flexural phonons with out-of-plane vibrations via the change in strength of flexural interatomic interactions and since these flexural phonons carry majority of the heat in bilayer graphene (65% at room temperature), the thermal conductivity is sensitive to layer twisting. Our study suggests that twisting will affect thermal transport only for those materials that have a major contribution from flexural phonon modes.

The thermal transport properties of single-wall carbon nanotubes (SWCNTs) are re-investigated using the iterative solution of the Boltzmann transport equation by including four-phonon scattering. Using only three-phonon scattering, the flexural and twisted phonon modes are found to remain non-scattered via Umklapp processes, resulting in the literature-reported divergence of thermal conductivity with tube length. However, with four-phonon scattering, while longitudinal modes remain unaffected, the otherwise non-scattered transverse modes undergo Umklapp scattering and result in a non-ballistic transport, thus settling a decades-long debate on the length dependence of thermal conductivity of SWCNTs. The predicted thermal conductivity of SWCNTs using both three- and four-phonon scatterings is 3700 W/m-K at 300 K and stays highest amongst all known materials.

We study the extended version of the non-uniform, capacitated facility location problem with multiple fulfilment channels between the facilities and clients, each with their own channel capacities and service cost. Though the problem has been extensively studied in the literature, all the prior works assume a single channel of fulfilment, and the existing methods based on linear programming, primal-dual relationships, local search heuristics etc. do not scale for a large supply chain system involving millions of decision variables. Using the concepts of sub-modularity and optimal transport theory, we present a scalable algorithm for determining the set of facilities to be opened under a cardinality constraint. By introducing various schemes such as: (i) iterative facility selection using incremental gain, (ii) approximation of the linear program using novel multi-stage Sinkhorn iterations, (iii) creation of facilities one for each fulfilment channel etc., we develop a fast but a tight approximate solution, requiring $\mathcal{O}\left(\frac{3+k}{m}ln\left(\frac{1}{\epsilon}\right)\right)$ instances of optimal transport problems to select k facilities from m options, each solvable in linear time. Our algorithm is implicitly endowed with all the theoretical guarantees enjoyed by submodular maximisation problems and the Sinkhorn distances. When compared against the state-of-the-art commercial MILP solvers, we obtain a 100-fold speedup in computation, while the difference in objective values lies within a narrow range of 3%.

With Reinforcement Learning (RL) for inventory management (IM) being a nascent field of research, approaches tend to be limited to simple, linear environments with implementations that are minor modifications of off-the-shelf RL algorithms. Scaling these simplistic environments to a real-world supply chain comes with a few challenges such as: minimizing the computational requirements of the environment, specifying agent configurations that are representative of dynamics at real world stores and warehouses, and specifying a reward framework that encourages desirable behavior across the whole supply chain. In this work, we present a system with a custom GPU-parallelized environment that consists of one warehouse and multiple stores, a novel architecture for agent-environment dynamics incorporating enhanced state and action spaces, and a shared reward specification that seeks to optimize for a large retailer's supply chain needs. Each vertex in the supply chain graph is an independent agent that, based on its own inventory, able to place replenishment orders to the vertex upstream. The warehouse agent, aside from placing orders from the supplier, has the special property of also being able to constrain replenishment to stores downstream, which results in it learning an additional allocation sub-policy. We achieve a system that outperforms standard inventory control policies such as a base-stock policy and other RL-based specifications for 1 product, and lay out a future direction of work for multiple products., Comment: 14 pages, 5 figures

Detection of fake news is crucial to ensure the authenticity of information and maintain the news ecosystems reliability. Recently, there has been an increase in fake news content due to the recent proliferation of social media and fake content generation techniques such as Deep Fake. The majority of the existing modalities of fake news detection focus on content based approaches. However, most of these techniques fail to deal with ultra realistic synthesized media produced by generative models. Our recent studies find that the propagation characteristics of authentic and fake news are distinguishable, irrespective of their modalities. In this regard, we have investigated the auxiliary information based on social context to detect fake news. This paper has analyzed the social context of fake news detection with a hybrid graph neural network based approach. This hybrid model is based on integrating a graph neural network on the propagation of news and bi directional encoder representations from the transformers model on news content to learn the text features. Thus this proposed approach learns the content as well as the context features and hence able to outperform the baseline models with an f1 score of 0.91 on PolitiFact and 0.93 on the Gossipcop dataset, respectively

We performed ab-initio driven density functional theory-based high throughput computations to search for materials with low thermal conductivity and high thermal transport anisotropy. We shortlisted a pool of 429 stable ternary semiconductors from the Materials Project and obtained phonon thermal conductivity by solving the Boltzmann transport equation on 225 materials. We found the lowest thermal conductivity of 0.16 W/m-K in SbRbK 2 and 40 materials with a thermal conductivity lower than 1 W/m-K at 300 K. For anisotropic thermal transport, we have identified six materials with anisotropy larger than 5 and 20 with thermal transport anisotropy higher than the largest reported literature value.