-
Optimizing KV Cache Eviction in LLMs: Adaptive Allocation for Enhanced Budget Utilization
Authors:
Yuan Feng,
Junlin Lv,
Yukun Cao,
Xike Xie,
S. Kevin Zhou
Abstract:
Large Language Models have excelled in various fields but encounter efficiency limitations due to the extensive KV cache required for long sequences inference. Many efforts try to evict non-critical cache elements during runtime, thereby reducing cache size within a given memory budget while preserving generation quality. Our reexamination of their underlying principles discerns that prevailing st…
▽ More
Large Language Models have excelled in various fields but encounter efficiency limitations due to the extensive KV cache required for long sequences inference. Many efforts try to evict non-critical cache elements during runtime, thereby reducing cache size within a given memory budget while preserving generation quality. Our reexamination of their underlying principles discerns that prevailing strategies essentially aim to minimize an upper bound of eviction loss within a specific budget allocation. However, we observe that the current practice of uniformly allocating budgets across different attention heads during the eviction procedure tends to degrade the quality of generation posten-eviction. In light of these findings, we propose a simple yet effective adaptive allocation algorithm that not only theoretically ensures its loss upper bound does not exceed that of previous uniform allocation methods, but also effectively aligns with the characteristics of the self-attention mechanism, thus practically reducing the upper bound. Further, integrating this algorithm with two of the most advanced methods yields Ada-SnapKV and Ada-Pyramid. Extensive experimental validation across 16 datasets and the Needle-in-a-Haystack test confirm that Ada-SnapKV and Ada-Pyramid achieve further enhancements, establishing new benchmarks in state-of-the-art performance.
△ Less
Submitted 16 July, 2024;
originally announced July 2024.
-
Improving Graph Out-of-distribution Generalization on Real-world Data
Authors:
Can Xu,
Yao Cheng,
Jianxiang Yu,
Haosen Wang,
Jingsong Lv,
Xiang Li
Abstract:
Existing methods for graph out-of-distribution (OOD) generalization primarily rely on empirical studies on synthetic datasets. Such approaches tend to overemphasize the causal relationships between invariant sub-graphs and labels, thereby neglecting the non-negligible role of environment in real-world scenarios. In contrast to previous studies that impose rigid independence assumptions on environm…
▽ More
Existing methods for graph out-of-distribution (OOD) generalization primarily rely on empirical studies on synthetic datasets. Such approaches tend to overemphasize the causal relationships between invariant sub-graphs and labels, thereby neglecting the non-negligible role of environment in real-world scenarios. In contrast to previous studies that impose rigid independence assumptions on environments and invariant sub-graphs, this paper presents the theorems of environment-label dependency and mutable rationale invariance, where the former characterizes the usefulness of environments in determining graph labels while the latter refers to the mutable importance of graph rationales. Based on analytic investigations, a novel variational inference based method named ``Probability Dependency on Environments and Rationales for OOD Graphs on Real-world Data'' (DEROG) is introduced. To alleviate the adverse effect of unknown prior knowledge on environments and rationales, DEROG utilizes generalized Bayesian inference. Further, DEROG employs an EM-based algorithm for optimization. Finally, extensive experiments on real-world datasets under different distribution shifts are conducted to show the superiority of DEROG. Our code is publicly available at https://anonymous.4open.science/r/DEROG-536B.
△ Less
Submitted 14 July, 2024;
originally announced July 2024.
-
Combinatorial Constructions of Optimal Quaternary Additive Codes
Authors:
Chaofeng Guan,
Jingjie Lv,
Gaojun Luo,
Zhi Ma
Abstract:
This paper aims to construct optimal quaternary additive codes with non-integer dimensions. Firstly, we propose combinatorial constructions of quaternary additive constant-weight codes, alongside additive anticode construction. Subsequently, we propose generalized Construction X, which facilitates the construction of non-integer dimensional optimal additive codes from linear codes. Then, we constr…
▽ More
This paper aims to construct optimal quaternary additive codes with non-integer dimensions. Firstly, we propose combinatorial constructions of quaternary additive constant-weight codes, alongside additive anticode construction. Subsequently, we propose generalized Construction X, which facilitates the construction of non-integer dimensional optimal additive codes from linear codes. Then, we construct ten classes of optimal quaternary non-integer dimensional additive codes through these two methods. As an application, we also determine the optimal additive $[n,3.5,n-t]_4$ codes for all $t$ with variable $n$, except for $t=6,7,12$.
△ Less
Submitted 4 July, 2024;
originally announced July 2024.
-
Self Adaptive Threshold Pseudo-labeling and Unreliable Sample Contrastive Loss for Semi-supervised Image Classification
Authors:
Xuerong Zhang,
Li Huang,
Jing Lv,
Ming Yang
Abstract:
Semi-supervised learning is attracting blooming attention, due to its success in combining unlabeled data. However, pseudo-labeling-based semi-supervised approaches suffer from two problems in image classification: (1) Existing methods might fail to adopt suitable thresholds since they either use a pre-defined/fixed threshold or an ad-hoc threshold adjusting scheme, resulting in inferior performan…
▽ More
Semi-supervised learning is attracting blooming attention, due to its success in combining unlabeled data. However, pseudo-labeling-based semi-supervised approaches suffer from two problems in image classification: (1) Existing methods might fail to adopt suitable thresholds since they either use a pre-defined/fixed threshold or an ad-hoc threshold adjusting scheme, resulting in inferior performance and slow convergence. (2) Discarding unlabeled data with confidence below the thresholds results in the loss of discriminating information. To solve these issues, we develop an effective method to make sufficient use of unlabeled data. Specifically, we design a self adaptive threshold pseudo-labeling strategy, which thresholds for each class can be dynamically adjusted to increase the number of reliable samples. Meanwhile, in order to effectively utilise unlabeled data with confidence below the thresholds, we propose an unreliable sample contrastive loss to mine the discriminative information in low-confidence samples by learning the similarities and differences between sample features. We evaluate our method on several classification benchmarks under partially labeled settings and demonstrate its superiority over the other approaches.
△ Less
Submitted 3 July, 2024;
originally announced July 2024.
-
TieBot: Learning to Knot a Tie from Visual Demonstration through a Real-to-Sim-to-Real Approach
Authors:
Weikun Peng,
Jun Lv,
Yuwei Zeng,
Haonan Chen,
Siheng Zhao,
Jichen Sun,
Cewu Lu,
Lin Shao
Abstract:
The tie-knotting task is highly challenging due to the tie's high deformation and long-horizon manipulation actions. This work presents TieBot, a Real-to-Sim-to-Real learning from visual demonstration system for the robots to learn to knot a tie. We introduce the Hierarchical Feature Matching approach to estimate a sequence of tie's meshes from the demonstration video. With these estimated meshes…
▽ More
The tie-knotting task is highly challenging due to the tie's high deformation and long-horizon manipulation actions. This work presents TieBot, a Real-to-Sim-to-Real learning from visual demonstration system for the robots to learn to knot a tie. We introduce the Hierarchical Feature Matching approach to estimate a sequence of tie's meshes from the demonstration video. With these estimated meshes used as subgoals, we first learn a teacher policy using privileged information. Then, we learn a student policy with point cloud observation by imitating teacher policy. Lastly, our pipeline learns a residual policy when the learned policy is applied to real-world execution, mitigating the Sim2Real gap. We demonstrate the effectiveness of TieBot in simulation and the real world. In the real-world experiment, a dual-arm robot successfully knots a tie, achieving 50% success rate among 10 trials. Videos can be found https://tiebots.github.io/.
△ Less
Submitted 3 July, 2024; v1 submitted 3 July, 2024;
originally announced July 2024.
-
Human-Agent Joint Learning for Efficient Robot Manipulation Skill Acquisition
Authors:
Shengcheng Luo,
Quanquan Peng,
Jun Lv,
Kaiwen Hong,
Katherine Rose Driggs-Campbell,
Cewu Lu,
Yong-Lu Li
Abstract:
Employing a teleoperation system for gathering demonstrations offers the potential for more efficient learning of robot manipulation. However, teleoperating a robot arm equipped with a dexterous hand or gripper, via a teleoperation system poses significant challenges due to its high dimensionality, complex motions, and differences in physiological structure.
In this study, we introduce a novel s…
▽ More
Employing a teleoperation system for gathering demonstrations offers the potential for more efficient learning of robot manipulation. However, teleoperating a robot arm equipped with a dexterous hand or gripper, via a teleoperation system poses significant challenges due to its high dimensionality, complex motions, and differences in physiological structure.
In this study, we introduce a novel system for joint learning between human operators and robots, that enables human operators to share control of a robot end-effector with a learned assistive agent, facilitating simultaneous human demonstration collection and robot manipulation teaching. In this setup, as data accumulates, the assistive agent gradually learns. Consequently, less human effort and attention are required, enhancing the efficiency of the data collection process. It also allows the human operator to adjust the control ratio to achieve a trade-off between manual and automated control.
We conducted experiments in both simulated environments and physical real-world settings. Through user studies and quantitative evaluations, it is evident that the proposed system could enhance data collection efficiency and reduce the need for human adaptation while ensuring the collected data is of sufficient quality for downstream tasks. Videos are available at https://norweig1an.github.io/human-agent-joint-learning.github.io/.
△ Less
Submitted 2 July, 2024; v1 submitted 28 June, 2024;
originally announced July 2024.
-
IMDL-BenCo: A Comprehensive Benchmark and Codebase for Image Manipulation Detection & Localization
Authors:
Xiaochen Ma,
Xuekang Zhu,
Lei Su,
Bo Du,
Zhuohang Jiang,
Bingkui Tong,
Zeyu Lei,
Xinyu Yang,
Chi-Man Pun,
Jiancheng Lv,
Jizhe Zhou
Abstract:
A comprehensive benchmark is yet to be established in the Image Manipulation Detection \& Localization (IMDL) field. The absence of such a benchmark leads to insufficient and misleading model evaluations, severely undermining the development of this field. However, the scarcity of open-sourced baseline models and inconsistent training and evaluation protocols make conducting rigorous experiments a…
▽ More
A comprehensive benchmark is yet to be established in the Image Manipulation Detection \& Localization (IMDL) field. The absence of such a benchmark leads to insufficient and misleading model evaluations, severely undermining the development of this field. However, the scarcity of open-sourced baseline models and inconsistent training and evaluation protocols make conducting rigorous experiments and faithful comparisons among IMDL models challenging. To address these challenges, we introduce IMDL-BenCo, the first comprehensive IMDL benchmark and modular codebase. IMDL-BenCo:~\textbf{i)} decomposes the IMDL framework into standardized, reusable components and revises the model construction pipeline, improving coding efficiency and customization flexibility;~\textbf{ii)} fully implements or incorporates training code for state-of-the-art models to establish a comprehensive IMDL benchmark; and~\textbf{iii)} conducts deep analysis based on the established benchmark and codebase, offering new insights into IMDL model architecture, dataset characteristics, and evaluation standards. Specifically, IMDL-BenCo includes common processing algorithms, 8 state-of-the-art IMDL models (1 of which are reproduced from scratch), 2 sets of standard training and evaluation protocols, 15 GPU-accelerated evaluation metrics, and 3 kinds of robustness evaluation. This benchmark and codebase represent a significant leap forward in calibrating the current progress in the IMDL field and inspiring future breakthroughs. Code is available at: https://github.com/scu-zjz/IMDLBenCo
△ Less
Submitted 15 June, 2024;
originally announced June 2024.
-
Global tensor polarization of spin $3/2$ hadrons and quark spin correlations in relativistic heavy ion collisions
Authors:
Zhe Zhang,
Ji-peng Lv,
Zi-han Yu,
Zuo-tang Liang
Abstract:
We study the global polarization of spin-$3/2$ hadrons in relativistic heavy ion collisions. We show in particular that the global tensor polarizations of rank two or three for spin-$3/2$ hadrons are sensitive to the local two or three quark spin correlations respectively in the quark gluon plasma produced in the collision processes. We present the relationships between these measurable tensor pol…
▽ More
We study the global polarization of spin-$3/2$ hadrons in relativistic heavy ion collisions. We show in particular that the global tensor polarizations of rank two or three for spin-$3/2$ hadrons are sensitive to the local two or three quark spin correlations respectively in the quark gluon plasma produced in the collision processes. We present the relationships between these measurable tensor polarizations and quark spin correlations in the quark matter system.
△ Less
Submitted 6 June, 2024;
originally announced June 2024.
-
Dishonesty in Helpful and Harmless Alignment
Authors:
Youcheng Huang,
Jingkun Tang,
Duanyu Feng,
Zheng Zhang,
Wenqiang Lei,
Jiancheng Lv,
Anthony G. Cohn
Abstract:
People tell lies when seeking rewards. Large language models (LLMs) are aligned to human values with reinforcement learning where they get rewards if they satisfy human preference. We find that this also induces dishonesty in helpful and harmless alignment where LLMs tell lies in generating harmless responses. Using the latest interpreting tools, we detect dishonesty, show how LLMs can be harmful…
▽ More
People tell lies when seeking rewards. Large language models (LLMs) are aligned to human values with reinforcement learning where they get rewards if they satisfy human preference. We find that this also induces dishonesty in helpful and harmless alignment where LLMs tell lies in generating harmless responses. Using the latest interpreting tools, we detect dishonesty, show how LLMs can be harmful if their honesty is increased, and analyze such conflicts at the parameter-level. Given these preliminaries and the hypothesis that reward-seeking stimulates dishonesty, we theoretically show that the dishonesty can in-turn decrease the alignment performances and augment reward-seeking alignment with representation regularization. Extensive results, including GPT-4 annotated win-rates, perplexities, and cases studies demonstrate that we can train more honest, helpful, and harmless LLMs. We will make all our codes and results be open-sourced upon this paper's acceptance.
△ Less
Submitted 5 June, 2024; v1 submitted 3 June, 2024;
originally announced June 2024.
-
Selective Annotation via Data Allocation: These Data Should Be Triaged to Experts for Annotation Rather Than the Model
Authors:
Chen Huang,
Yang Deng,
Wenqiang Lei,
Jiancheng Lv,
Ido Dagan
Abstract:
To obtain high-quality annotations under limited budget, semi-automatic annotation methods are commonly used, where a portion of the data is annotated by experts and a model is then trained to complete the annotations for the remaining data. However, these methods mainly focus on selecting informative data for expert annotations to improve the model predictive ability (i.e., triage-to-human data),…
▽ More
To obtain high-quality annotations under limited budget, semi-automatic annotation methods are commonly used, where a portion of the data is annotated by experts and a model is then trained to complete the annotations for the remaining data. However, these methods mainly focus on selecting informative data for expert annotations to improve the model predictive ability (i.e., triage-to-human data), while the rest of the data is indiscriminately assigned to model annotation (i.e., triage-to-model data). This may lead to inefficiencies in budget allocation for annotations, as easy data that the model could accurately annotate may be unnecessarily assigned to the expert, and hard data may be misclassified by the model. As a result, the overall annotation quality may be compromised. To address this issue, we propose a selective annotation framework called SANT. It effectively takes advantage of both the triage-to-human and triage-to-model data through the proposed error-aware triage and bi-weighting mechanisms. As such, informative or hard data is assigned to the expert for annotation, while easy data is handled by the model. Experimental results show that SANT consistently outperforms other baselines, leading to higher-quality annotation through its proper allocation of data to both expert and model workers. We provide pioneering work on data annotation within budget constraints, establishing a landmark for future triage-based annotation studies.
△ Less
Submitted 20 May, 2024;
originally announced May 2024.
-
ARAIDA: Analogical Reasoning-Augmented Interactive Data Annotation
Authors:
Chen Huang,
Yiping Jin,
Ilija Ilievski,
Wenqiang Lei,
Jiancheng Lv
Abstract:
Human annotation is a time-consuming task that requires a significant amount of effort. To address this issue, interactive data annotation utilizes an annotation model to provide suggestions for humans to approve or correct. However, annotation models trained with limited labeled data are prone to generating incorrect suggestions, leading to extra human correction effort. To tackle this challenge,…
▽ More
Human annotation is a time-consuming task that requires a significant amount of effort. To address this issue, interactive data annotation utilizes an annotation model to provide suggestions for humans to approve or correct. However, annotation models trained with limited labeled data are prone to generating incorrect suggestions, leading to extra human correction effort. To tackle this challenge, we propose Araida, an analogical reasoning-based approach that enhances automatic annotation accuracy in the interactive data annotation setting and reduces the need for human corrections. Araida involves an error-aware integration strategy that dynamically coordinates an annotation model and a k-nearest neighbors (KNN) model, giving more importance to KNN's predictions when predictions from the annotation model are deemed inaccurate. Empirical studies demonstrate that Araida is adaptable to different annotation tasks and models. On average, it reduces human correction labor by 11.02% compared to vanilla interactive data annotation methods.
△ Less
Submitted 1 June, 2024; v1 submitted 20 May, 2024;
originally announced May 2024.
-
A Versatile Framework for Analyzing Galaxy Image Data by Implanting Human-in-the-loop on a Large Vision Model
Authors:
Mingxiang Fu,
Yu Song,
Jiameng Lv,
Liang Cao,
Peng Jia,
Nan Li,
Xiangru Li,
Jifeng Liu,
A-Li Luo,
Bo Qiu,
Shiyin Shen,
Liangping Tu,
Lili Wang,
Shoulin Wei,
Haifeng Yang,
Zhenping Yi,
Zhiqiang Zou
Abstract:
The exponential growth of astronomical datasets provides an unprecedented opportunity for humans to gain insight into the Universe. However, effectively analyzing this vast amount of data poses a significant challenge. Astronomers are turning to deep learning techniques to address this, but the methods are limited by their specific training sets, leading to considerable duplicate workloads too. He…
▽ More
The exponential growth of astronomical datasets provides an unprecedented opportunity for humans to gain insight into the Universe. However, effectively analyzing this vast amount of data poses a significant challenge. Astronomers are turning to deep learning techniques to address this, but the methods are limited by their specific training sets, leading to considerable duplicate workloads too. Hence, as an example to present how to overcome the issue, we built a framework for general analysis of galaxy images, based on a large vision model (LVM) plus downstream tasks (DST), including galaxy morphological classification, image restoration, object detection, parameter extraction, and more. Considering the low signal-to-noise ratio of galaxy images and the imbalanced distribution of galaxy categories, we have incorporated a Human-in-the-loop (HITL) module into our large vision model, which leverages human knowledge to enhance the reliability and interpretability of processing galaxy images interactively. The proposed framework exhibits notable few-shot learning capabilities and versatile adaptability to all the abovementioned tasks on galaxy images in the DESI legacy imaging surveys. Expressly, for object detection, trained by 1000 data points, our DST upon the LVM achieves an accuracy of 96.7%, while ResNet50 plus Mask R-CNN gives an accuracy of 93.1%; for morphology classification, to obtain AUC ~0.9, LVM plus DST and HITL only requests 1/50 training sets compared to ResNet18. Expectedly, multimodal data can be integrated similarly, which opens up possibilities for conducting joint analyses with datasets spanning diverse domains in the era of multi-message astronomy.
△ Less
Submitted 17 May, 2024;
originally announced May 2024.
-
Co-Matching: Towards Human-Machine Collaborative Legal Case Matching
Authors:
Chen Huang,
Xinwei Yang,
Yang Deng,
Wenqiang Lei,
JianCheng Lv,
Tat-Seng Chua
Abstract:
Recent efforts have aimed to improve AI machines in legal case matching by integrating legal domain knowledge. However, successful legal case matching requires the tacit knowledge of legal practitioners, which is difficult to verbalize and encode into machines. This emphasizes the crucial role of involving legal practitioners in high-stakes legal case matching. To address this, we propose a collab…
▽ More
Recent efforts have aimed to improve AI machines in legal case matching by integrating legal domain knowledge. However, successful legal case matching requires the tacit knowledge of legal practitioners, which is difficult to verbalize and encode into machines. This emphasizes the crucial role of involving legal practitioners in high-stakes legal case matching. To address this, we propose a collaborative matching framework called Co-Matching, which encourages both the machine and the legal practitioner to participate in the matching process, integrating tacit knowledge. Unlike existing methods that rely solely on the machine, Co-Matching allows both the legal practitioner and the machine to determine key sentences and then combine them probabilistically. Co-Matching introduces a method called ProtoEM to estimate human decision uncertainty, facilitating the probabilistic combination. Experimental results demonstrate that Co-Matching consistently outperforms existing legal case matching methods, delivering significant performance improvements over human- and machine-based matching in isolation (on average, +5.51% and +8.71%, respectively). Further analysis shows that Co-Matching also ensures better human-machine collaboration effectiveness. Our study represents a pioneering effort in human-machine collaboration for the matching task, marking a milestone for future collaborative matching studies.
△ Less
Submitted 16 May, 2024;
originally announced May 2024.
-
Current Views on Mechanisms of the FLASH Effect in Cancer Radiotherapy
Authors:
Yuqi Ma,
Ziming Zhao,
Wenkang Zhang,
Jianfeng Lv,
Junyi Chen,
Xueqin Yan,
XiaoJi Lin,
Junlong Zhang,
Bingwu Wang,
Song Gao,
Jie Xiao,
Gen Yang
Abstract:
FLASH radiotherapy (FLASH-RT) is a new modality of radiotherapy by delivering doses with ultra-high dose rates. FLASH-RT has the ability to suppress tumor growth while sparing normal tissues, known as the FLASH effect. Although FLASH effect has proved valid in various models by different ionizing radiations, the exact underlying mechanism is still unclear. This article summarizes mainstream hypoth…
▽ More
FLASH radiotherapy (FLASH-RT) is a new modality of radiotherapy by delivering doses with ultra-high dose rates. FLASH-RT has the ability to suppress tumor growth while sparing normal tissues, known as the FLASH effect. Although FLASH effect has proved valid in various models by different ionizing radiations, the exact underlying mechanism is still unclear. This article summarizes mainstream hypotheses of FLASH effect at physicochemical and biological levels, including oxygen depletion and free radical reactions, nuclear and mitochondria damage, as well as immune response. These hypotheses contribute reasonable explanations to the FLASH effect, and are interconnected according to the chronological order of the organism's response to ionizing radiation. By collating the existing consensus, evidence, and hypotheses, this article provides a comprehensive overview of potential mechanisms of FLASH effect and practical guidance for future investigation in the field of FLASH-RT.
△ Less
Submitted 16 May, 2024;
originally announced May 2024.
-
DiffGen: Robot Demonstration Generation via Differentiable Physics Simulation, Differentiable Rendering, and Vision-Language Model
Authors:
Yang Jin,
Jun Lv,
Shuqiang Jiang,
Cewu Lu
Abstract:
Generating robot demonstrations through simulation is widely recognized as an effective way to scale up robot data. Previous work often trained reinforcement learning agents to generate expert policies, but this approach lacks sample efficiency. Recently, a line of work has attempted to generate robot demonstrations via differentiable simulation, which is promising but heavily relies on reward des…
▽ More
Generating robot demonstrations through simulation is widely recognized as an effective way to scale up robot data. Previous work often trained reinforcement learning agents to generate expert policies, but this approach lacks sample efficiency. Recently, a line of work has attempted to generate robot demonstrations via differentiable simulation, which is promising but heavily relies on reward design, a labor-intensive process. In this paper, we propose DiffGen, a novel framework that integrates differentiable physics simulation, differentiable rendering, and a vision-language model to enable automatic and efficient generation of robot demonstrations. Given a simulated robot manipulation scenario and a natural language instruction, DiffGen can generate realistic robot demonstrations by minimizing the distance between the embedding of the language instruction and the embedding of the simulated observation after manipulation. The embeddings are obtained from the vision-language model, and the optimization is achieved by calculating and descending gradients through the differentiable simulation, differentiable rendering, and vision-language model components, thereby accomplishing the specified task. Experiments demonstrate that with DiffGen, we could efficiently and effectively generate robot data with minimal human effort or training time.
△ Less
Submitted 12 May, 2024;
originally announced May 2024.
-
DrugLLM: Open Large Language Model for Few-shot Molecule Generation
Authors:
Xianggen Liu,
Yan Guo,
Haoran Li,
Jin Liu,
Shudong Huang,
Bowen Ke,
Jiancheng Lv
Abstract:
Large Language Models (LLMs) have made great strides in areas such as language processing and computer vision. Despite the emergence of diverse techniques to improve few-shot learning capacity, current LLMs fall short in handling the languages in biology and chemistry. For example, they are struggling to capture the relationship between molecule structure and pharmacochemical properties. Consequen…
▽ More
Large Language Models (LLMs) have made great strides in areas such as language processing and computer vision. Despite the emergence of diverse techniques to improve few-shot learning capacity, current LLMs fall short in handling the languages in biology and chemistry. For example, they are struggling to capture the relationship between molecule structure and pharmacochemical properties. Consequently, the few-shot learning capacity of small-molecule drug modification remains impeded. In this work, we introduced DrugLLM, a LLM tailored for drug design. During the training process, we employed Group-based Molecular Representation (GMR) to represent molecules, arranging them in sequences that reflect modifications aimed at enhancing specific molecular properties. DrugLLM learns how to modify molecules in drug discovery by predicting the next molecule based on past modifications. Extensive computational experiments demonstrate that DrugLLM can generate new molecules with expected properties based on limited examples, presenting a powerful few-shot molecule generation capacity.
△ Less
Submitted 7 May, 2024;
originally announced May 2024.
-
An Image Quality Evaluation and Masking Algorithm Based On Pre-trained Deep Neural Networks
Authors:
Peng Jia,
Yu Song,
Jiameng Lv,
Runyu Ning
Abstract:
With the growing amount of astronomical data, there is an increasing need for automated data processing pipelines, which can extract scientific information from observation data without human interventions. A critical aspect of these pipelines is the image quality evaluation and masking algorithm, which evaluates image qualities based on various factors such as cloud coverage, sky brightness, scat…
▽ More
With the growing amount of astronomical data, there is an increasing need for automated data processing pipelines, which can extract scientific information from observation data without human interventions. A critical aspect of these pipelines is the image quality evaluation and masking algorithm, which evaluates image qualities based on various factors such as cloud coverage, sky brightness, scattering light from the optical system, point spread function size and shape, and read-out noise. Occasionally, the algorithm requires masking of areas severely affected by noise. However, the algorithm often necessitates significant human interventions, reducing data processing efficiency. In this study, we present a deep learning based image quality evaluation algorithm that uses an autoencoder to learn features of high quality astronomical images. The trained autoencoder enables automatic evaluation of image quality and masking of noise affected areas. We have evaluated the performance of our algorithm using two test cases: images with point spread functions of varying full width half magnitude, and images with complex backgrounds. In the first scenario, our algorithm could effectively identify variations of the point spread functions, which can provide valuable reference information for photometry. In the second scenario, our method could successfully mask regions affected by complex regions, which could significantly increase the photometry accuracy. Our algorithm can be employed to automatically evaluate image quality obtained by different sky surveying projects, further increasing the speed and robustness of data processing pipelines.
△ Less
Submitted 6 May, 2024;
originally announced May 2024.
-
StyleSeg V2: Towards Robust One-shot Segmentation of Brain Tissue via Optimization-free Registration Error Perception
Authors:
Zhiwei Wang,
Xiaoyu Zeng,
Chongwei Wu,
Jinxin lv,
Xu Zhang,
Wei Fang,
Qiang Li
Abstract:
One-shot segmentation of brain tissue requires training registration-segmentation (reg-seg) dual-model iteratively, where reg-model aims to provide pseudo masks of unlabeled images for seg-model by warping a carefully-labeled atlas. However, the imperfect reg-model induces image-mask misalignment, poisoning the seg-model subsequently. Recent StyleSeg bypasses this bottleneck by replacing the unlab…
▽ More
One-shot segmentation of brain tissue requires training registration-segmentation (reg-seg) dual-model iteratively, where reg-model aims to provide pseudo masks of unlabeled images for seg-model by warping a carefully-labeled atlas. However, the imperfect reg-model induces image-mask misalignment, poisoning the seg-model subsequently. Recent StyleSeg bypasses this bottleneck by replacing the unlabeled images with their warped copies of atlas, but needs to borrow the diverse image patterns via style transformation. Here, we present StyleSeg V2, inherited from StyleSeg but granted the ability of perceiving the registration errors. The motivation is that good registration behaves in a mirrored fashion for mirrored images. Therefore, almost at no cost, StyleSeg V2 can have reg-model itself "speak out" incorrectly-aligned regions by simply mirroring (symmetrically flipping the brain) its input, and the registration errors are symmetric inconsistencies between the outputs of original and mirrored inputs. Consequently, StyleSeg V2 allows the seg-model to make use of correctly-aligned regions of unlabeled images and also enhances the fidelity of style-transformed warped atlas image by weighting the local transformation strength according to registration errors. The experimental results on three public datasets demonstrate that our proposed StyleSeg V2 outperforms other state-of-the-arts by considerable margins, and exceeds StyleSeg by increasing the average Dice by at least 2.4%.
△ Less
Submitted 18 May, 2024; v1 submitted 6 May, 2024;
originally announced May 2024.
-
Real-Time 4K Super-Resolution of Compressed AVIF Images. AIS 2024 Challenge Survey
Authors:
Marcos V. Conde,
Zhijun Lei,
Wen Li,
Cosmin Stejerean,
Ioannis Katsavounidis,
Radu Timofte,
Kihwan Yoon,
Ganzorig Gankhuyag,
Jiangtao Lv,
Long Sun,
Jinshan Pan,
Jiangxin Dong,
Jinhui Tang,
Zhiyuan Li,
Hao Wei,
Chenyang Ge,
Dongyang Zhang,
Tianle Liu,
Huaian Chen,
Yi Jin,
Menghan Zhou,
Yiqiang Yan,
Si Gao,
Biao Wu,
Shaoli Liu
, et al. (50 additional authors not shown)
Abstract:
This paper introduces a novel benchmark as part of the AIS 2024 Real-Time Image Super-Resolution (RTSR) Challenge, which aims to upscale compressed images from 540p to 4K resolution (4x factor) in real-time on commercial GPUs. For this, we use a diverse test set containing a variety of 4K images ranging from digital art to gaming and photography. The images are compressed using the modern AVIF cod…
▽ More
This paper introduces a novel benchmark as part of the AIS 2024 Real-Time Image Super-Resolution (RTSR) Challenge, which aims to upscale compressed images from 540p to 4K resolution (4x factor) in real-time on commercial GPUs. For this, we use a diverse test set containing a variety of 4K images ranging from digital art to gaming and photography. The images are compressed using the modern AVIF codec, instead of JPEG. All the proposed methods improve PSNR fidelity over Lanczos interpolation, and process images under 10ms. Out of the 160 participants, 25 teams submitted their code and models. The solutions present novel designs tailored for memory-efficiency and runtime on edge devices. This survey describes the best solutions for real-time SR of compressed high-resolution images.
△ Less
Submitted 25 April, 2024;
originally announced April 2024.
-
Gaussian-LIC: Photo-realistic LiDAR-Inertial-Camera SLAM with 3D Gaussian Splatting
Authors:
Xiaolei Lang,
Laijian Li,
Hang Zhang,
Feng Xiong,
Mu Xu,
Yong Liu,
Xingxing Zuo,
Jiajun Lv
Abstract:
We present a real-time LiDAR-Inertial-Camera SLAM system with 3D Gaussian Splatting as the mapping backend. Leveraging robust pose estimates from our LiDAR-Inertial-Camera odometry, Coco-LIC, an incremental photo-realistic mapping system is proposed in this paper. We initialize 3D Gaussians from colorized LiDAR points and optimize them using differentiable rendering powered by 3D Gaussian Splattin…
▽ More
We present a real-time LiDAR-Inertial-Camera SLAM system with 3D Gaussian Splatting as the mapping backend. Leveraging robust pose estimates from our LiDAR-Inertial-Camera odometry, Coco-LIC, an incremental photo-realistic mapping system is proposed in this paper. We initialize 3D Gaussians from colorized LiDAR points and optimize them using differentiable rendering powered by 3D Gaussian Splatting. Meticulously designed strategies are employed to incrementally expand the Gaussian map and adaptively control its density, ensuring high-quality mapping with real-time capability. Experiments conducted in diverse scenarios demonstrate the superior performance of our method compared to existing radiance-field-based SLAM systems.
△ Less
Submitted 10 April, 2024;
originally announced April 2024.
-
DeepLINK-T: deep learning inference for time series data using knockoffs and LSTM
Authors:
Wenxuan Zuo,
Zifan Zhu,
Yuxuan Du,
Yi-Chun Yeh,
Jed A. Fuhrman,
Jinchi Lv,
Yingying Fan,
Fengzhu Sun
Abstract:
High-dimensional longitudinal time series data is prevalent across various real-world applications. Many such applications can be modeled as regression problems with high-dimensional time series covariates. Deep learning has been a popular and powerful tool for fitting these regression models. Yet, the development of interpretable and reproducible deep-learning models is challenging and remains un…
▽ More
High-dimensional longitudinal time series data is prevalent across various real-world applications. Many such applications can be modeled as regression problems with high-dimensional time series covariates. Deep learning has been a popular and powerful tool for fitting these regression models. Yet, the development of interpretable and reproducible deep-learning models is challenging and remains underexplored. This study introduces a novel method, Deep Learning Inference using Knockoffs for Time series data (DeepLINK-T), focusing on the selection of significant time series variables in regression while controlling the false discovery rate (FDR) at a predetermined level. DeepLINK-T combines deep learning with knockoff inference to control FDR in feature selection for time series models, accommodating a wide variety of feature distributions. It addresses dependencies across time and features by leveraging a time-varying latent factor structure in time series covariates. Three key ingredients for DeepLINK-T are 1) a Long Short-Term Memory (LSTM) autoencoder for generating time series knockoff variables, 2) an LSTM prediction network using both original and knockoff variables, and 3) the application of the knockoffs framework for variable selection with FDR control. Extensive simulation studies have been conducted to evaluate DeepLINK-T's performance, showing its capability to control FDR effectively while demonstrating superior feature selection power for high-dimensional longitudinal time series data compared to its non-time series counterpart. DeepLINK-T is further applied to three metagenomic data sets, validating its practical utility and effectiveness, and underscoring its potential in real-world applications.
△ Less
Submitted 5 April, 2024;
originally announced April 2024.
-
Concept -- An Evaluation Protocol on Conversational Recommender Systems with System-centric and User-centric Factors
Authors:
Chen Huang,
Peixin Qin,
Yang Deng,
Wenqiang Lei,
Jiancheng Lv,
Tat-Seng Chua
Abstract:
The conversational recommendation system (CRS) has been criticized regarding its user experience in real-world scenarios, despite recent significant progress achieved in academia. Existing evaluation protocols for CRS may prioritize system-centric factors such as effectiveness and fluency in conversation while neglecting user-centric aspects. Thus, we propose a new and inclusive evaluation protoco…
▽ More
The conversational recommendation system (CRS) has been criticized regarding its user experience in real-world scenarios, despite recent significant progress achieved in academia. Existing evaluation protocols for CRS may prioritize system-centric factors such as effectiveness and fluency in conversation while neglecting user-centric aspects. Thus, we propose a new and inclusive evaluation protocol, Concept, which integrates both system- and user-centric factors. We conceptualise three key characteristics in representing such factors and further divide them into six primary abilities. To implement Concept, we adopt a LLM-based user simulator and evaluator with scoring rubrics that are tailored for each primary ability. Our protocol, Concept, serves a dual purpose. First, it provides an overview of the pros and cons in current CRS models. Second, it pinpoints the problem of low usability in the "omnipotent" ChatGPT and offers a comprehensive reference guide for evaluating CRS, thereby setting the foundation for CRS improvement.
△ Less
Submitted 6 May, 2024; v1 submitted 4 April, 2024;
originally announced April 2024.
-
CSST Strong Lensing Preparation: a Framework for Detecting Strong Lenses in the Multi-color Imaging Survey by the China Survey Space Telescope (CSST)
Authors:
Xu Li,
Ruiqi Sun,
Jiameng Lv,
Peng Jia,
Nan Li,
Chengliang Wei,
Zou Hu,
Xinzhong Er,
Yun Chen,
Zhang Ban,
Yuedong Fang,
Qi Guo,
Dezi Liu,
Guoliang Li,
Lin Lin,
Ming Li,
Ran Li,
Xiaobo Li,
Yu Luo,
Xianmin Meng,
Jundan Nie,
Zhaoxiang Qi,
Yisheng Qiu,
Li Shao,
Hao Tian
, et al. (7 additional authors not shown)
Abstract:
Strong gravitational lensing is a powerful tool for investigating dark matter and dark energy properties. With the advent of large-scale sky surveys, we can discover strong lensing systems on an unprecedented scale, which requires efficient tools to extract them from billions of astronomical objects. The existing mainstream lens-finding tools are based on machine learning algorithms and applied to…
▽ More
Strong gravitational lensing is a powerful tool for investigating dark matter and dark energy properties. With the advent of large-scale sky surveys, we can discover strong lensing systems on an unprecedented scale, which requires efficient tools to extract them from billions of astronomical objects. The existing mainstream lens-finding tools are based on machine learning algorithms and applied to cut-out-centered galaxies. However, according to the design and survey strategy of optical surveys by CSST, preparing cutouts with multiple bands requires considerable efforts. To overcome these challenges, we have developed a framework based on a hierarchical visual Transformer with a sliding window technique to search for strong lensing systems within entire images. Moreover, given that multi-color images of strong lensing systems can provide insights into their physical characteristics, our framework is specifically crafted to identify strong lensing systems in images with any number of channels. As evaluated using CSST mock data based on an Semi-Analytic Model named CosmoDC2, our framework achieves precision and recall rates of 0.98 and 0.90, respectively. To evaluate the effectiveness of our method in real observations, we have applied it to a subset of images from the DESI Legacy Imaging Surveys and media images from Euclid Early Release Observations. 61 new strong lensing system candidates are discovered by our method. However, we also identified false positives arising primarily from the simplified galaxy morphology assumptions within the simulation. This underscores the practical limitations of our approach while simultaneously highlighting potential avenues for future improvements.
△ Less
Submitted 2 April, 2024;
originally announced April 2024.
-
Space Group Informed Transformer for Crystalline Materials Generation
Authors:
Zhendong Cao,
Xiaoshan Luo,
Jian Lv,
Lei Wang
Abstract:
We introduce CrystalFormer, a transformer-based autoregressive model specifically designed for space group-controlled generation of crystalline materials. The space group symmetry significantly simplifies the crystal space, which is crucial for data and compute efficient generative modeling of crystalline materials. Leveraging the prominent discrete and sequential nature of the Wyckoff positions,…
▽ More
We introduce CrystalFormer, a transformer-based autoregressive model specifically designed for space group-controlled generation of crystalline materials. The space group symmetry significantly simplifies the crystal space, which is crucial for data and compute efficient generative modeling of crystalline materials. Leveraging the prominent discrete and sequential nature of the Wyckoff positions, CrystalFormer learns to generate crystals by directly predicting the species and locations of symmetry-inequivalent atoms in the unit cell. Our results demonstrate that CrystalFormer matches state-of-the-art performance on standard benchmarks for both validity, novelty, and stability of the generated crystalline materials. Our analysis also shows that CrystalFormer ingests sensible solid-state chemistry information from data for generative modeling. The CrystalFormer unifies symmetry-based structure search and generative pre-training in the realm of crystalline materials. The simplicity, generality, and flexibility of CrystalFormer position it as a promising architecture to be the foundational model of the entire crystalline materials space, heralding a new era in materials modeling and discovery.
△ Less
Submitted 23 March, 2024;
originally announced March 2024.
-
Genetic Auto-prompt Learning for Pre-trained Code Intelligence Language Models
Authors:
Chengzhe Feng,
Yanan Sun,
Ke Li,
Pan Zhou,
Jiancheng Lv,
Aojun Lu
Abstract:
As Pre-trained Language Models (PLMs), a popular approach for code intelligence, continue to grow in size, the computational cost of their usage has become prohibitively expensive. Prompt learning, a recent development in the field of natural language processing, emerges as a potential solution to address this challenge. In this paper, we investigate the effectiveness of prompt learning in code in…
▽ More
As Pre-trained Language Models (PLMs), a popular approach for code intelligence, continue to grow in size, the computational cost of their usage has become prohibitively expensive. Prompt learning, a recent development in the field of natural language processing, emerges as a potential solution to address this challenge. In this paper, we investigate the effectiveness of prompt learning in code intelligence tasks. We unveil its reliance on manually designed prompts, which often require significant human effort and expertise. Moreover, we discover existing automatic prompt design methods are very limited to code intelligence tasks due to factors including gradient dependence, high computational demands, and limited applicability. To effectively address both issues, we propose Genetic Auto Prompt (GenAP), which utilizes an elaborate genetic algorithm to automatically design prompts. With GenAP, non-experts can effortlessly generate superior prompts compared to meticulously manual-designed ones. GenAP operates without the need for gradients or additional computational costs, rendering it gradient-free and cost-effective. Moreover, GenAP supports both understanding and generation types of code intelligence tasks, exhibiting great applicability. We conduct GenAP on three popular code intelligence PLMs with three canonical code intelligence tasks including defect prediction, code summarization, and code translation. The results suggest that GenAP can effectively automate the process of designing prompts. Specifically, GenAP outperforms all other methods across all three tasks (e.g., improving accuracy by an average of 2.13% for defect prediction). To the best of our knowledge, GenAP is the first work to automatically design prompts for code intelligence PLMs.
△ Less
Submitted 20 March, 2024;
originally announced March 2024.
-
High Performance Graphene Integrated Photonics Platform Enabled by Gold-assisted Transfer
Authors:
Xiaoxuan Wu,
Zhengyi Cao,
Tianxiang Zhao,
Yun Wu,
Zhonghui Li,
Spyros Doukas,
Elefterios Lidorikis,
Yu Xue,
Liu Liu,
Omid Ghaebi,
Giancarlo Soavi,
Junpeng Lv,
Zhenghua Ni,
Junjia Wang
Abstract:
Graphene is promising for nanoscale, efficient, ultra-fast photo- and opto-electronic devices because of its remarkable electrical and optical properties, such as fast electron relaxation and heat dissipation. Here, we realize high-performance graphene integrated photonics platform enabled by gold-assisted transfer. Thanks to our optimized transfer technique, we fabricate and demonstrate (1) a mic…
▽ More
Graphene is promising for nanoscale, efficient, ultra-fast photo- and opto-electronic devices because of its remarkable electrical and optical properties, such as fast electron relaxation and heat dissipation. Here, we realize high-performance graphene integrated photonics platform enabled by gold-assisted transfer. Thanks to our optimized transfer technique, we fabricate and demonstrate (1) a microscale thermo-optic modulator with a tuning efficiency of 0.037 nm/mW and high heating performance of 67.4 K$μm^{3}mW^{-1}$ on a small active area of 7.54 $μm^{2}$ and (2) a graphene electro-absorption modulator featuring an high modulation bandwidth up to 26.8 GHz and a high-speed data rate reaching 48 Gb/s, and (3) a graphene Mach-Zehnder interferometer modulator with a high normalized modulation efficiency of 0.027 dBV$^{-1}μm^{-1}$. Our graphene integrated photonics platform has far superior performances compared to state of the art in terms of efficiency, low process complexity, and compact device footage. Thus, our approach and results provide the background for the realization of high-performance integrated photonic circuits with CMOS compatibility.
△ Less
Submitted 17 March, 2024;
originally announced March 2024.
-
Deep learning generative model for crystal structure prediction
Authors:
Xiaoshan Luo,
Zhenyu Wang,
Pengyue Gao,
Jian Lv,
Yanchao Wang,
Changfeng Chen,
Yanming Ma
Abstract:
Recent advances in deep learning generative models (GMs) have created unprecedented high capabilities in accessing and assessing complex high-dimensional data, allowing superior efficiency in navigating vast material configuration space in search of viable structures. Coupling such capabilities with physically significant data to construct trained models for materials discovery is crucial to movin…
▽ More
Recent advances in deep learning generative models (GMs) have created unprecedented high capabilities in accessing and assessing complex high-dimensional data, allowing superior efficiency in navigating vast material configuration space in search of viable structures. Coupling such capabilities with physically significant data to construct trained models for materials discovery is crucial to moving this emerging field forward. Here, we present a universal GM for crystal structure prediction (CSP) via a conditional crystal diffusion variational autoencoder (Cond-CDVAE) approach, which is tailored to allow user-defined material and physical parameters such as composition and pressure. This model is trained on an expansive dataset containing over 670,000 local minimum structures, including a rich spectrum of high-pressure structures, along with ambient-pressure structures in Materials Project database. We demonstrate that the Cond-CDVAE model can generate physically plausible structures with high fidelity under diverse pressure conditions without necessitating local optimization, accurately predicting 59.3% of the 3,547 unseen ambient-pressure experimental structures within 800 structure samplings, with the accuracy rate climbing to 83.2% for structures comprising fewer than 20 atoms per unit cell. These results meet or exceed those achieved via conventional CSP methods based on global optimization. The present findings showcase substantial potential of GMs in the realm of CSP.
△ Less
Submitted 16 March, 2024;
originally announced March 2024.
-
An Empirical Study of Parameter Efficient Fine-tuning on Vision-Language Pre-train Model
Authors:
Yuxin Tian,
Mouxing Yang,
Yunfan Li,
Dayiheng Liu,
Xingzhang Ren,
Xi Peng,
Jiancheng Lv
Abstract:
Recent studies applied Parameter Efficient Fine-Tuning techniques (PEFTs) to efficiently narrow the performance gap between pre-training and downstream. There are two important factors for various PEFTs, namely, the accessible data size and fine-tunable parameter size. A natural expectation for PEFTs is that the performance of various PEFTs is positively related to the data size and fine-tunable p…
▽ More
Recent studies applied Parameter Efficient Fine-Tuning techniques (PEFTs) to efficiently narrow the performance gap between pre-training and downstream. There are two important factors for various PEFTs, namely, the accessible data size and fine-tunable parameter size. A natural expectation for PEFTs is that the performance of various PEFTs is positively related to the data size and fine-tunable parameter size. However, according to the evaluation of five PEFTs on two downstream vision-language (VL) tasks, we find that such an intuition holds only if the downstream data and task are not consistent with pre-training. For downstream fine-tuning consistent with pre-training, data size no longer affects the performance, while the influence of fine-tunable parameter size is not monotonous. We believe such an observation could guide the choice of training strategy for various PEFTs.
△ Less
Submitted 18 May, 2024; v1 submitted 13 March, 2024;
originally announced March 2024.
-
Zero-Shot Aerial Object Detection with Visual Description Regularization
Authors:
Zhengqing Zang,
Chenyu Lin,
Chenwei Tang,
Tao Wang,
Jiancheng Lv
Abstract:
Existing object detection models are mainly trained on large-scale labeled datasets. However, annotating data for novel aerial object classes is expensive since it is time-consuming and may require expert knowledge. Thus, it is desirable to study label-efficient object detection methods on aerial images. In this work, we propose a zero-shot method for aerial object detection named visual Descripti…
▽ More
Existing object detection models are mainly trained on large-scale labeled datasets. However, annotating data for novel aerial object classes is expensive since it is time-consuming and may require expert knowledge. Thus, it is desirable to study label-efficient object detection methods on aerial images. In this work, we propose a zero-shot method for aerial object detection named visual Description Regularization, or DescReg. Concretely, we identify the weak semantic-visual correlation of the aerial objects and aim to address the challenge with prior descriptions of their visual appearance. Instead of directly encoding the descriptions into class embedding space which suffers from the representation gap problem, we propose to infuse the prior inter-class visual similarity conveyed in the descriptions into the embedding learning. The infusion process is accomplished with a newly designed similarity-aware triplet loss which incorporates structured regularization on the representation space. We conduct extensive experiments with three challenging aerial object detection datasets, including DIOR, xView, and DOTA. The results demonstrate that DescReg significantly outperforms the state-of-the-art ZSD methods with complex projection designs and generative frameworks, e.g., DescReg outperforms best reported ZSD method on DIOR by 4.5 mAP on unseen classes and 8.1 in HM. We further show the generalizability of DescReg by integrating it into generative ZSD methods as well as varying the detection architecture.
△ Less
Submitted 1 March, 2024; v1 submitted 28 February, 2024;
originally announced February 2024.
-
Diffusion Model-Based Image Editing: A Survey
Authors:
Yi Huang,
Jiancheng Huang,
Yifan Liu,
Mingfu Yan,
Jiaxi Lv,
Jianzhuang Liu,
Wei Xiong,
He Zhang,
Shifeng Chen,
Liangliang Cao
Abstract:
Denoising diffusion models have emerged as a powerful tool for various image generation and editing tasks, facilitating the synthesis of visual content in an unconditional or input-conditional manner. The core idea behind them is learning to reverse the process of gradually adding noise to images, allowing them to generate high-quality samples from a complex distribution. In this survey, we provid…
▽ More
Denoising diffusion models have emerged as a powerful tool for various image generation and editing tasks, facilitating the synthesis of visual content in an unconditional or input-conditional manner. The core idea behind them is learning to reverse the process of gradually adding noise to images, allowing them to generate high-quality samples from a complex distribution. In this survey, we provide an exhaustive overview of existing methods using diffusion models for image editing, covering both theoretical and practical aspects in the field. We delve into a thorough analysis and categorization of these works from multiple perspectives, including learning strategies, user-input conditions, and the array of specific editing tasks that can be accomplished. In addition, we pay special attention to image inpainting and outpainting, and explore both earlier traditional context-driven and current multimodal conditional methods, offering a comprehensive analysis of their methodologies. To further evaluate the performance of text-guided image editing algorithms, we propose a systematic benchmark, EditEval, featuring an innovative metric, LMM Score. Finally, we address current limitations and envision some potential directions for future research. The accompanying repository is released at https://github.com/SiatMMLab/Awesome-Diffusion-Model-Based-Image-Editing-Methods.
△ Less
Submitted 16 March, 2024; v1 submitted 27 February, 2024;
originally announced February 2024.
-
Diffusion Posterior Proximal Sampling for Image Restoration
Authors:
Hongjie Wu,
Linchao He,
Mingqin Zhang,
Dongdong Chen,
Kunming Luo,
Mengting Luo,
Ji-Zhe Zhou,
Hu Chen,
Jiancheng Lv
Abstract:
Diffusion models have demonstrated remarkable efficacy in generating high-quality samples. Existing diffusion-based image restoration algorithms exploit pre-trained diffusion models to leverage data priors, yet they still preserve elements inherited from the unconditional generation paradigm. These strategies initiate the denoising process with pure white noise and incorporate random noise at each…
▽ More
Diffusion models have demonstrated remarkable efficacy in generating high-quality samples. Existing diffusion-based image restoration algorithms exploit pre-trained diffusion models to leverage data priors, yet they still preserve elements inherited from the unconditional generation paradigm. These strategies initiate the denoising process with pure white noise and incorporate random noise at each generative step, leading to over-smoothed results. In this paper, we introduce a refined paradigm for diffusion-based image restoration. Specifically, we opt for a sample consistent with the measurement identity at each generative step, exploiting the sampling selection as an avenue for output stability and enhancement. Besides, we start the restoration process with an initialization combined with the measurement signal, providing supplementary information to better align the generative process. Extensive experimental results and analyses validate the effectiveness of our proposed approach across diverse image restoration tasks.
△ Less
Submitted 24 February, 2024;
originally announced February 2024.
-
Ultra-short lifetime isomer studies from photonuclear reactions using laser-driven ultra-intense γ-ray
Authors:
Di Wu,
Haoyang Lan,
Jiaxing Liu,
Huangang Lu,
Jianyao Zhang,
Jianfeng Lv,
Xuezhi Wu,
Hui Zhang,
Yadong Xia,
Qiangyou He,
Jie Cai,
Qianyi Ma,
Yuhui Xia,
Zhenan Wang,
Meizhi Wang,
Zhiyan Yang,
Xinlu Xu,
Yixing Geng,
Chen Lin,
Wenjun Ma,
Yanying Zhao,
Haoran Wang,
Fulong Liu,
Chuangye He,
Jinqing Yu
, et al. (7 additional authors not shown)
Abstract:
Isomers, ubiquitous populations of relatively long-lived nuclear excited states, play a crucial role in nuclear physics. However, isomers with half-life times of several seconds or less barely had experimental cross section data due to the lack of a suitable measuring method. We report a method of online γ spectroscopy for ultra-short-lived isomers from photonuclear reactions using laser-driven ul…
▽ More
Isomers, ubiquitous populations of relatively long-lived nuclear excited states, play a crucial role in nuclear physics. However, isomers with half-life times of several seconds or less barely had experimental cross section data due to the lack of a suitable measuring method. We report a method of online γ spectroscopy for ultra-short-lived isomers from photonuclear reactions using laser-driven ultra-intense γ-rays. The fastest time resolution can reach sub-ps level with γ-ray intensities >10^{19}/s ({\geqslant} 8 MeV). The ^{115}In(γ, n)^{114m2}In reaction (T_{1/2} = 43.1 ms) was first measured in the high-energy region which shed light on the nuclear structure studies of In element. Simulations showed it would be an efficient way to study ^{229m}Th (T_{1/2} = 7 μs), which is believed to be the next generation of nuclear clock. This work offered a unique way of gaining insight into ultra-short lifetimes and promised an effective way to fill the gap in relevant experimental data.
△ Less
Submitted 23 February, 2024;
originally announced February 2024.
-
Brain-inspired Distributed Memorization Learning for Efficient Feature-free Unsupervised Domain Adaptation
Authors:
Jianming Lv,
Depin Liang,
Zequan Liang,
Yaobin Zhang,
Sijun Xia
Abstract:
Compared with gradient based artificial neural networks, biological neural networks usually show a more powerful generalization ability to quickly adapt to unknown environments without using any gradient back-propagation procedure. Inspired by the distributed memory mechanism of human brains, we propose a novel gradient-free Distributed Memorization Learning mechanism, namely DML, to support quick…
▽ More
Compared with gradient based artificial neural networks, biological neural networks usually show a more powerful generalization ability to quickly adapt to unknown environments without using any gradient back-propagation procedure. Inspired by the distributed memory mechanism of human brains, we propose a novel gradient-free Distributed Memorization Learning mechanism, namely DML, to support quick domain adaptation of transferred models. In particular, DML adopts randomly connected neurons to memorize the association of input signals, which are propagated as impulses, and makes the final decision by associating the distributed memories based on their confidence. More importantly, DML is able to perform reinforced memorization based on unlabeled data to quickly adapt to a new domain without heavy fine-tuning of deep features, which makes it very suitable for deploying on edge devices. Experiments based on four cross-domain real-world datasets show that DML can achieve superior performance of real-time domain adaptation compared with traditional gradient based MLP with more than 10% improvement of accuracy while reducing 87% of the timing cost of optimization.
△ Less
Submitted 4 February, 2024;
originally announced February 2024.
-
Global quark spin correlations in relativistic heavy ion collisions
Authors:
Ji-peng Lv,
Zi-han Yu,
Zuo-tang Liang,
Qun Wang,
Xin-Nian Wang
Abstract:
The observation of the vector meson's global spin alignment by the STAR Collaboration reveals that strong spin correlations may exist for quarks and antiquarks in relativistic heavy-ion collisions in the normal direction of the reaction plane. We propose a systematic method to describe such correlations in the quark matter. The correlations can be classified as local and long range types. We show…
▽ More
The observation of the vector meson's global spin alignment by the STAR Collaboration reveals that strong spin correlations may exist for quarks and antiquarks in relativistic heavy-ion collisions in the normal direction of the reaction plane. We propose a systematic method to describe such correlations in the quark matter. The correlations can be classified as local and long range types. We show in particular that the effective quark spin correlations contain the genuine spin correlations originated directly from the dynamical process as well as those induced by averaging over other degrees of freedom. We also show that such correlations can be studied by measuring the vector meson's spin density matrix and hyperon-hyperon and hyperon-anti-hyperon spin correlations. We present the relationships between these measurable quantities and spin correlations of quarks and antiquarks.
△ Less
Submitted 25 February, 2024; v1 submitted 21 February, 2024;
originally announced February 2024.
-
Unlocking Insights: Semantic Search in Jupyter Notebooks
Authors:
Lan Li,
Jinpeng Lv
Abstract:
Semantic search, a process aimed at delivering highly relevant search results by comprehending the searcher's intent and the contextual meaning of terms within a searchable dataspace, plays a pivotal role in information retrieval. In this paper, we investigate the application of large language models to enhance semantic search capabilities, specifically tailored for the domain of Jupyter Notebooks…
▽ More
Semantic search, a process aimed at delivering highly relevant search results by comprehending the searcher's intent and the contextual meaning of terms within a searchable dataspace, plays a pivotal role in information retrieval. In this paper, we investigate the application of large language models to enhance semantic search capabilities, specifically tailored for the domain of Jupyter Notebooks. Our objective is to retrieve generated outputs, such as figures or tables, associated functions and methods, and other pertinent information.
We demonstrate a semantic search framework that achieves a comprehensive semantic understanding of the entire notebook's contents, enabling it to effectively handle various types of user queries. Key components of this framework include:
1). A data preprocessor is designed to handle diverse types of cells within Jupyter Notebooks, encompassing both markdown and code cells. 2). An innovative methodology is devised to address token size limitations that arise with code-type cells. We implement a finer-grained approach to data input, transitioning from the cell level to the function level, effectively resolving these issues.
△ Less
Submitted 20 February, 2024;
originally announced February 2024.
-
EasyFS: an Efficient Model-free Feature Selection Framework via Elastic Transformation of Features
Authors:
Jianming Lv,
Sijun Xia,
Depin Liang,
Wei Chen
Abstract:
Traditional model-free feature selection methods treat each feature independently while disregarding the interrelationships among features, which leads to relatively poor performance compared with the model-aware methods. To address this challenge, we propose an efficient model-free feature selection framework via elastic expansion and compression of the features, namely EasyFS, to achieve better…
▽ More
Traditional model-free feature selection methods treat each feature independently while disregarding the interrelationships among features, which leads to relatively poor performance compared with the model-aware methods. To address this challenge, we propose an efficient model-free feature selection framework via elastic expansion and compression of the features, namely EasyFS, to achieve better performance than state-of-the-art model-aware methods while sharing the characters of efficiency and flexibility with the existing model-free methods. In particular, EasyFS expands the feature space by using the random non-linear projection network to achieve the non-linear combinations of the original features, so as to model the interrelationships among the features and discover most correlated features. Meanwhile, a novel redundancy measurement based on the change of coding rate is proposed for efficient filtering of redundant features. Comprehensive experiments on 21 different datasets show that EasyFS outperforms state-of-the art methods up to 10.9\% in the regression tasks and 5.7\% in the classification tasks while saving more than 94\% of the time.
△ Less
Submitted 4 February, 2024;
originally announced February 2024.
-
Cross-Space Adaptive Filter: Integrating Graph Topology and Node Attributes for Alleviating the Over-smoothing Problem
Authors:
Chen Huang,
Haoyang Li,
Yifan Zhang,
Wenqiang Lei,
Jiancheng Lv
Abstract:
The vanilla Graph Convolutional Network (GCN) uses a low-pass filter to extract low-frequency signals from graph topology, which may lead to the over-smoothing problem when GCN goes deep. To this end, various methods have been proposed to create an adaptive filter by incorporating an extra filter (e.g., a high-pass filter) extracted from the graph topology. However, these methods heavily rely on t…
▽ More
The vanilla Graph Convolutional Network (GCN) uses a low-pass filter to extract low-frequency signals from graph topology, which may lead to the over-smoothing problem when GCN goes deep. To this end, various methods have been proposed to create an adaptive filter by incorporating an extra filter (e.g., a high-pass filter) extracted from the graph topology. However, these methods heavily rely on topological information and ignore the node attribute space, which severely sacrifices the expressive power of the deep GCNs, especially when dealing with disassortative graphs. In this paper, we propose a cross-space adaptive filter, called CSF, to produce the adaptive-frequency information extracted from both the topology and attribute spaces. Specifically, we first derive a tailored attribute-based high-pass filter that can be interpreted theoretically as a minimizer for semi-supervised kernel ridge regression. Then, we cast the topology-based low-pass filter as a Mercer's kernel within the context of GCNs. This serves as a foundation for combining it with the attribute-based filter to capture the adaptive-frequency information. Finally, we derive the cross-space filter via an effective multiple-kernel learning strategy, which unifies the attribute-based high-pass filter and the topology-based low-pass filter. This helps to address the over-smoothing problem while maintaining effectiveness. Extensive experiments demonstrate that CSF not only successfully alleviates the over-smoothing problem but also promotes the effectiveness of the node classification task.
△ Less
Submitted 10 February, 2024; v1 submitted 26 January, 2024;
originally announced January 2024.
-
Ultra-broadband near-field Josephson microwave microscopy
Authors:
Ping Zhang,
Yang-Yang Lyu,
Jingjing Lv,
Zihan Wei,
Shixian Chen,
Chenguang Wang,
Hongmei Du,
Dingding Li,
Zixi Wang,
Shoucheng Hou,
Runfeng Su,
Hancong Sun,
Yuan Du,
Li Du,
Liming Gao,
Yong-Lei Wang,
Huabing Wang,
Peiheng Wu
Abstract:
Advanced microwave technologies constitute the foundation of a wide range of modern sciences, including quantum computing, microwave photonics, spintronics, etc. To facilitate the design of chip-based microwave devices, there is an increasing demand for state-of-the-art microscopic techniques capable of characterizing the near-field microwave distribution and performance. In this work, we integrat…
▽ More
Advanced microwave technologies constitute the foundation of a wide range of modern sciences, including quantum computing, microwave photonics, spintronics, etc. To facilitate the design of chip-based microwave devices, there is an increasing demand for state-of-the-art microscopic techniques capable of characterizing the near-field microwave distribution and performance. In this work, we integrate Josephson junctions onto a nano-sized quartz tip, forming a highly sensitive microwave mixer on-tip. This allows us to conduct spectroscopic imaging of near-field microwave distributions with high spatial resolution. Leveraging its microwave-sensitive characteristics, our Josephson microscope achieves a broad detecting bandwidth of up to 200 GHz with remarkable frequency and intensity sensitivities. Our work emphasizes the benefits of utilizing the Josephson microscope as a real-time, non-destructive technique to advance integrated microwave electronics.
△ Less
Submitted 23 January, 2024;
originally announced January 2024.
-
DREditor: An Time-efficient Approach for Building a Domain-specific Dense Retrieval Model
Authors:
Chen Huang,
Duanyu Feng,
Wenqiang Lei,
Jiancheng Lv
Abstract:
Deploying dense retrieval models efficiently is becoming increasingly important across various industries. This is especially true for enterprise search services, where customizing search engines to meet the time demands of different enterprises in different domains is crucial. Motivated by this, we develop a time-efficient approach called DREditor to edit the matching rule of an off-the-shelf den…
▽ More
Deploying dense retrieval models efficiently is becoming increasingly important across various industries. This is especially true for enterprise search services, where customizing search engines to meet the time demands of different enterprises in different domains is crucial. Motivated by this, we develop a time-efficient approach called DREditor to edit the matching rule of an off-the-shelf dense retrieval model to suit a specific domain. This is achieved by directly calibrating the output embeddings of the model using an efficient and effective linear mapping. This mapping is powered by an edit operator that is obtained by solving a specially constructed least squares problem. Compared to implicit rule modification via long-time finetuning, our experimental results show that DREditor provides significant advantages on different domain-specific datasets, dataset sources, retrieval models, and computing devices. It consistently enhances time efficiency by 100-300 times while maintaining comparable or even superior retrieval performance. In a broader context, we take the first step to introduce a novel embedding calibration approach for the retrieval task, filling the technical blank in the current field of embedding calibration. This approach also paves the way for building domain-specific dense retrieval models efficiently and inexpensively.
△ Less
Submitted 23 January, 2024;
originally announced January 2024.
-
Importance-Aware Image Segmentation-based Semantic Communication for Autonomous Driving
Authors:
Jie Lv,
Haonan Tong,
Qiang Pan,
Zhilong Zhang,
Xinxin He,
Tao Luo,
Changchuan Yin
Abstract:
This article studies the problem of image segmentation-based semantic communication in autonomous driving. In real traffic scenes, detecting the key objects (e.g., vehicles, pedestrians and obstacles) is more crucial than that of other objects to guarantee driving safety. Therefore, we propose a vehicular image segmentation-oriented semantic communication system, termed VIS-SemCom, where image seg…
▽ More
This article studies the problem of image segmentation-based semantic communication in autonomous driving. In real traffic scenes, detecting the key objects (e.g., vehicles, pedestrians and obstacles) is more crucial than that of other objects to guarantee driving safety. Therefore, we propose a vehicular image segmentation-oriented semantic communication system, termed VIS-SemCom, where image segmentation features of important objects are transmitted to reduce transmission redundancy. First, to accurately extract image semantics, we develop a semantic codec based on Swin Transformer architecture, which expands the perceptual field thus improving the segmentation accuracy. Next, we propose a multi-scale semantic extraction scheme via assigning the number of Swin Transformer blocks for diverse resolution features, thus highlighting the important objects' accuracy. Furthermore, the importance-aware loss is invoked to emphasize the important objects, and an online hard sample mining (OHEM) strategy is proposed to handle small sample issues in the dataset. Experimental results demonstrate that the proposed VIS-SemCom can achieve a coding gain of nearly 6 dB with a 60% mean intersection over union (mIoU), reduce the transmitted data amount by up to 70% with a 60% mIoU, and improve the segmentation intersection over union (IoU) of important objects by 4%, compared to traditional transmission scheme.
△ Less
Submitted 16 January, 2024;
originally announced January 2024.
-
See the Unseen: Better Context-Consistent Knowledge-Editing by Noises
Authors:
Youcheng Huang,
Wenqiang Lei,
Zheng Zhang,
Jiancheng Lv,
Shuicheng Yan
Abstract:
Knowledge-editing updates knowledge of large language models (LLMs) and contributes to the interpretability and application of LLMs. However, knowledge applying is context-consistent: LLMs can recall the same knowledge in different contexts. Existing works ignore this property and the editing lacks generalization. In this paper, we empirically find that the effects of different contexts upon LLMs…
▽ More
Knowledge-editing updates knowledge of large language models (LLMs) and contributes to the interpretability and application of LLMs. However, knowledge applying is context-consistent: LLMs can recall the same knowledge in different contexts. Existing works ignore this property and the editing lacks generalization. In this paper, we empirically find that the effects of different contexts upon LLMs in recalling the same knowledge follow a Gaussian-like distribution. We then sample Gaussian noises to simulate the effects of different contexts when updating LLMs. By such, we can make LLMs see the unseen contexts where the edited knowledge will be applied, therefore improving the editing generalization. Experimental results on three LLMs demonstrate the effectiveness of our methods and also distinguish our methods from the others of fine-tuning LLMs by noises.
△ Less
Submitted 17 January, 2024; v1 submitted 15 January, 2024;
originally announced January 2024.
-
DPA-2: Towards a universal large atomic model for molecular and material simulation
Authors:
Duo Zhang,
Xinzijian Liu,
Xiangyu Zhang,
Chengqian Zhang,
Chun Cai,
Hangrui Bi,
Yiming Du,
Xuejian Qin,
Jiameng Huang,
Bowen Li,
Yifan Shan,
Jinzhe Zeng,
Yuzhi Zhang,
Siyuan Liu,
Yifan Li,
Junhan Chang,
Xinyan Wang,
Shuo Zhou,
Jianchuan Liu,
Xiaoshan Luo,
Zhenyu Wang,
Wanrun Jiang,
Jing Wu,
Yudi Yang,
Jiyuan Yang
, et al. (17 additional authors not shown)
Abstract:
The rapid development of artificial intelligence (AI) is driving significant changes in the field of atomic modeling, simulation, and design. AI-based potential energy models have been successfully used to perform large-scale and long-time simulations with the accuracy of ab initio electronic structure methods. However, the model generation process still hinders applications at scale. We envision…
▽ More
The rapid development of artificial intelligence (AI) is driving significant changes in the field of atomic modeling, simulation, and design. AI-based potential energy models have been successfully used to perform large-scale and long-time simulations with the accuracy of ab initio electronic structure methods. However, the model generation process still hinders applications at scale. We envision that the next stage would be a model-centric ecosystem, in which a large atomic model (LAM), pre-trained with as many atomic datasets as possible and can be efficiently fine-tuned and distilled to downstream tasks, would serve the new infrastructure of the field of molecular modeling. We propose DPA-2, a novel architecture for a LAM, and develop a comprehensive pipeline for model fine-tuning, distillation, and application, associated with automatic workflows. We show that DPA-2 can accurately represent a diverse range of chemical systems and materials, enabling high-quality simulations and predictions with significantly reduced efforts compared to traditional methods. Our approach paves the way for a universal large atomic model that can be widely applied in molecular and material simulation research, opening new opportunities for scientific discoveries and industrial applications.
△ Less
Submitted 24 December, 2023;
originally announced December 2023.
-
Critical quantum geometric tensors of parametrically-driven nonlinear resonators
Authors:
Hao-Long Zhang,
Jia-Hao Lv,
Ken Chen,
Xue-Jia Yu,
Fan Wu,
Zhen-Biao Yang,
Shi-Biao Zheng
Abstract:
Parametrically driven nonlinear resonators represent a building block for realizing fault-tolerant quantum computation and are useful for critical quantum sensing. From a fundamental viewpoint, the most intriguing feature of such a system is perhaps the critical phenomena, which can occur without interaction with any other quantum system. The non-analytic behaviors of its eigenspectrum have been s…
▽ More
Parametrically driven nonlinear resonators represent a building block for realizing fault-tolerant quantum computation and are useful for critical quantum sensing. From a fundamental viewpoint, the most intriguing feature of such a system is perhaps the critical phenomena, which can occur without interaction with any other quantum system. The non-analytic behaviors of its eigenspectrum have been substantially investigated, but those associated with the ground state wavefunction have largely remained unexplored. Using the quantum ground state geometric tensor as an indicator, we comprehensively establish a phase diagram involving the driving parameter $\varepsilon$ and phase $φ$. The results reveal that with the increase in $\varepsilon$, the system undergoes a quantum phase transition from the normal to the superradiant phase, with the critical point unaffected by $φ$. Furthermore, the critical exponent and scaling dimension are obtained by an exact numerical method, which is consistent with previous works. Our numerical results show that the phase transition falls within the universality class of the quantum Rabi model. This work reveals that the quantum metric and Berry curvature display diverging behaviors across the quantum phase transition.
△ Less
Submitted 21 December, 2023;
originally announced December 2023.
-
Generate E-commerce Product Background by Integrating Category Commonality and Personalized Style
Authors:
Haohan Wang,
Wei Feng,
Yang Lu,
Yaoyu Li,
Zheng Zhang,
Jingjing Lv,
Xin Zhu,
Junjie Shen,
Zhangang Lin,
Lixing Bo,
Jingping Shao
Abstract:
The state-of-the-art methods for e-commerce product background generation suffer from the inefficiency of designing product-wise prompts when scaling up the production, as well as the ineffectiveness of describing fine-grained styles when customizing personalized backgrounds for some specific brands. To address these obstacles, we integrate the category commonality and personalized style into diff…
▽ More
The state-of-the-art methods for e-commerce product background generation suffer from the inefficiency of designing product-wise prompts when scaling up the production, as well as the ineffectiveness of describing fine-grained styles when customizing personalized backgrounds for some specific brands. To address these obstacles, we integrate the category commonality and personalized style into diffusion models. Concretely, we propose a Category-Wise Generator to enable large-scale background generation for the first time. A unique identifier in the prompt is assigned to each category, whose attention is located on the background by a mask-guided cross attention layer to learn the category-wise style. Furthermore, for products with specific and fine-grained requirements in layout, elements, etc, a Personality-Wise Generator is devised to learn such personalized style directly from a reference image to resolve textual ambiguities, and is trained in a self-supervised manner for more efficient training data usage. To advance research in this field, the first large-scale e-commerce product background generation dataset BG60k is constructed, which covers more than 60k product images from over 2k categories. Experiments demonstrate that our method could generate high-quality backgrounds for different categories, and maintain the personalized background style of reference images. The link to BG60k and codes will be available soon.
△ Less
Submitted 19 December, 2023;
originally announced December 2023.
-
Near-Optimal Resilient Aggregation Rules for Distributed Learning Using 1-Center and 1-Mean Clustering with Outliers
Authors:
Yuhao Yi,
Ronghui You,
Hong Liu,
Changxin Liu,
Yuan Wang,
Jiancheng Lv
Abstract:
Byzantine machine learning has garnered considerable attention in light of the unpredictable faults that can occur in large-scale distributed learning systems. The key to secure resilience against Byzantine machines in distributed learning is resilient aggregation mechanisms. Although abundant resilient aggregation rules have been proposed, they are designed in ad-hoc manners, imposing extra barri…
▽ More
Byzantine machine learning has garnered considerable attention in light of the unpredictable faults that can occur in large-scale distributed learning systems. The key to secure resilience against Byzantine machines in distributed learning is resilient aggregation mechanisms. Although abundant resilient aggregation rules have been proposed, they are designed in ad-hoc manners, imposing extra barriers on comparing, analyzing, and improving the rules across performance criteria. This paper studies near-optimal aggregation rules using clustering in the presence of outliers. Our outlier-robust clustering approach utilizes geometric properties of the update vectors provided by workers. Our analysis show that constant approximations to the 1-center and 1-mean clustering problems with outliers provide near-optimal resilient aggregators for metric-based criteria, which have been proven to be crucial in the homogeneous and heterogeneous cases respectively. In addition, we discuss two contradicting types of attacks under which no single aggregation rule is guaranteed to improve upon the naive average. Based on the discussion, we propose a two-phase resilient aggregation framework. We run experiments for image classification using a non-convex loss function. The proposed algorithms outperform previously known aggregation rules by a large margin with both homogeneous and heterogeneous data distributions among non-faulty workers. Code and appendix are available at https://github.com/jerry907/AAAI24-RASHB.
△ Less
Submitted 31 March, 2024; v1 submitted 20 December, 2023;
originally announced December 2023.
-
Planning and Rendering: Towards End-to-End Product Poster Generation
Authors:
Zhaochen Li,
Fengheng Li,
Wei Feng,
Honghe Zhu,
An Liu,
Yaoyu Li,
Zheng Zhang,
Jingjing Lv,
Xin Zhu,
Junjie Shen,
Zhangang Lin,
Jingping Shao,
Zhenglu Yang
Abstract:
End-to-end product poster generation significantly optimizes design efficiency and reduces production costs. Prevailing methods predominantly rely on image-inpainting methods to generate clean background images for given products. Subsequently, poster layout generation methods are employed to produce corresponding layout results. However, the background images may not be suitable for accommodating…
▽ More
End-to-end product poster generation significantly optimizes design efficiency and reduces production costs. Prevailing methods predominantly rely on image-inpainting methods to generate clean background images for given products. Subsequently, poster layout generation methods are employed to produce corresponding layout results. However, the background images may not be suitable for accommodating textual content due to their complexity, and the fixed location of products limits the diversity of layout results. To alleviate these issues, we propose a novel product poster generation framework named P\&R. The P\&R draws inspiration from the workflow of designers in creating posters, which consists of two stages: Planning and Rendering. At the planning stage, we propose a PlanNet to generate the layout of the product and other visual components considering both the appearance features of the product and semantic features of the text, which improves the diversity and rationality of the layouts. At the rendering stage, we propose a RenderNet to generate the background for the product while considering the generated layout, where a spatial fusion module is introduced to fuse the layout of different visual components. To foster the advancement of this field, we propose the first end-to-end product poster generation dataset PPG30k, comprising 30k exquisite product poster images along with comprehensive image and text annotations. Our method outperforms the state-of-the-art product poster generation methods on PPG30k. The PPG30k will be released soon.
△ Less
Submitted 14 December, 2023;
originally announced December 2023.
-
Towards Equipping Transformer with the Ability of Systematic Compositionality
Authors:
Chen Huang,
Peixin Qin,
Wenqiang Lei,
Jiancheng Lv
Abstract:
One of the key factors in language productivity and human cognition is the ability of systematic compositionality, which refers to understanding composed unseen examples of seen primitives. However, recent evidence reveals that the Transformers have difficulty generalizing the composed context based on the seen primitives. To this end, we take the first step to propose a compositionality-aware Tra…
▽ More
One of the key factors in language productivity and human cognition is the ability of systematic compositionality, which refers to understanding composed unseen examples of seen primitives. However, recent evidence reveals that the Transformers have difficulty generalizing the composed context based on the seen primitives. To this end, we take the first step to propose a compositionality-aware Transformer called CAT and two novel pre-training tasks to facilitate systematic compositionality. We tentatively provide a successful implementation of a multi-layer CAT on the basis of the especially popular BERT. The experimental results demonstrate that CAT outperforms baselines on compositionality-aware tasks with minimal impact on the effectiveness on standardized language understanding tasks.
△ Less
Submitted 12 December, 2023;
originally announced December 2023.
-
Jointly spatial-temporal representation learning for individual trajectories
Authors:
Fei Huang,
Jianrong Lv,
Yang Yue
Abstract:
Individual trajectories, rich in human-environment interaction information across space and time, serve as vital inputs for geospatial foundation models (GeoFMs). However, existing attempts at learning trajectory representations have overlooked the implicit spatial-temporal dependency within trajectories, failing to encode such dependency in a deep learning-friendly format. That poses a challenge…
▽ More
Individual trajectories, rich in human-environment interaction information across space and time, serve as vital inputs for geospatial foundation models (GeoFMs). However, existing attempts at learning trajectory representations have overlooked the implicit spatial-temporal dependency within trajectories, failing to encode such dependency in a deep learning-friendly format. That poses a challenge in obtaining general-purpose trajectory representations. Therefore, this paper proposes a spatial-temporal joint representation learning method (ST-GraphRL) to formalize learnable spatial-temporal dependencies into trajectory representations. The proposed ST-GraphRL consists of three compositions: (i) a weighted directed spatial-temporal graph to explicitly construct mobility interactions in both space and time dimensions; (ii) a two-stage jointly encoder (i.e., decoupling and fusion), to learn entangled spatial-temporal dependencies by independently decomposing and jointly aggregating space and time information; (iii) a decoder guides ST-GraphRL to learn explicit mobility regularities by simulating the spatial-temporal distributions of trajectories. Tested on three real-world human mobility datasets, the proposed ST-GraphRL outperformed all the baseline models in predicting movement spatial-temporal distributions and preserving trajectory similarity with high spatial-temporal correlations. Analyzing spatial-temporal features presented in latent space validates that ST-GraphRL understands spatial-temporal patterns. This study may also benefit representation learnings of other geospatial data to achieve general-purpose data representations and advance GeoFMs development.
△ Less
Submitted 11 December, 2023; v1 submitted 7 December, 2023;
originally announced December 2023.
-
RTQ: Rethinking Video-language Understanding Based on Image-text Model
Authors:
Xiao Wang,
Yaoyu Li,
Tian Gan,
Zheng Zhang,
Jingjing Lv,
Liqiang Nie
Abstract:
Recent advancements in video-language understanding have been established on the foundation of image-text models, resulting in promising outcomes due to the shared knowledge between images and videos. However, video-language understanding presents unique challenges due to the inclusion of highly complex semantic details, which result in information redundancy, temporal dependency, and scene comple…
▽ More
Recent advancements in video-language understanding have been established on the foundation of image-text models, resulting in promising outcomes due to the shared knowledge between images and videos. However, video-language understanding presents unique challenges due to the inclusion of highly complex semantic details, which result in information redundancy, temporal dependency, and scene complexity. Current techniques have only partially tackled these issues, and our quantitative analysis indicates that some of these methods are complementary. In light of this, we propose a novel framework called RTQ (Refine, Temporal model, and Query), which addresses these challenges simultaneously. The approach involves refining redundant information within frames, modeling temporal relations among frames, and querying task-specific information from the videos. Remarkably, our model demonstrates outstanding performance even in the absence of video-language pre-training, and the results are comparable with or superior to those achieved by state-of-the-art pre-training methods. Code is available at https://github.com/SCZwangxiao/RTQ-MM2023.
△ Less
Submitted 17 December, 2023; v1 submitted 30 November, 2023;
originally announced December 2023.
-
Perception of Misalignment States for Sky Survey Telescopes with the Digital Twin and the Deep Neural Networks
Authors:
Miao Zhang,
Peng Jia,
Zhengyang Li,
Wennan Xiang,
Jiameng Lv,
Rui Sun
Abstract:
Sky survey telescopes play a critical role in modern astronomy, but misalignment of their optical elements can introduce significant variations in point spread functions, leading to reduced data quality. To address this, we need a method to obtain misalignment states, aiding in the reconstruction of accurate point spread functions for data processing methods or facilitating adjustments of optical…
▽ More
Sky survey telescopes play a critical role in modern astronomy, but misalignment of their optical elements can introduce significant variations in point spread functions, leading to reduced data quality. To address this, we need a method to obtain misalignment states, aiding in the reconstruction of accurate point spread functions for data processing methods or facilitating adjustments of optical components for improved image quality. Since sky survey telescopes consist of many optical elements, they result in a vast array of potential misalignment states, some of which are intricately coupled, posing detection challenges. However, by continuously adjusting the misalignment states of optical elements, we can disentangle coupled states. Based on this principle, we propose a deep neural network to extract misalignment states from continuously varying point spread functions in different field of views. To ensure sufficient and diverse training data, we recommend employing a digital twin to obtain data for neural network training. Additionally, we introduce the state graph to store misalignment data and explore complex relationships between misalignment states and corresponding point spread functions, guiding the generation of training data from experiments. Once trained, the neural network estimates misalignment states from observation data, regardless of the impacts caused by atmospheric turbulence, noise, and limited spatial sampling rates in the detector. The method proposed in this paper could be used to provide prior information for the active optics system and the optical system alignment.
△ Less
Submitted 29 November, 2023;
originally announced November 2023.