-
A Comprehensive Evaluation of Large Language Models on Temporal Event Forecasting
Authors:
He Chang,
Chenchen Ye,
Zhulin Tao,
Jie Wu,
Zhengmao Yang,
Yunshan Ma,
Xianglin Huang,
Tat-Seng Chua
Abstract:
Recently, Large Language Models (LLMs) have demonstrated great potential in various data mining tasks, such as knowledge question answering, mathematical reasoning, and commonsense reasoning. However, the reasoning capability of LLMs on temporal event forecasting has been under-explored. To systematically investigate their abilities in temporal event forecasting, we conduct a comprehensive evaluat…
▽ More
Recently, Large Language Models (LLMs) have demonstrated great potential in various data mining tasks, such as knowledge question answering, mathematical reasoning, and commonsense reasoning. However, the reasoning capability of LLMs on temporal event forecasting has been under-explored. To systematically investigate their abilities in temporal event forecasting, we conduct a comprehensive evaluation of LLM-based methods for temporal event forecasting. Due to the lack of a high-quality dataset that involves both graph and textual data, we first construct a benchmark dataset, named MidEast-TE-mini. Based on this dataset, we design a series of baseline methods, characterized by various input formats and retrieval augmented generation(RAG) modules. From extensive experiments, we find that directly integrating raw texts into the input of LLMs does not enhance zero-shot extrapolation performance. In contrast, incorporating raw texts in specific complex events and fine-tuning LLMs significantly improves performance. Moreover, enhanced with retrieval modules, LLM can effectively capture temporal relational patterns hidden in historical events. Meanwhile, issues such as popularity bias and the long-tail problem still persist in LLMs, particularly in the RAG-based method. These findings not only deepen our understanding of LLM-based event forecasting methods but also highlight several promising research directions.We consider that this comprehensive evaluation, along with the identified research opportunities, will significantly contribute to future research on temporal event forecasting through LLMs.
△ Less
Submitted 16 July, 2024;
originally announced July 2024.
-
Self-Guided Generation of Minority Samples Using Diffusion Models
Authors:
Soobin Um,
Jong Chul Ye
Abstract:
We present a novel approach for generating minority samples that live on low-density regions of a data manifold. Our framework is built upon diffusion models, leveraging the principle of guided sampling that incorporates an arbitrary energy-based guidance during inference time. The key defining feature of our sampler lies in its \emph{self-contained} nature, \ie, implementable solely with a pretra…
▽ More
We present a novel approach for generating minority samples that live on low-density regions of a data manifold. Our framework is built upon diffusion models, leveraging the principle of guided sampling that incorporates an arbitrary energy-based guidance during inference time. The key defining feature of our sampler lies in its \emph{self-contained} nature, \ie, implementable solely with a pretrained model. This distinguishes our sampler from existing techniques that require expensive additional components (like external classifiers) for minority generation. Specifically, we first estimate the likelihood of features within an intermediate latent sample by evaluating a reconstruction loss w.r.t. its posterior mean. The generation then proceeds with the minimization of the estimated likelihood, thereby encouraging the emergence of minority features in the latent samples of subsequent timesteps. To further improve the performance of our sampler, we provide several time-scheduling techniques that properly manage the influence of guidance over inference steps. Experiments on benchmark real datasets demonstrate that our approach can greatly improve the capability of creating realistic low-likelihood minority instances over the existing techniques without the reliance on costly additional elements. Code is available at \url{https://github.com/soobin-um/sg-minority}.
△ Less
Submitted 16 July, 2024;
originally announced July 2024.
-
Genomic Language Models: Opportunities and Challenges
Authors:
Gonzalo Benegas,
Chengzhong Ye,
Carlos Albors,
Jianan Canal Li,
Yun S. Song
Abstract:
Large language models (LLMs) are having transformative impacts across a wide range of scientific fields, particularly in the biomedical sciences. Just as the goal of Natural Language Processing is to understand sequences of words, a major objective in biology is to understand biological sequences. Genomic Language Models (gLMs), which are LLMs trained on DNA sequences, have the potential to signif…
▽ More
Large language models (LLMs) are having transformative impacts across a wide range of scientific fields, particularly in the biomedical sciences. Just as the goal of Natural Language Processing is to understand sequences of words, a major objective in biology is to understand biological sequences. Genomic Language Models (gLMs), which are LLMs trained on DNA sequences, have the potential to significantly advance our understanding of genomes and how DNA elements at various scales interact to give rise to complex functions. In this review, we showcase this potential by highlighting key applications of gLMs, including fitness prediction, sequence design, and transfer learning. Despite notable recent progress, however, developing effective and efficient gLMs presents numerous challenges, especially for species with large, complex genomes. We discuss major considerations for developing and evaluating gLMs.
△ Less
Submitted 16 July, 2024;
originally announced July 2024.
-
(Deep) Generative Geodesics
Authors:
Beomsu Kim,
Michael Puthawala,
Jong Chul Ye,
Emanuele Sansone
Abstract:
In this work, we propose to study the global geometrical properties of generative models. We introduce a new Riemannian metric to assess the similarity between any two data points. Importantly, our metric is agnostic to the parametrization of the generative model and requires only the evaluation of its data likelihood. Moreover, the metric leads to the conceptual definition of generative distances…
▽ More
In this work, we propose to study the global geometrical properties of generative models. We introduce a new Riemannian metric to assess the similarity between any two data points. Importantly, our metric is agnostic to the parametrization of the generative model and requires only the evaluation of its data likelihood. Moreover, the metric leads to the conceptual definition of generative distances and generative geodesics, whose computation can be done efficiently in the data space. Their approximations are proven to converge to their true values under mild conditions. We showcase three proof-of-concept applications of this global metric, including clustering, data visualization, and data interpolation, thus providing new tools to support the geometrical understanding of generative models.
△ Less
Submitted 15 July, 2024;
originally announced July 2024.
-
A Network Lens on Social Costs: Demolishing a Historic Street for a New Subway Station
Authors:
Xiaofan Liang,
Lu Chen,
Manying Lyu,
Yun Tian,
Changdong Ye
Abstract:
Urban redevelopment often involves a delicate balance between enhancing regional connectivity and preserving local social fabric. Through a case study in Guangzhou, China, we argue that demolishing a historic street to construct a new subway station shows competing interests between local government's priority to facilitate spatial connectivity and locals' priority to maintain a place for social i…
▽ More
Urban redevelopment often involves a delicate balance between enhancing regional connectivity and preserving local social fabric. Through a case study in Guangzhou, China, we argue that demolishing a historic street to construct a new subway station shows competing interests between local government's priority to facilitate spatial connectivity and locals' priority to maintain a place for social interaction and memories. We measure the social costs of the new subway station through a network lens, focusing on the loss of social ties and memories and low travel benefits of the new station for the local populations. We find that 1) the demolition will remove many small businesses that support locals' daily activities, social ties, and memories, and 2) the new station reduces travel distance and increases route options for passengers from other areas of the city more than locals nearby the demolition site. Our results contribute to a network-based framework and methodology to understand and contest inequality in expanding transportation network infrastructure in cities.
△ Less
Submitted 15 July, 2024;
originally announced July 2024.
-
Deep Diffusion Image Prior for Efficient OOD Adaptation in 3D Inverse Problems
Authors:
Hyungjin Chung,
Jong Chul Ye
Abstract:
Recent inverse problem solvers that leverage generative diffusion priors have garnered significant attention due to their exceptional quality. However, adaptation of the prior is necessary when there exists a discrepancy between the training and testing distributions. In this work, we propose deep diffusion image prior (DDIP), which generalizes the recent adaptation method of SCD by introducing a…
▽ More
Recent inverse problem solvers that leverage generative diffusion priors have garnered significant attention due to their exceptional quality. However, adaptation of the prior is necessary when there exists a discrepancy between the training and testing distributions. In this work, we propose deep diffusion image prior (DDIP), which generalizes the recent adaptation method of SCD by introducing a formal connection to the deep image prior. Under this framework, we propose an efficient adaptation method dubbed D3IP, specified for 3D measurements, which accelerates DDIP by orders of magnitude while achieving superior performance. D3IP enables seamless integration of 3D inverse solvers and thus leads to coherent 3D reconstruction. Moreover, we show that meta-learning techniques can also be applied to yield even better performance. We show that our method is capable of solving diverse 3D reconstructive tasks from the generative prior trained only with phantom images that are vastly different from the training set, opening up new opportunities of applying diffusion inverse solvers even when training with gold standard data is impossible. Code: https://github.com/HJ-harry/DDIP3D
△ Less
Submitted 15 July, 2024;
originally announced July 2024.
-
Fine-Grained Optimality of Partially Dynamic Shortest Paths and More
Authors:
Barna Saha,
Virginia Vassilevska Williams,
Yinzhan Xu,
Christopher Ye
Abstract:
Single Source Shortest Paths ($\textrm{SSSP}$) is among the most well-studied problems in computer science. In the incremental (resp. decremental) setting, the goal is to maintain distances from a fixed source in a graph undergoing edge insertions (resp. deletions). A long line of research culminated in a near-optimal deterministic $(1 + \varepsilon)$-approximate data structure with…
▽ More
Single Source Shortest Paths ($\textrm{SSSP}$) is among the most well-studied problems in computer science. In the incremental (resp. decremental) setting, the goal is to maintain distances from a fixed source in a graph undergoing edge insertions (resp. deletions). A long line of research culminated in a near-optimal deterministic $(1 + \varepsilon)$-approximate data structure with $m^{1 + o(1)}$ total update time over all $m$ updates by Bernstein, Probst Gutenberg and Saranurak [FOCS 2021]. However, there has been remarkably little progress on the exact $\textrm{SSSP}$ problem beyond Even and Shiloach's algorithm [J. ACM 1981] for unweighted graphs. For weighted graphs, there are no exact algorithms beyond recomputing $\textrm{SSSP}$ from scratch in $\widetilde{O}(m^2)$ total update time, even for the simpler Single-Source Single-Target Shortest Path problem ($\textrm{stSP}$). Despite this lack of progress, known (conditional) lower bounds only rule out algorithms with amortized update time better than $m^{1/2 - o(1)}$ in dense graphs.
In this paper, we give a tight (conditional) lower bound: any partially dynamic exact $\textrm{stSP}$ algorithm requires $m^{2 - o(1)}$ total update time for any sparsity $m$. We thus resolve the complexity of partially dynamic shortest paths, and separate the hardness of exact and approximate shortest paths, giving evidence as to why no non-trivial exact algorithms have been obtained while fast approximation algorithms are known.
Moreover, we give tight bounds on the complexity of combinatorial algorithms for several path problems that have been studied in the static setting since early sixties: Node-weighted shortest paths (studied alongside edge-weighted shortest paths), bottleneck paths (early work dates back to 1960), and earliest arrivals (early work dates back to 1958).
△ Less
Submitted 12 July, 2024;
originally announced July 2024.
-
Robust Multiscale Methods for Helmholtz equations in high contrast heterogeneous media
Authors:
Xingguang Jin,
Changqing Ye,
Eric T. Chung
Abstract:
In this paper, we provide the constraint energy minimization generalized multiscale finite element method (CEM-GMsFEM) to solve Helmholtz equations in heterogeneous medium. This novel multiscale method is specifically designed to overcome problems related to pollution effect, high-contrast coefficients, and the loss of hermiticity of operators. We establish the inf-sup stability and give an a prio…
▽ More
In this paper, we provide the constraint energy minimization generalized multiscale finite element method (CEM-GMsFEM) to solve Helmholtz equations in heterogeneous medium. This novel multiscale method is specifically designed to overcome problems related to pollution effect, high-contrast coefficients, and the loss of hermiticity of operators. We establish the inf-sup stability and give an a priori error estimate for this method under a number of established assumptions and resolution conditions. The theoretical results are validated by a set of numerical tests, which further show that the multiscale technique can effectively capture pertinent physical phenomena.
△ Less
Submitted 8 July, 2024; v1 submitted 5 July, 2024;
originally announced July 2024.
-
MIRAI: Evaluating LLM Agents for Event Forecasting
Authors:
Chenchen Ye,
Ziniu Hu,
Yihe Deng,
Zijie Huang,
Mingyu Derek Ma,
Yanqiao Zhu,
Wei Wang
Abstract:
Recent advancements in Large Language Models (LLMs) have empowered LLM agents to autonomously collect world information, over which to conduct reasoning to solve complex problems. Given this capability, increasing interests have been put into employing LLM agents for predicting international events, which can influence decision-making and shape policy development on an international scale. Despite…
▽ More
Recent advancements in Large Language Models (LLMs) have empowered LLM agents to autonomously collect world information, over which to conduct reasoning to solve complex problems. Given this capability, increasing interests have been put into employing LLM agents for predicting international events, which can influence decision-making and shape policy development on an international scale. Despite such a growing interest, there is a lack of a rigorous benchmark of LLM agents' forecasting capability and reliability. To address this gap, we introduce MIRAI, a novel benchmark designed to systematically evaluate LLM agents as temporal forecasters in the context of international events. Our benchmark features an agentic environment with tools for accessing an extensive database of historical, structured events and textual news articles. We refine the GDELT event database with careful cleaning and parsing to curate a series of relational prediction tasks with varying forecasting horizons, assessing LLM agents' abilities from short-term to long-term forecasting. We further implement APIs to enable LLM agents to utilize different tools via a code-based interface. In summary, MIRAI comprehensively evaluates the agents' capabilities in three dimensions: 1) autonomously source and integrate critical information from large global databases; 2) write codes using domain-specific APIs and libraries for tool-use; and 3) jointly reason over historical knowledge from diverse formats and time to accurately predict future events. Through comprehensive benchmarking, we aim to establish a reliable framework for assessing the capabilities of LLM agents in forecasting international events, thereby contributing to the development of more accurate and trustworthy models for international relation analysis.
△ Less
Submitted 1 July, 2024;
originally announced July 2024.
-
StableNormal: Reducing Diffusion Variance for Stable and Sharp Normal
Authors:
Chongjie Ye,
Lingteng Qiu,
Xiaodong Gu,
Qi Zuo,
Yushuang Wu,
Zilong Dong,
Liefeng Bo,
Yuliang Xiu,
Xiaoguang Han
Abstract:
This work addresses the challenge of high-quality surface normal estimation from monocular colored inputs (i.e., images and videos), a field which has recently been revolutionized by repurposing diffusion priors. However, previous attempts still struggle with stochastic inference, conflicting with the deterministic nature of the Image2Normal task, and costly ensembling step, which slows down the e…
▽ More
This work addresses the challenge of high-quality surface normal estimation from monocular colored inputs (i.e., images and videos), a field which has recently been revolutionized by repurposing diffusion priors. However, previous attempts still struggle with stochastic inference, conflicting with the deterministic nature of the Image2Normal task, and costly ensembling step, which slows down the estimation process. Our method, StableNormal, mitigates the stochasticity of the diffusion process by reducing inference variance, thus producing "Stable-and-Sharp" normal estimates without any additional ensembling process. StableNormal works robustly under challenging imaging conditions, such as extreme lighting, blurring, and low quality. It is also robust against transparent and reflective surfaces, as well as cluttered scenes with numerous objects. Specifically, StableNormal employs a coarse-to-fine strategy, which starts with a one-step normal estimator (YOSO) to derive an initial normal guess, that is relatively coarse but reliable, then followed by a semantic-guided refinement process (SG-DRN) that refines the normals to recover geometric details. The effectiveness of StableNormal is demonstrated through competitive performance in standard datasets such as DIODE-indoor, iBims, ScannetV2 and NYUv2, and also in various downstream tasks, such as surface reconstruction and normal enhancement. These results evidence that StableNormal retains both the "stability" and "sharpness" for accurate normal estimation. StableNormal represents a baby attempt to repurpose diffusion priors for deterministic estimation. To democratize this, code and models have been publicly available in hf.co/Stable-X
△ Less
Submitted 24 June, 2024;
originally announced June 2024.
-
A hybrid quantum-classical framework for computational fluid dynamics
Authors:
Chuang-Chao Ye,
Ning-Bo An,
Teng-Yang Ma,
Meng-Han Dou,
Wen Bai,
Zhao-Yun Chen,
Guo-Ping Guo
Abstract:
Great progress has been made in quantum computing in recent years, providing opportunities to overcome computation resource poverty in many scientific computations like computational fluid dynamics (CFD). In this work, efforts are made to exploit quantum potentialities in CFD, and a hybrid classical and quantum computing CFD framework is proposed to release the power of current quantum computing.…
▽ More
Great progress has been made in quantum computing in recent years, providing opportunities to overcome computation resource poverty in many scientific computations like computational fluid dynamics (CFD). In this work, efforts are made to exploit quantum potentialities in CFD, and a hybrid classical and quantum computing CFD framework is proposed to release the power of current quantum computing. In this framework, the traditional CFD solvers are coupled with quantum linear algebra libraries in weak form to achieve collaborative computation between classical and quantum computing. The quantum linear solver provides high-precision solutions and scalable problem sizes for linear systems and is designed to be easily callable for solving linear algebra systems similar to classical linear libraries, thus enabling seamless integration into existing CFD solvers. Some typical cases are performed to validate the feasibility of the proposed framework and the correctness of quantum linear algorithms in CFD.
△ Less
Submitted 24 June, 2024;
originally announced June 2024.
-
CAVM: Conditional Autoregressive Vision Model for Contrast-Enhanced Brain Tumor MRI Synthesis
Authors:
Lujun Gui,
Chuyang Ye,
Tianyi Yan
Abstract:
Contrast-enhanced magnetic resonance imaging (MRI) is pivotal in the pipeline of brain tumor segmentation and analysis. Gadolinium-based contrast agents, as the most commonly used contrast agents, are expensive and may have potential side effects, and it is desired to obtain contrast-enhanced brain tumor MRI scans without the actual use of contrast agents. Deep learning methods have been applied t…
▽ More
Contrast-enhanced magnetic resonance imaging (MRI) is pivotal in the pipeline of brain tumor segmentation and analysis. Gadolinium-based contrast agents, as the most commonly used contrast agents, are expensive and may have potential side effects, and it is desired to obtain contrast-enhanced brain tumor MRI scans without the actual use of contrast agents. Deep learning methods have been applied to synthesize virtual contrast-enhanced MRI scans from non-contrast images. However, as this synthesis problem is inherently ill-posed, these methods fall short in producing high-quality results. In this work, we propose Conditional Autoregressive Vision Model (CAVM) for improving the synthesis of contrast-enhanced brain tumor MRI. As the enhancement of image intensity grows with a higher dose of contrast agents, we assume that it is less challenging to synthesize a virtual image with a lower dose, where the difference between the contrast-enhanced and non-contrast images is smaller. Thus, CAVM gradually increases the contrast agent dosage and produces higher-dose images based on previous lower-dose ones until the final desired dose is achieved. Inspired by the resemblance between the gradual dose increase and the Chain-of-Thought approach in natural language processing, CAVM uses an autoregressive strategy with a decomposition tokenizer and a decoder. Specifically, the tokenizer is applied to obtain a more compact image representation for computational efficiency, and it decomposes the image into dose-variant and dose-invariant tokens. Then, a masked self-attention mechanism is developed for autoregression that gradually increases the dose of the virtual image based on the dose-variant tokens. Finally, the updated dose-variant tokens corresponding to the desired dose are decoded together with dose-invariant tokens to produce the final contrast-enhanced MRI.
△ Less
Submitted 23 June, 2024;
originally announced June 2024.
-
Split Federated Learning Empowered Vehicular Edge Intelligence: Adaptive Parellel Design and Future Directions
Authors:
Xianke Qiang,
Zheng Chang,
Chaoxiong Ye,
Timo Hamalainen,
Geyong Min
Abstract:
To realize ubiquitous intelligence of future vehicular networks, artificial intelligence (AI) is critical since it can mine knowledge from vehicular data to improve the quality of many AI driven vehicular services. By combining AI techniques with vehicular networks, Vehicular Edge Intelligence (VEI) can utilize the computing, storage, and communication resources of vehicles to train the AI models.…
▽ More
To realize ubiquitous intelligence of future vehicular networks, artificial intelligence (AI) is critical since it can mine knowledge from vehicular data to improve the quality of many AI driven vehicular services. By combining AI techniques with vehicular networks, Vehicular Edge Intelligence (VEI) can utilize the computing, storage, and communication resources of vehicles to train the AI models. Nevertheless, when executing the model training, the traditional centralized learning paradigm requires vehicles to upload their raw data to a central server, which results in significant communication overheads and the risk of privacy leakage. In this article, we first overview the system architectures, performance metrics and challenges ahead of VEI design. Then we propose to utilize distribute machine learning scheme, namely split federated learning (SFL), to boost the development of VEI. We present a novel adaptive and parellel SFL scheme and conduct corresponding analysis on its performance. Future research directions are highlighted to shed light on the efficient design of SFL.
△ Less
Submitted 27 June, 2024; v1 submitted 22 June, 2024;
originally announced June 2024.
-
The projected sensitivity of SCEP experiment to Magnetic Monopole
Authors:
Changqing Ye,
Beige Liu,
Zhe Cao,
Lingzhi Han,
Xinming Huang,
Min Jiang,
Dong Liu,
Qing Lin,
Shitian Wan,
Yusheng Wu,
Lei Zhao,
Yue Zhang,
Xinhua Peng,
Zhengguo Zhao
Abstract:
The investigation of beyond-Standard-Model particles is a compelling direction in the pursuit of new physics. One such hypothetical particle, the magnetic monopole, has garnered considerable attention due to its strong theoretical motivation and potential to unveil profound physical phenomena. The magnetic monopole is intricately linked to the long-standing enigma surrounding the quantization of e…
▽ More
The investigation of beyond-Standard-Model particles is a compelling direction in the pursuit of new physics. One such hypothetical particle, the magnetic monopole, has garnered considerable attention due to its strong theoretical motivation and potential to unveil profound physical phenomena. The magnetic monopole is intricately linked to the long-standing enigma surrounding the quantization of electric charge. In this manuscript, we propose a novel detection scenario for magnetic monopoles by employing a coincidence measurement technique that combines a room-temperature magnetometer with plastic scintillators. This setup allows for the collection of both the induction and scintillation signals generated by the passage of a monopole. The estimation of the sensitivity using a simple benchmark setup is given.
△ Less
Submitted 19 June, 2024; v1 submitted 18 June, 2024;
originally announced June 2024.
-
CliBench: Multifaceted Evaluation of Large Language Models in Clinical Decisions on Diagnoses, Procedures, Lab Tests Orders and Prescriptions
Authors:
Mingyu Derek Ma,
Chenchen Ye,
Yu Yan,
Xiaoxuan Wang,
Peipei Ping,
Timothy S Chang,
Wei Wang
Abstract:
The integration of Artificial Intelligence (AI), especially Large Language Models (LLMs), into the clinical diagnosis process offers significant potential to improve the efficiency and accessibility of medical care. While LLMs have shown some promise in the medical domain, their application in clinical diagnosis remains underexplored, especially in real-world clinical practice, where highly sophis…
▽ More
The integration of Artificial Intelligence (AI), especially Large Language Models (LLMs), into the clinical diagnosis process offers significant potential to improve the efficiency and accessibility of medical care. While LLMs have shown some promise in the medical domain, their application in clinical diagnosis remains underexplored, especially in real-world clinical practice, where highly sophisticated, patient-specific decisions need to be made. Current evaluations of LLMs in this field are often narrow in scope, focusing on specific diseases or specialties and employing simplified diagnostic tasks. To bridge this gap, we introduce CliBench, a novel benchmark developed from the MIMIC IV dataset, offering a comprehensive and realistic assessment of LLMs' capabilities in clinical diagnosis. This benchmark not only covers diagnoses from a diverse range of medical cases across various specialties but also incorporates tasks of clinical significance: treatment procedure identification, lab test ordering and medication prescriptions. Supported by structured output ontologies, CliBench enables a precise and multi-granular evaluation, offering an in-depth understanding of LLM's capability on diverse clinical tasks of desired granularity. We conduct a zero-shot evaluation of leading LLMs to assess their proficiency in clinical decision-making. Our preliminary results shed light on the potential and limitations of current LLMs in clinical settings, providing valuable insights for future advancements in LLM-powered healthcare.
△ Less
Submitted 14 June, 2024;
originally announced June 2024.
-
CFG++: Manifold-constrained Classifier Free Guidance for Diffusion Models
Authors:
Hyungjin Chung,
Jeongsol Kim,
Geon Yeong Park,
Hyelin Nam,
Jong Chul Ye
Abstract:
Classifier-free guidance (CFG) is a fundamental tool in modern diffusion models for text-guided generation. Although effective, CFG has notable drawbacks. For instance, DDIM with CFG lacks invertibility, complicating image editing; furthermore, high guidance scales, essential for high-quality outputs, frequently result in issues like mode collapse. Contrary to the widespread belief that these are…
▽ More
Classifier-free guidance (CFG) is a fundamental tool in modern diffusion models for text-guided generation. Although effective, CFG has notable drawbacks. For instance, DDIM with CFG lacks invertibility, complicating image editing; furthermore, high guidance scales, essential for high-quality outputs, frequently result in issues like mode collapse. Contrary to the widespread belief that these are inherent limitations of diffusion models, this paper reveals that the problems actually stem from the off-manifold phenomenon associated with CFG, rather than the diffusion models themselves. More specifically, inspired by the recent advancements of diffusion model-based inverse problem solvers (DIS), we reformulate text-guidance as an inverse problem with a text-conditioned score matching loss, and develop CFG++, a novel approach that tackles the off-manifold challenges inherent in traditional CFG. CFG++ features a surprisingly simple fix to CFG, yet it offers significant improvements, including better sample quality for text-to-image generation, invertibility, smaller guidance scales, reduced mode collapse, etc. Furthermore, CFG++ enables seamless interpolation between unconditional and conditional sampling at lower guidance scales, consistently outperforming traditional CFG at all scales. Experimental results confirm that our method significantly enhances performance in text-to-image generation, DDIM inversion, editing, and solving inverse problems, suggesting a wide-ranging impact and potential applications in various fields that utilize text guidance. Project Page: https://cfgpp-diffusion.github.io/.
△ Less
Submitted 12 June, 2024;
originally announced June 2024.
-
Enabling Large-Scale and High-Precision Fluid Simulations on Near-Term Quantum Computers
Authors:
Zhao-Yun Chen,
Teng-Yang Ma,
Chuang-Chao Ye,
Liang Xu,
Ming-Yang Tan,
Xi-Ning Zhuang,
Xiao-Fan Xu,
Yun-Jie Wang,
Tai-Ping Sun,
Yong Chen,
Lei Du,
Liang-Liang Guo,
Hai-Feng Zhang,
Hao-Ran Tao,
Tian-Le Wang,
Xiao-Yan Yang,
Ze-An Zhao,
Peng Wang,
Sheng Zhang,
Chi Zhang,
Ren-Ze Zhao,
Zhi-Long Jia,
Wei-Cheng Kong,
Meng-Han Dou,
Jun-Chao Wang
, et al. (7 additional authors not shown)
Abstract:
Quantum computational fluid dynamics (QCFD) offers a promising alternative to classical computational fluid dynamics (CFD) by leveraging quantum algorithms for higher efficiency. This paper introduces a comprehensive QCFD method, including an iterative method "Iterative-QLS" that suppresses error in quantum linear solver, and a subspace method to scale the solution to a larger size. We implement o…
▽ More
Quantum computational fluid dynamics (QCFD) offers a promising alternative to classical computational fluid dynamics (CFD) by leveraging quantum algorithms for higher efficiency. This paper introduces a comprehensive QCFD method, including an iterative method "Iterative-QLS" that suppresses error in quantum linear solver, and a subspace method to scale the solution to a larger size. We implement our method on a superconducting quantum computer, demonstrating successful simulations of steady Poiseuille flow and unsteady acoustic wave propagation. The Poiseuille flow simulation achieved a relative error of less than $0.2\%$, and the unsteady acoustic wave simulation solved a 5043-dimensional matrix. We emphasize the utilization of the quantum-classical hybrid approach in applications of near-term quantum computers. By adapting to quantum hardware constraints and offering scalable solutions for large-scale CFD problems, our method paves the way for practical applications of near-term quantum computers in computational science.
△ Less
Submitted 19 June, 2024; v1 submitted 10 June, 2024;
originally announced June 2024.
-
Discover Your Neighbors: Advanced Stable Test-Time Adaptation in Dynamic World
Authors:
Qinting Jiang,
Chuyang Ye,
Dongyan Wei,
Yuan Xue,
Jingyan Jiang,
Zhi Wang
Abstract:
Despite progress, deep neural networks still suffer performance declines under distribution shifts between training and test domains, leading to a substantial decrease in Quality of Experience (QoE) for multimedia applications. Existing test-time adaptation (TTA) methods are challenged by dynamic, multiple test distributions within batches. This work provides a new perspective on analyzing batch n…
▽ More
Despite progress, deep neural networks still suffer performance declines under distribution shifts between training and test domains, leading to a substantial decrease in Quality of Experience (QoE) for multimedia applications. Existing test-time adaptation (TTA) methods are challenged by dynamic, multiple test distributions within batches. This work provides a new perspective on analyzing batch normalization techniques through class-related and class-irrelevant features, our observations reveal combining source and test batch normalization statistics robustly characterizes target distributions. However, test statistics must have high similarity. We thus propose Discover Your Neighbours (DYN), the first backward-free approach specialized for dynamic TTA. The core innovation is identifying similar samples via instance normalization statistics and clustering into groups which provides consistent class-irrelevant representations. Specifically, Our DYN consists of layer-wise instance statistics clustering (LISC) and cluster-aware batch normalization (CABN). In LISC, we perform layer-wise clustering of approximate feature samples at each BN layer by calculating the cosine similarity of instance normalization statistics across the batch. CABN then aggregates SBN and TCN statistics to collaboratively characterize the target distribution, enabling more robust representations. Experimental results validate DYN's robustness and effectiveness, demonstrating maintained performance under dynamic data stream patterns.
△ Less
Submitted 8 June, 2024;
originally announced June 2024.
-
An iterative constraint energy minimizing generalized multiscale finite element method for contact problem
Authors:
Zishang Li,
Changqing Ye,
Eric T. Chung
Abstract:
This work presents an Iterative Constraint Energy Minimizing Generalized Multiscale Finite Element Method (ICEM-GMsFEM) for solving the contact problem with high contrast coefficients. The model problem can be characterized by a variational inequality, where we add a penalty term to convert this problem into a non-smooth and non-linear unconstrained minimizing problem. The characterization of the…
▽ More
This work presents an Iterative Constraint Energy Minimizing Generalized Multiscale Finite Element Method (ICEM-GMsFEM) for solving the contact problem with high contrast coefficients. The model problem can be characterized by a variational inequality, where we add a penalty term to convert this problem into a non-smooth and non-linear unconstrained minimizing problem. The characterization of the minimizer satisfies the variational form of a mixed Dirilect-Neumann-Robin boundary value problem. So we apply CEM-GMsFEM iteratively and introduce special boundary correctors along with multiscale spaces to achieve an optimal convergence rate. Numerical results are conducted for different highly heterogeneous permeability fields, validating the fast convergence of the CEM-GMsFEM iteration in handling the contact boundary and illustrating the stability of the proposed method with different sets of parameters. We also prove the fast convergence of the proposed iterative CEM-GMsFEM method and provide an error estimate of the multiscale solution under a mild assumption.
△ Less
Submitted 5 June, 2024;
originally announced June 2024.
-
E-ICL: Enhancing Fine-Grained Emotion Recognition through the Lens of Prototype Theory
Authors:
Zhou Yang,
Zhaochun Ren,
Chenglong Ye,
Yufeng Wang,
Haizhou Sun,
Chao Chen,
Xiaofei Zhu,
Yunbing Wu,
Xiangwen Liao
Abstract:
In-context learning (ICL) achieves remarkable performance in various domains such as knowledge acquisition, commonsense reasoning, and semantic understanding. However, its performance significantly deteriorates for emotion detection tasks, especially fine-grained emotion recognition. The underlying reasons for this remain unclear. In this paper, we identify the reasons behind ICL's poor performanc…
▽ More
In-context learning (ICL) achieves remarkable performance in various domains such as knowledge acquisition, commonsense reasoning, and semantic understanding. However, its performance significantly deteriorates for emotion detection tasks, especially fine-grained emotion recognition. The underlying reasons for this remain unclear. In this paper, we identify the reasons behind ICL's poor performance from the perspective of prototype theory and propose a method to address this issue. Specifically, we conduct extensive pilot experiments and find that ICL conforms to the prototype theory on fine-grained emotion recognition. Based on this theory, we uncover the following deficiencies in ICL: (1) It relies on prototypes (example-label pairs) that are semantically similar but emotionally inaccurate to predict emotions. (2) It is prone to interference from irrelevant categories, affecting the accuracy and robustness of the predictions. To address these issues, we propose an Emotion Context Learning method (E-ICL) on fine-grained emotion recognition. E-ICL relies on more emotionally accurate prototypes to predict categories by referring to emotionally similar examples with dynamic labels. Simultaneously, E-ICL employs an exclusionary emotion prediction strategy to avoid interference from irrelevant categories, thereby increasing its accuracy and robustness. Note that the entire process is accomplished with the assistance of a plug-and-play emotion auxiliary model, without additional training. Experiments on the fine-grained emotion datasets EDOS, Empathetic-Dialogues, EmpatheticIntent, and GoEmotions show that E-ICL achieves superior emotion prediction performance. Furthermore, even when the emotion auxiliary model used is lower than 10% of the LLMs, E-ICL can still boost the performance of LLMs by over 4% on multiple datasets.
△ Less
Submitted 4 June, 2024;
originally announced June 2024.
-
Replicability in High Dimensional Statistics
Authors:
Max Hopkins,
Russell Impagliazzo,
Daniel Kane,
Sihan Liu,
Christopher Ye
Abstract:
The replicability crisis is a major issue across nearly all areas of empirical science, calling for the formal study of replicability in statistics. Motivated in this context, [Impagliazzo, Lei, Pitassi, and Sorrell STOC 2022] introduced the notion of replicable learning algorithms, and gave basic procedures for $1$-dimensional tasks including statistical queries. In this work, we study the comput…
▽ More
The replicability crisis is a major issue across nearly all areas of empirical science, calling for the formal study of replicability in statistics. Motivated in this context, [Impagliazzo, Lei, Pitassi, and Sorrell STOC 2022] introduced the notion of replicable learning algorithms, and gave basic procedures for $1$-dimensional tasks including statistical queries. In this work, we study the computational and statistical cost of replicability for several fundamental high dimensional statistical tasks, including multi-hypothesis testing and mean estimation.
Our main contribution establishes a computational and statistical equivalence between optimal replicable algorithms and high dimensional isoperimetric tilings. As a consequence, we obtain matching sample complexity upper and lower bounds for replicable mean estimation of distributions with bounded covariance, resolving an open problem of [Bun, Gaboardi, Hopkins, Impagliazzo, Lei, Pitassi, Sivakumar, and Sorrell, STOC2023] and for the $N$-Coin Problem, resolving a problem of [Karbasi, Velegkas, Yang, and Zhou, NeurIPS2023] up to log factors.
While our equivalence is computational, allowing us to shave log factors in sample complexity from the best known efficient algorithms, efficient isoperimetric tilings are not known. To circumvent this, we introduce several relaxed paradigms that do allow for sample and computationally efficient algorithms, including allowing pre-processing, adaptivity, and approximate replicability. In these cases we give efficient algorithms matching or beating the best known sample complexity for mean estimation and the coin problem, including a generic procedure that reduces the standard quadratic overhead of replicability to linear in expectation.
△ Less
Submitted 3 June, 2024;
originally announced June 2024.
-
Analyzing Temporal Complex Events with Large Language Models? A Benchmark towards Temporal, Long Context Understanding
Authors:
Zhihan Zhang,
Yixin Cao,
Chenchen Ye,
Yunshan Ma,
Lizi Liao,
Tat-Seng Chua
Abstract:
The digital landscape is rapidly evolving with an ever-increasing volume of online news, emphasizing the need for swift and precise analysis of complex events. We refer to the complex events composed of many news articles over an extended period as Temporal Complex Event (TCE). This paper proposes a novel approach using Large Language Models (LLMs) to systematically extract and analyze the event c…
▽ More
The digital landscape is rapidly evolving with an ever-increasing volume of online news, emphasizing the need for swift and precise analysis of complex events. We refer to the complex events composed of many news articles over an extended period as Temporal Complex Event (TCE). This paper proposes a novel approach using Large Language Models (LLMs) to systematically extract and analyze the event chain within TCE, characterized by their key points and timestamps. We establish a benchmark, named TCELongBench, to evaluate the proficiency of LLMs in handling temporal dynamics and understanding extensive text. This benchmark encompasses three distinct tasks - reading comprehension, temporal sequencing, and future event forecasting. In the experiment, we leverage retrieval-augmented generation (RAG) method and LLMs with long context window to deal with lengthy news articles of TCE. Our findings indicate that models with suitable retrievers exhibit comparable performance with those utilizing long context window.
△ Less
Submitted 4 June, 2024;
originally announced June 2024.
-
Exploring Mathematical Extrapolation of Large Language Models with Synthetic Data
Authors:
Haolong Li,
Yu Ma,
Yinqi Zhang,
Chen Ye,
Jie Chen
Abstract:
Large Language Models (LLMs) have shown excellent performance in language understanding, text generation, code synthesis, and many other tasks, while they still struggle in complex multi-step reasoning problems, such as mathematical reasoning. In this paper, through a newly proposed arithmetical puzzle problem, we show that the model can perform well on multi-step reasoning tasks via fine-tuning o…
▽ More
Large Language Models (LLMs) have shown excellent performance in language understanding, text generation, code synthesis, and many other tasks, while they still struggle in complex multi-step reasoning problems, such as mathematical reasoning. In this paper, through a newly proposed arithmetical puzzle problem, we show that the model can perform well on multi-step reasoning tasks via fine-tuning on high-quality synthetic data. Experimental results with the open-llama-3B model on three different test datasets show that not only the model can reach a zero-shot pass@1 at 0.44 on the in-domain dataset, it also demonstrates certain generalization capabilities on the out-of-domain datasets. Specifically, this paper has designed two out-of-domain datasets in the form of extending the numerical range and the composing components of the arithmetical puzzle problem separately. The fine-tuned models have shown encouraging performance on these two far more difficult tasks with the zero-shot pass@1 at 0.33 and 0.35, respectively.
△ Less
Submitted 4 June, 2024;
originally announced June 2024.
-
Designing an Evaluation Framework for Large Language Models in Astronomy Research
Authors:
John F. Wu,
Alina Hyk,
Kiera McCormick,
Christine Ye,
Simone Astarita,
Elina Baral,
Jo Ciuca,
Jesse Cranney,
Anjalie Field,
Kartheik Iyer,
Philipp Koehn,
Jenn Kotler,
Sandor Kruk,
Michelle Ntampaka,
Charles O'Neill,
Joshua E. G. Peek,
Sanjib Sharma,
Mikaeel Yunus
Abstract:
Large Language Models (LLMs) are shifting how scientific research is done. It is imperative to understand how researchers interact with these models and how scientific sub-communities like astronomy might benefit from them. However, there is currently no standard for evaluating the use of LLMs in astronomy. Therefore, we present the experimental design for an evaluation study on how astronomy rese…
▽ More
Large Language Models (LLMs) are shifting how scientific research is done. It is imperative to understand how researchers interact with these models and how scientific sub-communities like astronomy might benefit from them. However, there is currently no standard for evaluating the use of LLMs in astronomy. Therefore, we present the experimental design for an evaluation study on how astronomy researchers interact with LLMs. We deploy a Slack chatbot that can answer queries from users via Retrieval-Augmented Generation (RAG); these responses are grounded in astronomy papers from arXiv. We record and anonymize user questions and chatbot answers, user upvotes and downvotes to LLM responses, user feedback to the LLM, and retrieved documents and similarity scores with the query. Our data collection method will enable future dynamic evaluations of LLM tools for astronomy.
△ Less
Submitted 30 May, 2024;
originally announced May 2024.
-
LDMol: Text-Conditioned Molecule Diffusion Model Leveraging Chemically Informative Latent Space
Authors:
Jinho Chang,
Jong Chul Ye
Abstract:
With the emergence of diffusion models as the frontline of generative models, many researchers have proposed molecule generation techniques using conditional diffusion models. However, due to the fundamental nature of a molecule, which carries highly entangled correlations within a small number of atoms and bonds, it becomes difficult for a model to connect raw data with the conditions when the co…
▽ More
With the emergence of diffusion models as the frontline of generative models, many researchers have proposed molecule generation techniques using conditional diffusion models. However, due to the fundamental nature of a molecule, which carries highly entangled correlations within a small number of atoms and bonds, it becomes difficult for a model to connect raw data with the conditions when the conditions become more complex as natural language. To address this, here we present a novel latent diffusion model dubbed LDMol, which enables a natural text-conditioned molecule generation. Specifically, LDMol is composed of three building blocks: a molecule encoder that produces a chemically informative feature space, a natural language-conditioned latent diffusion model using a Diffusion Transformer (DiT), and an autoregressive decoder for molecule re. In particular, recognizing that multiple SMILES notations can represent the same molecule, we employ a contrastive learning strategy to extract the chemical informative feature space. LDMol not only beats the existing baselines on the text-to-molecule generation benchmark but is also capable of zero-shot inference with unseen scenarios. Furthermore, we show that LDMol can be applied to downstream tasks such as molecule-to-text retrieval and text-driven molecule editing, demonstrating its versatility as a diffusion model.
△ Less
Submitted 28 May, 2024;
originally announced May 2024.
-
MindFormer: A Transformer Architecture for Multi-Subject Brain Decoding via fMRI
Authors:
Inhwa Han,
Jaayeon Lee,
Jong Chul Ye
Abstract:
Research efforts to understand neural signals have been ongoing for many years, with visual decoding from fMRI signals attracting considerable attention. Particularly, the advent of image diffusion models has advanced the reconstruction of images from fMRI data significantly. However, existing approaches often introduce inter- and intra- subject variations in the reconstructed images, which can co…
▽ More
Research efforts to understand neural signals have been ongoing for many years, with visual decoding from fMRI signals attracting considerable attention. Particularly, the advent of image diffusion models has advanced the reconstruction of images from fMRI data significantly. However, existing approaches often introduce inter- and intra- subject variations in the reconstructed images, which can compromise accuracy. To address current limitations in multi-subject brain decoding, we introduce a new Transformer architecture called MindFormer. This model is specifically designed to generate fMRI-conditioned feature vectors that can be used for conditioning Stable Diffusion model. More specifically, MindFormer incorporates two key innovations: 1) a novel training strategy based on the IP-Adapter to extract semantically meaningful features from fMRI signals, and 2) a subject specific token and linear layer that effectively capture individual differences in fMRI signals while synergistically combines multi subject fMRI data for training. Our experimental results demonstrate that Stable Diffusion, when integrated with MindFormer, produces semantically consistent images across different subjects. This capability significantly surpasses existing models in multi-subject brain decoding. Such advancements not only improve the accuracy of our reconstructions but also deepen our understanding of neural processing variations among individuals.
△ Less
Submitted 27 May, 2024;
originally announced May 2024.
-
Unified Editing of Panorama, 3D Scenes, and Videos Through Disentangled Self-Attention Injection
Authors:
Gihyun Kwon,
Jangho Park,
Jong Chul Ye
Abstract:
While text-to-image models have achieved impressive capabilities in image generation and editing, their application across various modalities often necessitates training separate models. Inspired by existing method of single image editing with self attention injection and video editing with shared attention, we propose a novel unified editing framework that combines the strengths of both approache…
▽ More
While text-to-image models have achieved impressive capabilities in image generation and editing, their application across various modalities often necessitates training separate models. Inspired by existing method of single image editing with self attention injection and video editing with shared attention, we propose a novel unified editing framework that combines the strengths of both approaches by utilizing only a basic 2D image text-to-image (T2I) diffusion model. Specifically, we design a sampling method that facilitates editing consecutive images while maintaining semantic consistency utilizing shared self-attention features during both reference and consecutive image sampling processes. Experimental results confirm that our method enables editing across diverse modalities including 3D scenes, videos, and panorama images.
△ Less
Submitted 27 May, 2024;
originally announced May 2024.
-
First-principles studies of fermiology in topological phases of bulk ZrTe$_5$
Authors:
Chao Chen Ye,
Yuliia Kreminska,
Jianting Ye,
Jagoda Sławińska
Abstract:
Topological insulators have been studied intensively over the last decades. Among these materials, three-dimensional (3D) zirconium pentatelluride (ZrTe$_5$) stands out as one of the most intriguing for both theoretical and experimental studies because of its diverse range of distinct topological phases. In this work, we employ density functional theory to study the electronic structure and quantu…
▽ More
Topological insulators have been studied intensively over the last decades. Among these materials, three-dimensional (3D) zirconium pentatelluride (ZrTe$_5$) stands out as one of the most intriguing for both theoretical and experimental studies because of its diverse range of distinct topological phases. In this work, we employ density functional theory to study the electronic structure and quantum oscillations exhibited by various topological phases of 3D bulk ZrTe$_5$. We have discovered that by analyzing combined patterns in band structures, Fermi surfaces, and Shubnikov-de Haas (SdH) oscillations we can determine the corresponding topological phase without relying on the conventional calculation of topological invariants or boundary state contributions. This approach facilitates the identification of topological phases in ZrTe$_5$ directly from experimental quantum oscillation measurements. Using this method, we have analyzed the entire process of topological phase transition, revealing changes in the topology of the Fermi pockets and validating the shapes deduced from the experimental data for the topological phases.
△ Less
Submitted 27 May, 2024; v1 submitted 24 May, 2024;
originally announced May 2024.
-
Biathlon: Harnessing Model Resilience for Accelerating ML Inference Pipelines
Authors:
Chaokun Chang,
Eric Lo,
Chunxiao Ye
Abstract:
Machine learning inference pipelines commonly encountered in data science and industries often require real-time responsiveness due to their user-facing nature. However, meeting this requirement becomes particularly challenging when certain input features require aggregating a large volume of data online. Recent literature on interpretable machine learning reveals that most machine learning models…
▽ More
Machine learning inference pipelines commonly encountered in data science and industries often require real-time responsiveness due to their user-facing nature. However, meeting this requirement becomes particularly challenging when certain input features require aggregating a large volume of data online. Recent literature on interpretable machine learning reveals that most machine learning models exhibit a notable degree of resilience to variations in input. This suggests that machine learning models can effectively accommodate approximate input features with minimal discernible impact on accuracy. In this paper, we introduce Biathlon, a novel ML serving system that leverages the inherent resilience of models and determines the optimal degree of approximation for each aggregation feature. This approach enables maximum speedup while ensuring a guaranteed bound on accuracy loss. We evaluate Biathlon on real pipelines from both industry applications and data science competitions, demonstrating its ability to meet real-time latency requirements by achieving 5.3x to 16.6x speedup with almost no accuracy loss.
△ Less
Submitted 18 May, 2024;
originally announced May 2024.
-
A Foundation Model for Brain Lesion Segmentation with Mixture of Modality Experts
Authors:
Xinru Zhang,
Ni Ou,
Berke Doga Basaran,
Marco Visentin,
Mengyun Qiao,
Renyang Gu,
Cheng Ouyang,
Yaou Liu,
Paul M. Matthew,
Chuyang Ye,
Wenjia Bai
Abstract:
Brain lesion segmentation plays an essential role in neurological research and diagnosis. As brain lesions can be caused by various pathological alterations, different types of brain lesions tend to manifest with different characteristics on different imaging modalities. Due to this complexity, brain lesion segmentation methods are often developed in a task-specific manner. A specific segmentation…
▽ More
Brain lesion segmentation plays an essential role in neurological research and diagnosis. As brain lesions can be caused by various pathological alterations, different types of brain lesions tend to manifest with different characteristics on different imaging modalities. Due to this complexity, brain lesion segmentation methods are often developed in a task-specific manner. A specific segmentation model is developed for a particular lesion type and imaging modality. However, the use of task-specific models requires predetermination of the lesion type and imaging modality, which complicates their deployment in real-world scenarios. In this work, we propose a universal foundation model for 3D brain lesion segmentation, which can automatically segment different types of brain lesions for input data of various imaging modalities. We formulate a novel Mixture of Modality Experts (MoME) framework with multiple expert networks attending to different imaging modalities. A hierarchical gating network combines the expert predictions and fosters expertise collaboration. Furthermore, we introduce a curriculum learning strategy during training to avoid the degeneration of each expert network and preserve their specialization. We evaluated the proposed method on nine brain lesion datasets, encompassing five imaging modalities and eight lesion types. The results show that our model outperforms state-of-the-art universal models and provides promising generalization to unseen datasets.
△ Less
Submitted 16 July, 2024; v1 submitted 16 May, 2024;
originally announced May 2024.
-
POWQMIX: Weighted Value Factorization with Potentially Optimal Joint Actions Recognition for Cooperative Multi-Agent Reinforcement Learning
Authors:
Chang Huang,
Junqiao Zhao,
Shatong Zhu,
Hongtu Zhou,
Chen Ye,
Tiantian Feng,
Changjun Jiang
Abstract:
Value function factorization methods are commonly used in cooperative multi-agent reinforcement learning, with QMIX receiving significant attention. Many QMIX-based methods introduce monotonicity constraints between the joint action value and individual action values to achieve decentralized execution. However, such constraints limit the representation capacity of value factorization, restricting…
▽ More
Value function factorization methods are commonly used in cooperative multi-agent reinforcement learning, with QMIX receiving significant attention. Many QMIX-based methods introduce monotonicity constraints between the joint action value and individual action values to achieve decentralized execution. However, such constraints limit the representation capacity of value factorization, restricting the joint action values it can represent and hindering the learning of the optimal policy. To address this challenge, we propose the Potentially Optimal joint actions Weighted QMIX (POWQMIX) algorithm, which recognizes the potentially optimal joint actions and assigns higher weights to the corresponding losses of these joint actions during training. We theoretically prove that with such a weighted training approach the optimal policy is guaranteed to be recovered. Experiments in matrix games, predator-prey, and StarCraft II Multi-Agent Challenge environments demonstrate that our algorithm outperforms the state-of-the-art value-based multi-agent reinforcement learning methods.
△ Less
Submitted 15 May, 2024; v1 submitted 12 May, 2024;
originally announced May 2024.
-
LOG-LIO2: A LiDAR-Inertial Odometry with Efficient Uncertainty Analysis
Authors:
Kai Huang,
Junqiao Zhao,
Jiaye Lin,
Zhongyang Zhu,
Shuangfu Song,
Chen Ye,
Tiantian Feng
Abstract:
Uncertainty in LiDAR measurements, stemming from factors such as range sensing, is crucial for LIO (LiDAR-Inertial Odometry) systems as it affects the accurate weighting in the loss function. While recent LIO systems address uncertainty related to range sensing, the impact of incident angle on uncertainty is often overlooked by the community. Moreover, the existing uncertainty propagation methods…
▽ More
Uncertainty in LiDAR measurements, stemming from factors such as range sensing, is crucial for LIO (LiDAR-Inertial Odometry) systems as it affects the accurate weighting in the loss function. While recent LIO systems address uncertainty related to range sensing, the impact of incident angle on uncertainty is often overlooked by the community. Moreover, the existing uncertainty propagation methods suffer from computational inefficiency. This paper proposes a comprehensive point uncertainty model that accounts for both the uncertainties from LiDAR measurements and surface characteristics, along with an efficient local uncertainty analytical method for LiDAR-based state estimation problem. We employ a projection operator that separates the uncertainty into the ray direction and its orthogonal plane. Then, we derive incremental Jacobian matrices of eigenvalues and eigenvectors w.r.t. points, which enables a fast approximation of uncertainty propagation. This approach eliminates the requirement for redundant traversal of points, significantly reducing the time complexity of uncertainty propagation from $\mathcal{O} (n)$ to $\mathcal{O} (1)$ when a new point is added. Simulations and experiments on public datasets are conducted to validate the accuracy and efficiency of our formulations. The proposed methods have been integrated into a LIO system, which is available at https://github.com/tiev-tongji/LOG-LIO2.
△ Less
Submitted 2 May, 2024;
originally announced May 2024.
-
MobileNetV4 -- Universal Models for the Mobile Ecosystem
Authors:
Danfeng Qin,
Chas Leichner,
Manolis Delakis,
Marco Fornoni,
Shixin Luo,
Fan Yang,
Weijun Wang,
Colby Banbury,
Chengxi Ye,
Berkin Akin,
Vaibhav Aggarwal,
Tenghui Zhu,
Daniele Moro,
Andrew Howard
Abstract:
We present the latest generation of MobileNets, known as MobileNetV4 (MNv4), featuring universally efficient architecture designs for mobile devices. At its core, we introduce the Universal Inverted Bottleneck (UIB) search block, a unified and flexible structure that merges Inverted Bottleneck (IB), ConvNext, Feed Forward Network (FFN), and a novel Extra Depthwise (ExtraDW) variant. Alongside UIB,…
▽ More
We present the latest generation of MobileNets, known as MobileNetV4 (MNv4), featuring universally efficient architecture designs for mobile devices. At its core, we introduce the Universal Inverted Bottleneck (UIB) search block, a unified and flexible structure that merges Inverted Bottleneck (IB), ConvNext, Feed Forward Network (FFN), and a novel Extra Depthwise (ExtraDW) variant. Alongside UIB, we present Mobile MQA, an attention block tailored for mobile accelerators, delivering a significant 39% speedup. An optimized neural architecture search (NAS) recipe is also introduced which improves MNv4 search effectiveness. The integration of UIB, Mobile MQA and the refined NAS recipe results in a new suite of MNv4 models that are mostly Pareto optimal across mobile CPUs, DSPs, GPUs, as well as specialized accelerators like Apple Neural Engine and Google Pixel EdgeTPU - a characteristic not found in any other models tested. Finally, to further boost accuracy, we introduce a novel distillation technique. Enhanced by this technique, our MNv4-Hybrid-Large model delivers 87% ImageNet-1K accuracy, with a Pixel 8 EdgeTPU runtime of just 3.8ms.
△ Less
Submitted 16 April, 2024;
originally announced April 2024.
-
Latent-based Diffusion Model for Long-tailed Recognition
Authors:
Pengxiao Han,
Changkun Ye,
Jieming Zhou,
Jing Zhang,
Jie Hong,
Xuesong Li
Abstract:
Long-tailed imbalance distribution is a common issue in practical computer vision applications. Previous works proposed methods to address this problem, which can be categorized into several classes: re-sampling, re-weighting, transfer learning, and feature augmentation. In recent years, diffusion models have shown an impressive generation ability in many sub-problems of deep computer vision. Howe…
▽ More
Long-tailed imbalance distribution is a common issue in practical computer vision applications. Previous works proposed methods to address this problem, which can be categorized into several classes: re-sampling, re-weighting, transfer learning, and feature augmentation. In recent years, diffusion models have shown an impressive generation ability in many sub-problems of deep computer vision. However, its powerful generation has not been explored in long-tailed problems. We propose a new approach, the Latent-based Diffusion Model for Long-tailed Recognition (LDMLR), as a feature augmentation method to tackle the issue. First, we encode the imbalanced dataset into features using the baseline model. Then, we train a Denoising Diffusion Implicit Model (DDIM) using these encoded features to generate pseudo-features. Finally, we train the classifier using the encoded and pseudo-features from the previous two steps. The model's accuracy shows an improvement on the CIFAR-LT and ImageNet-LT datasets by using the proposed method.
△ Less
Submitted 23 April, 2024; v1 submitted 6 April, 2024;
originally announced April 2024.
-
Concept Weaver: Enabling Multi-Concept Fusion in Text-to-Image Models
Authors:
Gihyun Kwon,
Simon Jenni,
Dingzeyu Li,
Joon-Young Lee,
Jong Chul Ye,
Fabian Caba Heilbron
Abstract:
While there has been significant progress in customizing text-to-image generation models, generating images that combine multiple personalized concepts remains challenging. In this work, we introduce Concept Weaver, a method for composing customized text-to-image diffusion models at inference time. Specifically, the method breaks the process into two steps: creating a template image aligned with t…
▽ More
While there has been significant progress in customizing text-to-image generation models, generating images that combine multiple personalized concepts remains challenging. In this work, we introduce Concept Weaver, a method for composing customized text-to-image diffusion models at inference time. Specifically, the method breaks the process into two steps: creating a template image aligned with the semantics of input prompts, and then personalizing the template using a concept fusion strategy. The fusion strategy incorporates the appearance of the target concepts into the template image while retaining its structural details. The results indicate that our method can generate multiple custom concepts with higher identity fidelity compared to alternative approaches. Furthermore, the method is shown to seamlessly handle more than two concepts and closely follow the semantic meaning of the input prompt without blending appearances across different subjects.
△ Less
Submitted 5 April, 2024;
originally announced April 2024.
-
Jailbreaking Prompt Attack: A Controllable Adversarial Attack against Diffusion Models
Authors:
Jiachen Ma,
Anda Cao,
Zhiqing Xiao,
Jie Zhang,
Chao Ye,
Junbo Zhao
Abstract:
Text-to-Image (T2I) models have received widespread attention due to their remarkable generation capabilities. However, concerns have been raised about the ethical implications of the models in generating Not Safe for Work (NSFW) images because NSFW images may cause discomfort to people or be used for illegal purposes. To mitigate the generation of such images, T2I models deploy various types of s…
▽ More
Text-to-Image (T2I) models have received widespread attention due to their remarkable generation capabilities. However, concerns have been raised about the ethical implications of the models in generating Not Safe for Work (NSFW) images because NSFW images may cause discomfort to people or be used for illegal purposes. To mitigate the generation of such images, T2I models deploy various types of safety checkers. However, they still cannot completely prevent the generation of NSFW images. In this paper, we propose the Jailbreak Prompt Attack (JPA) - an automatic attack framework. We aim to maintain prompts that bypass safety checkers while preserving the semantics of the original images. Specifically, we aim to find prompts that can bypass safety checkers because of the robustness of the text space. Our evaluation demonstrates that JPA successfully bypasses both online services with closed-box safety checkers and offline defenses safety checkers to generate NSFW images.
△ Less
Submitted 2 June, 2024; v1 submitted 2 April, 2024;
originally announced April 2024.
-
A fast cosine transformation accelerated method for predicting effective thermal conductivity
Authors:
Changqing Ye,
Shubin Fu,
Eric T. Chung
Abstract:
Predicting effective thermal conductivity by solving a Partial Differential Equation (PDE) defined on a high-resolution Representative Volume Element (RVE) is a computationally intensive task. In this paper, we tackle the task by proposing an efficient and implementation-friendly computational method that can fully leverage the computing power offered by hardware accelerators, namely, graphical pr…
▽ More
Predicting effective thermal conductivity by solving a Partial Differential Equation (PDE) defined on a high-resolution Representative Volume Element (RVE) is a computationally intensive task. In this paper, we tackle the task by proposing an efficient and implementation-friendly computational method that can fully leverage the computing power offered by hardware accelerators, namely, graphical processing units (GPUs). We first employ the Two-Point Flux-Approximation scheme to discretize the PDE and then utilize the preconditioned conjugate gradient method to solve the resulting algebraic linear system. The construction of the preconditioner originates from FFT-based homogenization methods, and an engineered linear programming technique is utilized to determine the homogeneous reference parameters. The fundamental observation presented in this paper is that the preconditioner system can be effectively solved using multiple Fast Cosine Transformations (FCT) and parallel tridiagonal matrix solvers. Regarding the fact that default multiple FCTs are unavailable on the CUDA platform, we detail how to derive FCTs from FFTs with nearly optimal memory usage. Numerical experiments including the stability comparison with standard preconditioners are conducted for 3D RVEs. Our performance reports indicate that the proposed method can achieve a $5$-fold acceleration on the GPU platform over the pure CPU platform and solve the problems with $512^3$ degrees of freedom and reasonable contrast ratios in less than $30$ seconds.
△ Less
Submitted 2 April, 2024;
originally announced April 2024.
-
GauStudio: A Modular Framework for 3D Gaussian Splatting and Beyond
Authors:
Chongjie Ye,
Yinyu Nie,
Jiahao Chang,
Yuantao Chen,
Yihao Zhi,
Xiaoguang Han
Abstract:
We present GauStudio, a novel modular framework for modeling 3D Gaussian Splatting (3DGS) to provide standardized, plug-and-play components for users to easily customize and implement a 3DGS pipeline. Supported by our framework, we propose a hybrid Gaussian representation with foreground and skyball background models. Experiments demonstrate this representation reduces artifacts in unbounded outdo…
▽ More
We present GauStudio, a novel modular framework for modeling 3D Gaussian Splatting (3DGS) to provide standardized, plug-and-play components for users to easily customize and implement a 3DGS pipeline. Supported by our framework, we propose a hybrid Gaussian representation with foreground and skyball background models. Experiments demonstrate this representation reduces artifacts in unbounded outdoor scenes and improves novel view synthesis. Finally, we propose Gaussian Splatting Surface Reconstruction (GauS), a novel render-then-fuse approach for high-fidelity mesh reconstruction from 3DGS inputs without fine-tuning. Overall, our GauStudio framework, hybrid representation, and GauS approach enhance 3DGS modeling and rendering capabilities, enabling higher-quality novel view synthesis and surface reconstruction.
△ Less
Submitted 28 March, 2024;
originally announced March 2024.
-
A robust two-level overlapping preconditioner for Darcy flow in high-contrast media
Authors:
Changqing Ye,
Shubin Fu,
Eric T. Chung,
Jizu Huang
Abstract:
In this article, a two-level overlapping domain decomposition preconditioner is developed for solving linear algebraic systems obtained from simulating Darcy flow in high-contrast media. Our preconditioner starts at a mixed finite element method for discretizing the partial differential equation by Darcy's law with the no-flux boundary condition and is then followed by a velocity elimination techn…
▽ More
In this article, a two-level overlapping domain decomposition preconditioner is developed for solving linear algebraic systems obtained from simulating Darcy flow in high-contrast media. Our preconditioner starts at a mixed finite element method for discretizing the partial differential equation by Darcy's law with the no-flux boundary condition and is then followed by a velocity elimination technique to yield a linear algebraic system with only unknowns of pressure. Then, our main objective is to design a robust and efficient domain decomposition preconditioner for this system, which is accomplished by engineering a multiscale coarse space that is capable of characterizing high-contrast features of the permeability field. A generalized eigenvalue problem is solved in each non-overlapping coarse element in a communication-free manner to form the global solver, which is accompanied by local solvers originated from additive Schwarz methods but with a non-Galerkin discretization to derive the two-level preconditioner. We provide a rigorous analysis that indicates that the condition number of the preconditioned system could be bounded above with several assumptions. Extensive numerical experiments with various types of three-dimensional high-contrast models are exhibited. In particular, we study the robustness against the contrast of the media as well as the influences of numbers of eigenfunctions, oversampling sizes, and subdomain partitions on the efficiency of the proposed preconditioner. Besides, strong and weak scalability performances are also examined.
△ Less
Submitted 28 March, 2024;
originally announced March 2024.
-
An efficient multiscale multigrid preconditioner for Darcy flow in high-contrast media
Authors:
Changqing Ye,
Shubin Fu,
Eric T. Chung,
Jizu Huang
Abstract:
In this paper, we develop a multigrid preconditioner to solve Darcy flow in highly heterogeneous porous media. The key component of the preconditioner is to construct a sequence of nested subspaces $W_{\mathcal{L}}\subset W_{\mathcal{L}-1}\subset\cdots\subset W_1=W_h$. An appropriate spectral problem is defined in the space of $W_{i-1}$, then the eigenfunctions of the spectral problems are utilize…
▽ More
In this paper, we develop a multigrid preconditioner to solve Darcy flow in highly heterogeneous porous media. The key component of the preconditioner is to construct a sequence of nested subspaces $W_{\mathcal{L}}\subset W_{\mathcal{L}-1}\subset\cdots\subset W_1=W_h$. An appropriate spectral problem is defined in the space of $W_{i-1}$, then the eigenfunctions of the spectral problems are utilized to form $W_i$. The preconditioner is applied to solve a positive semidefinite linear system which results from discretizing the Darcy flow equation with the lowest order Raviart-Thomas spaces and adopting a trapezoidal quadrature rule. Theoretical analysis and numerical investigations of this preconditioner will be presented. In particular, we will consider several typical highly heterogeneous permeability fields whose resolutions are up to $1024^3$ and examine the computational performance of the preconditioner in several aspects, such as strong scalability, weak scalability, and robustness against the contrast of the media. We also demonstrate an application of this preconditioner for solving a two-phase flow benchmark problem.
△ Less
Submitted 28 March, 2024;
originally announced March 2024.
-
Numerical analysis of a FE/SAV scheme for a Caginalp phase field model with mechanical effects in stereolithography
Authors:
Xingguang Jin,
Kei Fong Lam,
Changqing Ye
Abstract:
In this work we propose a phase field model based on a Caginalp system with mechanical effects to study the underlying physical and chemical processes behind stereolithography, which is an additive manufacturing (3D printing) technique that builds objects in a layer-by-layer fashion by using an ultraviolet laser to solidify liquid polymer resins. Existence of weak solutions is established by demon…
▽ More
In this work we propose a phase field model based on a Caginalp system with mechanical effects to study the underlying physical and chemical processes behind stereolithography, which is an additive manufacturing (3D printing) technique that builds objects in a layer-by-layer fashion by using an ultraviolet laser to solidify liquid polymer resins. Existence of weak solutions is established by demonstrating the convergence of a numerical scheme based on a first order scalar auxiliary variable temporal discretization and a finite element spatial discretization. We further establish uniqueness and regularity of solutions, as well as optimal error estimates for the Caginalp system that are supported by numerical simulations. We also present some qualitative two-dimensional simulations of the stereolithography processes captured by the model.
△ Less
Submitted 26 March, 2024;
originally announced March 2024.
-
Spectral Motion Alignment for Video Motion Transfer using Diffusion Models
Authors:
Geon Yeong Park,
Hyeonho Jeong,
Sang Wan Lee,
Jong Chul Ye
Abstract:
The evolution of diffusion models has greatly impacted video generation and understanding. Particularly, text-to-video diffusion models (VDMs) have significantly facilitated the customization of input video with target appearance, motion, etc. Despite these advances, challenges persist in accurately distilling motion information from video frames. While existing works leverage the consecutive fram…
▽ More
The evolution of diffusion models has greatly impacted video generation and understanding. Particularly, text-to-video diffusion models (VDMs) have significantly facilitated the customization of input video with target appearance, motion, etc. Despite these advances, challenges persist in accurately distilling motion information from video frames. While existing works leverage the consecutive frame residual as the target motion vector, they inherently lack global motion context and are vulnerable to frame-wise distortions. To address this, we present Spectral Motion Alignment (SMA), a novel framework that refines and aligns motion vectors using Fourier and wavelet transforms. SMA learns motion patterns by incorporating frequency-domain regularization, facilitating the learning of whole-frame global motion dynamics, and mitigating spatial artifacts. Extensive experiments demonstrate SMA's efficacy in improving motion transfer while maintaining computational efficiency and compatibility across various video customization frameworks.
△ Less
Submitted 22 March, 2024;
originally announced March 2024.
-
Deep Clustering Evaluation: How to Validate Internal Clustering Validation Measures
Authors:
Zeya Wang,
Chenglong Ye
Abstract:
Deep clustering, a method for partitioning complex, high-dimensional data using deep neural networks, presents unique evaluation challenges. Traditional clustering validation measures, designed for low-dimensional spaces, are problematic for deep clustering, which involves projecting data into lower-dimensional embeddings before partitioning. Two key issues are identified: 1) the curse of dimensio…
▽ More
Deep clustering, a method for partitioning complex, high-dimensional data using deep neural networks, presents unique evaluation challenges. Traditional clustering validation measures, designed for low-dimensional spaces, are problematic for deep clustering, which involves projecting data into lower-dimensional embeddings before partitioning. Two key issues are identified: 1) the curse of dimensionality when applying these measures to raw data, and 2) the unreliable comparison of clustering results across different embedding spaces stemming from variations in training procedures and parameter settings in different clustering models. This paper addresses these challenges in evaluating clustering quality in deep learning. We present a theoretical framework to highlight ineffectiveness arising from using internal validation measures on raw and embedded data and propose a systematic approach to applying clustering validity indices in deep clustering contexts. Experiments show that this framework aligns better with external validation measures, effectively reducing the misguidance from the improper use of clustering validity indices in deep learning.
△ Less
Submitted 21 March, 2024;
originally announced March 2024.
-
OTSeg: Multi-prompt Sinkhorn Attention for Zero-Shot Semantic Segmentation
Authors:
Kwanyoung Kim,
Yujin Oh,
Jong Chul Ye
Abstract:
The recent success of CLIP has demonstrated promising results in zero-shot semantic segmentation by transferring muiltimodal knowledge to pixel-level classification. However, leveraging pre-trained CLIP knowledge to closely align text embeddings with pixel embeddings still has limitations in existing approaches. To address this issue, we propose OTSeg, a novel multimodal attention mechanism aimed…
▽ More
The recent success of CLIP has demonstrated promising results in zero-shot semantic segmentation by transferring muiltimodal knowledge to pixel-level classification. However, leveraging pre-trained CLIP knowledge to closely align text embeddings with pixel embeddings still has limitations in existing approaches. To address this issue, we propose OTSeg, a novel multimodal attention mechanism aimed at enhancing the potential of multiple text prompts for matching associated pixel embeddings. We first propose Multi-Prompts Sinkhorn (MPS) based on the Optimal Transport (OT) algorithm, which leads multiple text prompts to selectively focus on various semantic features within image pixels. Moreover, inspired by the success of Sinkformers in unimodal settings, we introduce the extension of MPS, called Multi-Prompts Sinkhorn Attention (MPSA) , which effectively replaces cross-attention mechanisms within Transformer framework in multimodal settings. Through extensive experiments, we demonstrate that OTSeg achieves state-of-the-art (SOTA) performance with significant gains on Zero-Shot Semantic Segmentation (ZS3) tasks across three benchmark datasets.
△ Less
Submitted 11 July, 2024; v1 submitted 21 March, 2024;
originally announced March 2024.
-
Ground-A-Score: Scaling Up the Score Distillation for Multi-Attribute Editing
Authors:
Hangeol Chang,
Jinho Chang,
Jong Chul Ye
Abstract:
Despite recent advancements in text-to-image diffusion models facilitating various image editing techniques, complex text prompts often lead to an oversight of some requests due to a bottleneck in processing text information. To tackle this challenge, we present Ground-A-Score, a simple yet powerful model-agnostic image editing method by incorporating grounding during score distillation. This appr…
▽ More
Despite recent advancements in text-to-image diffusion models facilitating various image editing techniques, complex text prompts often lead to an oversight of some requests due to a bottleneck in processing text information. To tackle this challenge, we present Ground-A-Score, a simple yet powerful model-agnostic image editing method by incorporating grounding during score distillation. This approach ensures a precise reflection of intricate prompt requirements in the editing outcomes, taking into account the prior knowledge of the object locations within the image. Moreover, the selective application with a new penalty coefficient and contrastive loss helps to precisely target editing areas while preserving the integrity of the objects in the source image. Both qualitative assessments and quantitative analyses confirm that Ground-A-Score successfully adheres to the intricate details of extended and multifaceted prompts, ensuring high-quality outcomes that respect the original image attributes.
△ Less
Submitted 20 March, 2024;
originally announced March 2024.
-
Generalized Consistency Trajectory Models for Image Manipulation
Authors:
Beomsu Kim,
Jaemin Kim,
Jeongsol Kim,
Jong Chul Ye
Abstract:
Diffusion-based generative models excel in unconditional generation, as well as on applied tasks such as image editing and restoration. The success of diffusion models lies in the iterative nature of diffusion: diffusion breaks down the complex process of mapping noise to data into a sequence of simple denoising tasks. Moreover, we are able to exert fine-grained control over the generation process…
▽ More
Diffusion-based generative models excel in unconditional generation, as well as on applied tasks such as image editing and restoration. The success of diffusion models lies in the iterative nature of diffusion: diffusion breaks down the complex process of mapping noise to data into a sequence of simple denoising tasks. Moreover, we are able to exert fine-grained control over the generation process by injecting guidance terms into each denoising step. However, the iterative process is also computationally intensive, often taking from tens up to thousands of function evaluations. Although consistency trajectory models (CTMs) enable traversal between any time points along the probability flow ODE (PFODE) and score inference with a single function evaluation, CTMs only allow translation from Gaussian noise to data. Thus, this work aims to unlock the full potential of CTMs by proposing generalized CTMs (GCTMs), which translate between arbitrary distributions via ODEs. We discuss the design space of GCTMs and demonstrate their efficacy in various image manipulation tasks such as image-to-image translation, restoration, and editing. Code: \url{https://github.com/1202kbs/GCTM}
△ Less
Submitted 19 March, 2024;
originally announced March 2024.
-
Con-CDVAE: A method for the conditional generation of crystal structures
Authors:
Cai-Yuan Ye,
Hong-Ming Weng,
Quan-Sheng Wu
Abstract:
In recent years, progress has been made in generating new crystalline materials using generative machine learning models, though gaps remain in efficiently generating crystals based on target properties. This paper proposes the Con-CDVAE model, an extension of the Crystal Diffusion Variational Autoencoder (CDVAE), for conditional crystal generation. We introduce innovative components, design a two…
▽ More
In recent years, progress has been made in generating new crystalline materials using generative machine learning models, though gaps remain in efficiently generating crystals based on target properties. This paper proposes the Con-CDVAE model, an extension of the Crystal Diffusion Variational Autoencoder (CDVAE), for conditional crystal generation. We introduce innovative components, design a two-step training method, and develop three unique generation strategies to enhance model performance. The effectiveness of Con-CDVAE is demonstrated through extensive testing under various conditions, including both single and combined property targets. Ablation studies further underscore the critical role of the new components in achieving our model's performance. Additionally, we validate the physical credibility of the generated crystals through Density Functional Theory (DFT) calculations, confirming Con-CDVAE's potential in material science research.
△ Less
Submitted 19 March, 2024;
originally announced March 2024.
-
DreamMotion: Space-Time Self-Similar Score Distillation for Zero-Shot Video Editing
Authors:
Hyeonho Jeong,
Jinho Chang,
Geon Yeong Park,
Jong Chul Ye
Abstract:
Text-driven diffusion-based video editing presents a unique challenge not encountered in image editing literature: establishing real-world motion. Unlike existing video editing approaches, here we focus on score distillation sampling to circumvent the standard reverse diffusion process and initiate optimization from videos that already exhibit natural motion. Our analysis reveals that while video…
▽ More
Text-driven diffusion-based video editing presents a unique challenge not encountered in image editing literature: establishing real-world motion. Unlike existing video editing approaches, here we focus on score distillation sampling to circumvent the standard reverse diffusion process and initiate optimization from videos that already exhibit natural motion. Our analysis reveals that while video score distillation can effectively introduce new content indicated by target text, it can also cause significant structure and motion deviation. To counteract this, we propose to match space-time self-similarities of the original video and the edited video during the score distillation. Thanks to the use of score distillation, our approach is model-agnostic, which can be applied for both cascaded and non-cascaded video diffusion frameworks. Through extensive comparisons with leading methods, our approach demonstrates its superiority in altering appearances while accurately preserving the original structure and motion.
△ Less
Submitted 15 July, 2024; v1 submitted 18 March, 2024;
originally announced March 2024.
-
DreamSampler: Unifying Diffusion Sampling and Score Distillation for Image Manipulation
Authors:
Jeongsol Kim,
Geon Yeong Park,
Jong Chul Ye
Abstract:
Reverse sampling and score-distillation have emerged as main workhorses in recent years for image manipulation using latent diffusion models (LDMs). While reverse diffusion sampling often requires adjustments of LDM architecture or feature engineering, score distillation offers a simple yet powerful model-agnostic approach, but it is often prone to mode-collapsing. To address these limitations and…
▽ More
Reverse sampling and score-distillation have emerged as main workhorses in recent years for image manipulation using latent diffusion models (LDMs). While reverse diffusion sampling often requires adjustments of LDM architecture or feature engineering, score distillation offers a simple yet powerful model-agnostic approach, but it is often prone to mode-collapsing. To address these limitations and leverage the strengths of both approaches, here we introduce a novel framework called {\em DreamSampler}, which seamlessly integrates these two distinct approaches through the lens of regularized latent optimization. Similar to score-distillation, DreamSampler is a model-agnostic approach applicable to any LDM architecture, but it allows both distillation and reverse sampling with additional guidance for image editing and reconstruction. Through experiments involving image editing, SVG reconstruction and etc, we demonstrate the competitive performance of DreamSampler compared to existing approaches, while providing new applications.
△ Less
Submitted 17 March, 2024;
originally announced March 2024.
-
Generating Synthetic Computed Tomography for Radiotherapy: SynthRAD2023 Challenge Report
Authors:
Evi M. C. Huijben,
Maarten L. Terpstra,
Arthur Jr. Galapon,
Suraj Pai,
Adrian Thummerer,
Peter Koopmans,
Manya Afonso,
Maureen van Eijnatten,
Oliver Gurney-Champion,
Zeli Chen,
Yiwen Zhang,
Kaiyi Zheng,
Chuanpu Li,
Haowen Pang,
Chuyang Ye,
Runqi Wang,
Tao Song,
Fuxin Fan,
Jingna Qiu,
Yixing Huang,
Juhyung Ha,
Jong Sung Park,
Alexandra Alain-Beaudoin,
Silvain Bériault,
Pengxin Yu
, et al. (34 additional authors not shown)
Abstract:
Radiation therapy plays a crucial role in cancer treatment, necessitating precise delivery of radiation to tumors while sparing healthy tissues over multiple days. Computed tomography (CT) is integral for treatment planning, offering electron density data crucial for accurate dose calculations. However, accurately representing patient anatomy is challenging, especially in adaptive radiotherapy, wh…
▽ More
Radiation therapy plays a crucial role in cancer treatment, necessitating precise delivery of radiation to tumors while sparing healthy tissues over multiple days. Computed tomography (CT) is integral for treatment planning, offering electron density data crucial for accurate dose calculations. However, accurately representing patient anatomy is challenging, especially in adaptive radiotherapy, where CT is not acquired daily. Magnetic resonance imaging (MRI) provides superior soft-tissue contrast. Still, it lacks electron density information while cone beam CT (CBCT) lacks direct electron density calibration and is mainly used for patient positioning. Adopting MRI-only or CBCT-based adaptive radiotherapy eliminates the need for CT planning but presents challenges. Synthetic CT (sCT) generation techniques aim to address these challenges by using image synthesis to bridge the gap between MRI, CBCT, and CT. The SynthRAD2023 challenge was organized to compare synthetic CT generation methods using multi-center ground truth data from 1080 patients, divided into two tasks: 1) MRI-to-CT and 2) CBCT-to-CT. The evaluation included image similarity and dose-based metrics from proton and photon plans. The challenge attracted significant participation, with 617 registrations and 22/17 valid submissions for tasks 1/2. Top-performing teams achieved high structural similarity indices (>0.87/0.90) and gamma pass rates for photon (>98.1%/99.0%) and proton (>97.3%/97.0%) plans. However, no significant correlation was found between image similarity metrics and dose accuracy, emphasizing the need for dose evaluation when assessing the clinical applicability of sCT. SynthRAD2023 facilitated the investigation and benchmarking of sCT generation techniques, providing insights for developing MRI-only and CBCT-based adaptive radiotherapy.
△ Less
Submitted 11 June, 2024; v1 submitted 13 March, 2024;
originally announced March 2024.