Showing 1–50 of 188 results for author: Müller, J

Dynamical Measure Transport and Neural PDE Solvers for Sampling

Authors: Jingtong Sun, Julius Berner, Lorenz Richter, Marius Zeinhofer, Johannes Müller, Kamyar Azizzadenesheli, Anima Anandkumar

Abstract: The task of sampling from a probability density can be approached as transporting a tractable density function to the target, known as dynamical measure transport. In this work, we tackle it through a principled unified framework using deterministic or stochastic evolutions described by partial differential equations (PDEs). This framework incorporates prior trajectory-based sampling methods, such… ▽ More The task of sampling from a probability density can be approached as transporting a tractable density function to the target, known as dynamical measure transport. In this work, we tackle it through a principled unified framework using deterministic or stochastic evolutions described by partial differential equations (PDEs). This framework incorporates prior trajectory-based sampling methods, such as diffusion models or Schrödinger bridges, without relying on the concept of time-reversals. Moreover, it allows us to propose novel numerical methods for solving the transport task and thus sampling from complicated targets without the need for the normalization constant or data samples. We employ physics-informed neural networks (PINNs) to approximate the respective PDE solutions, implying both conceptional and computational advantages. In particular, PINNs allow for simulation- and discretization-free optimization and can be trained very efficiently, leading to significantly better mode coverage in the sampling task compared to alternative methods. Moreover, they can readily be fine-tuned with Gauss-Newton methods to achieve high accuracy in sampling. △ Less

Submitted 10 July, 2024; originally announced July 2024.

AI Driven Laser Parameter Search: Inverse Design of Photonic Surfaces using Greedy Surrogate-based Optimization

Authors: Luka Grbcic, Minok Park, Juliane Müller, Vassilia Zorba, Wibe Albert de Jong

Abstract: Photonic surfaces designed with specific optical characteristics are becoming increasingly important for use in in various energy harvesting and storage systems. , In this study, we develop a surrogate-based optimization approach for designing such surfaces. The surrogate-based optimization framework employs the Random Forest algorithm and uses a greedy, prediction-based exploration strategy to id… ▽ More Photonic surfaces designed with specific optical characteristics are becoming increasingly important for use in in various energy harvesting and storage systems. , In this study, we develop a surrogate-based optimization approach for designing such surfaces. The surrogate-based optimization framework employs the Random Forest algorithm and uses a greedy, prediction-based exploration strategy to identify the laser fabrication parameters that minimize the discrepancy relative to a user-defined target optical characteristics. We demonstrate the approach on two synthetic benchmarks and two specific cases of photonic surface inverse design targets. It exhibits superior performance when compared to other optimization algorithms across all benchmarks. Additionally, we demonstrate a technique of inverse design warm starting for changed target optical characteristics which enhances the performance of the introduced approach. △ Less

Submitted 20 June, 2024; originally announced July 2024.

Meta-experiments: Improving experimentation through experimentation

Authors: Melanie J. I. Müller

Abstract: A/B testing is widexly used in the industry to optimize customer facing websites. Many companies employ experimentation specialists to facilitate and improve the process of A/B testing. Here, we present the application of A/B testing to this improvement effort itself, by running experiments on the experimentation process, which we call 'meta-experiments'. We discuss the challenges of this approach… ▽ More A/B testing is widexly used in the industry to optimize customer facing websites. Many companies employ experimentation specialists to facilitate and improve the process of A/B testing. Here, we present the application of A/B testing to this improvement effort itself, by running experiments on the experimentation process, which we call 'meta-experiments'. We discuss the challenges of this approach using the example of one of our meta-experiments, which helped experimenters to run more sufficiently powered A/B tests. We also point out the benefits of 'dog fooding' for the experimentation specialists when running their own experiments. △ Less

Submitted 24 June, 2024; originally announced June 2024.

Comments: 6 pages, 2 figures, 1 table

Deciphering the Definition of Adversarial Robustness for post-hoc OOD Detectors

Authors: Peter Lorenz, Mario Fernandez, Jens Müller, Ullrich Köthe

Abstract: Detecting out-of-distribution (OOD) inputs is critical for safely deploying deep learning models in real-world scenarios. In recent years, many OOD detectors have been developed, and even the benchmarking has been standardized, i.e. OpenOOD. The number of post-hoc detectors is growing fast and showing an option to protect a pre-trained classifier against natural distribution shifts, claiming to be… ▽ More Detecting out-of-distribution (OOD) inputs is critical for safely deploying deep learning models in real-world scenarios. In recent years, many OOD detectors have been developed, and even the benchmarking has been standardized, i.e. OpenOOD. The number of post-hoc detectors is growing fast and showing an option to protect a pre-trained classifier against natural distribution shifts, claiming to be ready for real-world scenarios. However, its efficacy in handling adversarial examples has been neglected in the majority of studies. This paper investigates the adversarial robustness of the 16 post-hoc detectors on several evasion attacks and discuss a roadmap towards adversarial defense in OOD detectors. △ Less

Submitted 28 June, 2024; v1 submitted 21 June, 2024; originally announced June 2024.

Resource-efficient Medical Image Analysis with Self-adapting Forward-Forward Networks

Authors: Johanna P. Müller, Bernhard Kainz

Abstract: We introduce a fast Self-adapting Forward-Forward Network (SaFF-Net) for medical imaging analysis, mitigating power consumption and resource limitations, which currently primarily stem from the prevalent reliance on back-propagation for model training and fine-tuning. Building upon the recently proposed Forward-Forward Algorithm (FFA), we introduce the Convolutional Forward-Forward Algorithm (CFFA… ▽ More We introduce a fast Self-adapting Forward-Forward Network (SaFF-Net) for medical imaging analysis, mitigating power consumption and resource limitations, which currently primarily stem from the prevalent reliance on back-propagation for model training and fine-tuning. Building upon the recently proposed Forward-Forward Algorithm (FFA), we introduce the Convolutional Forward-Forward Algorithm (CFFA), a parameter-efficient reformulation that is suitable for advanced image analysis and overcomes the speed and generalisation constraints of the original FFA. To address hyper-parameter sensitivity of FFAs we are also introducing a self-adapting framework SaFF-Net fine-tuning parameters during warmup and training in parallel. Our approach enables more effective model training and eliminates the previously essential requirement for an arbitrarily chosen Goodness function in FFA. We evaluate our approach on several benchmarking datasets in comparison with standard Back-Propagation (BP) neural networks showing that FFA-based networks with notably fewer parameters and function evaluations can compete with standard models, especially, in one-shot scenarios and large batch sizes. The code will be available at the time of the conference. △ Less

Submitted 20 June, 2024; originally announced June 2024.

Comments: Under Review

Demystifying Higher-Order Graph Neural Networks

Authors: Maciej Besta, Florian Scheidl, Lukas Gianinazzi, Shachar Klaiman, Jürgen Müller, Torsten Hoefler

Abstract: Higher-order graph neural networks (HOGNNs) are an important class of GNN models that harness polyadic relations between vertices beyond plain edges. They have been used to eliminate issues such as over-smoothing or over-squashing, to significantly enhance the accuracy of GNN predictions, to improve the expressiveness of GNN architectures, and for numerous other goals. A plethora of HOGNN models h… ▽ More Higher-order graph neural networks (HOGNNs) are an important class of GNN models that harness polyadic relations between vertices beyond plain edges. They have been used to eliminate issues such as over-smoothing or over-squashing, to significantly enhance the accuracy of GNN predictions, to improve the expressiveness of GNN architectures, and for numerous other goals. A plethora of HOGNN models have been introduced, and they come with diverse neural architectures, and even with different notions of what the "higher-order" means. This richness makes it very challenging to appropriately analyze and compare HOGNN models, and to decide in what scenario to use specific ones. To alleviate this, we first design an in-depth taxonomy and a blueprint for HOGNNs. This facilitates designing models that maximize performance. Then, we use our taxonomy to analyze and compare the available HOGNN models. The outcomes of our analysis are synthesized in a set of insights that help to select the most beneficial GNN model in a given scenario, and a comprehensive list of challenges and opportunities for further research into more powerful HOGNNs. △ Less

Submitted 18 June, 2024; originally announced June 2024.

Multi-Head RAG: Solving Multi-Aspect Problems with LLMs

Authors: Maciej Besta, Ales Kubicek, Roman Niggli, Robert Gerstenberger, Lucas Weitzendorf, Mingyuan Chi, Patrick Iff, Joanna Gajda, Piotr Nyczyk, Jürgen Müller, Hubert Niewiadomski, Marcin Chrapek, Michał Podstawski, Torsten Hoefler

Abstract: Retrieval Augmented Generation (RAG) enhances the abilities of Large Language Models (LLMs) by enabling the retrieval of documents into the LLM context to provide more accurate and relevant responses. Existing RAG solutions do not focus on queries that may require fetching multiple documents with substantially different contents. Such queries occur frequently, but are challenging because the embed… ▽ More Retrieval Augmented Generation (RAG) enhances the abilities of Large Language Models (LLMs) by enabling the retrieval of documents into the LLM context to provide more accurate and relevant responses. Existing RAG solutions do not focus on queries that may require fetching multiple documents with substantially different contents. Such queries occur frequently, but are challenging because the embeddings of these documents may be distant in the embedding space, making it hard to retrieve them all. This paper introduces Multi-Head RAG (MRAG), a novel scheme designed to address this gap with a simple yet powerful idea: leveraging activations of Transformer's multi-head attention layer, instead of the decoder layer, as keys for fetching multi-aspect documents. The driving motivation is that different attention heads can learn to capture different data aspects. Harnessing the corresponding activations results in embeddings that represent various facets of data items and queries, improving the retrieval accuracy for complex queries. We provide an evaluation methodology and metrics, synthetic datasets, and real-world use cases to demonstrate MRAG's effectiveness, showing improvements of up to 20% in relevance over standard RAG baselines. MRAG can be seamlessly integrated with existing RAG frameworks and benchmarking tools like RAGAS as well as different classes of data stores. △ Less

Submitted 7 June, 2024; originally announced June 2024.

Essentially Sharp Estimates on the Entropy Regularization Error in Discrete Discounted Markov Decision Processes

Authors: Johannes Müller, Semih Cayci

Abstract: We study the error introduced by entropy regularization of infinite-horizon discrete discounted Markov decision processes. We show that this error decreases exponentially in the inverse regularization strength both in a weighted KL-divergence and in value with a problem-specific exponent. We provide a lower bound matching our upper bound up to a polynomial factor. Our proof relies on the correspon… ▽ More We study the error introduced by entropy regularization of infinite-horizon discrete discounted Markov decision processes. We show that this error decreases exponentially in the inverse regularization strength both in a weighted KL-divergence and in value with a problem-specific exponent. We provide a lower bound matching our upper bound up to a polynomial factor. Our proof relies on the correspondence of the solutions of entropy-regularized Markov decision processes with gradient flows of the unregularized reward with respect to a Riemannian metric common in natural policy gradient methods. Further, this correspondence allows us to identify the limit of the gradient flow as the generalized maximum entropy optimal policy, thereby characterizing the implicit bias of the Kakade gradient flow which corresponds to a time-continuous version of the natural policy gradient method. We use this to show that for entropy-regularized natural policy gradient methods the overall error decays exponentially in the square root of the number of iterations improving existing sublinear guarantees. △ Less

Submitted 25 June, 2024; v1 submitted 6 June, 2024; originally announced June 2024.

Comments: 26 pages, 1 figure

MSC Class: 37N40; 65K05; 90C05; 90C40; 90C53

Inverse design of photonic surfaces on Inconel via multi-fidelity machine learning ensemble framework and high throughput femtosecond laser processing

Authors: Luka Grbcic, Minok Park, Mahmoud Elzouka, Ravi Prasher, Juliane Müller, Costas P. Grigoropoulos, Sean D. Lubner, Vassilia Zorba, Wibe Albert de Jong

Abstract: We demonstrate a multi-fidelity (MF) machine learning ensemble framework for the inverse design of photonic surfaces, trained on a dataset of 11,759 samples that we fabricate using high throughput femtosecond laser processing. The MF ensemble combines an initial low fidelity model for generating design solutions, with a high fidelity model that refines these solutions through local optimization. T… ▽ More We demonstrate a multi-fidelity (MF) machine learning ensemble framework for the inverse design of photonic surfaces, trained on a dataset of 11,759 samples that we fabricate using high throughput femtosecond laser processing. The MF ensemble combines an initial low fidelity model for generating design solutions, with a high fidelity model that refines these solutions through local optimization. The combined MF ensemble can generate multiple disparate sets of laser-processing parameters that can each produce the same target input spectral emissivity with high accuracy (root mean squared errors < 2%). SHapley Additive exPlanations analysis shows transparent model interpretability of the complex relationship between laser parameters and spectral emissivity. Finally, the MF ensemble is experimentally validated by fabricating and evaluating photonic surface designs that it generates for improved efficiency energy harvesting devices. Our approach provides a powerful tool for advancing the inverse design of photonic surfaces in energy harvesting applications. △ Less

Submitted 3 June, 2024; originally announced June 2024.

ADESSE: Advice Explanations in Complex Repeated Decision-Making Environments

Authors: Sören Schleibaum, Lu Feng, Sarit Kraus, Jörg P. Müller

Abstract: In the evolving landscape of human-centered AI, fostering a synergistic relationship between humans and AI agents in decision-making processes stands as a paramount challenge. This work considers a problem setup where an intelligent agent comprising a neural network-based prediction component and a deep reinforcement learning component provides advice to a human decision-maker in complex repeated… ▽ More In the evolving landscape of human-centered AI, fostering a synergistic relationship between humans and AI agents in decision-making processes stands as a paramount challenge. This work considers a problem setup where an intelligent agent comprising a neural network-based prediction component and a deep reinforcement learning component provides advice to a human decision-maker in complex repeated decision-making environments. Whether the human decision-maker would follow the agent's advice depends on their beliefs and trust in the agent and on their understanding of the advice itself. To this end, we developed an approach named ADESSE to generate explanations about the adviser agent to improve human trust and decision-making. Computational experiments on a range of environments with varying model sizes demonstrate the applicability and scalability of ADESSE. Furthermore, an interactive game-based user study shows that participants were significantly more satisfied, achieved a higher reward in the game, and took less time to select an action when presented with explanations generated by ADESSE. These findings illuminate the critical role of tailored, human-centered explanations in AI-assisted decision-making. △ Less

Submitted 31 May, 2024; originally announced May 2024.

Kronecker-Factored Approximate Curvature for Physics-Informed Neural Networks

Authors: Felix Dangel, Johannes Müller, Marius Zeinhofer

Abstract: Physics-informed neural networks (PINNs) are infamous for being hard to train. Recently, second-order methods based on natural gradient and Gauss-Newton methods have shown promising performance, improving the accuracy achieved by first-order methods by several orders of magnitude. While promising, the proposed methods only scale to networks with a few thousand parameters due to the high computatio… ▽ More Physics-informed neural networks (PINNs) are infamous for being hard to train. Recently, second-order methods based on natural gradient and Gauss-Newton methods have shown promising performance, improving the accuracy achieved by first-order methods by several orders of magnitude. While promising, the proposed methods only scale to networks with a few thousand parameters due to the high computational cost to evaluate, store, and invert the curvature matrix. We propose Kronecker-factored approximate curvature (KFAC) for PINN losses that greatly reduces the computational cost and allows scaling to much larger networks. Our approach goes beyond the established KFAC for traditional deep learning problems as it captures contributions from a PDE's differential operator that are crucial for optimization. To establish KFAC for such losses, we use Taylor-mode automatic differentiation to describe the differential operator's computation graph as a forward network with shared weights. This allows us to apply KFAC thanks to a recently-developed general formulation for networks with weight sharing. Empirically, we find that our KFAC-based optimizers are competitive with expensive second-order methods on small problems, scale more favorably to higher-dimensional neural networks and PDEs, and consistently outperform first-order methods and LBFGS. △ Less

Submitted 27 May, 2024; v1 submitted 24 May, 2024; originally announced May 2024.

Effective Quadratic Error Bounds for Floating-Point Algorithms Computing the Hypotenuse Function

Authors: Jean-Michel Muller, Bruno Salvy

Abstract: We provide tools to help automate the error analysis of algorithms that evaluate simple functions over the floating-point numbers. The aim is to obtain tight relative error bounds for these algorithms, expressed as a function of the unit round-off. Due to the discrete nature of the set of floating-point numbers, the largest errors are often intrinsically "arithmetic" in the sense that their appear… ▽ More We provide tools to help automate the error analysis of algorithms that evaluate simple functions over the floating-point numbers. The aim is to obtain tight relative error bounds for these algorithms, expressed as a function of the unit round-off. Due to the discrete nature of the set of floating-point numbers, the largest errors are often intrinsically "arithmetic" in the sense that their appearance may depend on specific bit patterns in the binary representations of intermediate variables, which may be present only for some precisions. We focus on generic (i.e., parameterized by the precision) and analytic over-estimations that still capture the correlations between the errors made at each step of the algorithms. Using methods from computer algebra, which we adapt to the particular structure of the polynomial systems that encode the errors, we obtain bounds with a linear term in the unit round-off that is sharp in manycases. An explicit quadratic bound is given, rather than the $O()$-estimate that is more common in this area. This is particularly important when using low precision formats, which are increasingly common in modern processors. Using this approach, we compare five algorithms for computing the hypotenuse function, ranging from elementary to quite challenging. △ Less

Submitted 6 May, 2024; originally announced May 2024.

SIM2VR: Towards Automated Biomechanical Testing in VR

Authors: Florian Fischer, Aleksi Ikkala, Markus Klar, Arthur Fleig, Miroslav Bachinski, Roderick Murray-Smith, Perttu Hämäläinen, Antti Oulasvirta, Jörg Müller

Abstract: Automated biomechanical testing has great potential for the development of VR applications, as initial insights into user behaviour can be gained in silico early in the design process. In particular, it allows prediction of user movements and ergonomic variables, such as fatigue, prior to conducting user studies. However, there is a fundamental disconnect between simulators hosting state-of-the-ar… ▽ More Automated biomechanical testing has great potential for the development of VR applications, as initial insights into user behaviour can be gained in silico early in the design process. In particular, it allows prediction of user movements and ergonomic variables, such as fatigue, prior to conducting user studies. However, there is a fundamental disconnect between simulators hosting state-of-the-art biomechanical user models and simulators used to develop and run VR applications. Existing user simulators often struggle to capture the intricacies and nuances of real-world VR applications, reducing ecological validity of user predictions. In this paper, we introduce SIM2VR, a system that aligns user simulation with a given VR application by establishing a continuous closed loop between the two processes. This, for the first time, enables training simulated users directly in the same VR application that real users interact with. We demonstrate that SIM2VR can predict differences in user performance, ergonomics and strategies in a fast-paced, dynamic arcade game. In order to expand the scope of automated biomechanical testing beyond simple visuomotor tasks, advances in cognitive models and reward function design will be needed. △ Less

Submitted 26 April, 2024; originally announced April 2024.

Comments: 22 pages, 10 figures, 2 tables; Supplementary Material: 7 pages, 4 figures, 1 table

Implications of the AI Act for Non-Discrimination Law and Algorithmic Fairness

Authors: Luca Deck, Jan-Laurin Müller, Conradin Braun, Domenique Zipperling, Niklas Kühl

Abstract: The topic of fairness in AI, as debated in the FATE (Fairness, Accountability, Transparency, and Ethics in AI) communities, has sparked meaningful discussions in the past years. However, from a legal perspective, particularly from the perspective of European Union law, many open questions remain. Whereas algorithmic fairness aims to mitigate structural inequalities at design-level, European non-di… ▽ More The topic of fairness in AI, as debated in the FATE (Fairness, Accountability, Transparency, and Ethics in AI) communities, has sparked meaningful discussions in the past years. However, from a legal perspective, particularly from the perspective of European Union law, many open questions remain. Whereas algorithmic fairness aims to mitigate structural inequalities at design-level, European non-discrimination law is tailored to individual cases of discrimination after an AI model has been deployed. The AI Act might present a tremendous step towards bridging these two approaches by shifting non-discrimination responsibilities into the design stage of AI models. Based on an integrative reading of the AI Act, we comment on legal as well as technical enforcement problems and propose practical implications on bias detection and bias correction in order to specify and comply with specific technical requirements. △ Less

Submitted 26 June, 2024; v1 submitted 29 March, 2024; originally announced March 2024.

Fisher-Rao Gradient Flows of Linear Programs and State-Action Natural Policy Gradients

Authors: Johannes Müller, Semih Çaycı, Guido Montúfar

Abstract: Kakade's natural policy gradient method has been studied extensively in the last years showing linear convergence with and without regularization. We study another natural gradient method which is based on the Fisher information matrix of the state-action distributions and has received little attention from the theoretical side. Here, the state-action distributions follow the Fisher-Rao gradient f… ▽ More Kakade's natural policy gradient method has been studied extensively in the last years showing linear convergence with and without regularization. We study another natural gradient method which is based on the Fisher information matrix of the state-action distributions and has received little attention from the theoretical side. Here, the state-action distributions follow the Fisher-Rao gradient flow inside the state-action polytope with respect to a linear potential. Therefore, we study Fisher-Rao gradient flows of linear programs more generally and show linear convergence with a rate that depends on the geometry of the linear program. Equivalently, this yields an estimate on the error induced by entropic regularization of the linear program which improves existing results. We extend these results and show sublinear convergence for perturbed Fisher-Rao gradient flows and natural gradient flows up to an approximation error. In particular, these general results cover the case of state-action natural policy gradients. △ Less

Submitted 28 March, 2024; originally announced March 2024.

Comments: 27 pages, 4 figures, under review

MSC Class: 65K05; 90C05; 90C08; 90C40; 90C53

Automated Data Curation for Robust Language Model Fine-Tuning

Authors: Jiuhai Chen, Jonas Mueller

Abstract: Large Language Models have become the de facto approach to sequence-to-sequence text generation tasks, but for specialized tasks/domains, a pretrained LLM lacks specific capabilities to produce accurate or well-formatted responses. Supervised fine-tuning specializes a LLM by training it on dataset of example prompts with target responses, but real-world data tends to be noisy. While many fine-tuni… ▽ More Large Language Models have become the de facto approach to sequence-to-sequence text generation tasks, but for specialized tasks/domains, a pretrained LLM lacks specific capabilities to produce accurate or well-formatted responses. Supervised fine-tuning specializes a LLM by training it on dataset of example prompts with target responses, but real-world data tends to be noisy. While many fine-tuning algorithms exist, here we consider a \emph{data-centric AI} perspective on LLM fine-tuning, studying how to \emph{systematically} curate the training dataset to improve the LLM produced via \emph{any} fine-tuning algorithm. We introduce an automated data curation pipeline CLEAR (Confidence-based LLM Evaluation And Rectification) for instruction tuning datasets, that can be used with any LLM and fine-tuning procedure. CLEAR estimates which training data is low-quality and either filters or corrects it. Automatically identifying which data to filter or correct is done via LLM-derived confidence estimates, to ensure only confident modifications to the dataset. Unlike existing data curation techniques, CLEAR is a comprehensive framework that can improve a dataset (and trained model outputs) without additional fine-tuning computations. We don't assume access to a stronger LLM than the model being fine-tuned (e.g.\ relying on GPT-4 when fine-tuning GPT-3.5), to see whether CLEAR can meaningfully improve the capabilities of any LLM. Experiments reveal that CLEAR consistently improves the performance of fine-tuned models across many datasets and models (like GPT-3.5 and Llama2). △ Less

Submitted 19 March, 2024; originally announced March 2024.

Scaling Rectified Flow Transformers for High-Resolution Image Synthesis

Authors: Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhorn, Zion English, Kyle Lacey, Alex Goodwin, Yannik Marek, Robin Rombach

Abstract: Diffusion models create data from noise by inverting the forward paths of data towards noise and have emerged as a powerful generative modeling technique for high-dimensional, perceptual data such as images and videos. Rectified flow is a recent generative model formulation that connects data and noise in a straight line. Despite its better theoretical properties and conceptual simplicity, it is n… ▽ More Diffusion models create data from noise by inverting the forward paths of data towards noise and have emerged as a powerful generative modeling technique for high-dimensional, perceptual data such as images and videos. Rectified flow is a recent generative model formulation that connects data and noise in a straight line. Despite its better theoretical properties and conceptual simplicity, it is not yet decisively established as standard practice. In this work, we improve existing noise sampling techniques for training rectified flow models by biasing them towards perceptually relevant scales. Through a large-scale study, we demonstrate the superior performance of this approach compared to established diffusion formulations for high-resolution text-to-image synthesis. Additionally, we present a novel transformer-based architecture for text-to-image generation that uses separate weights for the two modalities and enables a bidirectional flow of information between image and text tokens, improving text comprehension, typography, and human preference ratings. We demonstrate that this architecture follows predictable scaling trends and correlates lower validation loss to improved text-to-image synthesis as measured by various metrics and human evaluations. Our largest models outperform state-of-the-art models, and we will make our experimental data, code, and model weights publicly available. △ Less

Submitted 5 March, 2024; originally announced March 2024.

Efficient and Interpretable Traffic Destination Prediction using Explainable Boosting Machines

Authors: Yasin Yousif, Jörg Müller

Abstract: Developing accurate models for traffic trajectory predictions is crucial for achieving fully autonomous driving. Various deep neural network models have been employed to address this challenge, but their black-box nature hinders transparency and debugging capabilities in a deployed system. Glass-box models offer a solution by providing full interpretability through methods like \ac{GAM}. In this s… ▽ More Developing accurate models for traffic trajectory predictions is crucial for achieving fully autonomous driving. Various deep neural network models have been employed to address this challenge, but their black-box nature hinders transparency and debugging capabilities in a deployed system. Glass-box models offer a solution by providing full interpretability through methods like \ac{GAM}. In this study, we evaluate an efficient additive model called \ac{EBM} for traffic prediction on three popular mixed traffic datasets: \ac{SDD}, \ac{InD}, and Argoverse. Our results show that the \ac{EBM} models perform competitively in predicting pedestrian destinations within \ac{SDD} and \ac{InD} while providing modest predictions for vehicle-dominant Argoverse dataset. Additionally, our transparent trained models allow us to analyse feature importance and interactions, as well as provide qualitative examples of predictions explanation. The full training code will be made public upon publication. △ Less

Submitted 5 February, 2024; originally announced February 2024.

Creating a Synthesizer from Schrödinger's Equation

Authors: Arthur Freye, Jannis Müller

Abstract: Our project offers an alternative approach to the sensory perception of the Schrödinger equation (an elementary model of quantum phenomena) by interpreting it as a sound wave. We are building a synthesizer plugin that simulates a quantum mechanical state that evolves over time. Thus, our tool allows the creation of unique sounds that are in motion and feel alive. These can be used in professional… ▽ More Our project offers an alternative approach to the sensory perception of the Schrödinger equation (an elementary model of quantum phenomena) by interpreting it as a sound wave. We are building a synthesizer plugin that simulates a quantum mechanical state that evolves over time. Thus, our tool allows the creation of unique sounds that are in motion and feel alive. These can be used in professional music production without any knowledge of physics, while at the same time providing insight into a chapter of quantum mechanics. The goal is to lower the threshold for entering complex theory by first developing an intuition for the subject; but the tool can also be used purely as a musical instrument. The user is encouraged, but not forced, to learn more about the underlying physics. Simulation parameters are adjustable in real-time, allowing intuitive experimentation. Despite the approximate calculations, real physical effects such as quantum tunneling can be observed acoustically and visually. △ Less

Submitted 1 February, 2024; originally announced February 2024.

Journal ref: Proceedings of the 28th International Conference on Auditory Display (ICAD 2023), 2023, pp. 179-182

Hold Tight: Identifying Behavioral Patterns During Prolonged Work in VR through Video Analysis

Authors: Verena Biener, Forouzan Farzinnejad, Rinaldo Schuster, Seyedmasih Tabaei, Leon Lindlein, Jinghui Hu, Negar Nouri, John J. Dudley, Per Ola Kristensson, Jörg Müller, Jens Grubert

Abstract: VR devices have recently been actively promoted as tools for knowledge workers and prior work has demonstrated that VR can support some knowledge worker tasks. However, only a few studies have explored the effects of prolonged use of VR such as a study observing 16 participant working in VR and a physical environment for one work-week each and reporting mainly on subjective feedback. As a nuanced… ▽ More VR devices have recently been actively promoted as tools for knowledge workers and prior work has demonstrated that VR can support some knowledge worker tasks. However, only a few studies have explored the effects of prolonged use of VR such as a study observing 16 participant working in VR and a physical environment for one work-week each and reporting mainly on subjective feedback. As a nuanced understanding of participants' behavior in VR and how it evolves over time is still missing, we report on the results from an analysis of 559 hours of video material obtained in this prior study. Among other findings, we report that (1) the frequency of actions related to adjusting the headset reduced by 46% and the frequency of actions related to supporting the headset reduced by 42% over the five days; (2) the HMD was removed 31% less frequently over the five days but for 41% longer periods; (3) wearing an HMD is disruptive to normal patterns of eating and drinking, but not to social interactions, such as talking. The combined findings in this work demonstrate the value of long-term studies of deployed VR systems and can be used to inform the design of better, more ergonomic VR systems as tools for knowledge workers. △ Less

Submitted 29 January, 2024; v1 submitted 26 January, 2024; originally announced January 2024.

Demystifying Chains, Trees, and Graphs of Thoughts

Authors: Maciej Besta, Florim Memedi, Zhenyu Zhang, Robert Gerstenberger, Guangyuan Piao, Nils Blach, Piotr Nyczyk, Marcin Copik, Grzegorz Kwaśniewski, Jürgen Müller, Lukas Gianinazzi, Ales Kubicek, Hubert Niewiadomski, Aidan O'Mahony, Onur Mutlu, Torsten Hoefler

Abstract: The field of natural language processing (NLP) has witnessed significant progress in recent years, with a notable focus on improving large language models' (LLM) performance through innovative prompting techniques. Among these, prompt engineering coupled with structures has emerged as a promising paradigm, with designs such as Chain-of-Thought, Tree of Thoughts, or Graph of Thoughts, in which the… ▽ More The field of natural language processing (NLP) has witnessed significant progress in recent years, with a notable focus on improving large language models' (LLM) performance through innovative prompting techniques. Among these, prompt engineering coupled with structures has emerged as a promising paradigm, with designs such as Chain-of-Thought, Tree of Thoughts, or Graph of Thoughts, in which the overall LLM reasoning is guided by a structure such as a graph. As illustrated with numerous examples, this paradigm significantly enhances the LLM's capability to solve numerous tasks, ranging from logical or mathematical reasoning to planning or creative writing. To facilitate the understanding of this growing field and pave the way for future developments, we devise a general blueprint for effective and efficient LLM reasoning schemes. For this, we conduct an in-depth analysis of the prompt execution pipeline, clarifying and clearly defining different concepts. We then build the first taxonomy of structure-enhanced LLM reasoning schemes. We focus on identifying fundamental classes of harnessed structures, and we analyze the representations of these structures, algorithms executed with these structures, and many others. We refer to these structures as reasoning topologies, because their representation becomes to a degree spatial, as they are contained within the LLM context. Our study compares existing prompting schemes using the proposed taxonomy, discussing how certain design choices lead to different patterns in performance and cost. We also outline theoretical underpinnings, relationships between prompting and other parts of the LLM ecosystem such as knowledge bases, and the associated research challenges. Our work will help to advance future prompt engineering techniques. △ Less

Submitted 5 April, 2024; v1 submitted 25 January, 2024; originally announced January 2024.

Capitalizing on Next-Generation Optical Communication Systems with Proactive Multi-Period Network Planning

Authors: Jasper Müller, Sai Kireet Patri, Gabriele Di Rosa, Achim Autenrieth, Jörg-Peter Elbers, Carmen Mas-Machuca

Abstract: Optical transport network operators typically follow a pay-as-you-grow strategy for their network deployment. We propose a proactive multi-period planning approach based on heuristic network planning, supporting this deployment strategy while enabling efficient network utilization through next-generation technology. We report 60% less provisioned lightpaths. Optical transport network operators typically follow a pay-as-you-grow strategy for their network deployment. We propose a proactive multi-period planning approach based on heuristic network planning, supporting this deployment strategy while enabling efficient network utilization through next-generation technology. We report 60% less provisioned lightpaths. △ Less

Submitted 22 December, 2023; originally announced December 2023.

Comments: The work has been partially funded by the German Federal Ministry of Education and Research in the project STARFALL (16KIS1418K)

Journal ref: European Conference on Optical Communications (ECOC) 2023

On the Benefits of Rate-Adaptive Transceivers: A Network Planning Study

Authors: Jasper Müller, Gabriele Di Rosa, Tobias Fehenberger, Mario Wenning, Sai Kireet Patri, Jörg-Peter Elbers, Carmen Mas-Machuca

Abstract: Flexible-grid Elastic Optical Networks (EONs) have been widely deployed in recent years to support the growing demand for bandwidth-intensive applications. To address this cost-efficiently, optimized utilization of EONs is required. Next-generation bandwidth-variable transceivers (BVTs) will offer increased adaptivity in symbol rate as well as modulation through probabilistic constellation shaping… ▽ More Flexible-grid Elastic Optical Networks (EONs) have been widely deployed in recent years to support the growing demand for bandwidth-intensive applications. To address this cost-efficiently, optimized utilization of EONs is required. Next-generation bandwidth-variable transceivers (BVTs) will offer increased adaptivity in symbol rate as well as modulation through probabilistic constellation shaping. In this work, we therefore investigate the impact of increased configuration granularity on various aspects of optical networks. We account for practical implementation considerations of BVT configurations for the estimation of the required signal-to-noise ratio. Additionally, an optimization algorithm is presented that selects the most efficient configuration for each considered data rate and bandwidth combination. Based on the advanced transceiver configurations, we conduct a network planning study using a physical-layer-aware algorithm for flexible-grid EONs, and present results for a national and a continental optical backbone network topology. Our research demonstrates that a rise in modulation rate adaptivity results in substantial savings in resources, decreasing the number of necessary lightpaths by as much as 20% in EONs. In contrast, increased symbol rate granularity only results in minor savings. △ Less

Submitted 18 December, 2023; originally announced December 2023.

Comments: Copyright 2023 IEEE. This work has been partially funded in the framework of the CELTIC-NEXT project AI-NET-PROTECT (Project ID C2019/3-4) (#16KIS1279K) and in the programme "Souverän. Digital. Vernetzt." joint project 6G-life (#16KISK002) by the German Federal Ministry of Education and Research

Towards Context-Aware Domain Generalization: Understanding the Benefits and Limits of Marginal Transfer Learning

Authors: Jens Müller, Lars Kühmichel, Martin Rohbeck, Stefan T. Radev, Ullrich Köthe

Abstract: In this work, we analyze the conditions under which information about the context of an input $X$ can improve the predictions of deep learning models in new domains. Following work in marginal transfer learning in Domain Generalization (DG), we formalize the notion of context as a permutation-invariant representation of a set of data points that originate from the same domain as the input itself.… ▽ More In this work, we analyze the conditions under which information about the context of an input $X$ can improve the predictions of deep learning models in new domains. Following work in marginal transfer learning in Domain Generalization (DG), we formalize the notion of context as a permutation-invariant representation of a set of data points that originate from the same domain as the input itself. We offer a theoretical analysis of the conditions under which this approach can, in principle, yield benefits, and formulate two necessary criteria that can be easily verified in practice. Additionally, we contribute insights into the kind of distribution shifts for which the marginal transfer learning approach promises robustness. Empirical analysis shows that our criteria are effective in discerning both favorable and unfavorable scenarios. Finally, we demonstrate that we can reliably detect scenarios where a model is tasked with unwarranted extrapolation in out-of-distribution (OOD) domains, identifying potential failure cases. Consequently, we showcase a method to select between the most predictive and the most robust model, circumventing the well-known trade-off between predictive performance and robustness. △ Less

Submitted 21 February, 2024; v1 submitted 15 December, 2023; originally announced December 2023.

A Golden-Free Formal Method for Trojan Detection in Non-Interfering Accelerators

Authors: Anna Lena Duque Antón, Johannes Müller, Lucas Deutschmann, Mohammad Rahmani Fadiheh, Dominik Stoffel, Wolfgang Kunz

Abstract: The threat of hardware Trojans (HTs) in security-critical IPs like cryptographic accelerators poses severe security risks. The HT detection methods available today mostly rely on golden models and detailed circuit specifications. Often they are specific to certain HT payload types, making pre-silicon verification difficult and leading to security gaps. We propose a novel formal verification method… ▽ More The threat of hardware Trojans (HTs) in security-critical IPs like cryptographic accelerators poses severe security risks. The HT detection methods available today mostly rely on golden models and detailed circuit specifications. Often they are specific to certain HT payload types, making pre-silicon verification difficult and leading to security gaps. We propose a novel formal verification method for HT detection in non-interfering accelerators at the Register Transfer Level (RTL), employing standard formal property checking. Our method guarantees the exhaustive detection of any sequential HT independently of its payload behavior, including physical side channels. It does not require a golden model or a functional specification of the design. The experimental results demonstrate efficient and effective detection of all sequential HTs in accelerators available on Trust-Hub, including those with complex triggers and payloads. △ Less

Submitted 11 December, 2023; originally announced December 2023.

Comments: Accepted for publication at the 27th Design, Automation and Test in Europe Conference (DATE'24), Mar 25-27 2024, Valencia, Spain

Efficient Inverse Design Optimization through Multi-fidelity Simulations, Machine Learning, and Search Space Reduction Strategies

Authors: Luka Grbcic, Juliane Müller, Wibe Albert de Jong

Abstract: This paper introduces a methodology designed to augment the inverse design optimization process in scenarios constrained by limited compute, through the strategic synergy of multi-fidelity evaluations, machine learning models, and optimization algorithms. The proposed methodology is analyzed on two distinct engineering inverse design problems: airfoil inverse design and the scalar field reconstruc… ▽ More This paper introduces a methodology designed to augment the inverse design optimization process in scenarios constrained by limited compute, through the strategic synergy of multi-fidelity evaluations, machine learning models, and optimization algorithms. The proposed methodology is analyzed on two distinct engineering inverse design problems: airfoil inverse design and the scalar field reconstruction problem. It leverages a machine learning model trained with low-fidelity simulation data, in each optimization cycle, thereby proficiently predicting a target variable and discerning whether a high-fidelity simulation is necessitated, which notably conserves computational resources. Additionally, the machine learning model is strategically deployed prior to optimization to compress the design space boundaries, thereby further accelerating convergence toward the optimal solution. The methodology has been employed to enhance two optimization algorithms, namely Differential Evolution and Particle Swarm Optimization. Comparative analyses illustrate performance improvements across both algorithms. Notably, this method is adaptable across any inverse design application, facilitating a synergy between a representative low-fidelity ML model, and high-fidelity simulation, and can be seamlessly applied across any variety of population-based optimization algorithms.} △ Less

Submitted 3 June, 2024; v1 submitted 6 December, 2023; originally announced December 2023.

Stochastic Vision Transformers with Wasserstein Distance-Aware Attention

Authors: Franciskus Xaverius Erick, Mina Rezaei, Johanna Paula Müller, Bernhard Kainz

Abstract: Self-supervised learning is one of the most promising approaches to acquiring knowledge from limited labeled data. Despite the substantial advancements made in recent years, self-supervised models have posed a challenge to practitioners, as they do not readily provide insight into the model's confidence and uncertainty. Tackling this issue is no simple feat, primarily due to the complexity involve… ▽ More Self-supervised learning is one of the most promising approaches to acquiring knowledge from limited labeled data. Despite the substantial advancements made in recent years, self-supervised models have posed a challenge to practitioners, as they do not readily provide insight into the model's confidence and uncertainty. Tackling this issue is no simple feat, primarily due to the complexity involved in implementing techniques that can make use of the latent representations learned during pre-training without relying on explicit labels. Motivated by this, we introduce a new stochastic vision transformer that integrates uncertainty and distance awareness into self-supervised learning (SSL) pipelines. Instead of the conventional deterministic vector embedding, our novel stochastic vision transformer encodes image patches into elliptical Gaussian distributional embeddings. Notably, the attention matrices of these stochastic representational embeddings are computed using Wasserstein distance-based attention, effectively capitalizing on the distributional nature of these embeddings. Additionally, we propose a regularization term based on Wasserstein distance for both pre-training and fine-tuning processes, thereby incorporating distance awareness into latent representations. We perform extensive experiments across different tasks such as in-distribution generalization, out-of-distribution detection, dataset corruption, semi-supervised settings, and transfer learning to other datasets and tasks. Our proposed method achieves superior accuracy and calibration, surpassing the self-supervised baseline in a wide range of experiments on a variety of datasets. △ Less

Submitted 30 November, 2023; originally announced November 2023.

TraM-NeRF: Tracing Mirror and Near-Perfect Specular Reflections through Neural Radiance Fields

Authors: Leif Van Holland, Ruben Bliersbach, Jan U. Müller, Patrick Stotko, Reinhard Klein

Abstract: Implicit representations like Neural Radiance Fields (NeRF) showed impressive results for photorealistic rendering of complex scenes with fine details. However, ideal or near-perfectly specular reflecting objects such as mirrors, which are often encountered in various indoor scenes, impose ambiguities and inconsistencies in the representation of the reconstructed scene leading to severe artifacts… ▽ More Implicit representations like Neural Radiance Fields (NeRF) showed impressive results for photorealistic rendering of complex scenes with fine details. However, ideal or near-perfectly specular reflecting objects such as mirrors, which are often encountered in various indoor scenes, impose ambiguities and inconsistencies in the representation of the reconstructed scene leading to severe artifacts in the synthesized renderings. In this paper, we present a novel reflection tracing method tailored for the involved volume rendering within NeRF that takes these mirror-like objects into account while avoiding the cost of straightforward but expensive extensions through standard path tracing. By explicitly modeling the reflection behavior using physically plausible materials and estimating the reflected radiance with Monte-Carlo methods within the volume rendering formulation, we derive efficient strategies for importance sampling and the transmittance computation along rays from only few samples. We show that our novel method enables the training of consistent representations of such challenging scenes and achieves superior results in comparison to previous state-of-the-art approaches. △ Less

Submitted 16 October, 2023; originally announced October 2023.

Working with XR in Public: Effects on Users and Bystanders

Authors: Verena Biener, Snehanjali Kalamkar, John J Dudley, Jinghui Hu, Per Ola Kristensson, Jörg Müller, Jens Grubert

Abstract: Recent commercial off-the-shelf virtual and augmented reality devices have been promoted as tools for knowledge work and research findings show how this kind of work can benefit from the affordances of extended reality (XR). One major advantage that XR can provide is the enlarged display space that can be used to display virtual screens which is a feature already readily available in many commerci… ▽ More Recent commercial off-the-shelf virtual and augmented reality devices have been promoted as tools for knowledge work and research findings show how this kind of work can benefit from the affordances of extended reality (XR). One major advantage that XR can provide is the enlarged display space that can be used to display virtual screens which is a feature already readily available in many commercial devices. This could be especially helpful in mobile contexts, in which users might not have access to their optimal physical work setup. Such situations often occur in a public setting, for example when working on a train while traveling to a business meeting. At the same time, the use of XR devices is still uncommon in public, which might impact both users and bystanders. Hence, there is a need to better understand the implications of using XR devices for work in public both on the user itself, as well as on bystanders. We report the results of a study in a university cafeteria in which participants used three different systems. In one setup they only used a laptop with a single screen, in a second setup, they combined the laptop with an optical see-through AR headset, and in the third, they combined the laptop with an immersive VR headset. In addition, we also collected 231 responses from bystanders through a questionnaire. The combined results indicate that (1) users feel safer if they can see their physical surroundings; (2) current use of XR in public makes users stand out; and (3) prior XR experience can influence how users feel when using XR in public. △ Less

Submitted 15 October, 2023; originally announced October 2023.

Whole Slide Multiple Instance Learning for Predicting Axillary Lymph Node Metastasis

Authors: Glejdis Shkëmbi, Johanna P. Müller, Zhe Li, Katharina Breininger, Peter Schüffler, Bernhard Kainz

Abstract: Breast cancer is a major concern for women's health globally, with axillary lymph node (ALN) metastasis identification being critical for prognosis evaluation and treatment guidance. This paper presents a deep learning (DL) classification pipeline for quantifying clinical information from digital core-needle biopsy (CNB) images, with one step less than existing methods. A publicly available datase… ▽ More Breast cancer is a major concern for women's health globally, with axillary lymph node (ALN) metastasis identification being critical for prognosis evaluation and treatment guidance. This paper presents a deep learning (DL) classification pipeline for quantifying clinical information from digital core-needle biopsy (CNB) images, with one step less than existing methods. A publicly available dataset of 1058 patients was used to evaluate the performance of different baseline state-of-the-art (SOTA) DL models in classifying ALN metastatic status based on CNB images. An extensive ablation study of various data augmentation techniques was also conducted. Finally, the manual tumor segmentation and annotation step performed by the pathologists was assessed. △ Less

Submitted 6 October, 2023; originally announced October 2023.

Comments: Accepted for MICCAI DEMI Workshop 2023

Journal ref: Data Engineering in Medical Imaging. DEMI 2023. Lecture Notes in Computer Science, vol 14314. Springer, Cham

A New Security Threat in MCUs -- SoC-wide timing side channels and how to find them

Authors: Johannes Müller, Anna Lena Duque Antón, Lucas Deutschmann, Dino Mehmedagić, Mohammad Rahmani Fadiheh, Dominik Stoffel, Wolfgang Kunz

Abstract: Microarchitectural timing side channels have been thoroughly investigated as a security threat in hardware designs featuring shared buffers (e.g., caches) and/or parallelism between attacker and victim task execution. Contradicting common intuitions, recent activities demonstrate, however, that this threat is real also in microcontroller SoCs without such features. In this paper, we describe SoC-w… ▽ More Microarchitectural timing side channels have been thoroughly investigated as a security threat in hardware designs featuring shared buffers (e.g., caches) and/or parallelism between attacker and victim task execution. Contradicting common intuitions, recent activities demonstrate, however, that this threat is real also in microcontroller SoCs without such features. In this paper, we describe SoC-wide timing side channels previously neglected by security analysis and present a new formal method to close this gap. In a case study with the RISC-V Pulpissimo SoC platform, our method found a vulnerability to a so far unknown attack variant that allows an attacker to obtain information about a victim's memory access behavior. After implementing a conservative fix, we were able to verify that the SoC is now secure w.r.t. timing side channels. △ Less

Submitted 22 September, 2023; originally announced September 2023.

ObjectLab: Automated Diagnosis of Mislabeled Images in Object Detection Data

Authors: Ulyana Tkachenko, Aditya Thyagarajan, Jonas Mueller

Abstract: Despite powering sensitive systems like autonomous vehicles, object detection remains fairly brittle in part due to annotation errors that plague most real-world training datasets. We propose ObjectLab, a straightforward algorithm to detect diverse errors in object detection labels, including: overlooked bounding boxes, badly located boxes, and incorrect class label assignments. ObjectLab utilizes… ▽ More Despite powering sensitive systems like autonomous vehicles, object detection remains fairly brittle in part due to annotation errors that plague most real-world training datasets. We propose ObjectLab, a straightforward algorithm to detect diverse errors in object detection labels, including: overlooked bounding boxes, badly located boxes, and incorrect class label assignments. ObjectLab utilizes any trained object detection model to score the label quality of each image, such that mislabeled images can be automatically prioritized for label review/correction. Properly handling erroneous data enables training a better version of the same object detection model, without any change in existing modeling code. Across different object detection datasets (including COCO) and different models (including Detectron-X101 and Faster-RCNN), ObjectLab consistently detects annotation errors with much better precision/recall compared to other label quality scores. △ Less

Submitted 2 September, 2023; originally announced September 2023.

Comments: ICML Workshop on Data-centric Machine Learning Research

Quantifying Uncertainty in Answers from any Language Model and Enhancing their Trustworthiness

Authors: Jiuhai Chen, Jonas Mueller

Abstract: We introduce BSDetector, a method for detecting bad and speculative answers from a pretrained Large Language Model by estimating a numeric confidence score for any output it generated. Our uncertainty quantification technique works for any LLM accessible only via a black-box API, whose training data remains unknown. By expending a bit of extra computation, users of any LLM API can now get the same… ▽ More We introduce BSDetector, a method for detecting bad and speculative answers from a pretrained Large Language Model by estimating a numeric confidence score for any output it generated. Our uncertainty quantification technique works for any LLM accessible only via a black-box API, whose training data remains unknown. By expending a bit of extra computation, users of any LLM API can now get the same response as they would ordinarily, as well as a confidence estimate that cautions when not to trust this response. Experiments on both closed and open-form Question-Answer benchmarks reveal that BSDetector more accurately identifies incorrect LLM responses than alternative uncertainty estimation procedures (for both GPT-3 and ChatGPT). By sampling multiple responses from the LLM and considering the one with the highest confidence score, we can additionally obtain more accurate responses from the same LLM, without any extra training steps. In applications involving automated evaluation with LLMs, accounting for our confidence scores leads to more reliable evaluation in both human-in-the-loop and fully-automated settings (across both GPT 3.5 and 4). △ Less

Submitted 4 October, 2023; v1 submitted 30 August, 2023; originally announced August 2023.

Video Analysis of Behavioral Patterns During Prolonged Work in VR

Authors: Verena Biener, Forouzan Farzinnejad, Rinaldo Schuster, Seyedmasih Tabaei, Leon Lindlein, Jinghui Hu, Negar Nouri, John J. Dudley, Per Ola Kristensson, Jörg Müller, Jens Grubert

Abstract: VR has recently been promoted as a tool for knowledge workers and studies have shown that it has the potential to improve knowledge work. However, studies on its prolonged use have been scarce. A prior study compared working in VR for one week to working in a physical environment, focusing on performance measures and subjective feedback. However, a nuanced understanding and comparison of participa… ▽ More VR has recently been promoted as a tool for knowledge workers and studies have shown that it has the potential to improve knowledge work. However, studies on its prolonged use have been scarce. A prior study compared working in VR for one week to working in a physical environment, focusing on performance measures and subjective feedback. However, a nuanced understanding and comparison of participants' behavior in VR and the physical environment is still missing. To this end, we analyzed video material made available from this previously conducted experiment, carried out over a working week, and present our findings on comparing the behavior of participants while working in VR and in a physical environment. △ Less

Submitted 23 August, 2023; originally announced August 2023.

ACM Class: I.3.7

Feel the Breeze: Promoting Relaxation in Virtual Reality using Mid-Air Haptics

Authors: Naga Sai Surya Vamsy Malladi, Viktorija Paneva, Jörg Müller

Abstract: Mid-air haptic interfaces employ focused ultrasound waves to generate touchless haptic sensations on the skin. Prior studies have demonstrated the potential positive impact of mid-air haptic feedback on virtual experiences, enhancing aspects such as enjoyment, immersion, and sense of agency. As a highly immersive environment, Virtual Reality (VR) is being explored as a tool for stress management a… ▽ More Mid-air haptic interfaces employ focused ultrasound waves to generate touchless haptic sensations on the skin. Prior studies have demonstrated the potential positive impact of mid-air haptic feedback on virtual experiences, enhancing aspects such as enjoyment, immersion, and sense of agency. As a highly immersive environment, Virtual Reality (VR) is being explored as a tool for stress management and relaxation in current research. However, the impact of incorporating mid-air haptic stimuli into relaxing experiences in VR has not been studied thus far. In this paper, for the first time, we design a mid-air haptic stimulation that is congruent with a relaxing scene in VR, and conduct a user study investigating the effectiveness of this experience. Our user study encompasses three different conditions: a control group with no relaxation intervention, a VR-only relaxation experience, and a VR+Haptics relaxation experience that includes the mid-air haptic feedback. While we did not find any significant differences between the conditions, a trend suggesting that the VR+Haptics condition might be associated with greater pleasure emerged, requiring further validation with a larger sample size. These initial findings set the foundation for future investigations into leveraging multimodal interventions in VR, utilising mid-air haptics to potentially enhance relaxation experiences. △ Less

Submitted 18 August, 2023; originally announced August 2023.

Comments: 5 pages, 6 figures. This is the author's version. Final version of records is to appear in 2023 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct)

Enumerating Tarski fixed points on lattices of binary relations

Authors: Julian Müller

Abstract: We study the problem of enumerating Tarski fixed points, focusing on the relational lattices of equivalences, quasiorders and binary relations. We present a polynomial space enumeration algorithm for Tarski fixed points on these lattices and other lattices of polynomial height. It achieves polynomial delay when enumerating fixed points of increasing isotone maps on all three lattices, as well as d… ▽ More We study the problem of enumerating Tarski fixed points, focusing on the relational lattices of equivalences, quasiorders and binary relations. We present a polynomial space enumeration algorithm for Tarski fixed points on these lattices and other lattices of polynomial height. It achieves polynomial delay when enumerating fixed points of increasing isotone maps on all three lattices, as well as decreasing isotone maps on the lattice of binary relations. In those cases in which the enumeration algorithm does not guarantee polynomial delay on the three relational lattices on the other hand, we prove exponential lower bounds for deciding the existence of three fixed points when the isotone map is given as an oracle, and that it is NP-hard to find three or more Tarski fixed points. More generally, we show that any deterministic or bounded-error randomized algorithm must perform a number of queries asymptotically at least as large as the lattice width to decide the existence of three fixed points when the isotone map is given as an oracle. Finally, we demonstrate that our findings yield a polynomial delay and space algorithm for listing bisimulations and instances of some related models of behavioral or role equivalence. △ Less

Submitted 15 August, 2023; originally announced August 2023.

A Scalable Formal Verification Methodology for Data-Oblivious Hardware

Authors: Lucas Deutschmann, Johannes Mueller, Mohammad Rahmani Fadiheh, Dominik Stoffel, Wolfgang Kunz

Abstract: The importance of preventing microarchitectural timing side channels in security-critical applications has surged in recent years. Constant-time programming has emerged as a best-practice technique for preventing the leakage of secret information through timing. It is based on the assumption that the timing of certain basic machine instructions is independent of their respective input data. Howeve… ▽ More The importance of preventing microarchitectural timing side channels in security-critical applications has surged in recent years. Constant-time programming has emerged as a best-practice technique for preventing the leakage of secret information through timing. It is based on the assumption that the timing of certain basic machine instructions is independent of their respective input data. However, whether or not an instruction satisfies this data-independent timing criterion varies between individual processor microarchitectures. In this paper, we propose a novel methodology to formally verify data-oblivious behavior in hardware using standard property checking techniques. The proposed methodology is based on an inductive property that enables scalability even to complex out-of-order cores. We show that proving this inductive property is sufficient to exhaustively verify data-obliviousness at the microarchitectural level. In addition, the paper discusses several techniques that can be used to make the verification process easier and faster. We demonstrate the feasibility of the proposed methodology through case studies on several open-source designs. One case study uncovered a data-dependent timing violation in the extensively verified and highly secure IBEX RISC-V core. In addition to several hardware accelerators and in-order processors, our experiments also include RISC-V BOOM, a complex out-of-order processor, highlighting the scalability of the approach. △ Less

Submitted 11 March, 2024; v1 submitted 15 August, 2023; originally announced August 2023.

The Impact of Different Virtual Work Environments on Flow, Performance, User Emotions, and Preferences

Authors: Alicja Kiluk, Viktorija Paneva, Sofia Seinfeld, Jörg Müller

Abstract: This research explores how different virtual work environments, differing in the type and amount of elements they include, impact users' flow, performance, emotional state, and preferences. Pre-study interviews were conducted to inform the design of three VR work environments: the Dark Room, the Empty Room, and the Furnished Room. Fifteen participants took part in a user study where they engaged i… ▽ More This research explores how different virtual work environments, differing in the type and amount of elements they include, impact users' flow, performance, emotional state, and preferences. Pre-study interviews were conducted to inform the design of three VR work environments: the Dark Room, the Empty Room, and the Furnished Room. Fifteen participants took part in a user study where they engaged in a logic-based task simulating deep work while experiencing each environment. The findings suggest that while objective performance measures did not differ significantly, subjective experiences and perceptions varied across the environments. Participants reported feeling less distracted and more focused in the Dark Room and the Empty Room compared to the Furnished Room. The Empty Room was associated with the highest levels of relaxation and calmness, while the Furnished Room was perceived as visually appealing yet more distracting. These findings highlight the variability of user preferences and emphasise the importance of considering user comfort and well-being in the design of virtual work environments. The study contributes to the better understanding of virtual workspaces and provides insights for designing environments that promote flow, productivity, and user well-being. △ Less

Submitted 14 August, 2023; originally announced August 2023.

Comments: 6 pages, 8 figures. This is the author's version. Final version of records is to appear in 2023 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct)

Estimating label quality and errors in semantic segmentation data via any model

Authors: Vedang Lad, Jonas Mueller

Abstract: The labor-intensive annotation process of semantic segmentation datasets is often prone to errors, since humans struggle to label every pixel correctly. We study algorithms to automatically detect such annotation errors, in particular methods to score label quality, such that the images with the lowest scores are least likely to be correctly labeled. This helps prioritize what data to review in or… ▽ More The labor-intensive annotation process of semantic segmentation datasets is often prone to errors, since humans struggle to label every pixel correctly. We study algorithms to automatically detect such annotation errors, in particular methods to score label quality, such that the images with the lowest scores are least likely to be correctly labeled. This helps prioritize what data to review in order to ensure a high-quality training/evaluation dataset, which is critical in sensitive applications such as medical imaging and autonomous vehicles. Widely applicable, our label quality scores rely on probabilistic predictions from a trained segmentation model -- any model architecture and training procedure can be utilized. Here we study 7 different label quality scoring methods used in conjunction with a DeepLabV3+ or a FPN segmentation model to detect annotation errors in a version of the SYNTHIA dataset. Precision-recall evaluations reveal a score -- the soft-minimum of the model-estimated likelihoods of each pixel's annotated class -- that is particularly effective to identify images that are mislabeled, across multiple types of annotation error. △ Less

Submitted 11 July, 2023; originally announced July 2023.

Comments: ICML Workshop on Data-centric Machine Learning Research 2023

SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis

Authors: Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, Robin Rombach

Abstract: We present SDXL, a latent diffusion model for text-to-image synthesis. Compared to previous versions of Stable Diffusion, SDXL leverages a three times larger UNet backbone: The increase of model parameters is mainly due to more attention blocks and a larger cross-attention context as SDXL uses a second text encoder. We design multiple novel conditioning schemes and train SDXL on multiple aspect ra… ▽ More We present SDXL, a latent diffusion model for text-to-image synthesis. Compared to previous versions of Stable Diffusion, SDXL leverages a three times larger UNet backbone: The increase of model parameters is mainly due to more attention blocks and a larger cross-attention context as SDXL uses a second text encoder. We design multiple novel conditioning schemes and train SDXL on multiple aspect ratios. We also introduce a refinement model which is used to improve the visual fidelity of samples generated by SDXL using a post-hoc image-to-image technique. We demonstrate that SDXL shows drastically improved performance compared the previous versions of Stable Diffusion and achieves results competitive with those of black-box state-of-the-art image generators. In the spirit of promoting open research and fostering transparency in large model training and evaluation, we provide access to code and model weights at https://github.com/Stability-AI/generative-models △ Less

Submitted 4 July, 2023; originally announced July 2023.

Many tasks make light work: Learning to localise medical anomalies from multiple synthetic tasks

Authors: Matthew Baugh, Jeremy Tan, Johanna P. Müller, Mischa Dombrowski, James Batten, Bernhard Kainz

Abstract: There is a growing interest in single-class modelling and out-of-distribution detection as fully supervised machine learning models cannot reliably identify classes not included in their training. The long tail of infinitely many out-of-distribution classes in real-world scenarios, e.g., for screening, triage, and quality control, means that it is often necessary to train single-class models that… ▽ More There is a growing interest in single-class modelling and out-of-distribution detection as fully supervised machine learning models cannot reliably identify classes not included in their training. The long tail of infinitely many out-of-distribution classes in real-world scenarios, e.g., for screening, triage, and quality control, means that it is often necessary to train single-class models that represent an expected feature distribution, e.g., from only strictly healthy volunteer data. Conventional supervised machine learning would require the collection of datasets that contain enough samples of all possible diseases in every imaging modality, which is not realistic. Self-supervised learning methods with synthetic anomalies are currently amongst the most promising approaches, alongside generative auto-encoders that analyse the residual reconstruction error. However, all methods suffer from a lack of structured validation, which makes calibration for deployment difficult and dataset-dependant. Our method alleviates this by making use of multiple visually-distinct synthetic anomaly learning tasks for both training and validation. This enables more robust training and generalisation. With our approach we can readily outperform state-of-the-art methods, which we demonstrate on exemplars in brain MRI and chest X-rays. Code is available at https://github.com/matt-baugh/many-tasks-make-light-work . △ Less

Submitted 3 July, 2023; originally announced July 2023.

Comments: Early accepted to MICCAI 2023

On the Convergence Rate of Gaussianization with Random Rotations

Authors: Felix Draxler, Lars Kühmichel, Armand Rousselot, Jens Müller, Christoph Schnörr, Ullrich Köthe

Abstract: Gaussianization is a simple generative model that can be trained without backpropagation. It has shown compelling performance on low dimensional data. As the dimension increases, however, it has been observed that the convergence speed slows down. We show analytically that the number of required layers scales linearly with the dimension for Gaussian input. We argue that this is because the model i… ▽ More Gaussianization is a simple generative model that can be trained without backpropagation. It has shown compelling performance on low dimensional data. As the dimension increases, however, it has been observed that the convergence speed slows down. We show analytically that the number of required layers scales linearly with the dimension for Gaussian input. We argue that this is because the model is unable to capture dependencies between dimensions. Empirically, we find the same linear increase in cost for arbitrary input $p(x)$, but observe favorable scaling for some distributions. We explore potential speed-ups and formulate challenges for further research. △ Less

Submitted 23 June, 2023; originally announced June 2023.

Map Point Selection for Visual SLAM

Authors: Christiaan J. Müller, Corné E. van Daalen

Abstract: Simultaneous localisation and mapping (SLAM) play a vital role in autonomous robotics. Robotic platforms are often resource-constrained, and this limitation motivates resource-efficient SLAM implementations. While sparse visual SLAM algorithms offer good accuracy for modest hardware requirements, even these more scalable sparse approaches face limitations when applied to large-scale and long-term… ▽ More Simultaneous localisation and mapping (SLAM) play a vital role in autonomous robotics. Robotic platforms are often resource-constrained, and this limitation motivates resource-efficient SLAM implementations. While sparse visual SLAM algorithms offer good accuracy for modest hardware requirements, even these more scalable sparse approaches face limitations when applied to large-scale and long-term scenarios. A contributing factor is that the point clouds resulting from SLAM are inefficient to use and contain significant redundancy. This paper proposes the use of subset selection algorithms to reduce the map produced by sparse visual SLAM algorithms. Information-theoretic techniques have been applied to simpler related problems before, but they do not scale if applied to the full visual SLAM problem. This paper proposes a number of novel information\hyp{}theoretic utility functions for map point selection and optimises these functions using greedy algorithms. The reduced maps are evaluated using practical data alongside an existing visual SLAM implementation (ORB-SLAM 2). Approximate selection techniques proposed in this paper achieve trajectory accuracy comparable to an offline baseline while being suitable for online use. These techniques enable the practical reduction of maps for visual SLAM with competitive trajectory accuracy. Results also demonstrate that SLAM front-end performance can significantly impact the performance of map point selection. This shows the importance of testing map point selection with a front-end implementation. To exploit this, this paper proposes an approach that includes a model of the front-end in the utility function when additional information is available. This approach outperforms alternatives on applicable datasets and highlights future research directions. △ Less

Submitted 22 June, 2023; originally announced June 2023.

Zero-Shot Anomaly Detection with Pre-trained Segmentation Models

Authors: Matthew Baugh, James Batten, Johanna P. Müller, Bernhard Kainz

Abstract: This technical report outlines our submission to the zero-shot track of the Visual Anomaly and Novelty Detection (VAND) 2023 Challenge. Building on the performance of the WINCLIP framework, we aim to enhance the system's localization capabilities by integrating zero-shot segmentation models. In addition, we perform foreground instance segmentation which enables the model to focus on the relevant p… ▽ More This technical report outlines our submission to the zero-shot track of the Visual Anomaly and Novelty Detection (VAND) 2023 Challenge. Building on the performance of the WINCLIP framework, we aim to enhance the system's localization capabilities by integrating zero-shot segmentation models. In addition, we perform foreground instance segmentation which enables the model to focus on the relevant parts of the image, thus allowing the models to better identify small or subtle deviations. Our pipeline requires no external data or information, allowing for it to be directly applied to new datasets. Our team (Variance Vigilance Vanguard) ranked third in the zero-shot track of the VAND challenge, and achieve an average F1-max score of 81.5/24.2 at a sample/pixel level on the VisA dataset. △ Less

Submitted 15 June, 2023; originally announced June 2023.

Comments: Ranked 3rd in zero-shot track of the Visual Anomaly and Novelty Detection (VAND) 2023 Challenge

Detecting Errors in a Numerical Response via any Regression Model

Authors: Hang Zhou, Jonas Mueller, Mayank Kumar, Jane-Ling Wang, Jing Lei

Abstract: Noise plagues many numerical datasets, where the recorded values in the data may fail to match the true underlying values due to reasons including: erroneous sensors, data entry/processing mistakes, or imperfect human estimates. We consider general regression settings with covariates and a potentially corrupted response whose observed values may contain errors. By accounting for various uncertaint… ▽ More Noise plagues many numerical datasets, where the recorded values in the data may fail to match the true underlying values due to reasons including: erroneous sensors, data entry/processing mistakes, or imperfect human estimates. We consider general regression settings with covariates and a potentially corrupted response whose observed values may contain errors. By accounting for various uncertainties, we introduced veracity scores that distinguish between genuine errors and natural data fluctuations, conditioned on the available covariate information in the dataset. We propose a simple yet efficient filtering procedure for eliminating potential errors, and establish theoretical guarantees for our method. We also contribute a new error detection benchmark involving 5 regression datasets with real-world numerical errors (for which the true values are also known). In this benchmark and additional simulation studies, our method identifies incorrect values with better precision/recall than other approaches. △ Less

Submitted 12 March, 2024; v1 submitted 25 May, 2023; originally announced May 2023.

Detecting Dataset Drift and Non-IID Sampling via k-Nearest Neighbors

Authors: Jesse Cummings, Elías Snorrason, Jonas Mueller

Abstract: We present a straightforward statistical test to detect certain violations of the assumption that the data are Independent and Identically Distributed (IID). The specific form of violation considered is common across real-world applications: whether the examples are ordered in the dataset such that almost adjacent examples tend to have more similar feature values (e.g. due to distributional drift,… ▽ More We present a straightforward statistical test to detect certain violations of the assumption that the data are Independent and Identically Distributed (IID). The specific form of violation considered is common across real-world applications: whether the examples are ordered in the dataset such that almost adjacent examples tend to have more similar feature values (e.g. due to distributional drift, or attractive interactions between datapoints). Based on a k-Nearest Neighbors estimate, our approach can be used to audit any multivariate numeric data as well as other data types (image, text, audio, etc.) that can be numerically represented, perhaps with model embeddings. Compared with existing methods to detect drift or auto-correlation, our approach is both applicable to more types of data and also able to detect a wider variety of IID violations in practice. Code: https://github.com/cleanlab/cleanlab △ Less

Submitted 25 May, 2023; originally announced May 2023.

The Graph Database Interface: Scaling Online Transactional and Analytical Graph Workloads to Hundreds of Thousands of Cores

Authors: Maciej Besta, Robert Gerstenberger, Marc Fischer, Michał Podstawski, Nils Blach, Berke Egeli, Georgy Mitenkov, Wojciech Chlapek, Marek Michalewicz, Hubert Niewiadomski, Jürgen Müller, Torsten Hoefler

Abstract: Graph databases (GDBs) are crucial in academic and industry applications. The key challenges in developing GDBs are achieving high performance, scalability, programmability, and portability. To tackle these challenges, we harness established practices from the HPC landscape to build a system that outperforms all past GDBs presented in the literature by orders of magnitude, for both OLTP and OLAP w… ▽ More Graph databases (GDBs) are crucial in academic and industry applications. The key challenges in developing GDBs are achieving high performance, scalability, programmability, and portability. To tackle these challenges, we harness established practices from the HPC landscape to build a system that outperforms all past GDBs presented in the literature by orders of magnitude, for both OLTP and OLAP workloads. For this, we first identify and crystallize performance-critical building blocks in the GDB design, and abstract them into a portable and programmable API specification, called the Graph Database Interface (GDI), inspired by the best practices of MPI. We then use GDI to design a GDB for distributed-memory RDMA architectures. Our implementation harnesses one-sided RDMA communication and collective operations, and it offers architecture-independent theoretical performance guarantees. The resulting design achieves extreme scales of more than a hundred thousand cores. Our work will facilitate the development of next-generation extreme-scale graph databases. △ Less

Submitted 20 November, 2023; v1 submitted 18 May, 2023; originally announced May 2023.

Comments: Best Paper Finalist at ACM Supercomputing '23 (SC '23)

Journal ref: Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis, 2023 (SC '23)

QoT estimation using EGN-assisted machine learning for multi-period network planning

Authors: Jasper Müller, Sai Kireet Patri, Tobias Fehenberger, Helmut Griesser, Jörg-Peter Elbers, Carmen Mas-Machuca

Abstract: The rapidly growing traffic demands in fiber-optical networks require flexibility and accuracy in configuring lightpaths, for which fast and accurate quality of transmission (QoT) estimation is of pivotal importance. This paper introduces a machine learning (ML)-based QoT estimation approach that meets these requirements. The proposed gradient-boosting ML model uses precomputed per-channel self-ch… ▽ More The rapidly growing traffic demands in fiber-optical networks require flexibility and accuracy in configuring lightpaths, for which fast and accurate quality of transmission (QoT) estimation is of pivotal importance. This paper introduces a machine learning (ML)-based QoT estimation approach that meets these requirements. The proposed gradient-boosting ML model uses precomputed per-channel self-channel-interference values as representative and condensed features to estimate non-linear interference in a flexible-grid network. With an enhanced Gaussian noise (GN) model simulation as the baseline, the ML model achieves a mean absolute signal-to-noise ratio error of approximately 0.1 dB, which is an improvement over the GN model. For three different network topologies and network planning approaches of varying complexities, a multi-period network planning study is performed in which ML and GN are compared as path computation elements (PCEs). The results show that the ML PCE is capable of matching or slightly improving the performance of the GN PCE on all topologies while reducing significantly the computation time of network planning by up to 70%. △ Less

Submitted 12 May, 2023; originally announced May 2023.

Comments: This work has been partially funded in the framework of the CELTIC-NEXT project AI-NET-PROTECT (Project ID C2019/3-4) by the German Federal Ministry of Education and Research (#16KIS1279K). Carmen Mas-Machuca acknowledges the support by the Federal Ministry of Education and Research of Germany (BMBF) in the programme of "Souverän. Digital. Vernetzt." joint project 6G-life (#16KISK002)

Journal ref: Journal of Optical Communications and Networking (JOCN) 2022

Multi-Wavelength Transponders for High-capacity Optical Networks: A Physical-layer-aware Network Planning Study

Authors: Jasper Müller, Ognjen Jovanovic, Tobias Fehenberger, Gabriele Di Rosa, Jörg-Peter Elbers, Carmen Mas-Machuca

Abstract: Continued cost- and power-efficient capacity scaling in optical networks is imperative to keep pace with ever-increasing traffic demands. In this paper, we investigate multi-wavelength transponders as a potential way forward. Suitable system architectures and realistic specifications of multi-wavelength transponders are identified and analyzed in terms of transmit OSNR penalties and spectral const… ▽ More Continued cost- and power-efficient capacity scaling in optical networks is imperative to keep pace with ever-increasing traffic demands. In this paper, we investigate multi-wavelength transponders as a potential way forward. Suitable system architectures and realistic specifications of multi-wavelength transponders are identified and analyzed in terms of transmit OSNR penalties and spectral constraints. We investigate the performance for different specifications as compared to single-wavelength transponders in a network planning study on two network topologies, developing guidelines for multi-wavelength transponders specifications and their potential benefits. The studies show a reduction in the number of required lasers of up to 83% at the expense of a slight increase in number of lightpaths, demonstrating the potential for significant cost savings and efficiency improvements. △ Less

Submitted 12 May, 2023; originally announced May 2023.

Comments: The work has been partially funded by the German Federal Ministry of Education and Research in the project STARFALL (16KIS1418K) and the programme of "Souverän. Digital. Vernetzt." joint project 6G-life (16KISK002), as well as the European Research Council through the ERC-CoG FRECOM project (grant no. 771878)

Journal ref: Journal of Optical Communications and Networking (JOCN) 2023

A Protocol for Cast-as-Intended Verifiability with a Second Device

Authors: Johannes Müller, Tomasz Truderung

Abstract: Numerous institutions, such as companies, universities, or non-governmental organizations, employ Internet voting for remote elections. Since the main purpose of an election is to determine the voters' will, it is fundamentally important to ensure that the final election result correctly reflects the voters' votes. To this end, modern secure Internet voting schemes aim for what is called end-to-en… ▽ More Numerous institutions, such as companies, universities, or non-governmental organizations, employ Internet voting for remote elections. Since the main purpose of an election is to determine the voters' will, it is fundamentally important to ensure that the final election result correctly reflects the voters' votes. To this end, modern secure Internet voting schemes aim for what is called end-to-end verifiability. This fundamental security property ensures that the correctness of the final result can be verified, even if some of the computers or parties involved are malfunctioning or corrupted. A standard component in this approach is so called cast-as-intended verifiability which enables individual voters to verify that the ballots cast on their behalf contain their intended choices. Numerous approaches for cast-as-intended verifiability have been proposed in the literature, some of which have also been employed in real-life Internet elections. One of the well established approaches for cast-as-intended verifiability is to employ a second device which can be used by voters to audit their submitted ballots. This approach offers several advantages - including support for flexible ballot/election types and intuitive user experience - and it has been used in real-life elections, for instance in Estonia. In this work, we improve the existing solutions for cast-as-intended verifiability based on the use of a second device. We propose a solution which, while preserving the advantageous practical properties sketched above, provides tighter security guarantees. Our method does not increase the risk of vote-selling when compared to the underlying voting protocol being augmented and, to achieve this, it requires only comparatively weak trust assumptions. It can be combined with various voting protocols, including commitment-based systems offering everlasting privacy. △ Less

Submitted 19 April, 2023; originally announced April 2023.