-
Human Factors in Model-Driven Engineering: Future Research Goals and Initiatives for MDE
Authors:
Grischa Liebel,
Jil Klünder,
Regina Hebig,
Christopher Lazik,
Inês Nunes,
Isabella Graßl,
Jan-Philipp Steghöfer,
Joeri Exelmans,
Julian Oertel,
Kai Marquardt,
Katharina Juhnke,
Kurt Schneider,
Lucas Gren,
Lucia Happe,
Marc Herrmann,
Marvin Wyrich,
Matthias Tichy,
Miguel Goulão,
Rebekka Wohlrab,
Reyhaneh Kalantari,
Robert Heinrich,
Sandra Greiner,
Satrio Adi Rukmono,
Shalini Chakraborty,
Silvia Abrahão
, et al. (1 additional authors not shown)
Abstract:
Purpose: Software modelling and Model-Driven Engineering (MDE) is traditionally studied from a technical perspective. However, one of the core motivations behind the use of software models is inherently human-centred. Models aim to enable practitioners to communicate about software designs, make software understandable, or make software easier to write through domain-specific modelling languages.…
▽ More
Purpose: Software modelling and Model-Driven Engineering (MDE) is traditionally studied from a technical perspective. However, one of the core motivations behind the use of software models is inherently human-centred. Models aim to enable practitioners to communicate about software designs, make software understandable, or make software easier to write through domain-specific modelling languages. Several recent studies challenge the idea that these aims can always be reached and indicate that human factors play a role in the success of MDE. However, there is an under-representation of research focusing on human factors in modelling. Methods: During a GI-Dagstuhl seminar, topics related to human factors in modelling were discussed by 26 expert participants from research and industry. Results: In breakout groups, five topics were covered in depth, namely modelling human aspects, factors of modeller experience, diversity and inclusion in MDE, collaboration and MDE, and teaching human-aware MDE. Conclusion: We summarise our insights gained during the discussions on the five topics. We formulate research goals, questions, and propositions that support directing future initiatives towards an MDE community that is aware of and supportive of human factors and values.
△ Less
Submitted 29 April, 2024;
originally announced April 2024.
-
AudioProtoPNet: An interpretable deep learning model for bird sound classification
Authors:
René Heinrich,
Bernhard Sick,
Christoph Scholz
Abstract:
Recently, scientists have proposed several deep learning models to monitor the diversity of bird species. These models can detect bird species with high accuracy by analyzing acoustic signals. However, traditional deep learning algorithms are black-box models that provide no insight into their decision-making process. For domain experts, such as ornithologists, it is crucial that these models are…
▽ More
Recently, scientists have proposed several deep learning models to monitor the diversity of bird species. These models can detect bird species with high accuracy by analyzing acoustic signals. However, traditional deep learning algorithms are black-box models that provide no insight into their decision-making process. For domain experts, such as ornithologists, it is crucial that these models are not only efficient, but also interpretable in order to be used as assistive tools. In this study, we present an adaption of the Prototypical Part Network (ProtoPNet) for audio classification that provides inherent interpretability through its model architecture. Our approach is based on a ConvNeXt backbone architecture for feature extraction and learns prototypical patterns for each bird species using spectrograms of the training data. Classification of new data is done by comparison with these prototypes in latent space, which simultaneously serve as easily understandable explanations for the model's decisions. We evaluated the performance of our model on seven different datasets representing bird species from different geographical regions. In our experiments, the model showed excellent results, achieving an average AUROC of 0.82 and an average cmAP of 0.37 across the seven datasets, making it comparable to state-of-the-art black-box models for bird sound classification. Thus, this work demonstrates that even for the challenging task of bioacoustic bird classification, powerful yet interpretable deep learning models can be developed to provide valuable insights to domain experts.
△ Less
Submitted 29 May, 2024; v1 submitted 16 April, 2024;
originally announced April 2024.
-
BirdSet: A Dataset and Benchmark for Classification in Avian Bioacoustics
Authors:
Lukas Rauch,
Raphael Schwinger,
Moritz Wirth,
René Heinrich,
Denis Huseljic,
Jonas Lange,
Stefan Kahl,
Bernhard Sick,
Sven Tomforde,
Christoph Scholz
Abstract:
Deep learning (DL) models have emerged as a powerful tool in avian bioacoustics to assess environmental health. To maximize the potential of cost-effective and minimal-invasive passive acoustic monitoring (PAM), DL models must analyze bird vocalizations across a wide range of species and environmental conditions. However, data fragmentation challenges a comprehensive evaluation of generalization p…
▽ More
Deep learning (DL) models have emerged as a powerful tool in avian bioacoustics to assess environmental health. To maximize the potential of cost-effective and minimal-invasive passive acoustic monitoring (PAM), DL models must analyze bird vocalizations across a wide range of species and environmental conditions. However, data fragmentation challenges a comprehensive evaluation of generalization performance. Therefore, we introduce the BirdSet dataset, comprising approximately 520,000 global bird recordings for training and over 400 hours of PAM recordings for testing. Our benchmark offers baselines for several DL models to enhance comparability and consolidate research across studies, along with code implementations that include comprehensive training and evaluation protocols.
△ Less
Submitted 17 June, 2024; v1 submitted 15 March, 2024;
originally announced March 2024.
-
An Extensible Framework for Architecture-Based Data Flow Analysis for Information Security
Authors:
Nicolas Boltz,
Sebastian Hahner,
Christopher Gerking,
Robert Heinrich
Abstract:
The growing interconnection between software systems increases the need for security already at design time. Security-related properties like confidentiality are often analyzed based on data flow diagrams (DFDs). However, manually analyzing DFDs of large software systems is bothersome and error-prone, and adjusting an already deployed software is costly. Additionally, closed analysis ecosystems li…
▽ More
The growing interconnection between software systems increases the need for security already at design time. Security-related properties like confidentiality are often analyzed based on data flow diagrams (DFDs). However, manually analyzing DFDs of large software systems is bothersome and error-prone, and adjusting an already deployed software is costly. Additionally, closed analysis ecosystems limit the reuse of modeled information and impede comprehensive statements about a system's security. In this paper, we present an open and extensible framework for data flow analysis. The central element of our framework is our new implementation of a well-validated data-flow-based analysis approach. The framework is compatible with DFDs and can also extract data flows from the Palladio architectural description language. We showcase the extensibility with multiple model and analysis extensions. Our evaluation indicates that we can analyze similar scenarios while achieving higher scalability compared to previous implementations.
△ Less
Submitted 14 March, 2024;
originally announced March 2024.
-
COSTREAM: Learned Cost Models for Operator Placement in Edge-Cloud Environments
Authors:
Roman Heinrich,
Carsten Binnig,
Harald Kornmayer,
Manisha Luthra
Abstract:
In this work, we present COSTREAM, a novel learned cost model for Distributed Stream Processing Systems that provides accurate predictions of the execution costs of a streaming query in an edge-cloud environment. The cost model can be used to find an initial placement of operators across heterogeneous hardware, which is particularly important in these environments. In our evaluation, we demonstrat…
▽ More
In this work, we present COSTREAM, a novel learned cost model for Distributed Stream Processing Systems that provides accurate predictions of the execution costs of a streaming query in an edge-cloud environment. The cost model can be used to find an initial placement of operators across heterogeneous hardware, which is particularly important in these environments. In our evaluation, we demonstrate that COSTREAM can produce highly accurate cost estimates for the initial operator placement and even generalize to unseen placements, queries, and hardware. When using COSTREAM to optimize the placements of streaming operators, a median speed-up of around 21x can be achieved compared to baselines.
△ Less
Submitted 13 March, 2024;
originally announced March 2024.
-
Quantifying and combining uncertainty for improving the behavior of Digital Twin Systems
Authors:
Julien Deantoni,
Paula Muñoz,
Cláudio Gomes,
Clark Verbrugge,
Rakshit Mittal,
Robert Heinrich,
Stijn Bellis,
Antonio Vallecillo
Abstract:
Uncertainty is an inherent property of any complex system, especially those that integrate physical parts or operate in real environments. In this paper, we focus on the Digital Twins of adaptive systems, which are particularly complex to design, verify, and optimize. One of the problems of having two systems (the physical one and its digital replica) is that their behavior may not always be consi…
▽ More
Uncertainty is an inherent property of any complex system, especially those that integrate physical parts or operate in real environments. In this paper, we focus on the Digital Twins of adaptive systems, which are particularly complex to design, verify, and optimize. One of the problems of having two systems (the physical one and its digital replica) is that their behavior may not always be consistent. In addition, both twins are normally subject to different types of uncertainties, which complicates their comparison. In this paper we propose the explicit representation and treatment of the uncertainty of both twins, and show how this enables a more accurate comparison of their behaviors. Furthermore, this allows us to reduce the overall system uncertainty and improve its behavior by properly averaging the individual uncertainties of the two twins. An exemplary incubator system is used to illustrate and validate our proposal.
△ Less
Submitted 16 February, 2024;
originally announced February 2024.
-
Quantifying Software Correctness by Combining Architecture Modeling and Formal Program Analysis
Authors:
Florian Lanzinger,
Christian Martin,
Frederik Reiche,
Samuel Teuber,
Robert Heinrich,
Alexander Weigl
Abstract:
Most formal methods see the correctness of a software system as a binary decision. However, proving the correctness of complex systems completely is difficult because they are composed of multiple components, usage scenarios, and environments. We present QuAC, a modular approach for quantifying the correctness of service-oriented software systems by combining software architecture modeling with de…
▽ More
Most formal methods see the correctness of a software system as a binary decision. However, proving the correctness of complex systems completely is difficult because they are composed of multiple components, usage scenarios, and environments. We present QuAC, a modular approach for quantifying the correctness of service-oriented software systems by combining software architecture modeling with deductive verification. Our approach is based on a model of the service-oriented architecture and the probabilistic usage scenarios of the system. The correctness of a single service is approximated by a coverage region, which is a formula describing which inputs for that service are proven to not lead to an erroneous execution. The coverage regions can be determined by a combination of various analyses, e.g., formal verification, expert estimations, or testing. The coverage regions and the software model are then combined into a probabilistic program. From this, we can compute the probability that under a given usage profile no service is called outside its coverage region. If the coverage region is large enough, then instead of attempting to get 100% coverage, which may be prohibitively expensive, run-time verification or testing approaches may be used to deal with inputs outside the coverage region. We also present an implementation of QuAC for Java using the modeling tool Palladio and the deductive verification tool KeY. We demonstrate its usability by applying it to a software simulation of an energy system.
△ Less
Submitted 25 January, 2024;
originally announced January 2024.
-
Tool-Supported Architecture-Based Data Flow Analysis for Confidentiality
Authors:
Felix Schwickerath,
Nicolas Boltz,
Sebastian Hahner,
Maximilian Walter,
Christopher Gerking,
Robert Heinrich
Abstract:
Through the increasing interconnection between various systems, the need for confidential systems is increasing. Confidential systems share data only with authorized entities. However, estimating the confidentiality of a system is complex, and adjusting an already deployed software is costly. Thus, it is helpful to have confidentiality analyses, which can estimate the confidentiality already at de…
▽ More
Through the increasing interconnection between various systems, the need for confidential systems is increasing. Confidential systems share data only with authorized entities. However, estimating the confidentiality of a system is complex, and adjusting an already deployed software is costly. Thus, it is helpful to have confidentiality analyses, which can estimate the confidentiality already at design time. Based on an existing data-flow-based confidentiality analysis concept, we reimplemented a data flow analysis as a Java-based tool. The tool uses the software architecture to identify access violations based on the data flow. The evaluation for our tool indicates that we can analyze similar scenarios and scale for certain scenarios better than the existing analysis.
△ Less
Submitted 3 August, 2023;
originally announced August 2023.
-
Targeted Adversarial Attacks on Wind Power Forecasts
Authors:
René Heinrich,
Christoph Scholz,
Stephan Vogt,
Malte Lehna
Abstract:
In recent years, researchers proposed a variety of deep learning models for wind power forecasting. These models predict the wind power generation of wind farms or entire regions more accurately than traditional machine learning algorithms or physical models. However, latest research has shown that deep learning models can often be manipulated by adversarial attacks. Since wind power forecasts are…
▽ More
In recent years, researchers proposed a variety of deep learning models for wind power forecasting. These models predict the wind power generation of wind farms or entire regions more accurately than traditional machine learning algorithms or physical models. However, latest research has shown that deep learning models can often be manipulated by adversarial attacks. Since wind power forecasts are essential for the stability of modern power systems, it is important to protect them from this threat. In this work, we investigate the vulnerability of two different forecasting models to targeted, semi-targeted, and untargeted adversarial attacks. We consider a Long Short-Term Memory (LSTM) network for predicting the power generation of individual wind farms and a Convolutional Neural Network (CNN) for forecasting the wind power generation throughout Germany. Moreover, we propose the Total Adversarial Robustness Score (TARS), an evaluation metric for quantifying the robustness of regression models to targeted and semi-targeted adversarial attacks. It assesses the impact of attacks on the model's performance, as well as the extent to which the attacker's goal was achieved, by assigning a score between 0 (very vulnerable) and 1 (very robust). In our experiments, the LSTM forecasting model was fairly robust and achieved a TARS value of over 0.78 for all adversarial attacks investigated. The CNN forecasting model only achieved TARS values below 0.10 when trained ordinarily, and was thus very vulnerable. Yet, its robustness could be significantly improved by adversarial training, which always resulted in a TARS above 0.46.
△ Less
Submitted 17 August, 2023; v1 submitted 29 March, 2023;
originally announced March 2023.
-
Zero-Shot Cost Models for Distributed Stream Processing
Authors:
Roman Heinrich,
Manisha Luthra,
Harald Kornmayer,
Carsten Binnig
Abstract:
This paper proposes a learned cost estimation model for Distributed Stream Processing Systems (DSPS) with an aim to provide accurate cost predictions of executing queries. A major premise of this work is that the proposed learned model can generalize to the dynamics of streaming workloads out-of-the-box. This means a model once trained can accurately predict performance metrics such as latency and…
▽ More
This paper proposes a learned cost estimation model for Distributed Stream Processing Systems (DSPS) with an aim to provide accurate cost predictions of executing queries. A major premise of this work is that the proposed learned model can generalize to the dynamics of streaming workloads out-of-the-box. This means a model once trained can accurately predict performance metrics such as latency and throughput even if the characteristics of the data and workload or the deployment of operators to hardware changes at runtime. That way, the model can be used to solve tasks such as optimizing the placement of operators to minimize the end-to-end latency of a streaming query or maximize its throughput even under varying conditions. Our evaluation on a well-known DSPS, Apache Storm, shows that the model can predict accurately for unseen workloads and queries while generalizing across real-world benchmarks.
△ Less
Submitted 8 July, 2022;
originally announced July 2022.
-
Towards fuzzification of adaptation rules in self-adaptive architectures
Authors:
Tomáš Bureš,
Petr Hnětynka,
Martin Kruliš,
Danylo Khalyeyev,
Sebastian Hahner,
Stephan Seifermann,
Maximilian Walter,
Robert Heinrich
Abstract:
In this paper, we focus on exploiting neural networks for the analysis and planning stage in self-adaptive architectures. The studied motivating cases in the paper involve existing (legacy) self-adaptive architectures and their adaptation logic, which has been specified by logical rules. We further assume that there is a need to endow these systems with the ability to learn based on examples of in…
▽ More
In this paper, we focus on exploiting neural networks for the analysis and planning stage in self-adaptive architectures. The studied motivating cases in the paper involve existing (legacy) self-adaptive architectures and their adaptation logic, which has been specified by logical rules. We further assume that there is a need to endow these systems with the ability to learn based on examples of inputs and expected outputs. One simple option to address such a need is to replace the reasoning based on logical rules with a neural network. However, this step brings several problems that often create at least a temporary regress. The reason is the logical rules typically represent a large and tested body of domain knowledge, which may be lost if the logical rules are replaced by a neural network. Further, the black-box nature of generic neural networks obfuscates how the systems work inside and consequently introduces more uncertainty. In this paper, we present a method that makes it possible to endow an existing self-adaptive architectures with the ability to learn using neural networks, while preserving domain knowledge existing in the logical rules. We introduce a continuum between the existing rule-based system and a system based on a generic neural network. We show how to navigate in this continuum and create a neural network architecture that naturally embeds the original logical rules and how to gradually scale the learning potential of the network, thus controlling the uncertainty inherent to all soft computing models. We showcase and evaluate the approach on representative excerpts from two larger real-life use cases.
△ Less
Submitted 17 December, 2021;
originally announced December 2021.
-
A Reinforcement Learning Approach for the Continuous Electricity Market of Germany: Trading from the Perspective of a Wind Park Operator
Authors:
Malte Lehna,
Björn Hoppmann,
René Heinrich,
Christoph Scholz
Abstract:
With the rising extension of renewable energies, the intraday electricity markets have recorded a growing popularity amongst traders as well as electric utilities to cope with the induced volatility of the energy supply. Through their short trading horizon and continuous nature, the intraday markets offer the ability to adjust trading decisions from the day-ahead market or reduce trading risk in a…
▽ More
With the rising extension of renewable energies, the intraday electricity markets have recorded a growing popularity amongst traders as well as electric utilities to cope with the induced volatility of the energy supply. Through their short trading horizon and continuous nature, the intraday markets offer the ability to adjust trading decisions from the day-ahead market or reduce trading risk in a short-term notice. Producers of renewable energies utilize the intraday market to lower their forecast risk, by modifying their provided capacities based on current forecasts. However, the market dynamics are complex due to the fact that the power grids have to remain stable and electricity is only partly storable. Consequently, robust and intelligent trading strategies are required that are capable to operate in the intraday market. In this work, we propose a novel autonomous trading approach based on Deep Reinforcement Learning (DRL) algorithms as a possible solution. For this purpose, we model the intraday trade as a Markov Decision Problem (MDP) and employ the Proximal Policy Optimization (PPO) algorithm as our DRL approach. A simulation framework is introduced that enables the trading of the continuous intraday price in a resolution of one minute steps. We test our framework in a case study from the perspective of a wind park operator. We include next to general trade information both price and wind forecasts. On a test scenario of German intraday trading results from 2018, we are able to outperform multiple baselines with at least 45.24% improvement, showing the advantage of the DRL algorithm. However, we also discuss limitations and enhancements of the DRL agent, in order to increase the performance in future works.
△ Less
Submitted 26 November, 2021;
originally announced November 2021.
-
How is Performance Addressed in DevOps? A Survey on Industrial Practices
Authors:
Cor-Paul Bezemer,
Simon Eismann,
Vincenzo Ferme,
Johannes Grohmann,
Robert Heinrich,
Pooyan Jamshidi,
Weiyi Shang,
André van Hoorn,
Monica Villaviencio,
Jürgen Walter,
Felix Willnecker
Abstract:
DevOps is a modern software engineering paradigm that is gaining widespread adoption in industry. The goal of DevOps is to bring software changes into production with a high frequency and fast feedback cycles. This conflicts with software quality assurance activities, particularly with respect to performance. For instance, performance evaluation activities -- such as load testing -- require a cons…
▽ More
DevOps is a modern software engineering paradigm that is gaining widespread adoption in industry. The goal of DevOps is to bring software changes into production with a high frequency and fast feedback cycles. This conflicts with software quality assurance activities, particularly with respect to performance. For instance, performance evaluation activities -- such as load testing -- require a considerable amount of time to get statistically significant results.
We conducted an industrial survey to get insights into how performance is addressed in industrial DevOps settings. In particular, we were interested in the frequency of executing performance evaluations, the tools being used, the granularity of the obtained performance data, and the use of model-based techniques. The survey responses, which come from a wide variety of participants from different industry sectors, indicate that the complexity of performance engineering approaches and tools is a barrier for wide-spread adoption of performance analysis in DevOps. The implication of our results is that performance analysis tools need to have a short learning curve, and should be easy to integrate into the DevOps pipeline.
△ Less
Submitted 21 August, 2018;
originally announced August 2018.