-
The Effects of Computational Resources on Flaky Tests
Authors:
Denini Silva,
Martin Gruber,
Satyajit Gokhale,
Ellen Arteca,
Alexi Turcotte,
Marcelo d'Amorim,
Wing Lam,
Stefan Winter,
Jonathan Bell
Abstract:
Flaky tests are tests that nondeterministically pass and fail in unchanged code. These tests can be detrimental to developers' productivity. Particularly when tests run in continuous integration environments, the tests may be competing for access to limited computational resources (CPUs, memory etc.), and we hypothesize that resource (in)availability may be a significant factor in the failure rate…
▽ More
Flaky tests are tests that nondeterministically pass and fail in unchanged code. These tests can be detrimental to developers' productivity. Particularly when tests run in continuous integration environments, the tests may be competing for access to limited computational resources (CPUs, memory etc.), and we hypothesize that resource (in)availability may be a significant factor in the failure rate of flaky tests. We present the first assessment of the impact that computational resources have on flaky tests, including a total of 52 projects written in Java, JavaScript and Python, and 27 different resource configurations. Using a rigorous statistical methodology, we determine which tests are RAFT (Resource-Affected Flaky Tests). We find that 46.5% of the flaky tests in our dataset are RAFT, indicating that a substantial proportion of flaky-test failures can be avoided by adjusting the resources available when running tests. We report RAFTs and configurations to avoid them to developers, and received interest to either fix the RAFTs or to improve the specifications of the projects so that tests would be run only in configurations that are unlikely to encounter RAFT failures. Our results also have implications for researchers attempting to detect flaky tests, e.g., reducing the resources available when running tests is a cost-effective approach to detect more flaky failures.
△ Less
Submitted 18 October, 2023;
originally announced October 2023.
-
npm-filter: Automating the mining of dynamic information from npm packages
Authors:
Ellen Arteca,
Alexi Turcotte
Abstract:
The static properties of code repositories, e.g., lines of code, dependents, dependencies, etc. can be readily scraped from code hosting platforms such as GitHub, and from package management systems such as npm for JavaScript; Although no less important, information related to the dynamic properties of programs, e.g., number of tests in a test suite that pass or fail, is less readily available. Th…
▽ More
The static properties of code repositories, e.g., lines of code, dependents, dependencies, etc. can be readily scraped from code hosting platforms such as GitHub, and from package management systems such as npm for JavaScript; Although no less important, information related to the dynamic properties of programs, e.g., number of tests in a test suite that pass or fail, is less readily available. The ability to easily collect this dynamic information could be immensely useful to researchers conducting corpus analyses, as they could differentiate projects based on properties that can only be observed by running them.
In this paper, we present npm-filter, an automated tool that can download, install, build, test, and run custom user scripts over the source code of JavaScript projects available on npm, the most popular JavaScript package manager. We outline this tool, describe its implementation, and show that npm-filter has already been useful in developing evaluation suites for multiple JavaScript tools.
△ Less
Submitted 29 March, 2022; v1 submitted 20 January, 2022;
originally announced January 2022.
-
Stubbifier: Debloating Dynamic Server-Side JavaScript Applications
Authors:
Alexi Turcotte,
Ellen Arteca,
Ashish Mishra,
Saba Alimadadi,
Frank Tip
Abstract:
JavaScript is an increasingly popular language for server-side development, thanks in part to the Node.js runtime environment and its vast ecosystem of modules. With the Node.js package manager npm, users are able to easily include external modules as dependencies in their projects. However, npm installs modules with all of their functionality, even if only a fraction is needed, which causes an un…
▽ More
JavaScript is an increasingly popular language for server-side development, thanks in part to the Node.js runtime environment and its vast ecosystem of modules. With the Node.js package manager npm, users are able to easily include external modules as dependencies in their projects. However, npm installs modules with all of their functionality, even if only a fraction is needed, which causes an undue increase in code size. Eliminating this unused functionality from distributions is desirable, but the sound analysis required to find unused code is difficult due to JavaScript's extreme dynamicity.
We present a fully automatic technique that identifies unused code by constructing static or dynamic call graphs from the application's tests, and replacing code deemed unreachable with either file- or function-level stubs. If a stub is called, it will fetch and execute the original code on-demand, thus relaxing the requirement that the call graph be sound. The technique also provides an optional guarded execution mode to guard application against injection vulnerabilities in untested code that resulted from stub expansion.
This technique is implemented in an open source tool called Stubbifier, which supports the ECMAScript 2019 standard. In an empirical evaluation on 15 Node.js applications and 75 clients of these applications, Stubbifier reduced application size by 56% on average while incurring only minor performance overhead. The evaluation also shows that Stubbifier's guarded execution mode is capable of preventing several known injection vulnerabilities that are manifested in stubbed-out code. Finally, Stubbifier can work alongside bundlers, popular JavaScript tools for bundling an application with its dependencies. For the considered subject applications, we measured an average size reduction of 37% in bundled distributions.
△ Less
Submitted 27 October, 2021;
originally announced October 2021.
-
Token Swapping on Trees
Authors:
Ahmad Biniaz,
Kshitij Jain,
Anna Lubiw,
Zuzana Masárová,
Tillmann Miltzow,
Debajyoti Mondal,
Anurag Murty Naredla,
Josef Tkadlec,
Alexi Turcotte
Abstract:
The input to the token swapping problem is a graph with vertices $v_1, v_2, \ldots, v_n$, and $n$ tokens with labels $1, 2, \ldots, n$, one on each vertex. The goal is to get token $i$ to vertex $v_i$ for all $i= 1, \ldots, n$ using a minimum number of swaps, where a swap exchanges the tokens on the endpoints of an edge. We present some results about token swapping on a tree, also known as "sortin…
▽ More
The input to the token swapping problem is a graph with vertices $v_1, v_2, \ldots, v_n$, and $n$ tokens with labels $1, 2, \ldots, n$, one on each vertex. The goal is to get token $i$ to vertex $v_i$ for all $i= 1, \ldots, n$ using a minimum number of swaps, where a swap exchanges the tokens on the endpoints of an edge. We present some results about token swapping on a tree, also known as "sorting with a transposition tree":
1. An optimum swap sequence may need to perform a swap on a leaf vertex that has the correct token (a "happy leaf"), disproving a conjecture of Vaughan.
2. Any algorithm that fixes happy leaves -- as all known approximation algorithms for the problem do -- has approximation factor at least $4/3$. Furthermore, the two best-known 2-approximation algorithms have approximation factor exactly 2.
3. A generalized problem -- weighted coloured token swapping -- is NP-complete on trees, even when they are restricted to be subdivided stars, but solvable in polynomial time on paths and stars. In this version, tokens and vertices have colours, and colours have weights. The goal is to get every token to a vertex of the same colour, and the cost of a swap is the sum of the weights of the two tokens involved.
△ Less
Submitted 12 January, 2023; v1 submitted 16 March, 2019;
originally announced March 2019.
