-
ACM MMSys 2024 Bandwidth Estimation in Real Time Communications Challenge
Authors:
Sami Khairy,
Gabriel Mittag,
Vishak Gopal,
Francis Y. Yan,
Zhixiong Niu,
Ezra Ameri,
Scott Inglis,
Mehrsa Golestaneh,
Ross Cutler
Abstract:
The quality of experience (QoE) delivered by video conferencing systems to end users depends in part on correctly estimating the capacity of the bottleneck link between the sender and the receiver over time. Bandwidth estimation for real-time communications (RTC) remains a significant challenge, primarily due to the continuously evolving heterogeneous network architectures and technologies. From t…
▽ More
The quality of experience (QoE) delivered by video conferencing systems to end users depends in part on correctly estimating the capacity of the bottleneck link between the sender and the receiver over time. Bandwidth estimation for real-time communications (RTC) remains a significant challenge, primarily due to the continuously evolving heterogeneous network architectures and technologies. From the first bandwidth estimation challenge which was hosted at ACM MMSys 2021, we learned that bandwidth estimation models trained with reinforcement learning (RL) in simulations to maximize network-based reward functions may not be optimal in reality due to the sim-to-real gap and the difficulty of aligning network-based rewards with user-perceived QoE. This grand challenge aims to advance bandwidth estimation model design by aligning reward maximization with user-perceived QoE optimization using offline RL and a real-world dataset with objective rewards which have high correlations with subjective audio/video quality in Microsoft Teams. All models submitted to the grand challenge underwent initial evaluation on our emulation platform. For a comprehensive evaluation under diverse network conditions with temporal fluctuations, top models were further evaluated on our geographically distributed testbed by using each model to conduct 600 calls within a 12-day period. The winning model is shown to deliver comparable performance to the top behavior policy in the released dataset. By leveraging real-world data and integrating objective audio/video quality scores as rewards, offline RL can therefore facilitate the development of competitive bandwidth estimators for RTC.
△ Less
Submitted 15 March, 2024; v1 submitted 10 March, 2024;
originally announced March 2024.
-
Underreporting of errors in NLG output, and what to do about it
Authors:
Emiel van Miltenburg,
Miruna-Adriana Clinciu,
Ondřej Dušek,
Dimitra Gkatzia,
Stephanie Inglis,
Leo Leppänen,
Saad Mahamood,
Emma Manning,
Stephanie Schoch,
Craig Thomson,
Luou Wen
Abstract:
We observe a severe under-reporting of the different kinds of errors that Natural Language Generation systems make. This is a problem, because mistakes are an important indicator of where systems should still be improved. If authors only report overall performance metrics, the research community is left in the dark about the specific weaknesses that are exhibited by `state-of-the-art' research. Ne…
▽ More
We observe a severe under-reporting of the different kinds of errors that Natural Language Generation systems make. This is a problem, because mistakes are an important indicator of where systems should still be improved. If authors only report overall performance metrics, the research community is left in the dark about the specific weaknesses that are exhibited by `state-of-the-art' research. Next to quantifying the extent of error under-reporting, this position paper provides recommendations for error identification, analysis and reporting.
△ Less
Submitted 8 August, 2021; v1 submitted 2 August, 2021;
originally announced August 2021.
-
Resonance: Replacing Software Constants with Context-Aware Models in Real-time Communication
Authors:
Jayant Gupchup,
Ashkan Aazami,
Yaran Fan,
Senja Filipi,
Tom Finley,
Scott Inglis,
Marcus Asteborg,
Luke Caroll,
Rajan Chari,
Markus Cozowicz,
Vishak Gopal,
Vinod Prakash,
Sasikanth Bendapudi,
Jack Gerrits,
Eric Lau,
Huazhou Liu,
Marco Rossi,
Dima Slobodianyk,
Dmitri Birjukov,
Matty Cooper,
Nilesh Javar,
Dmitriy Perednya,
Sriram Srinivasan,
John Langford,
Ross Cutler
, et al. (1 additional authors not shown)
Abstract:
Large software systems tune hundreds of 'constants' to optimize their runtime performance. These values are commonly derived through intuition, lab tests, or A/B tests. A 'one-size-fits-all' approach is often sub-optimal as the best value depends on runtime context. In this paper, we provide an experimental approach to replace constants with learned contextual functions for Skype - a widely used r…
▽ More
Large software systems tune hundreds of 'constants' to optimize their runtime performance. These values are commonly derived through intuition, lab tests, or A/B tests. A 'one-size-fits-all' approach is often sub-optimal as the best value depends on runtime context. In this paper, we provide an experimental approach to replace constants with learned contextual functions for Skype - a widely used real-time communication (RTC) application. We present Resonance, a system based on contextual bandits (CB). We describe experiences from three real-world experiments: applying it to the audio, video, and transport components in Skype. We surface a unique and practical challenge of performing machine learning (ML) inference in large software systems written using encapsulation principles. Finally, we open-source FeatureBroker, a library to reduce the friction in adopting ML models in such development environments
△ Less
Submitted 22 November, 2020;
originally announced November 2020.
-
Machine Learning at Microsoft with ML .NET
Authors:
Zeeshan Ahmed,
Saeed Amizadeh,
Mikhail Bilenko,
Rogan Carr,
Wei-Sheng Chin,
Yael Dekel,
Xavier Dupre,
Vadim Eksarevskiy,
Eric Erhardt,
Costin Eseanu,
Senja Filipi,
Tom Finley,
Abhishek Goswami,
Monte Hoover,
Scott Inglis,
Matteo Interlandi,
Shon Katzenberger,
Najeeb Kazmi,
Gleb Krivosheev,
Pete Luferenko,
Ivan Matantsev,
Sergiy Matusevych,
Shahab Moradi,
Gani Nazirov,
Justin Ormont
, et al. (9 additional authors not shown)
Abstract:
Machine Learning is transitioning from an art and science into a technology available to every developer. In the near future, every application on every platform will incorporate trained models to encode data-based decisions that would be impossible for developers to author. This presents a significant engineering challenge, since currently data science and modeling are largely decoupled from stan…
▽ More
Machine Learning is transitioning from an art and science into a technology available to every developer. In the near future, every application on every platform will incorporate trained models to encode data-based decisions that would be impossible for developers to author. This presents a significant engineering challenge, since currently data science and modeling are largely decoupled from standard software development processes. This separation makes incorporating machine learning capabilities inside applications unnecessarily costly and difficult, and furthermore discourage developers from embracing ML in first place. In this paper we present ML .NET, a framework developed at Microsoft over the last decade in response to the challenge of making it easy to ship machine learning models in large software applications. We present its architecture, and illuminate the application demands that shaped it. Specifically, we introduce DataView, the core data abstraction of ML .NET which allows it to capture full predictive pipelines efficiently and consistently across training and inference lifecycles. We close the paper with a surprisingly favorable performance study of ML .NET compared to more recent entrants, and a discussion of some lessons learned.
△ Less
Submitted 15 May, 2019; v1 submitted 14 May, 2019;
originally announced May 2019.
