-
Live Fitness Coaching as a Testbed for Situated Interaction
Authors:
Sunny Panchal,
Apratim Bhattacharyya,
Guillaume Berger,
Antoine Mercier,
Cornelius Bohm,
Florian Dietrichkeit,
Reza Pourreza,
Xuanlin Li,
Pulkit Madan,
Mingu Lee,
Mark Todorovich,
Ingo Bax,
Roland Memisevic
Abstract:
Tasks at the intersection of vision and language have had a profound impact in advancing the capabilities of vision-language models such as dialog-based assistants. However, models trained on existing tasks are largely limited to turn-based interactions, where each turn must be stepped (i.e., prompted) by the user. Open-ended, asynchronous interactions where an AI model may proactively deliver tim…
▽ More
Tasks at the intersection of vision and language have had a profound impact in advancing the capabilities of vision-language models such as dialog-based assistants. However, models trained on existing tasks are largely limited to turn-based interactions, where each turn must be stepped (i.e., prompted) by the user. Open-ended, asynchronous interactions where an AI model may proactively deliver timely responses or feedback based on the unfolding situation in real-time are an open challenge. In this work, we present the QEVD benchmark and dataset which explores human-AI interaction in the challenging, yet controlled, real-world domain of fitness coaching - a task which intrinsically requires monitoring live user activity and providing timely feedback. It is the first benchmark that requires assistive vision-language models to recognize complex human actions, identify mistakes grounded in those actions, and provide appropriate feedback. Our experiments reveal the limitations of existing state of the art vision-language models for such asynchronous situated interactions. Motivated by this, we propose a simple end-to-end streaming baseline that can respond asynchronously to human actions with appropriate feedbacks at the appropriate time.
△ Less
Submitted 10 July, 2024;
originally announced July 2024.
-
Is end-to-end learning enough for fitness activity recognition?
Authors:
Antoine Mercier,
Guillaume Berger,
Sunny Panchal,
Florian Letsch,
Cornelius Boehm,
Nahua Kang,
Ingo Bax,
Roland Memisevic
Abstract:
End-to-end learning has taken hold of many computer vision tasks, in particular, related to still images, with task-specific optimization yielding very strong performance. Nevertheless, human-centric action recognition is still largely dominated by hand-crafted pipelines, and only individual components are replaced by neural networks that typically operate on individual frames. As a testbed to stu…
▽ More
End-to-end learning has taken hold of many computer vision tasks, in particular, related to still images, with task-specific optimization yielding very strong performance. Nevertheless, human-centric action recognition is still largely dominated by hand-crafted pipelines, and only individual components are replaced by neural networks that typically operate on individual frames. As a testbed to study the relevance of such pipelines, we present a new fully annotated video dataset of fitness activities. Any recognition capabilities in this domain are almost exclusively a function of human poses and their temporal dynamics, so pose-based solutions should perform well. We show that, with this labelled data, end-to-end learning on raw pixels can compete with state-of-the-art action recognition pipelines based on pose estimation. We also show that end-to-end learning can support temporally fine-grained tasks such as real-time repetition counting.
△ Less
Submitted 14 May, 2023;
originally announced May 2023.
-
The "something something" video database for learning and evaluating visual common sense
Authors:
Raghav Goyal,
Samira Ebrahimi Kahou,
Vincent Michalski,
Joanna Materzyńska,
Susanne Westphal,
Heuna Kim,
Valentin Haenel,
Ingo Fruend,
Peter Yianilos,
Moritz Mueller-Freitag,
Florian Hoppe,
Christian Thurau,
Ingo Bax,
Roland Memisevic
Abstract:
Neural networks trained on datasets such as ImageNet have led to major advances in visual object classification. One obstacle that prevents networks from reasoning more deeply about complex scenes and situations, and from integrating visual knowledge with natural language, like humans do, is their lack of common sense knowledge about the physical world. Videos, unlike still images, contain a wealt…
▽ More
Neural networks trained on datasets such as ImageNet have led to major advances in visual object classification. One obstacle that prevents networks from reasoning more deeply about complex scenes and situations, and from integrating visual knowledge with natural language, like humans do, is their lack of common sense knowledge about the physical world. Videos, unlike still images, contain a wealth of detailed information about the physical world. However, most labelled video datasets represent high-level concepts rather than detailed physical aspects about actions and scenes. In this work, we describe our ongoing collection of the "something-something" database of video prediction tasks whose solutions require a common sense understanding of the depicted situation. The database currently contains more than 100,000 videos across 174 classes, which are defined as caption-templates. We also describe the challenges in crowd-sourcing this data at scale.
△ Less
Submitted 15 June, 2017; v1 submitted 13 June, 2017;
originally announced June 2017.
