-
Reviewing FID and SID Metrics on Generative Adversarial Networks
Authors:
Ricardo de Deijn,
Aishwarya Batra,
Brandon Koch,
Naseef Mansoor,
Hema Makkena
Abstract:
The growth of generative adversarial network (GAN) models has increased the ability of image processing and provides numerous industries with the technology to produce realistic image transformations. However, with the field being recently established there are new evaluation metrics that can further this research. Previous research has shown the Fréchet Inception Distance (FID) to be an effective…
▽ More
The growth of generative adversarial network (GAN) models has increased the ability of image processing and provides numerous industries with the technology to produce realistic image transformations. However, with the field being recently established there are new evaluation metrics that can further this research. Previous research has shown the Fréchet Inception Distance (FID) to be an effective metric when testing these image-to-image GANs in real-world applications. Signed Inception Distance (SID), a founded metric in 2023, expands on FID by allowing unsigned distances. This paper uses public datasets that consist of façades, cityscapes, and maps within Pix2Pix and CycleGAN models. After training these models are evaluated on both inception distance metrics which measure the generating performance of the trained models. Our findings indicate that usage of the metric SID incorporates an efficient and effective metric to complement, or even exceed the ability shown using the FID for the image-to-image GANs
△ Less
Submitted 5 February, 2024;
originally announced February 2024.
-
"It Can Relate to Real Lives": Attitudes and Expectations in Justice-Centered Data Structures & Algorithms for Non-Majors
Authors:
Anna Batra,
Iris Zhou,
Suh Young Choi,
Chongjiu Gao,
Yanbing Xiao,
Sonia Fereidooni,
Kevin Lin
Abstract:
Prior work has argued for a more justice-centered approach to postsecondary computing education by emphasizing ethics, identity, and political vision. In this experience report, we examine how postsecondary students of diverse gender and racial identities experience a justice-centered Data Structures and Algorithms designed for undergraduate non-computer science majors. Through a quantitative and…
▽ More
Prior work has argued for a more justice-centered approach to postsecondary computing education by emphasizing ethics, identity, and political vision. In this experience report, we examine how postsecondary students of diverse gender and racial identities experience a justice-centered Data Structures and Algorithms designed for undergraduate non-computer science majors. Through a quantitative and qualitative analysis of two quarters of student survey data collected at the start and end of each quarter, we report on student attitudes and expectations.
Across the class, we found a significant increase in the following attitudes: computing confidence and sense of belonging. While women, non-binary, and other students not identifying as men (WNB+) also increased in these areas, they still reported significantly lower confidence and sense of belonging than men at the end of the quarter. Black, Latinx, Middle Eastern and North African, Native American, and Pacific Islander (BLMNPI) students had no significant differences compared to white and Asian students.
We also analyzed end-of-quarter student self-reflections on their fulfillment of expectations prior to taking the course. While the majority of students reported a positive overall sentiment about the course and many students specifically appreciated the justice-centered approach, some desired more practice with program implementation and interview preparation. We discuss implications for practice and articulate a political vision for holding both appreciation for computing ethics and a desire for professional preparation together through iterative design.
△ Less
Submitted 15 March, 2024; v1 submitted 19 December, 2023;
originally announced December 2023.
-
Efficient Pre-training for Localized Instruction Generation of Videos
Authors:
Anil Batra,
Davide Moltisanti,
Laura Sevilla-Lara,
Marcus Rohrbach,
Frank Keller
Abstract:
Procedural videos, exemplified by recipe demonstrations, are instrumental in conveying step-by-step instructions. However, understanding such videos is challenging as it involves the precise localization of steps and the generation of textual instructions. Manually annotating steps and writing instructions is costly, which limits the size of current datasets and hinders effective learning. Leverag…
▽ More
Procedural videos, exemplified by recipe demonstrations, are instrumental in conveying step-by-step instructions. However, understanding such videos is challenging as it involves the precise localization of steps and the generation of textual instructions. Manually annotating steps and writing instructions is costly, which limits the size of current datasets and hinders effective learning. Leveraging large but noisy video-transcript datasets for pre-training can boost performance but demands significant computational resources. Furthermore, transcripts contain irrelevant content and differ in style from human-written instructions. To mitigate these issues, we propose a novel technique, Sieve-&-Swap, to automatically generate high quality training data for the recipe domain: (i) Sieve filters irrelevant transcripts and (ii) Swap acquires high quality text by replacing transcripts with human-written instruction from a text-only recipe dataset. The resulting dataset is three orders of magnitude smaller than current web-scale datasets but enables efficient training of large-scale models. Alongside Sieve-&-Swap, we propose Procedure Transformer (ProcX), a model for end-to-end step localization and instruction generation for procedural videos. When pre-trained on our curated dataset, this model achieves state-of-the-art performance on YouCook2 and Tasty while using a fraction of the training data. Our code and dataset will be publicly released.
△ Less
Submitted 23 May, 2024; v1 submitted 27 November, 2023;
originally announced November 2023.
-
Image generation with shortest path diffusion
Authors:
Ayan Das,
Stathi Fotiadis,
Anil Batra,
Farhang Nabiei,
FengTing Liao,
Sattar Vakili,
Da-shan Shiu,
Alberto Bernacchia
Abstract:
The field of image generation has made significant progress thanks to the introduction of Diffusion Models, which learn to progressively reverse a given image corruption. Recently, a few studies introduced alternative ways of corrupting images in Diffusion Models, with an emphasis on blurring. However, these studies are purely empirical and it remains unclear what is the optimal procedure for corr…
▽ More
The field of image generation has made significant progress thanks to the introduction of Diffusion Models, which learn to progressively reverse a given image corruption. Recently, a few studies introduced alternative ways of corrupting images in Diffusion Models, with an emphasis on blurring. However, these studies are purely empirical and it remains unclear what is the optimal procedure for corrupting an image. In this work, we hypothesize that the optimal procedure minimizes the length of the path taken when corrupting an image towards a given final state. We propose the Fisher metric for the path length, measured in the space of probability distributions. We compute the shortest path according to this metric, and we show that it corresponds to a combination of image sharpening, rather than blurring, and noise deblurring. While the corruption was chosen arbitrarily in previous work, our Shortest Path Diffusion (SPD) determines uniquely the entire spatiotemporal structure of the corruption. We show that SPD improves on strong baselines without any hyperparameter tuning, and outperforms all previous Diffusion Models based on image blurring. Furthermore, any small deviation from the shortest path leads to worse performance, suggesting that SPD provides the optimal procedure to corrupt images. Our work sheds new light on observations made in recent works and provides a new approach to improve diffusion models on images and other types of data.
△ Less
Submitted 1 June, 2023;
originally announced June 2023.
-
A Closer Look at Temporal Ordering in the Segmentation of Instructional Videos
Authors:
Anil Batra,
Shreyank N Gowda,
Frank Keller,
Laura Sevilla-Lara
Abstract:
Understanding the steps required to perform a task is an important skill for AI systems. Learning these steps from instructional videos involves two subproblems: (i) identifying the temporal boundary of sequentially occurring segments and (ii) summarizing these steps in natural language. We refer to this task as Procedure Segmentation and Summarization (PSS). In this paper, we take a closer look a…
▽ More
Understanding the steps required to perform a task is an important skill for AI systems. Learning these steps from instructional videos involves two subproblems: (i) identifying the temporal boundary of sequentially occurring segments and (ii) summarizing these steps in natural language. We refer to this task as Procedure Segmentation and Summarization (PSS). In this paper, we take a closer look at PSS and propose three fundamental improvements over current methods. The segmentation task is critical, as generating a correct summary requires each step of the procedure to be correctly identified. However, current segmentation metrics often overestimate the segmentation quality because they do not consider the temporal order of segments. In our first contribution, we propose a new segmentation metric that takes into account the order of segments, giving a more reliable measure of the accuracy of a given predicted segmentation. Current PSS methods are typically trained by proposing segments, matching them with the ground truth and computing a loss. However, much like segmentation metrics, existing matching algorithms do not consider the temporal order of the mapping between candidate segments and the ground truth. In our second contribution, we propose a matching algorithm that constrains the temporal order of segment mapping, and is also differentiable. Lastly, we introduce multi-modal feature training for PSS, which further improves segmentation. We evaluate our approach on two instructional video datasets (YouCook2 and Tasty) and observe an improvement over the state-of-the-art of $\sim7\%$ and $\sim2.5\%$ for procedure segmentation and summarization, respectively.
△ Less
Submitted 7 October, 2022; v1 submitted 30 September, 2022;
originally announced September 2022.
-
GesSure- A Robust Face-Authentication enabled Dynamic Gesture Recognition GUI Application
Authors:
Ankit Jha,
Ishita,
Pratham G. Shenwai,
Ayush Batra,
Siddharth Kotian,
Piyush Modi
Abstract:
Using physical interactive devices like mouse and keyboards hinders naturalistic human-machine interaction and increases the probability of surface contact during a pandemic. Existing gesture-recognition systems do not possess user authentication, making them unreliable. Static gestures in current gesture-recognition technology introduce long adaptation periods and reduce user compatibility. Our t…
▽ More
Using physical interactive devices like mouse and keyboards hinders naturalistic human-machine interaction and increases the probability of surface contact during a pandemic. Existing gesture-recognition systems do not possess user authentication, making them unreliable. Static gestures in current gesture-recognition technology introduce long adaptation periods and reduce user compatibility. Our technology places a strong emphasis on user recognition and safety. We use meaningful and relevant gestures for task operation, resulting in a better user experience. This paper aims to design a robust, face-verification-enabled gesture recognition system that utilizes a graphical user interface and primarily focuses on security through user recognition and authorization. The face model uses MTCNN and FaceNet to verify the user, and our LSTM-CNN architecture for gesture recognition, achieving an accuracy of 95% with five classes of gestures. The prototype developed through our research has successfully executed context-dependent tasks like save, print, control video-player operations and exit, and context-free operating system tasks like sleep, shut-down, and unlock intuitively. Our application and dataset are available as open source.
△ Less
Submitted 7 September, 2022; v1 submitted 22 July, 2022;
originally announced July 2022.
