-
VidLA: Video-Language Alignment at Scale
Authors:
Mamshad Nayeem Rizve,
Fan Fei,
Jayakrishnan Unnikrishnan,
Son Tran,
Benjamin Z. Yao,
Belinda Zeng,
Mubarak Shah,
Trishul Chilimbi
Abstract:
In this paper, we propose VidLA, an approach for video-language alignment at scale. There are two major limitations of previous video-language alignment approaches. First, they do not capture both short-range and long-range temporal dependencies and typically employ complex hierarchical deep network architectures that are hard to integrate with existing pretrained image-text foundation models. To…
▽ More
In this paper, we propose VidLA, an approach for video-language alignment at scale. There are two major limitations of previous video-language alignment approaches. First, they do not capture both short-range and long-range temporal dependencies and typically employ complex hierarchical deep network architectures that are hard to integrate with existing pretrained image-text foundation models. To effectively address this limitation, we instead keep the network architecture simple and use a set of data tokens that operate at different temporal resolutions in a hierarchical manner, accounting for the temporally hierarchical nature of videos. By employing a simple two-tower architecture, we are able to initialize our video-language model with pretrained image-text foundation models, thereby boosting the final performance. Second, existing video-language alignment works struggle due to the lack of semantically aligned large-scale training data. To overcome it, we leverage recent LLMs to curate the largest video-language dataset to date with better visual grounding. Furthermore, unlike existing video-text datasets which only contain short clips, our dataset is enriched with video clips of varying durations to aid our temporally hierarchical data tokens in extracting better representations at varying temporal scales. Overall, empirical results show that our proposed approach surpasses state-of-the-art methods on multiple retrieval benchmarks, especially on longer videos, and performs competitively on classification benchmarks.
△ Less
Submitted 21 March, 2024;
originally announced March 2024.
-
CRGC -- A Practical Framework for Constructing Reusable Garbled Circuits
Authors:
Christopher Harth-Kitzerow,
Georg Carle,
Fan Fei,
Andre Luckow,
Johannes Klepsch
Abstract:
In this work, we introduce two schemes to construct reusable garbled circuits (RGCs) in the semi-honest setting. Our completely reusable garbled circuit (CRGC) scheme allows the generator (party A) to construct and send an obfuscated boolean circuit along with an encoded input to the evaluator (party B). In contrast to Yao's Garbled Circuit protocol, B can securely evaluate the same CRGC with an a…
▽ More
In this work, we introduce two schemes to construct reusable garbled circuits (RGCs) in the semi-honest setting. Our completely reusable garbled circuit (CRGC) scheme allows the generator (party A) to construct and send an obfuscated boolean circuit along with an encoded input to the evaluator (party B). In contrast to Yao's Garbled Circuit protocol, B can securely evaluate the same CRGC with an arbitrary number of inputs. As a tradeoff, CRGCs predictably leak some input bits of A to B. We also propose a partially reusable garbled circuit (PRGC) scheme that divides a circuit into reusable and non-reusable sections. PRGCs do not leak input bits of A. We benchmark our CRGC implementation against the state-of-the-art garbled circuit libraries EMP SH2PC and TinyGarble2. Using our framework, evaluating a CRGC is up to twenty times faster, albeit with weaker privacy guarantees, than evaluating an equivalent garbled circuit constructed by the two existing libraries. Our open-source library can convert any C++ function to a CRGC at approx. 80 million gates per second and repeatedly evaluate a CRGC at approx. 350 million gates per second. Additionally, a compressed CRGC is approx. 75% smaller in file size than the unobfuscated boolean circuit.
△ Less
Submitted 6 May, 2022; v1 submitted 23 March, 2022;
originally announced March 2022.
-
Bio-Inspired Adversarial Attack Against Deep Neural Networks
Authors:
Bowei Xi,
Yujie Chen,
Fan Fei,
Zhan Tu,
Xinyan Deng
Abstract:
The paper develops a new adversarial attack against deep neural networks (DNN), based on applying bio-inspired design to moving physical objects. To the best of our knowledge, this is the first work to introduce physical attacks with a moving object. Instead of following the dominating attack strategy in the existing literature, i.e., to introduce minor perturbations to a digital input or a statio…
▽ More
The paper develops a new adversarial attack against deep neural networks (DNN), based on applying bio-inspired design to moving physical objects. To the best of our knowledge, this is the first work to introduce physical attacks with a moving object. Instead of following the dominating attack strategy in the existing literature, i.e., to introduce minor perturbations to a digital input or a stationary physical object, we show two new successful attack strategies in this paper. We show by superimposing several patterns onto one physical object, a DNN becomes confused and picks one of the patterns to assign a class label. Our experiment with three flapping wing robots demonstrates the possibility of developing an adversarial camouflage to cause a targeted mistake by DNN. We also show certain motion can reduce the dependency among consecutive frames in a video and make an object detector "blind", i.e., not able to detect an object exists in the video. Hence in a successful physical attack against DNN, targeted motion against the system should also be considered.
△ Less
Submitted 29 June, 2021;
originally announced July 2021.
-
Flappy Hummingbird: An Open Source Dynamic Simulation of Flapping Wing Robots and Animals
Authors:
Fan Fei,
Zhan Tu,
Yilun Yang,
Jian Zhang,
Xinyan Deng
Abstract:
Insects and hummingbirds exhibit extraordinary flight capabilities and can simultaneously master seemingly conflicting goals: stable hovering and aggressive maneuvering, unmatched by small scale man-made vehicles. Flapping Wing Micro Air Vehicles (FWMAVs) hold great promise for closing this performance gap. However, design and control of such systems remain challenging due to various constraints.…
▽ More
Insects and hummingbirds exhibit extraordinary flight capabilities and can simultaneously master seemingly conflicting goals: stable hovering and aggressive maneuvering, unmatched by small scale man-made vehicles. Flapping Wing Micro Air Vehicles (FWMAVs) hold great promise for closing this performance gap. However, design and control of such systems remain challenging due to various constraints. Here, we present an open source high fidelity dynamic simulation for FWMAVs to serve as a testbed for the design, optimization and flight control of FWMAVs. For simulation validation, we recreated the hummingbird-scale robot developed in our lab in the simulation. System identification was performed to obtain the model parameters. The force generation, open-loop and closed-loop dynamic response between simulated and experimental flights were compared and validated. The unsteady aerodynamics and the highly nonlinear flight dynamics present challenging control problems for conventional and learning control algorithms such as Reinforcement Learning. The interface of the simulation is fully compatible with OpenAI Gym environment. As a benchmark study, we present a linear controller for hovering stabilization and a Deep Reinforcement Learning control policy for goal-directed maneuvering. Finally, we demonstrate direct simulation-to-real transfer of both control policies onto the physical robot, further demonstrating the fidelity of the simulation.
△ Less
Submitted 25 February, 2019;
originally announced February 2019.
-
Learning Extreme Hummingbird Maneuvers on Flapping Wing Robots
Authors:
Fan Fei,
Zhan Tu,
Jian Zhang,
Xinyan Deng
Abstract:
Biological studies show that hummingbirds can perform extreme aerobatic maneuvers during fast escape. Given a sudden looming visual stimulus at hover, a hummingbird initiates a fast backward translation coupled with a 180-degree yaw turn, which is followed by instant posture stabilization in just under 10 wingbeats. Consider the wingbeat frequency of 40Hz, this aggressive maneuver is carried out i…
▽ More
Biological studies show that hummingbirds can perform extreme aerobatic maneuvers during fast escape. Given a sudden looming visual stimulus at hover, a hummingbird initiates a fast backward translation coupled with a 180-degree yaw turn, which is followed by instant posture stabilization in just under 10 wingbeats. Consider the wingbeat frequency of 40Hz, this aggressive maneuver is carried out in just 0.2 seconds. Inspired by the hummingbirds' near-maximal performance during such extreme maneuvers, we developed a flight control strategy and experimentally demonstrated that such maneuverability can be achieved by an at-scale 12-gram hummingbird robot equipped with just two actuators. The proposed hybrid control policy combines model-based nonlinear control with model-free reinforcement learning. We use model-based nonlinear control for nominal flight control, as the dynamic model is relatively accurate for these conditions. However, during extreme maneuver, the modeling error becomes unmanageable. A model-free reinforcement learning policy trained in simulation was optimized to 'destabilize' the system and maximize the performance during maneuvering. The hybrid policy manifests a maneuver that is close to that observed in hummingbirds. Direct simulation-to-real transfer is achieved, demonstrating the hummingbird-like fast evasive maneuvers on the at-scale hummingbird robot.
△ Less
Submitted 25 February, 2019;
originally announced February 2019.
-
Acting Is Seeing: Navigating Tight Space Using Flapping Wings
Authors:
Zhan Tu,
Fan Fei,
Jian Zhang,
Xinyan Deng
Abstract:
Wings of flying animals can not only generate lift and control torques but also can sense their surroundings. Such dual functions of sensing and actuation coupled in one element are particularly useful for small sized bio-inspired robotic flyers, whose weight, size, and power are under stringent constraint. In this work, we present the first flapping-wing robot using its flapping wings for environ…
▽ More
Wings of flying animals can not only generate lift and control torques but also can sense their surroundings. Such dual functions of sensing and actuation coupled in one element are particularly useful for small sized bio-inspired robotic flyers, whose weight, size, and power are under stringent constraint. In this work, we present the first flapping-wing robot using its flapping wings for environmental perception and navigation in tight space, without the need for any visual feedback. As the test platform, we introduce the Purdue Hummingbird, a flapping-wing robot with 17cm wingspan and 12 grams weight, with a pair of 30-40Hz flapping wings driven by only two actuators. By interpreting the wing loading feedback and its variations, the vehicle can detect the presence of environmental changes such as grounds, walls, stairs, obstacles and wind gust. The instantaneous wing loading can be obtained through the measurements and interpretation of the current feedback by the motors that actuate the wings. The effectiveness of the proposed approach is experimentally demonstrated on several challenging flight tasks without vision: terrain following, wall following and going through a narrow corridor. To ensure flight stability, a robust controller was designed for handling unforeseen disturbances during the flight. Sensing and navigating one's environment through actuator loading is a promising method for mobile robots, and it can serve as an alternative or complementary method to visual perception.
△ Less
Submitted 28 February, 2019; v1 submitted 22 February, 2019;
originally announced February 2019.
-
Flight Recovery of MAVs with Compromised IMU
Authors:
Zhan Tu,
Fan Fei,
Matthew Eagon,
Dongyan Xu,
Xinyan Deng
Abstract:
Micro Aerial Vehicles (MAVs) rely on onboard attitude and position sensors for autonomous flight. Due to their size, weight, and power (SWaP) constraints, most modern MAVs use miniaturized inertial measurement units (IMUs) to provide attitude feedback, which is critical for flight stabilization and control. However, recent adversarial attack studies have demonstrated that many commonly used IMUs a…
▽ More
Micro Aerial Vehicles (MAVs) rely on onboard attitude and position sensors for autonomous flight. Due to their size, weight, and power (SWaP) constraints, most modern MAVs use miniaturized inertial measurement units (IMUs) to provide attitude feedback, which is critical for flight stabilization and control. However, recent adversarial attack studies have demonstrated that many commonly used IMUs are vulnerable to attacks exploiting their physical characteristics. Conventional redundancy-based approaches are not effective against such attacks because redundant IMUs have the same or similar physical vulnerabilities. In this paper, we present a novel fault-tolerant solution for IMU compromised scenarios, using separate position and heading information to restore the failed attitude states. Rather than adding more IMU alternatives for recovery, the proposed method is intended to minimize any modifications to the existing system and control program. Thus, it is particularly useful for vehicles that have tight SWaP constraints while requiring simultaneous high performance and safety demands. To execute the recovery logic properly, a robust estimator was designed for fine-grained detection and isolation of the faulty sensors. The effectiveness of the proposed approach was validated on a quadcopter MAV through both simulation and experimental flight tests.
△ Less
Submitted 18 July, 2019; v1 submitted 30 November, 2018;
originally announced December 2018.
