Localizing Events in Videos with Multimodal Queries

Abstract

Video understanding is a pivotal task in the digital era, yet the dynamic and multi-event nature of videos makes them labor-intensive and computationally demanding to process. Thus, localizing a specific event given a semantic query has gained importance in both user-oriented applications like video search and academic research into video foundation models. A significant limitation in current research is that semantic queries are typically in natural language that depicts the semantics of the target event. This setting overlooks the potential for multimodal semantic queries composed of images and texts. To address this gap, we introduce a new benchmark, ICQ, for localizing events in videos with multimodal queries, along with a new evaluation dataset ICQ-Highlight. Our new benchmark aims to evaluate how well models can localize an event given a multimodal semantic query that consists of a reference image, which depicts the event, and a refinement text to adjust the images’ semantics. To systematically benchmark model performance, we include 4 styles of reference images and 5 types of refinement texts, allowing us to explore model performance across different domains. We propose 3 adaptation methods that tailor existing models to our new setting and evaluate 10 SOTA models, ranging from specialized to large-scale foundation models. We believe this benchmark is an initial step toward investigating multimodal queries in video event localization.

1 Introduction

Videos are the prevailing data medium on the Internet and a common multimodal interface when we interact with the world. User-centric applications, such as video search engines and video highlight/moment recommendations, are increasingly popular on streaming media and short video platforms. Moreover, large foundation models are expected to process videos as input data to understand surroundings and make decisions. Consequently, video understanding has been a long-standing research topic and has recently gained increased attention.

However, videos are inherently dynamic and contain multiple events [66, 75] that are sparsely distributed. This redundancy makes processing and understanding dense videos labor-intensive and computation-demanding for human users and deep-learning models. As a result, the need to localize events in videos becomes essential [28, 48].

Figure 1: Localizing events in videos with semantics queries: so far, the community has only focused on natural language query-based video event localization as in [29]. Our benchmark ICQ focuses on a more general scenario: localizing events in video with multimodal queries.

Localizing events in videos encompasses a broad spectrum of related tasks. From a practical perspective, particularly in user-centric applications like video search and recommendation, tasks including video moment retrieval [17, 18, 39] and highlight detection [2, 29, 45] focus on identifying and retrieving video segments of interest based on textual queries within extensive, long-range videos. For video foundation models that aim to understand and reason video content, video temporal grounding [11, 12, 13, 15, 24, 56] with a given natural language query not only can reduce the video processing duration but also elucidate the reasoning processes.

A series of benchmarks [4, 16, 29, 54] has been established for exploring video event localization using natural language queries as semantic queries. Building on these foundations, existing models have primarily focused on this natural language query setting [1, 6, 7, 8, 10, 9, 12, 13, 16, 19, 29, 60]. However, with the increasing need for human users to efficiently process massive video data online and the advent of large-scale foundation models in recent years, multimodal interaction with videos is a promising scenario. In other words, texts should not be the only possible query for localizing events in videos. As the saying goes, “A picture is worth a thousand words.”, images are illiteral language and can express rich semantic meaning and describe events in videos.

Multimodal queries, also known as composed queries [23, 57] for video event localization, bring practical benefits. From a pragmatic perspective, using queries such as user-input “scribble images” can facilitate a more natural human-computer interaction. As users, we often opt for writing brief and simple text queries rather than detailed and lengthy paragraphs for semantic search on videos, and thus, a text query can be ambiguous. Texts sometimes fail to deliver the message, while images are capable. In the meantime, grounding/localizing events in videos with multimodal queries heuristically serves as an important module of video foundation models as in temporal grounding and episodic memory search [21, 27, 50]. This relates to grounding an event stimulated by a similar scene, which is similar to a common cognitive phenomenon called Déjà vu.

Since using multimodal queries for semantically searching events in videos remains largely unexplored, this inspires us to propose a new task: localizing events in videos with multimodal queries. We introduce a new benchmark, ICQ, for localizing events in videos using Image-Text Composed Queries as multimodal queries. Our benchmark is targeted at evaluating model performance for localization events in videos with multimodal queries consisting of reference images and refinement texts. Alongside this benchmark, we propose a new evaluation dataset, ICQ-Highlight, as a testbed for our task. Given that reference images may have a significant distribution shift from the video data in styles and that refinement texts should alter the semantic meaning of reference images in various aspects, our dataset highlights 4 reference image styles and 5 refinement text types.

In ICQ, we evaluate a broad spectrum of existing video localization models, from specialized models to LLM-based video foundation models, on the ICQ. To bridge the gap between current natural language query-based models and multimodal queries, we propose 3 adaptation methods: Captioning, Summarization, and Visual Query Encoding. Our results demonstrate that existing models can be effectively adapted to our new benchmark with the aid of Multimodal Large Language Models (MLLMs) and Large Language Models (LLMs) despite performance decline and instability to a greater or lesser extent. They should serve as a solid baseline for future studies. Additionally, our findings reveal that multimodal semantic queries can successfully localize events in videos, suggesting multimodal queries have promising applications for video localization.

Our contributions are summarized as follows:

1. We introduce a new evaluation benchmark, ICQ, and a new evaluation dataset, ICQ-Highlight, for analyzing event localization in videos with multimodal queries;

2. We propose 3 adaptation methods and evaluate 10 models ranging from specialized models to large-scale video foundation models;

3. Our comprehensive experiments show that our adaptation method is a simple yet effective baseline method to adapt existing models to ICQ;

4. We claim that using multimodal queries for video event localization is a practical and feasible scenario with broad prospects.

2 Related Work

2.1 Localizing Event in Videos with Natural Language Queries

Query-based video temporal localization has been a long-standing research topic and is an umbrella of several related tasks. According to their scenarios and motivation, they can be further classified into several similar but slightly different tasks. Video moment retrieval [32, 38, 42, 43, 41, 68, 71, 74] aims to localize a video segment based on a textual caption query that describes events in the video. Video temporal grounding/localization [14, 22, 34, 35, 46, 47, 67, 70, 72] with natural language queries aims to determine the video segment that corresponds with textual description and usually serves downstream Question-answering task [3, 63, 69, 76] and aims to provide relevant segments in videos. Other similar yet less relevant tasks include video highlight detection [2, 29, 45, 54] and action detection; these tasks also involve localizing video segments but with an implicit query or a category-level action label. Our benchmark steps torwards localizing video events in multimodal query. This multimodal query underlines a composed query of images and text, which are different from other works, as a semantic search for events in videos described by multimodal queries.

Regarding the methodology, a line of works are focused on video moment retrieval/ video temporal grounding tasks: this includes two-stage (i.e. proposal-based) models [33] that firstly generates moment candidates and then filter out the matched moment based on the query and one-stage (i.e. proposal-free) models [7, 52, 70] that integrates the moment generation and moment localization into a unified framework. Within the one-stage models, DETR [5] has been widely employed in multiple models for video temporal localization as in [25, 29, 45, 44, 55, 64]. More recent works [31, 40, 65, 62] attempt to uniform multiple video localization tasks, including video moment retrieval and highlight detection in a single framework. This again shows the correlation of video temporal localization tasks. In addition, with the large-scale video foundation models and MLLMs gaining increasing attention, temporal grounding has also been a core module in models like SeViLA [69], InternVideo2 [61], VideoPrism [78], etc. [51, 73].

2.2 Multimodal Query for Image/Video Understanding

Using multimodal queries is a practical and important scenario for video/vision understanding scenarios [57, 58]. However, it is cruical to note that video event localization with multimodal queries differs from image/video retrieval tasks, which primarily involve instance-level similarity matching. Temporal localization requires dense video processing, significantly increasing the complexity of the task.

For video localization tasks, [77] is the first work to use image queries to localize unseen activities in videos. More recently, [20] proposes to ground videos spatio-temporally using images or texts, although their queries are still limited to object or action levels. To the best of our knowledge, our work is the first to attempt localizing events in videos using multimodal semantic queries.

3 ICQ: Video Event Localization with Multimodal Queries

In the following section, we will detail the benchmark ICQ and a new evaluation dataset ICQ-Highlight to benchmark video event localization with multimodal queries.

Figure 2: Examples of ICQ-Highlight: Multimodal queries consist of a reference image and a refinement text. We consider 4 different reference image styles: scribble, cartoon, cinematic, and realistic. They describe a target event that corresponds to moments or segments in original videos and are equivalent to natural language queries in the original dataset [29]. Refinement texts add either complementary information if reference images are minimal like for scribble images, or corrective information if reference images are more complicated.

3.1 Task Definition

We define the multimodal query $q_{m}$ as consisting of a reference image $v_{ref}$ accompanied by a refinement text $t_{ref}$ for minor adjustments to localize the target event in a video that corresponds to the query semantically. The reference image captures the broad semantics of the target event, while the refinement text provides supplementary information that can be either complementary or corrective. We believe that this setting is more adaptable and general in real-world applications.

Given the query $q_{m}$, the model predicts all the relevant segments or moments $\left[time_{start},time_{end}\right]$. Similar to the metrics used in common-setting video moment retrieval, we utilize recall R and mean Average Precision as the evaluation metrics for video moment retrieval.

Reference Image Styles Reference images $v_{ref}$ visually describe the semantics of an event in a video. They can be simple scribble images with minimal strokes that describe an event succinctly, effectively summarizing an event for non-verbal semantic queries in video localization or more detailed images that depict semantically relevant scenes in a video. As illustrated in Fig. 2, reference images describe semantically similar scenes yet might vary in details as target videos. In practice, visual queries can differ in style, which may impact model performance. Therefore, we explore multiple reference image styles, as detailed in the subsequent section, to assess whether the model maintains consistent performance across various styles as an indicator of model robustness.

Refinement Texts Refinement texts refer to simple phrases or sentences to complement or correct descriptions that are either missing or contradictory in the reference images. This is particularly practical in real-world applications, as reference images often do not semantically align perfectly with the target video event. We identify 5 different types of refinement texts that can be applied to various aspects of the reference image semantics: “object”, “action”, “relation”, “attribute”, “environment”, and “others” as shown in Fig. 3. This categorization is designed for elements of a semantic scene graph [26] and borrowed by us to summarize different semantic elements of the multimodal queries.

3.2 Dataset Construction

We introduce our new evaluation dataset, ICQ-Highlight, as a testbed for ICQ. This dataset is built upon the validation set of QVHighlight [29], a popular natural-language query-based video localization dataset. For each original query in QVHighlight, we construct multimodal semantic queries that incorporate reference images paired with refinement texts. Considering the reference image style distribution discussed earlier, ICQ-Highlight features 4 variants based on different image styles. In total, the dataset comprises 1515 videos and 1546 test samples on average for each style. The exact numbers may vary slightly across styles and are provided in the Appendix.

Reference Image Generation We generate reference images based on the original natural language queries and refinement texts using a suite of state-of-the-art Text-to-Image models, including DALL-E-2 and Stable Diffusion. For the reference image styles mentioned earlier, we select 4 representative styles: scribble, cartoon, cinematic, and realistic. These styles effectively capture a variety of real-world scenarios such as user inputs, book illustrations, television shows, and actual photographs, where images are often used as queries.

Data Annotation and Preprocessing We emphasize the meticulous crowd-sourced data curation and annotation effort applied to QVHighlight for 2 main reasons: (1) To introduce refinement texts, we purposefully modify the original semantics of text queries in QVHighlight to generate queries that are similar yet subtly different; (2) Given that the original queries in QVHighlight can be too simple and ambiguous to generate reasonable reference images, we add necessary annotations to ensure that the generated image queries are more relevant to the original video semantics. We employed human annotators to annotate and modify the natural language queries. Each query is annotated and reviewed by different annotators to ensure consistency. Further details can be found in the Appendix.

Data Curation and Quality check Image generation can suffer from significant imperfections in terms of semantic consistency and content safety. To address these issues, we implement a quality check in two stages: (1) We calculate the semantic similarity between the generated images and the text queries using BLIP2 [30] encoders, eliminating samples that score lower than 0.2; (2) We perform human sanity check to replace images that are: i) semantically misaligned with the text, ii) mismatched with the required reference image style, iii) containing sensitive or unpleasant content (e.g., violent, racial, sexual content), counterintuitive elements, or obvious generation artifacts.

Figure 3: Distribution of refinement text types. Refinement texts are designed to either complement or correct the original semantics of reference images. We identify 5 major types of refinement texts, each targeting different semantic aspects: object, action, relationship, attribute, environment, and others.

3.3 Baseline Selection

We have selected and benchmarked 10 models specifically designed for video event localization with natural language queries. We assess the zero-shot performance of these models using checkpoints that have been fine-tuned on the original QVHighlight dataset. This evaluation allows us to understand their effectiveness for multimodal queries straight out of the box.

Particularly, we categorize them as follows and compare the models in different dimensions in the Appendix: (1) Specialized models use natural language as a semantic query and are targeted at video moment retrieval tasks. We have selected a series of these models including Moment-DETR[29], QD-DETR[45], EaTR[25], CG-DETR[44], TR-DETR[55]; (2) Unified frameworks are aimed to solve multiple video localization tasks within one model, such as moment retrieval, highlight detection, and video summarization. We have selected UMT[40], UniVTG[31], and UVCOM[62] as strong baselines; (3) LLM-based Models features the power of Large Language Models, which prove to be a powerful and general head for varied video tasks. We have selected SeViLA [69] as a representative.

3.4 Adaptation Methods

Most existing video localization methods utilize natural language as input queries and are not readily adaptable to composed queries. Thus, we propose 3 adaptation methods: Captioning (Cap), Summarization(Sum), and Visual Query Encoding(VisEnc), as illustrated in Fig. 4. For Cap and Sum, we aim to leverage the power of LLMs and MLLMs to caption reference images $v_{ref}$ and integrate refinement texts $t_{ref}$: Cap uses MLLMs as a captioner to caption reference images and LLMs as a modifier to integrate refinement texts. In contrast, Sum uses MLLMs to directly summarize reference images and refinement texts in one step. Generated texts $t_{query}$ can be seamlessly used by existing models. For VisEnc, we explore using only reference images and employing visual encoders to embed the reference images as query embeddings $e_{query}$. This is based on the background that all models we have selected employ a dual-stream encoder that embeds image-text pairs in a joint feature space.

Figure 4: Adaptation methods: We propose 3 adaptation methods to bridge the current gap between natural language query-based models and our multimodal query-based benchmark: Captioning(Cap), Summarization(Sum), and Visual Query Encoding(VisEnc). For brevity, we refer to them by the abbreviation.

4 Experiments

4.1 Experimental Setup

Implementation We employ state-of-the-art MLLMs, LLaVA-mistral [36, 37], as a captioner and GPT-3.5 as a modifier in our Cap adaptation. For a fair comparison, we utilize LLaVA-mistral for Sum adaptation. We believe that the performance of these models is representative of the SOTA capabilities of MLLMs. For VisEnc, we utilize the corresponding CLIP [49] Visual Encoder, as all models typically employ the CLIP Text Encoder for text query encoding. In this adaptation method, we omit refinement texts and only use the reference image.

Evaluation Metrics We evaluate models on our new testbed ICQ-Highlight. For evaluation, we report both Recall R@1 with IoU thresholds 0.5 and 0.7, mean Average Precision with IoU threshold 0.5 and the average over multiple IoU thresholds [0.5:0.05:0.95] as standard metrics for video moment retrieval and localization [29, 69], where IoU (Intersection over Union) thresholds determine if a predicted temporal window is positive.

4.2 Results & Analysis

We present the pairwise performance of 10 models combined with 3 adaptation methods on ICQ in Tab. 4.2- 4.2. For Cap and Sum methods, we have conducted multiple runs with different prompts used for captioning and summarization and reported the average performance and standard deviation.

Table 1: Model performance (Recall) on ICQ. We highlight the best score in bold for each adaptation method and reference image style. For Cap and Sum, we also report the standard deviation of 3 runs with different prompts. ${\dagger}$ indicates the usage of additional audio modality.

llcccccccc[colortbl-like] 3-8,12,19-21,23-28,32 9-11 1 Model scribble cartoon cinematic realistic
R1@0.5 R1@0.7 R1@0.5 R1@0.7 R1@0.5 R1@0.7 R1@0.5 R1@0.7
Captioning Moment-DETR (2021) 44.83 (± 2.7) 27.97 (± 2.2) 46.02 (± 1.5) 29.36 (± 0.9) 46.89 (± 0.7) 30.35 (± 1.2) 47.16 (± 1.5) 30.53 (± 0.8)
QD-DETR (2023) 48.92 (± 4.1) 33.57 (± 3.3) 52.87 (± 0.8) 36.01 (± 1.3) 54.01 (± 0.7) 37.29 (± 0.5) 53.07 (± 0.8) 37.53 (± 1.1)
QD-DETR${\dagger}$ (2023) 50.15 (± 4.6) 34.67 (± 3.9) 53.53 (± 1.3) 38.30 (± 1.2) 53.37 (± 0.6) 37.93 (± 0.5) 53.39 (± 1.0) 38.47 (± 0.8)
EaTR (2023) 49.20 (± 3.2) 34.82 (± 3.5) 50.50 (± 0.6) 35.27 (± 0.7) 51.76 (± 0.5) 36.92 (± 0.7) 52.33 (± 0.5) 37.01 (± 0.3)
CG-DETR (2023) 50.65 (± 3.5) 36.37 (± 2.9) 56.26 (± 0.7) 40.82 (± 0.7) 54.53 (± 0.9) 39.32 (± 0.8) 56.72 (± 0.7) 41.79 (± 1.2)
TR-DETR (2024) 50.99 (± 3.3) 35.55 (± 3.7) 55.37 (± 1.0) 39.92 (± 2.0) 56.03 (± 1.0) 40.69 (± 0.9) 56.94 (± 0.5) 41.99 (± 0.3)
UMT${\dagger}$ (2022) 44.76 (± 3.5) 29.41 (± 3.0) 48.15 (± 1.7) 32.18 (± 1.6) 49.96 (± 0.9) 33.90 (± 0.9) 48.83 (± 1.0) 34.09 (± 1.2)
UniVTG (2023) 47.50 (± 3.1) 31.58 (± 3.0) 49.50 (± 0.8) 33.09 (± 1.1) 50.98 (± 0.2) 33.36 (± 0.6) 51.42 (± 1.1) 43.75 (± 0.2)
UVCOM (2023) 50.99 (± 3.6) 37.36 (± 3.1) 54.39 (± 0.5) 40.06 (± 1.0) 55.88 (± 0.7) 40.88 (± 0.5) 54.92 (± 0.9) 41.08 (± 0.9)
SeViLA (2023) 17.37 (± 1.3) 10.56 (± 0.8) 22.72 (± 0.8) 15.31 (± 0.7) 25.94 (± 0.1) 16.99 (± 0.3) 26.83 (± 0.8) 16.83 (± 0.6)
Summarization Moment-DETR (2021) 42.00 (± 3.3) 25.14 (± 3.0) 44.56 (± 2.4) 27.24 (± 2.1) 43.73 (± 2.0) 27.00 (± 1.8) 44.34 (± 2.6) 27.74 (± 2.0)
QD-DETR (2023) 45.56 (± 3.3) 30.44 (± 3.0) 49.09 (± 3.8) 33.64 (± 3.2) 48.89 (± 3.5) 32.66 (± 3.1) 47.83 (± 4.1) 32.86 (± 3.8)
QD-DETR${\dagger}$ (2023) 46.57 (± 3.8) 32.52 (± 3.6) 49.30 (± 4.3) 34.12 (± 4.2) 48.83 (± 3.2) 34.16 (± 3.4) 49.13 (± 4.4) 33.83 (± 3.1)
EaTR (2023) 45.79 (± 3.0) 32.67 (± 2.9) 48.45 (± 2.9) 32.96 (± 2.7) 48.24 (± 3.8) 33.35 (± 3.5) 48.69 (± 3.7) 33.85 (± 2.5)
CG-DETR (2023) 47.07 (± 4.2) 33.14 (± 4.1) 51.46 (± 3.1) 36.49 (± 2.7) 50.59 (± 3.4) 36.08 (± 3.6) 51.91 (± 3.5) 36.58 (± 2.4)
TR-DETR (2024) 46.44 (± 4.4) 33.23 (± 3.8) 51.35 (± 3.2) 36.14 (± 2.3) 51.92 (± 3.8) 36.29 (± 3.7) 52.87 (± 4.0) 36.77 (± 3.4)
UMT${\dagger}$ (2022) 43.88 (± 3.4) 29.28 (± 1.9) 45.39 (± 2.8) 29.98 (± 2.4) 45.37 (± 2.3) 30.01 (± 2.2) 46.35 (± 2.0) 30.27 (± 1.0)
UniVTG (2023) 44.98 (± 3.3) 27.99 (± 2.7) 46.19 (± 3.5) 30.37 (± 2.4) 47.22 (± 3.3) 29.90 (± 2.5) 50.39 (± 3.3) 30.33 (± 2.4)
UVCOM (2023) 46.62 (± 3.8) 33.40 (± 3.4) 51.48 (± 4.1) 36.92 (± 3.7) 50.91 (± 5.3) 36.58 (± 4.5) 51.18 (± 3.7) 36.23 (± 3.4)
SeViLA (2023) 17.89 (± 1.9) 10.65 (± 1.5) 27.47 (± 3.5) 16.98 (± 1.9) 27.76 (± 2.5) 17.77 (± 1.5) 28.61 (± 3.3) 17.30 (± 2.0)
Visual Query Enc. Moment-DETR (2021) 12.55 5.69 13.38 6.59 14.36 6.01 14.88 6.53
QD-DETR (2023) 15.91 9.12 14.88 8.62 13.90 8.49 14.62 8.36
QD-DETR${\dagger}$ (2023) 15.65 10.03 12.60 6.79 12.34 6.72 12.34 7.44
EaTR (2023) 19.86 13.00 19.91 12.99 21.15 13.45 21.48 13.38
CG-DETR (2023) 22.90 13.00 24.93 13.58 23.24 13.12 24.74 14.23
TR-DETR (2024) 17.92 11.19 17.36 11.10 15.14 9.86 15.60 9.53
UMT${\dagger}$ (2022) 5.43 2.85 4.77 2.09 5.22 2.35 4.57 2.42
UniVTG (2023) 21.93 13.00 23.89 13.64 22.78 13.19 22.52 12.79
UVCOM (2023) 17.08 9.77 16.78 10.97 17.36 11.68 17.10 11.23

Table 2: Model performance (mAP) on ICQ. We highlight the best score in bold for each adaptation method and reference image style. For Cap and Sum, we also report the standard deviation of 3 runs with different prompts. ${\dagger}$ indicates the usage of additional audio modality.

llcccccccc[colortbl-like] 3-8,12,19-21,23-28,32 9-11 1 Model scribble cartoon cinematic realistic
mAP@0.5 Avg. mAP@0.5 Avg. mAP@0.5 Avg. mAP@0.5 Avg.
Captioning Moment-DETR (2021) 46.98 (± 2.3) 26.15 (± 1.5) 48.14 (± 1.2) 27.22 (± 0.7) 48.98 (± 0.4) 27.96 (± 0.4) 49.00 (± 0.82) 27.72 (± 0.5)
QD-DETR (2023) 50.69 (± 3.1) 31.01 (± 2.4) 54.15 (± 0.9) 33.04 (± 0.9) 55.32 (± 0.9) 34.06 (± 0.7) 54.75 (± 0.7) 34.31 (± 0.7)
QD-DETR${\dagger}$ (2023) 50.78 (± 3.9) 31.44 (± 3.0) 53.91 (± 1.2) 33.94 (± 1.0) 54.06 (± 0.5) 34.67 (± 0.3) 53.82 (± 0.8) 34.18 (± 0.7)
EaTR (2023) 52.11 (± 2.8) 32.88 (± 2.6) 53.23 (± 0.7) 33.60 (± 0.7) 54.00 (± 0.7) 34.54 (± 0.3) 54.36 (± 0.8) 34.73 (± 0.3)
CG-DETR (2023) 51.13 (± 3.0) 32.13 (± 2.1) 56.15 (± 0.8) 36.08 (± 0.6) 55.15 (± 1.0) 35.22 (± 0.7) 56.63 (± 0.8) 36.57 (± 0.9)
TR-DETR (2024) 51.07 (± 2.5) 32.15 (± 2.1) 55.72 (± 1.1) 35.98 (± 1.2) 55.87 (± 0.8) 36.29 (± 0.5) 56.32 (± 0.4) 36.76 (± 0.5)
UMT${\dagger}$ (2022) 42.35 (± 2.7) 26.47 (± 2.0) 45.03 (± 1.3) 28.64 (± 1.0) 46.43 (± 0.8) 30.01 (± 0.7) 45.93 (± 0.8) 29.67 (± 0.8)
UniVTG (2023) 40.68 (± 2.5) 24.71 (± 1.9) 42.68 (± 0.7) 26.03 (± 0.6) 43.53 (± 0.4) 26.43 (± 0.5) 43.64 (± 0.8) 26.76 (± 0.5)
UVCOM (2023) 51.27 (± 3.2) 33.39 (± 2.5) 54.40 (± 0.7) 36.50 (± 0.7) 55.99 (± 0.7) 37.11 (± 0.3) 54.98 (± 0.8) 36.83 (± 0.6)
SeViLA (2023) 14.45 (± 0.8) 9.30 (± 0.6) 19.52 (± 0.5) 13.12 (± 0.4) 22.16 (± 0.3) 14.64 (± 0.4) 22.48 (± 0.6) 14.55 (± 0.5)
Summarization Moment-DETR (2021) 44.40 (± 2.5) 23.96 (± 1.8) 47.31 (± 2.1) 26.03 (± 1.4) 46.62 (± 1.9) 25.55 (± 1.3) 47.29 (± 2.2) 26.07 (± 1.3)
QD-DETR (2023) 47.09 (± 2.8) 28.27 (± 2.4) 51.06 (± 3.3) 30.90 (± 2.5) 50.89 (± 3.3) 30.52 (± 2.8) 50.05 (± 3.6) 30.49 (± 2.7)
QD-DETR${\dagger}$ (2023) 48.10 (± 3.2) 29.49 (± 2.9) 50.72 (± 3.3) 31.11 (± 3.0) 49.94 (± 2.8) 31.38 (± 2.4) 50.30 (± 3.8) 30.85 (± 2.6)
EaTR (2023) 49.07 (± 2.6) 30.92 (± 2.0) 50.82 (± 2.6) 31.38 (± 1.7) 50.71 (± 3.2) 31.34 (± 2.7) 51.37 (± 3.0) 32.02 (± 2.0)
CG-DETR (2023) 48.41 (± 3.5) 29.86 (± 2.9) 52.31 (± 2.9) 33.21 (± 2.3) 51.59 (± 2.8) 32.34 (± 2.5) 52.31 (± 3.1) 32.91 (± 2.0)
TR-DETR (2024) 46.69 (± 3.6) 29.72 (± 2.8) 52.41 (± 2.6) 33.48 (± 1.9) 52.39 (± 3.1) 33.14 (± 2.6) 52.87 (± 3.1) 33.57 (± 2.5)
UMT${\dagger}$ (2022) 40.99 (± 2.7) 25.88 (± 1.8) 43.03 (± 2.0) 27.02 (± 1.5) 42.88 (± 2.0) 26.73 (± 1.6) 43.89 (± 1.3) 27.38 (± 1.0)
UniVTG (2023) 38.86 (± 2.7) 22.76 (± 1.8) 40.13 (± 2.8) 24.43 (± 1.7) 40.73 (± 2.7) 24.02 (± 1.9) 40.20 (± 2.4) 24.11 (± 1.6)
UVCOM (2023) 47.33 (± 3.2) 30.75 (± 2.5) 52.22 (± 3.4) 34.00 (± 2.7) 51.37 (± 4.2) 33.36 (± 3.1) 51.64 (± 3.8) 33.52 (± 2.6)
SeViLA (2023) 14.54 (± 1.7) 9.24 (± 1.3) 22.13 (± 1.8) 14.07 (± 1.1) 22.17 (± 1.4) 14.52 (± 0.9) 22.87 (± 1.8) 14.45 (± 1.3)
Visual Query Enc. Moment-DETR (2021) 14.95 6.67 16.51 7.21 17.00 7.39 17.41 7.66
QD-DETR (2023) 19.48 10.11 19.57 10.18 18.07 9.54 18.88 9.94
QD-DETR${\dagger}$ (2023) 18.22 9.74 14.31 7.30 15.18 7.45 14.71 7.66
EaTR (2023) 25.27 13.98 25.95 14.21 26.83 14.70 26.65 14.49
CG-DETR (2023) 30.24 15.57 30.78 15.70 30.07 15.48 30.98 15.83
TR-DETR (2024) 21.09 11.67 20.87 11.71 19.62 11.02 19.72 10.76
UMT${\dagger}$ (2022) 5.57 2.81 4.66 1.96 5.60 2.46 4.59 2.23
UniVTG (2023) 24.30 13.02 20.80 11.56 19.85 10.99 19.42 10.95
UVCOM (2023) 20.13 11.15 20.19 11.96 20.67 12.37 20.73 12.03

Best adaptation methods We find that Cap can achieve the best performance and is more robust to different prompts compared to other adaptation methods by an average margin of 3.6% on all styles. We observe that both utilizing MLLMs for captioning reference images, Sum suffers more than Cap adaptation regarding performance and is more sensitive to prompts for all reference styles, which can be observed from the higher standard deviation, showing asking MLLMs to caption and summarize the refinement texts is less controllable. To conclude, captioning images is still a golden method since MLLMs and LLMs are powerful enough to generate faithful captions.

Comparing models Models that perform well in one adaptation method tend to perform well in others. For example, UVCOM and TR-DETR consistently show high performance across Cap, Sum, and VisEnc methods. We observe that more recent models keep their outperforming performance on our ICQ. Latest models, including UVCOM, TR-DETR, and CG-DETR, tend to perform better across different adaptation methods and reference image styles. In contrast, older models like Moment-DETR consistently show lower performance. LLM-based models, SeViLA, cannot compete with other specialized models; this aligns with their subpar performance on natural language query-based benchmarks. In the next section, we find that model performance on ICQ highly correlates with that on natural language query-based benchmark QVHighlight. This shows that (1) our multimodal queries share semantics with the original benchmark; (2) the adaptation methods and models could understand semantics from multimodal queries.

Comparing styles We find all adaptation methods perform consistently across different styles and therefore suggest that they could understand the multimodal semantics queries well, particularly for styles including cartoon, cinematic, and realistic; the model performance is close to each other. For scribble, all models show marginally worse performance, and even both Cap and Sum methods have more significant standard deviation, which reflects that it is heavily influenced by the prompts. This can be explained by the fact that scribble images are more minimal and abstract in semantics and more challenging to interpret. Surprisingly, in spite of being more abstract and simpler, the model performance on scribble reference images is close to other reference image styles. This demonstrates the potential of using scribble as multimodal queries in real-world video event localization applications like video search.

Comparing refinement text types In addition, we calculate the model performance on different subsets of refinement texts shown in Fig. 5. We conclude even though models have close performance across reference image styles, and they show varied performance on different refinement text types across styles. For scribble style, models perform in general for “relation” better than on other styles. For cartoon style, models demonstrate a more balanced performance across all types. The performance is notably higher for “environment” and “attribute” in cinematic style. Finally, for realistic style, the models yield better performance in “object” and “environment”.

(a) scribble

(b) cartoon

(c) cinematic

(d) realistic

Figure 5: Model performance on different subsets of refinement text types. We observe that model performance with different refinement text types varies across styles.

4.3 Ablation studies

Multimodal Query-based vs. Natural Language Query-based Performance We compare model performance on the multimodal query-based ICQ-Highlight and the original natural language query-based QVHighlight using Spearman’s rank correlation coefficient [53] on R1@0.5. For scribble, Spearman’s rank correlation coefficients are 0.89(Cap) and 0.93(Sum). The cartoon style yields coefficients of 0.98(Cap) and 0.94(Sum). The cinematic style shows coefficients of 0.93 for both Cap and Sum. Lastly, realistic has coefficients of 0.96(Cap) and 0.95(Sum). The high correlation scores indicate a strong positive correlation across benchmarks, suggesting queries of both benchmarks share the common semantics and yield the reliability of our benchmark.

Model Performance With vs. Without Refinement Texts To assess the impact of refinement texts on video event localization using multimodal queries, we have evaluated model performance using only reference images as queries, omitting refinement texts. We employ the Cap adaptation without a modifier for integrating refinement texts. As shown in Tab. 3, we present the model performance and their relative performance drop in percentage compared to those with refinement texts. Models have different scales of performance drop, which indicates that refinement texts help refine the semantics of reference images and localize the events. Additionally, we observe that for scribble images, the performance drop is less pronounced compared to other reference image styles in that these images are inherently minimalistic and less reliant on detailed semantics.

Table 3: Model performance without refinement texts. We employ Cap for methods without considering refinement texts. The performance drop highlighted in the parenthesis indicates that refinement texts in ICQ-Highlight can help refine the semantics of the reference images and localize the events better.

Model	scribble		cartoon		cinematic		realistic
	R1@0.5	R1@0.7	R1@0.5	R1@0.7	R1@0.5	R1@0.7	R1@0.5	R1@0.7
Moment-DETR	45.15 (-2.7$\%$)	28.72 (-3.3$\%$)	43.60 (-7.1$\%$)	27.94 (-5.8$\%$)	44.06 (-7.3$\%$)	29.70 (-2.8$\%$)	44.06 (-9.3$\%$)	28.98 (-6.5$\%$)
QD-DETR	49.81 (-4.0$\%$)	33.70 (-5.4$\%$)	49.87 (-6.6$\%$)	34.33 (-6.3$\%$)	49.67 (-9.3$\%$)	34.73 (-8.1$\%$)	50.52 (-5.7$\%$)	35.25 (-7.4$\%$)
QD-DETR${\dagger}$	51.29 (-3.9$\%$)	36.03 (-3.8$\%$)	48.69 (-10.8$\%$)	33.88 (-13.4$\%$)	49.48 (-8.5$\%$)	34.99 (-9.0$\%$)	49.93 (-7.5$\%$)	35.05 (-10.4$\%$)
EaTR	52.01 (+0.5$\%$)	37.77 (+1.2$\%$)	47.45 (-6.7$\%$)	33.09 (-8.0$\%$)	48.56 (-7.0)	34.33 (-5.1)	49.61 (-6.1$\%$)	35.64 (-3.0$\%$)
CG-DETR	51.42 (-4.0$\%$)	37.84 (-1.7$\%$)	49.35 (-13.0$\%$)	35.90 (-13.4$\%$)	48.89 (-10.3)	34.79 (-11.3)	51.04 (-10.5$\%$)	36.55 (-14.0$\%$)
TR-DETR	52.01 (-2.4$\%$)	37.19 (-2.9$\%$)	51.04 (-9.2$\%$)	36.62 (-11.2$\%$)	50.00 (-11.8)	36.03 (-12.5)	52.28 (-8.8$\%$)	37.53 (-10.6$\%$)
UMT${\dagger}$	46.25 (-3.0$\%$)	31.57 (-1.0$\%$)	45.82 (-6.9$\%$)	30.61 (-7.1$\%$)	46.34 (-8.6$\%$)	29.96 (-13.7$\%$)	46.08 (-6.2$\%$)	31.85 (-7.1$\%$)
UniVTG	47.87 (-3.8$\%$)	33.76 (-2.2$\%$)	45.56 (-9.4$\%$)	29.24 (-11.5$\%$)	45.43 (-11.2$\%$)	29.05 (-13.9$\%$)	46.80 (-9.3$\%$)	30.42 (-12.4$\%$)
UVCOM	52.26 (-1.7$\%$)	39.39 (+1.0$\%$)	51.50 (-6.1$\%$)	37.99 (-6.6$\%$)	50.98 (-9.4$\%$)	36.75 (-11.3$\%$)	51.70 (-7.6$\%$)	37.53 (-10.5$\%$)
SeViLA	13.15 (-30.3$\%$)	8.06 (-29.3$\%$)	11.89 (-49.8$\%$)	6.89 (-57.0$\%$)	13.26 (-49.0$\%$)	8.32 (-51.5$\%$)	13.65 (-49.1$\%$)	8.22 (-51.1$\%$)

5 Conclusion

Limitations and Future Work As the first multimodal query-based video event localization benchmark, ICQ still has several limitations: (1) Selection of LLM-based models is limited as of the time of the work due to the lack of more open source models; (2) Our current benchmark utilizes generated multimodal queries and, as a result, can suffer from generation artifacts. Additionally, although our benchmark serves as a practical testbed, fine-tuning models with unlabeled videos [43, 59] for this new setting remains an open question, particularly because of the lack of training data.

Societal Impacts We believe that using multimodal semantic queries for video event localization brings prospects in real-world applications, such as providing service for illiterate, pre-literate, or non-speakers in cross-lingual situations, as it allows them to interact with videos through simple scribble images as a more accessible and convenient approach. However, reference images could contain intentionally harmful content, which may pose new threats to AI safety and privacy.

In this work, we introduce a new benchmark, ICQ, marking an initial step towards using multimodal semantic queries for video event localization. We have found that MLLM/LLM-enhanced enhanced adaptation methods can accommodate conventional models to multimodal queries, serving as a simple yet effective baseline for this novel setting. Our findings confirm that using multimodal queries for video event localization is practical and feasible. Nonetheless, the field remains open to innovative model architectures and training paradigms for multimodal queries. We believe our work paves the way for real-world applications that leverage multimodal queries to interact with video content.

Appendix A Appendix

In this Appendix, we present the following:

• Additional information about the dataset ICQ-Highlight and licenses for the datasets and models we have used.

• Additional technical implementations including prompts of the benchmark ICQ;

• Extended experimental results due to page limits in the main part.

A.1 Dataset: ICQ-Highlight

A.1.1 License

The dataset and code are publicly accessible. We use standard licenses from the community and provide the following links to the non-commercial licenses for the datasets we used in this paper.

QVHighlights: https://github.com/jayleicn/moment_detr/blob/main/data/LICENSE

Stability Diffusion:https://github.com/Stability-AI/stablediffusion/blob/main/LICENSE

A.1.2 Construction Pipeline

We base our model on the original annotation from QVHighlights [29]. The whole pipeline as shown in Fig. 6 consists of (1) annotation: We further conduct a quality check on the annotations in the original dataset and filter out a few samples (details can be found in Sec. A.1.4). In order to generate more relevant reference images, we manually augment the original captions by adding new visual details based on three frames extracted from the raw videos. To introduce refinement texts, we purposely alter certain details of the captions to generate a new one. All annotations are carried out by two individuals and evaluated by a third party for accuracy. (2) We use the augmented and altered captions to generate reference images with a suite of Text-2-Image models, including DALL-E 2 and Stability Diffusion XL for 4 variants of styles. (3) We implement an additional quality check process for all generated images to eliminate and regenerate images that might contain unsafe or counterintuitive content. We employ BLIP2 [30] to filter out generated images that have lower semantic similarity with augmented captions than 0.2 and conduct a manual sanity check to control the image quality.

Figure 6: Dataset Construction Pipeline: We base our model with original annotations from QVHighlights and introduce a pipeline consisting of annotation, reference image generation, and quality check.

A.1.3 Statistics

Tab. 4 presents the statistics for various reference image styles in terms of the number of queries, videos, and the presence of refinement texts. Tab. 5 breaks down the statistics of refinement texts for different reference image styles across various query types: object, action, relation, attribute, environment, and others. The numbers of each type can vary slightly depending on the different styles.

Table 4: Statistics of Different Reference Image Styles

Reference Image Style	#Queries	#Videos	#With Refinement Texts	#Without Refinement Texts
scribble	1546	1515	/	5
cinematic	1532	1502	1445	5
cartoon	1532	1501	1444	5
realistic	1532	1501	1446	4

Table 5: Statistics of Refinement Texts

Reference Image Style	#Queries
	Object	Action	Relation	Attribute	Environment	Others
scribble	594	242	50	162	343	70
cinematic	588	239	50	162	343	66
cartoon	590	239	48	161	341	68
realistic	586	241	50	161	341	70

A.1.4 Details of Deleted Data

We removed four entries from the QVHighlight dataset that could cause violent, sexual, sensitive, or graphic content in generation in the original natural language query as listed:

• “A graph depicts penis size.” (qid: 9737)

• “People mess with the bull statues testicles.” (qid: 7787)

• “People butcher meat from a carcass.” (qid: 4023)

• “Woman films herself wearing black lingerie in the bathroom.” (qid: 7685)

A.2 Benchmark Details

A.2.1 Model Comparison

Tab. 6 compares our selected baseline models. Query encoder denotes the text encoder of each model to encode natural language queries. Source represents the modalities of the source data, while V and A refer to “Video” and “Audio” respectively. All models have been fine-tuned on QVHilight.

Table 6: Comparison of selected baseline models. ^∗We only list the model head for the localization task if the model has multiple heads for different tasks.

Model	Visual Encoder	Query Encoder	Localization Decoder∗	Source
Moment-DETR (2021)	ViT-B/32 + SlowFast	CLIP Text	DETR	V
QD-DETR (2023)	ViT-B/32 + SlowFast	CLIP Text	DETR	V, V+A
EaTR (2023)	ViT-B/32 + SlowFast	CLIP Text	DETR	V
CG-DETR (2023)	ViT-B/32 + SlowFast	CLIP Text	DETR	V
TR-DETR (2024)	ViT-B/32 + SlowFast	CLIP Text	DETR	V, V+A
UMT (2022)	ViT-B/32 + SlowFast	CLIP Text	Transformer	V+A
UniVTG (2023)	ViT-B/32 + SlowFast	CLIP Text	Conv. Heads	V
UVCOM (2023)	ViT-B/32 + SlowFast	CLIP Text	Transformer Heads	V, V+A
SeViLA (2023)	ViT-G	CLIP Text	Multimodal LLM (BLIP2)	V

A.2.2 Prompt Engineering

Since the performance may highly depend on the wording in a prompt, we use 3 different prompts for Cap and Sum adaptation methods. In Tab. 7, the prompts are divided into “Prompts For Style cartoon/cinematic/realistic” and “Prompts for scribble”. This distinction arises because refining scribble images with complementary texts involves adding new details, slightly differing from other scenarios. Despite this minor variation, the prompt style remains consistent, simulating 3 different user query styles.

Table 7: Prompts for Cap and Sum. We use 3 different prompts and report the average performance and standard derivation in other tables.

	Prompts For Style cartoon/cinematic/realistic	Prompts For Style scribble
1	I have a caption {INPUT DATA}, adjust the {MODIFICATION TYPE} from {MODIFIED DETAIL} to {ORIGINAL DETAIL}. The revised caption should remain coherent and logical without introducing any additional details.	I have a caption {INPUT DATA}. Modify it by adding {NEW TYPE} {NEW DETAIL}. The revised caption should remain coherent and logical without introducing any other additional details.
2	Read this {INPUT DATA}! Change the {MODIFICATION TYPE} from {MODIFIED DETAIL} to {ORIGINAL DETAIL}. Then, write a new caption that fits and doesn’t add new stuff. Only give the caption, no extra words.	Read this {INPUT DATA}! Add the {NEW TYPE} {NEW DETAIL} to it. Then, write a new caption that fits and doesn’t add new stuff. Only give the caption, no extra words.
3	Here’s a caption {INPUT DATA}. Can you change {MODIFICATION TYPE} from {MODIFIED DETAIL} to {ORIGINAL DETAIL}? After that, make a new caption that makes sense and doesn’t add anything extra. Just write the caption, no explanations needed.	Here’s a caption {INPUT DATA}. Can you add {NEW TYPE} {NEW DETAIL}? After that, make a new caption that makes sense and doesn’t add anything extra. Just write the caption, no explanations needed.

A.3 Extended Results

A.3.1 Model Performance with Different Prompts

We demonstrate the results of 3 prompts across different models and various metrics, including R1@0.5, R1@0.7, mAP@0.5, mAP@0.7, and Avg, showing both the Cap and Sum methods.

In Fig. 7, the results indicate that the performance is consistent across different metrics, demonstrating the robustness of the models when using Cap. The models generally maintain similar performance levels regardless of the specific metric, suggesting their stability and reliability in Cap. In contrast, Fig. 8 illustrates the performance of the same models but using Sum. It is evident that prompt 1 consistently outperforms prompts 2 and 3 across all metrics. This indicates that the models are more sensitive to the formulation of the prompt in Sum.

Figure 7: Model Performance of Cap on ICQ.

Figure 8: Model Performance of Sum on ICQ.

Table 8: Performance comparison between the original query and corrupted text. The performance drop highlighted in the parenthesis indicates that the modifications on natural language query are non-trivial. ${\dagger}$ indicates the usage of additional audio modality.

Method	original
	R1@0.5	R1@0.7	mAP@0.5	mAP@0.7	Avg.
Moment-DETR (2021)	54.92 (-4.6$\%$)	36.87 (-3.3$\%$)	55.95 (-4.2$\%$)	31.59 (-4.5$\%$)	32.54 (-3.8$\%$)
QD-DETR (2023)	62.87 (-8.6$\%$)	46.70 (-12.5$\%$)	62.66 (-7.6$\%$)	41.59 (-12.4$\%$)	41.23 (-10.3$\%$)
QD-DETR${\dagger}$ (2023)	63.71 (-6.2$\%$)	47.67 (-8.1$\%$)	62.9 (-5.6$\%$)	42.07 (-6.6$\%$)	41.73 (-6.4$\%$)
EaTR (2023)	60.93 (-8.0$\%$)	46.12 (-9.5$\%$)	62.01 (-5.9$\%$)	42.11 (-7.6$\%$)	41.39 (-6.7$\%$)
CG-DETR (2023)	67.27 (-8.9$\%$)	51.94 (-13.6$\%$)	65.48 (-7.6$\%$)	45.64 (-12.4$\%$)	44.88 (-11.3$\%$)
TR-DETR (2024)	67.08 (-7.5$\%$)	51.36 (-8.3$\%$)	66.20 (-7.3$\%$)	46.28 (-9.3$\%$)	44.99 (-8.1$\%$)
UMT${\dagger}$ (2022)	60.22 (-10.0$\%$)	44.24 (-14.1$\%$)	56.62 (-9.5$\%$)	39.85 (-15.2$\%$)	38.54 (-12.9$\%$)
UniVTG (2023)	59.70 (-8.7$\%$)	40.82 (-7.2$\%$)	51.22 (-8.0$\%$)	32.84 (-9.9$\%$)	32.53 (-9.0$\%$)
UVCOM (2023)	65.01 (-5.6$\%$)	51.75 (-8.0$\%$)	64.88 (-5.3$\%$)	46.96 (-9.0$\%$)	45.83 (-8.2$\%$)
SeViLA (2023)	56.57 (-56.2$\%$)	40.45 (-62.1$\%$)	47.14 (-56.8$\%$)	32.69 (-62.3$\%$)	33.10 (-60.6$\%$)

Table 9: Model performance (Recall) of Captioning without refinement text and Visual Query Enc. on ICQ. We highlight the best score in bold for both methods and reference image style.

llcccccccc[colortbl-like] Model scribble cartoon cinematic realistic
R1@0.5 R1@0.7 R1@0.5 R1@0.7 R1@0.5 R1@0.7 R1@0.5 R1@0.7
Captioning only Moment-DETR (2021) 45.15 28.72 43.60 27.94 44.06 29.70 44.06 28.98
QD-DETR (2023) 49.81 33.70 49.87 34.33 49.67 34.73 50.52 35.25
QD-DETR${\dagger}$ (2023) 51.29 36.03 48.69 33.88 49.48 34.99 49.93 35.05
EaTR (2023) 52.01 37.77 47.45 33.09 48.56 34.33 49.61 35.64
CG-DETR (2023) 51.42 37.84 49.35 35.90 48.89 34.79 51.04 36.55
TR-DETR (2024) 52.01 37.19 51.04 36.62 50.00 36.03 52.28 37.53
UMT${\dagger}$ (2022) 46.25 31.57 45.82 30.61 46.34 29.96 46.08 31.85
UniVTG (2023) 47.87 33.76 45.56 29.24 45.43 29.05 46.80 30.42
UVCOM (2023) 52.26 39.39 51.50 37.99 50.98 36.75 51.70 37.53
Visual Query Enc. Moment-DETR (2021) 12.55 5.69 13.38 6.59 14.36 6.01 14.88 6.53
QD-DETR (2023) 15.91 9.12 14.88 8.62 13.90 8.49 14.62 8.36
QD-DETR${\dagger}$ (2023) 15.65 10.03 12.60 6.79 12.34 6.72 12.34 7.44
EaTR (2023) 19.86 13.00 19.91 12.99 21.15 13.45 21.48 13.38
CG-DETR (2023) 22.90 13.00 24.93 13.58 23.24 13.12 24.74 14.23
TR-DETR (2024) 17.92 11.19 17.36 11.10 15.14 9.86 15.60 9.53
UMT${\dagger}$ (2022) 5.43 2.85 4.77 2.09 5.22 2.35 4.57 2.42
UniVTG (2023) 21.93 13.00 23.89 13.64 22.78 13.19 22.52 12.79
UVCOM (2023) 17.08 9.77 16.78 10.97 17.36 11.68 17.10 11.23

A.3.2 Captioning Without Refinement Text V.S. Visual Query Encoding

We compare the model performance between Cap without refinement text and VisEnc, as shown in Tab. A.3.1. Both methods only use reference images as queries without refinement texts. Overall, Cap without refinement texts still significantly outperforms pure VisEnc, highlighting the effectiveness of image captioning. Additionally, TR-DETR and UVCOM perform best across all styles.

A.3.3 Original vs. Corrupted Text in Our Annotation

We have evaluated the model performance based on the original queries in QVHighlights and our corrupted texts to assess the significance of the refinement texts and the sensitivity of different models to natural language queries. [45] points out that the impact of natural language query may be minimal for some existing models, such as Moment-DETR. As shown in Tab. 8, Moment-DETR exhibits relatively smaller drops across all metrics, supporting this claim. On the other hand, the latest models, such as CG-DETR and TR-DETR, experience larger performance drops, indicating a higher sensitivity to query modifications. Furthermore, SeViLA is extremely sensitive to query modifications, which is shown by severe declines in performance across all evaluated metrics. Overall, the considerable performance decline across various models demonstrates that our modifications significantly affect the original queries. This also shows that our introduced refinement texts are not semantically trivial for localizing with multimodal queries.