Short Film Dataset (SFD):
A Benchmark for Story-Level Video Understanding

Abstract

Recent advances in vision-language models have significantly propelled video understanding. Existing datasets and tasks, however, have notable limitations. Most datasets are confined to short videos with limited events and narrow narratives. For example, datasets with instructional and egocentric videos often document the activities of one person in a single scene. Although some movie datasets offer richer content, they are often limited to short-term tasks, lack publicly available videos and frequently encounter data leakage given the use of movie forums and other resources in LLM training. To address the above limitations, we propose the Short Film Dataset (SFD) with 1,078 publicly available amateur movies, a wide variety of genres and minimal data leakage issues. SFD offers long-term story-oriented video tasks in the form of multiple-choice and open-ended question answering. Our extensive experiments emphasize the need for long-term reasoning to solve SFD tasks. Notably, we find strong signals in movie transcripts leading to the on-par performance of people and LLMs. We also show significantly lower performance of current models compared to people when using vision data alone.

Figure 1: VideoQA examples from three video domains: instructional videos, egocentric videos and movies. While instructional and egocentric videos usually depict one or two people performing a single task, movies present time-extended stories with a rich variety in terms of scenes, characters and interactions.

https://shortfilmdataset.github.io

1 Introduction

Recent advances in vision-language models [33, 31, 9, 10, 61, 60, 16, 15, 11, 68] show significant promise for enhancing machine perception. Despite their remarkable progress, current datasets are still constrained by some limitations. Most of them consist of short videos [20, 1, 13, 69, 72, 17, 55, 27, 32, 19, 65], often under one minute, and primarily target short-term tasks e.g., action recognition and video retrieval, often requiring only a few seconds of visual content for problem-solving [39, 28]. For long-form video understanding, the community has turned to three video categories. (i) Egocentric videos, an emerging domain featuring extended first-person sequences that capture a person’s continuous actions [18, 12, 24, 34, 39, 52, 51] or chronicle daily activities, offering tasks from object manipulation to social interactions [18]. Nevertheless, while such datasets excel in video duration, they lack narrative depth, focusing mainly on immediate visuals rather than rich storytelling; (ii) Instructional videos [41, 71], which depict a wide variety of human tasks and focus on the procedural aspect, are often brief and deficient in storytelling; (iii) Movies, unlike egocentric and instructional videos, provide complex plots and extended duration, making them an ideal testing ground for developing models that can comprehend long-form video narratives, see Figure 1.

Several movie datasets have been proposed in the literature [48, 4, 56, 29, 53, 21, 64, 22, 59, 66, 49, 40, 44, 7, 6, 38] offering rich narrative-driven content and various tasks. These tasks include clip retrieval [4], video question answering [56, 29, 38], audio description [53, 21], semantic role labeling [49], scene segmentation [22, 44], and visual scene recognition [6]. However, current movie datasets suffer from three main limitations. (i) Accessibility: these datasets are not publicly available as they typically contain commercial copyrighted movies (see Figure 2). (ii) Clip duration: they often consist of incomplete and short clips [56, 29, 45, 64] (i.e., less than 4 minutes, see Figure 2), limiting their potential for long-form understanding with evolving storylines. Moreover, these datasets frequently suffer from ill-designed benchmark questions with insufficient narrative focus [54]. (iii) Data Leakage: as these datasets comprise well-known commercial movies, modern Large Language Models (LLMs) and Vision-Language Models (VLMs) have likely been exposed to some form of movie information (i.e., synopses, reviews, discussions, subtitles, blog posts). Figure 3 showcases this, where given only movie titles, modern LLMs can obtain high accuracy when answering questions from main movie datasets. This data leakage leads to biased benchmarks and ineffective training.

Figure 2: Comparison of SFD to other VQA datasets. The circle size indicates the number of QA pairs in each dataset.

Figure 3: Data leakage. When given only the movie title, higher zero-shot accuracy in question-answering by LLMs indicates greater data leakage. LLMs are ranked by MMLU.

To address the above limitations, we introduce the Short Film Dataset (SFD), a novel video benchmark comprising 1,078 short films, totalling over 243 hours of videos with an average duration of 13 minutes per film—significantly longer than existing datasets (see Figure 2). Unlike copyrighted films, SFD includes publicly accessible amateur films of diverse genres from YouTube. Although these films are shorter than traditional movies, they feature complex narratives that unfold through key sequences of events and multiple character interactions. Most importantly, with minimal exposure to LLMs, our dataset is less prone to data leakage issues plaguing other movie datasets (see Figure 3).

Aiming towards long-term story-level multimodal video understanding, we propose two question-answering tasks on SFD: (a) following recent trend [56, 45], we propose Multiple-Choice Question answering (MCQ); (b) We also introduce Open-Ended Question answering (OEQ), a more generative and unrestricted QA format that more realistically and objectively measures a model’s ability to comprehend long-form videos. All questions and answers are generated by LLMs from movie descriptions and undergo thorough manual curation, ensuring they accurately reflect the settings, characters, storylines, and themes of the movies.

Figure 4: Samples in the SFD dataset. SFD features diverse genres, characterized by distinct visual styles and storytelling. Each movie comes with a concise, high-level description known as a logline.

To validate the advantages of SFD, we conduct extensive experiments and quantify data leakage, the benefits of long-term reasoning and comparison to human performance while using state-of-the-art vision and language models. Our findings are: (i) Unlike existing VideoQA datasets, SFD provides story-level tasks defined on public data with minimal data leakage (Section 3.1); (ii) our long-term reasoning experiments in Section 3.2 confirm the advantage of movie-level video understanding for solving SFD tasks; (iii) the evaluation in Section 3 reveals a large gap between recent VideoQA methods and human performance in vision-only and multimodal settings. SFD, hence, provides a unique benchmark for evaluating and advancing new methods for story-level long-term video understanding. Our dataset, code, and models are publicly available at https://shortfilmdataset.github.io.

2 Short Film Dataset

Short films are motion pictures that typically last between 5 and 20 minutes. As illustrated in Figure 4, short films span a variety of styles (e.g., narrative fiction, documentary, animation, etc.) and genres (e.g., action, drama, comedy, horror, etc.). Filmmakers often use the short film format to explore new ideas, techniques, and storytelling styles. Short films, hence, have more of an experimental rather than commercial value and are often made publicly available. Samples from the dataset can be found in Appendix H.

The amount of publicly available short films has significantly increased over the last years (see Figure 6(c)). We take advantage of this development and create a Short Film Dataset (SFD) to foster research on story-level video understanding. In the remaining part of this section, we describe the procedure for collecting videos and corresponding metadata (Section 2.1) as well as generating question-answer pairs (Section 2.2). Section 2.3 presents the analysis and statistics of SFD. The overall pipeline for SFD creation is illustrated in Figure 5.

2.1 Data Collection

To build our dataset, we capitalize on the abundance of short films on video-sharing platforms. We specifically target the Omeleto YouTube channel and its subchannels, including @OmeletoComedy and @OmeletoDrama, among others. These channels promote high-quality and award-winning short films, some of which have been recognized by Oscars and BAFTA awards. We download videos, subtitles, and associated metadata from YouTube using yt-dlp.

As speech transcripts were originally available only for a portion of the dataset, we have used WhisperX [3] to extract missing transcripts. The metadata accompanying each short film includes the movie title, a logline which is a concise one-sentence summary of the plot, and additional movie details such as genre, release year, region of origin, and language. Each film also comes with a detailed description that offers a deeper understanding of various elements such as a synopsis providing a brief narrative overview, the director’s inspirations, storytelling approach, actors’ performances, and other external information. To maintain clarity and objectivity in our dataset, we retain only the synopsis and exclude other subjective or ambiguous elements of the description.

Figure 5: Data generation pipeline of SFD. For short films with available synopses, we use an LLM to generate Question-Answer Pairs and to create challenging distractors. Both Open-Ended and Multiple-Choice Questions undergo automatic filtering and manual curation. Annotation details, e.g., shots, clip info, face tracks and subtitles are also extracted and included in the SFD dataset.

2.2 Question and Answer Generation

To promote the development of long-form and story-oriented video understanding, we complement SFD with two Video Question Answering (VideoQA) tasks: Multiple-Choice Question answering (MCQ) and Open-Ended Question answering (OEQ). We first automatically generate question-answer pairs by prompting GPT-4 [42] with movie titles, loglines and synopses. We then manually curate questions and answers to ensure their relevance and correctness. We describe the generation process in more details below.

Automatic QA pair generation. For each film we obtain human-written information, i.e. movie titles, loglines and synopses from Omeleto YouTube channels. We then prompt GPT-4 [42] to propose relevant question-answer pairs from movie information, with dedicated prompts to avoid common mistakes. In particular, we emphasise direct and clear questions, with brief and specific answers to avoid speculation, repetition and ambiguity. Our example prompts are available in Appendix I. This stage results in 7,104 question-answer (QA) pairs that we use for the open-ended question-answering task.

Automatic Distractor Generation. To construct the MCQ task, we use GPT-4 [42] to complement each question with 4 distractors. The distractors are crafted to provide plausible yet incorrect answers that fit within the movie’s context. To this end, our prompts are designed with specific criteria: distractors should be syntactically similar to the correct answer while being semantically different. Our distractors also incorporate diverse misdirections relative to movies (character confusion, plot adjustments), consider the movie genre and plausible scenarios, and include accurate but irrelevant information from other parts of the synopsis. This stage results in 7,104 Multiple-Choice Questions (MCQs), with 5 options per question with one correct and four incorrect answers.

Manual QA Curation. To ensure the quality of questions and answers, we perform a thorough manual curation based on the following criteria. We remove questions that cannot be answered based on the synopsis, including factually incorrect, ambiguous, or subjective ones. Additionally, questions considered overly simplistic or answerable through external knowledge are excluded. This stage results in 4,885 MCQs from 1,078 movies.

2.3 Dataset Analysis and Statistics

Our SFD dataset contains a total of 1,078 unique films ranging in duration from 5 to 37 minutes and having an average length of 13 minutes (see Figure 6(a)). Overall, these movies amount to 243 hours and cover distinct genres, as illustrated in Figure 6(d). Each movie in the dataset is accompanied by a title, a logline which averages around 15 words, and a synopsis of about 97 words, equivalent to approximately 2 sentences, as shown in Figure 6(b). We use this metadata for question and answer generation as described in Section 2.2. The majority of films in SFD are in English and mainly originate from North America and Europe (see Figures 6(f) and 6(e)). Examples of the dataset’s diversity can be seen in Figure 4. Additional data preprocessing and statistics are available in Appendix C, including clips, captions, location tags, and face tracks.

In terms of video understanding tasks, SFD comprises 4,885 MCQs and OEQs, averaging 4.53 questions per film. Table 1 shows the comparison to both general video understanding (videoQA) and movie-oriented (movieQA) question-answering datasets. Notably, SFD exhibits the longest average video duration (821 seconds) among all datasets. Compared to videoQA datasets, SFD holds the properties of multimodality and long-term nature, similar to most movieQA datasets. Compared to other movieQA ones, SFD offers advantages in public availability of videos, data leakage (i.e., not exposed to LLMs), and story completeness (using full movies instead of trimmed clips, potentially impacting narrative coherence).

(a) Video duration

(b) Synopsis length

(c) Release year

(d) Movie genre

(e) Region of origin

(f) Language

Figure 6: Statistics of SFD.

Table 1: Comparison of VideoQA and MovieQA datasets.

Dataset	Venue	Annotation	Avg.	#QA Pairs	Multimodal	Long-Term	Accessible	Unknown	Full
			Length (s)					To LLMs	Movies
General VideoQA datasets
MSRVTT-QA [67] (test)	ACM 2017	Auto	15	72,820	✗	✗	✓	✓	-
MSVD-QA [8] (test)	ACM 2017	Auto	10	13,156	✗	✗	✓	✓	-
TGIF-QA [23] (test)	CVPR 2017	Auto	3	25,751	✗	✗	✓	✓	-
ActivityNet-QA [75] (test)	AAAI 2019	Manual	180	8,000	✗	✗	✓	✓	-
How2QA [50] (test)	EMNLP 2020	Manual	60	4,400	✗	✗	✓	✓	-
NeXT-QA [65] (test)	CVPR 2021	Manual	44	9,178	✗	✗	✓	✓	-
iVQA [35]	ICCV 2021	Manual	18	10,000	✗	✗	✓	✓	-
EgoSchema [39]	NeurIPS 2023	Manual + Auto	180	5,000	✗	✓	✓	✓	-
MovieQA datasets
MovieQA [56] (test)	CVPR 2016	Manual	203	6,462	✓	✓	✗	✗	✗
TVQA [29] (test)	EMNLP 2018	Manual	76	15,253	✓	✗	✗	✓	✗
LVU [64] (test)	CVPR 2021	Manual	220	1,223	✓	✓	✓	✗	✗
MovieChat [54] (test)	CVPR 2024	Manual	459	2,417	✓	✓	✗	✓	✗
CinePile [45] (test)	arXiv 2024	Manual + Auto	160	4,940	✓	✓	✓	✓	✗
SFD (Ours)		Manual + Auto	821	4,885	✓	✓	✓	✓	✓

Question Analysis. The generated question-answer pairs are long and diverse: the median answer length is 10 words and the vocabulary contains 8,928 different words. Furthermore, to ensure that the generated QAs cover various aspects of movies, we categorize them into four types (see Table 2):
1. Setting-related questions focus on the location and time of the movie. They may pertain to geographical locations (e.g., a specific city or country), a type of place (e.g., home, office), a historical period (e.g., a specific era), or a time of day (e.g., morning, night).
2. Character-related questions focus on the individuals portrayed in the movie. They may concern personal information (age, gender, profession), traits, motivations, and relationships (siblings, friends). Note that character development, as part of the narrative, is excluded from this category.
3. Story-related questions focus on the key sequence of events and interactions that compose the plot and narrative arc of the movie.
4. Theme-related questions focus on the underlying message or central idea that the movie explores and conveys to its audience.

Table 2: Samples from SFD by question type.

Question Type	Questions	Answers
Setting	Where do John and Emma have their conversation?	On the bus.
	Which city is the backdrop?	Liverpool.
	What decade does the movie take place in?	1990s.
Character	Who is the main character?	Josh.
	What is the nationality of the soldier who becomes a prisoner?	German.
	How is Mr. Jones described?	As stern and harsh.
Story	Can you name a specific tactic Tommy uses against Tiny?	He steals Tiny’s food at dinner and distracts her at a school-wide relay race.
	What is the imminent threat to Greg’s residence?	The house is in foreclosure and the bank is about to repossess it.
	What triggers Carla to believe her husband Frank will surprise her for their anniversary?	A bouquet left on her doorstep.
Theme	What is a major theme?	The theme revolves around reconnecting and the complexities of honesty between exes.
	What main theme is explored through Sarah’s experience?	The theme of romantic expectations versus present realities.
	What is the main theme?	The processing of grief and reconciliation between siblings.

3 Experiments

3.1 Data Leakage

Modern LLMs, pre-trained on vast internet data, risk being exposed during training to information about commercial films, such as synopses, reviews, scripts, and transcripts. This issue may result in models recalling answers directly without analyzing the video content, leading to biased benchmarks. In this section, we quantitatively assess the extent of data leakage on datasets. Specifically, we prompt LLMs to answer open-ended questions using only the movie title and compute the accuracy of responses, following [37]. We experiment with 3 movie datasets: MovieQA [56], LVU [64] and our SFD, 5 open-sourced models: Gemma 2B [57], Mistral 7B/8x7B [25], LLaMA-3 8B/70B [58] and 4 commercial models Claude 3 Haiku/Sonnet, and GPT-3.5/4 [42]. Figure 3 reports the results of the data leakage experiment and further details can be found in Appendix G.

We observe that both MovieQA and LVU suffer from data leakage, reaching up to 71.3% and 76.0% accuracies. In contrast, thanks to the low presence of amateur film on the internet, SFD exhibits a maximum accuracy of 36.0%, indicating a low leakage issue. Furthermore, as expected, the extent of data leakage correlates with the knowledge level reflected by MMLU score [14] (see Appendix section G for more experiments), indicating that LLMs with more knowledge exhibit higher levels of memorization, leading to worse leakage issues. For instance, for both MovieQA and LVU, the zero-shot accuracy increases from approximately random level (20%) to more than 70% as the model size rises. Meanwhile, SFD maintains stable and low accuracies ranging between 19.7% and 36.0%, regardless of LLM knowledge variation, further indicating its low data leakage. This experiment reveals that relying solely on existing datasets to evaluate new methods is insufficient. Instead, our SFD offers a more objective and reliable test bed for long-term video understanding.

3.2 Temporal Window Study

To confirm that our tasks require long-form video understanding, we conduct a temporal window study with benchmarks at three levels: shot-, scene-, and movie-level. A shot, defined as a continuous video clip, typically amounts to around 151 per movie. At the Shot-Level, the model uses data from a single shot, including partial subtitles and visual data, with predictions aggregated by taking the maximum logits. Scene-Level inference aggregates data from approximately 10 shots in a similar manner. Movie-Level inference utilizes the entire film’s data, as detailed in Section 3.3. To further analyze the behaviors of each modality, we ablate at each temporal level three combinations of modalities: Vision, Language, and multimodal Vision-Language. This study is conducted with two top-performing methods from Section 3.3: FrozenBiLM and LLoVi. Further details on the experiment can be found in Appendix E.

Figure 7: Accuracy across different temporal windows (shot-level, scene-level, and movie-level) across different input modalities.

Our results, shown in Figure 7, reveal the following trends: (1) Both Language-only and Vision-Language settings show substantial improvements with larger temporal windows. For instance, in the language-only setting, LLoVi’s accuracy increases from 38.5% at the shot-level to 64.2% at the movie-level, reflecting a 25.7% gain. Similarly, FrozenBiLM shows an increasing trend but with a smaller gain in accuracy (+7.3%). The multimodal setting of LLoVi’s accuracy increases from 50.1% to 55.6% at the scene-level but shows only marginal gains at the movie-level. This plateau may be attributed to LLoVi’s naive fusion approach, which combines visual captions with subtitles, limiting further improvements; (2) The Vision-only modality exhibits low and stagnant performance across all levels suggesting that the current handling of visual data is inadequate for this task. Additional results for other methods are presented in Appendix E.

In conclusion, broader temporal windows significantly enhance SFD task performance, especially in language-only settings, underscoring the importance of long-form understanding in our design.

3.3 Baseline Comparison

In this section, we benchmark recent videoQA methods on SFD and evaluate 7 open-source models in three categories: FrozenBiLM [5], mPLUG-Owl2 [74] and Video-LLaVA [33] are VLMs that project visual frames into embedding of an LLM with the help of adapters; MovieChat [54] and TimeChat [47] have more advanced and specific settings, the former introduces a memory mechanism to handle long sequences and the latter proposes a timestamp-aware frame encoder for time-sensitive tasks; LLoVi [79] and LangRepo [26] are text-based methods: they both rely on visual captions on top of an LLM. LangRepo proposes an extra summarization and chain-of-thoughts [62] reasoning prompting.

These models are tested in a zero-shot video question-answering setting, adapted for long-form video understanding by incorporating subtitles and sampling the maximum number of frames possible. In the Multiple-Choice Question answering (MCQ) benchmark, we calculate the accuracy score based on the logits of the predicted options where applicable [5, 33, 74]. For models that generate plain-text answers, we evaluate them using a method similar to the Open-Ended Question answering (OEQ) where we rely on GPT-3.5 to compute the similarity between the predicted and correct answers, as detailed in [37]. All methods are evaluated across three modalities to better assess the contribution of each: Vision-Only (V) with video frames, Language-Only (L) with subtitles, and Vision-Language (VL) combining both. See Appendix D for more details. The results are summarized in Table 3.

User Study. To verify the answerability of our multiple-choice questions and assess the upper limit of SFD, we conducted three user studies accordingly to the aforementioned modality definitions: (1) Vision-Language (VL),—full video with audio and subtitles; (2) Vision-only (V)-muted videos; (3) Language-only (L)-plain text subtitles. For each question, all participants were asked to select the correct answer. Table 3 (last row) reports the results. More details on the user study are available in Appendix F. We observe that when provided with the full multimodal information, participants answer questions with high accuracy (89.8%). As expected, removing modalities lowers accuracy. Specifically, when using only subtitles the performance is 70.9%, whereas the vision-only performance drops to 59.0%.

Table 3: Baselines. Multiple-choice question answering and Open-ended question answering performances when using Vision-Only (V), Language-Only (L) and Vision-Language (VL) information.

Method	Venue	% Accuracy
		Multiple-Choice QA			Open-Ended QA
		V	L	VL	V	L	VL
Random		20.0	20.0	20.0	-	-	-
FrozenBiLM [5]	NeurIPS 2021	23.4	38.2	38.6	-	-	-
mPLUG-Owl2 [74]	CVPR 2024	38.3	20.7	21.3	22.1	1.8	1.6
Video-LLaVA [33]	arXiv 2023	34.2	21.3	24.7	19.2	6.4	8.0
LLoVi [79]	arXiv 2023	30.8	64.2	55.6	16.2	40.3	24.7
LangRepo [26]	arXiv 2024	29.0	32.1	31.0	3.5	10.4	9.5
MovieChat [54]	CVPR 2024	8.4	6.4	8.0	14.0	15.7	11.8
TimeChat [47]	CVPR 2024	25.5	6.4	31.8	26.4	9.4	5.9
Human		59.0	70.9	89.8	-	-	-

Multiple-Choice Question answering (MCQ). These results highlight significant insights. Overall, model performance is generally poor, with LLoVi being a notable exception in the language-only setting (64.2%), driven largely by subtitles. Some models, such as FrozenBiLM, perform adequately in the vision-only setting (38.3%), but their performance drops when combining both modalities, indicating integration issues.

Compared to human performance, there is still substantial room for improvement. Accuracy is on par in the language-only setting, with only a 6.7% difference. However, as the best models mainly rely on subtitles, there is a significant gap in the multi-modal setting, where human performance reaches 89.8%. This discrepancy can be explained by the vision-only setting, where the gap reaches 20.7%. These results highlight that while models perform reasonably well with textual data, they struggle to effectively integrate visual information, underscoring the need for advancements in multi-modal fusion techniques to bridge the gap between human and model performance.

Open-Ended Question answering (OEQ). This task proves even more challenging, with lower accuracy across all models, ranging from 3.5% to 40.3%. Subtitles-only models, particularly LLoVi, lead in performance, highlighting the dominance of language processing in understanding and answering complex questions. These findings emphasize the difficulty of the task and the need for improved methods to handle and integrate multi-modal data effectively.

Overall, our results suggest that (1) text (in the form of subtitles) is a stronger cue than visual information for movie question-answering; (2) modern multimodal methods (max accuracy at 60.0%, see Table 3) fall behind human evaluation; and more importantly (3) there is a large room for improvement in the visual aspect, where the gap in performance between modern methods (average accuracy 38.3%) and user study (59.0% accuracy) is still very high.

4 Related Work

VideoQA benchmarks. Questions in such benchmarks can be framed to assess the reasoning, memory, and comprehension skills of both humans and algorithms. Hence, many datasets have been made available through the years, covering visual descriptions [69, 8], temporal action reasoning [65], compositional reasoning [19], social intelligence [76], instructional videos [71], egocentric videos [39, 24, 34], movies [56, 29, 38], among others [23, 67, 23, 32, 75, 63]. However, these datasets feature short videos and typically require reasoning on only a few frames to solve the task at hand, as shown by temporal certificates in [39].

VideoQA methods. To address video question answering, most modern approaches are based on large-scale pre-trained models, by deploying either multimodal contrastive learning [15, 31, 61, 78, 77, 10, 9] or visually-conditioned large language models [30, 11, 68, 74, 2, 70, 73]. For instance, mPLUG-Owl2 [68] enhances LLMs with visual capabilities, while FrozenBiLM [73], a frozen bidirectional language model, uses lightweight adapters and masked language modelling to excel in zero-shot tasks. FrozenBiLM excels in zero-shot video question-answering without using explicit supervision. More recently, Video-LLaVA [33] used a simple projection layer to connect a visual encoder and a large language model, fine-tuned with visual instruction tuning data.

Long-form video understanding. As new long-context models emerge [46], we need more complex benchmarks. Long-form video understanding evaluates the long-term reasoning capabilities of models. Some datasets have been introduced in recent years, mostly for egocentric videos [39] and movies [64, 53, 21, 56]. Instead, LVU [64] involves long movie clips but the subsequent tasks are relatively limited, focusing on irrelevant aspects like ’like’ ratio and view count prediction. [39] proposes multiple-choice questions on 3-minute videos that require on average 100 seconds of video watching to be answered correctly.  [53, 21] introduce an audio description task, which is currently tackled at the clip level but could be extended over the whole movie. Finally, closer to our work,  [56] and  [54] propose a movie question-answering datasets focused on movie plots, settings and characters. However, they do not release any video data because of copyright issues.

5 Discussion

We introduced the Short Film Dataset (SFD), a long-form video understanding benchmark featuring complete and narrative-driven amateur short movies. It includes two types of QA tasks, multiple-choice and open-ended, that are automatically generated by LLMs and manually curated by human annotators. Compared to other long-form video datasets, SFD stands out by offering richer, story-oriented content with tasks specifically tailored for long-term reasoning, as validated by our experiments. Unlike most movie datasets that use copyrighted commercial films prone to data leakage with LLMs, SFD relies on amateur films, which are publicly accessible and have a limited online presence. Furthermore, our analysis indicates that while language-based methods achieve performance levels comparable to humans, state-of-the-art vision-based and multimodal methods fall behind human evaluation and they still have considerable room for improvement, highlighting the need for further advancements. In conclusion, we believe that our SFD paves the way for accessible, and comprehensive long-term movie understanding that is not known a priori by LLMs, thus helping the community develop robust methods. Our dataset and code will be made publicly available.

6 Acknowledgement

This work was supported by ANR-22-CE23-0007, Hi!Paris grant and fellowship, and was granted access to the High-Performance Computing (HPC) resources of IDRIS under the allocations 2023-AD011014489 made by GENCI. We would like to thank Nicolas Dufour, Robin Courant, Pierre Vassal, and the anonymous reviewers for their insightful comments and suggestions. We also express our gratitude to the annotators of the human study for their help.

Appendix

In this appendix, we show additional materials of the paper Short Film Dataset (SFD): A Benchmark for Story-Level Video Understanding. We present the following items:

1. Checklist

2. Ethical considerations

3. Data Preprocessing and Statistics

4. Technical details on baselines

5. Technical details on the temporal window study

6. Human Study

7. Technical details on data leakage experiment

8. Samples from SFD

9. Prompts

10. Datasheet for Datasets

Appendix A Checklist

1. For all authors…

   1. Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope? [Yes]

   2. Did you describe the limitations of your work? [Yes]

   3. Did you discuss any potential negative societal impacts of your work? [Yes] See Supplementary Materials Section B

   4. Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes] See Supplementary Materials Section B

2. If you are including theoretical results…

   1. Did you state the full set of assumptions of all theoretical results? [N/A]

   2. Did you include complete proofs of all theoretical results? [N/A]

3. If you ran experiments (e.g. for benchmarks)…

   1. Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)? [Yes]  https://huggingface.co/datasets/rghermi/sfd

   2. Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Sections D,  E, and  G.

   3. Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)? [No] Our experiments consist of benchmarking various aspects of our dataset. For a given compute, instead of performing one experiment several times (with different random seeds), we chose to experiment with several different multimodal methods and multiple LLMs. We observed that they all report similar performances and overall follow the same trend; hence, we argue that our findings are generalizable.

   4. Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Section 3.3

4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets…

   1. If your work uses existing assets, did you cite the creators? [Yes] See Section B.

   2. Did you mention the license of the assets? [Yes] See Section B.

   3. Did you include any new assets either in the supplemental material or as a URL? [N/A]

   4. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? [Yes] See Section B.

   5. Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]

5. If you used crowdsourcing or conducted research with human subjects…

   1. Did you include the full text of instructions given to participants and screenshots, if applicable? [Yes] See Section F.

   2. Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]

   3. Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]

Appendix B Ethical considerations

1. Copyright and License:

   1. We do not distribute raw video content; instead, we provide only URLs redirecting to YouTube, where the full copyright and licensing rights of creators are acknowledged. Additionally, by sharing the URLs, YouTube’s Terms of Service regarding exclusive rights are respected..

   2. The dataset will be used solely for academic research purposes. To further protect the metadata and other information, we apply the CC BY-NC-SA 4.0 (Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International) license to our dataset.

2. Potential negative social impact:

   1. Bias: Collected movies are primarily from North America and Europe, mainly English-speaking, the under-representation of the content towards non-English-speaking population and culture, potentially leading to unbalanced, biased video understanding model training and testing.

   2. Impact on creative process: Progress on movie understanding may encourage filmmakers to tailor their work to optimize for what is favoured by algorithms, potentially stifling creative freedom and depriving innovation.

Appendix C Data Preprocessing and Statistics

Table 4: Additional statistics. Statistics for the clip and face tracks in all movies of SFD.

(a) Clip statistics

	SFD
#Movies	1,078
#Clips	161,321
#Clips / Movie	151
Avg. Clip Len. (s)	5.43
#Captions	161,321
#Location Tags	161,321

(b) Face tracks

	SFD
#Face Instances	56,265
#Unique Faces	2,598
#Unique Faces / Movie	2.41
#Subtitles	23,820

SFD, alongside the videos and subtitles (see Section 2), also offers a comprehensive set of annotations: clips (shots), shot captions and scales, location tags, and face tracks. These resources, together with tools for downloading, preprocessing, and annotation, will be made available as part of the dataset release. Below, we detail their creation processes.

Clips (Shots). In the context of movies, a clip corresponds to a shot, characterized by its composition, length, and camera movement. We use PySceneDetect to segment movies in SFD into 161,321 shots, with each movie containing an average of 151 shots, as shown in Table 5(a). The number of shots varies by genre; for example, comedies average 163 shots, while dramas have about 146.

Clip information. For each shot, we extract detailed shot-level information. We generate shot captions using image and video captioning tools (LLaVA [36], BLIP-2 [30], Video-LLaVA [33]), and we identify locations using CLIP features [43] based on a curated list of locations from MovieCLIP [6].

Face Tracks. We use an advanced facial recognition pipeline  to identify over 56k+ face instances in the dataset, corresponding to 2.6k+ unique characters, with an average of 2.41 characters per movie, as shown in Table 5(b).

These annotations enrich the dataset, providing deeper insights for tasks related to character analysis and interaction within the films.

(a) Movies (Ours)

(b) Instructional Videos (iVQA)

(c) Egocentric Videos (EgoSchema)

Figure 8: Comparative word cloud analysis across video domains: Movies, Instructional, and Egocentric Videos.

In Figure 8, we perform a word cloud analysis for three video domains commonly used in the literature: instructional videos [71], egocentric videos [39], and movies (ours). We notice that instructional and egocentric videos are focused on actions (verbs like make, go, cut, use, pick, put) and objects (e.g. table, sugar, oil, thing, stuff, table, paper), with an emphasis on body parts for egocentric videos (right hand, left hand). For movies, the vocabulary is more focused on characters (e.g. man, woman, boy, girl, young, old), relationships (e.g. mother, father, family, couple), time (e.g. day, night, year), and locations (e.g. home). These clearly show the story component of SFD.

Appendix D Technical details on baselines

In this section, we present additional details and results for the baselines discussed in Section 3.3 and Table 3 main paper, where we report the benchmark performance of the state-of-the-art video understanding methods on our SFD.

FrozenBiLM [5]. We use FrozenBiLM pre-trained on WebVid10M[5]. Following[39], we reformulate the question-answering task for this model with the following prompt: ‘Question: {question} Is it {option_i}?’. We get logits for the yes/no tokens for each option and take the maximum as the final prediction. We report results with and without subtitles. We compute the accuracy score, by comparing the predicted option index and the correct option index. As it is a masked language model, FrozenBiLM is not suited to open-ended question answering.

mPLUG-Owl2 [74]. We use the instruction-tuned version (‘MAGAer13/mplug-owl2-llama2-7b’), based on ViT-L and LLaMa-7b. Inference is the same as FrozenBiLM. In the open-ended question-answering setting, we only provide the question and record a free-form text output: ‘Question: {question} Please answer with a short sentence.’. Following [37], we use GPT-3.5 (‘gpt-3.5-turbo’) to compute the similarity between the predicted and the correct answer. We report this LLM-based accuracy.

Video-LLaVA [33]. We use the fine-tuned version (‘LanguageBind/Video-LLaVA-7B-hf’), based on LanguageBind and Vicuna-13B v1.5. Inference is the same as mPLUG-Owl2, in both multiple-choice and open-ended question-answering settings.

LLoVi [79]. Following the official implementation, we use LLaVA (‘llava-hf/llava-1.5-13b-hf’) to extract captions at 0.5 FPS and GPT-3.5 (‘gpt-3.5-turbo’) as a backbone LLM. We experiment with the prompting strategy proposed in the original paper [79]. The prompt for multiple-choice question answering is as follows:

‘Please provide a single-letter answer (A, B, C, D, E) to the following multiple-choice question, and your answer must be one of the letters (A, B, C, D, or E). You must not provide any other response or explanation. You are given some language descriptions of a video. The video is {duration} seconds long. Each sentence describes a {clip_length}s clip. The descriptions are sequential and non-overlapping which cover the whole video exactly. Here are the descriptions: {captions}. You are going to answer a multiple-choice question based on the descriptions, and your answer should be a single letter chosen from the choices. Here is the question: {question}. Here are the choices. A: {option_0} B: {option_1} C: {option_2} D: {option_3} E: {option_4}")’

LangRepo [26]. Similar to LLoVi, LangRepo is another LLM-backed video understanding method. We use the open-sourced official implementation, Mixtral 8$\times$7B (‘mistralai/Mixtral-8x7B-Instruct-v0.1’). Unlike LLoVi, LangRepo operates in two stages: The first is summarization, where LangRepo consolidates similar information from descriptions, captions, subtitles, or both, using specific prompts following the official implementation:

‘You are a helpful expert in video analysis. You are given a list of {num_to_rephrase} language descriptions for a video. Each sentence describes a {clip_length}s clip. Here are the descriptions as a list:{memory}. Please summarize and rephrase each item in the list as a single sentence of {num_words_in_rephrase} words. Keep the same original subject. Keep all information intact without leaving anything out. Return only the rephrased list of {num_to_rephrase} descriptions in the same order, without additional details.’

In the second stage, the question-answering is based on the rephrased summarization with prompts:

‘Here are the descriptions: {narration}. You are going to answer a multiple-choice question based on the descriptions. Here is the question: {question}? You must select one of these choices as the answer: A: {option_0} B: {option_1} C: {option_2} D: {option_3} E: {option_4} The correct answer is:’

MovieChat [54]. We use the official implementation of MovieChat (‘rese1f/MovieChat’, which incorporates LLaMa2-7B (‘meta-llama/Llama-2-7b’ and corresponding Video-LLaMa-7B (‘DAMO-NLP-SG/Video-LLaMA-2-7B-Finetuned’). Similarly, we use GPT-3.5 (‘gpt-3.5-turbo’) [42] to compute the accuracy of responses. Following the official implementation, we use the following prompt:

‘By watching the video and combining it with the caption/subtitle from video , you are required to answer a question with multiple options: Question: {question}; Option A: {option_0}; Option B: {option_1}; Option C: {option_2}; Option D: {option_3}; Option E: {option_4}; Answer to the multiple-choice question based on the textual information extracted from the movie. If you don’t know the answer, please output ‘Unknown’. Your output should be just one of A, B, C, D, E and nothing else.’

TimeChat [47]. We use the official implementation of TimeChat (‘ShuhuaiRen/TimeChat-7b’). Following the official implementation, the prompt and the accuracy computation details are the same as those in the previous MovieChat model.

For all models, we also incorporate subtitles as input. For that, we concatenate each query with the following text: ‘Subtitles: {subtitles}’. Subtitles are formatted as follows: ‘{start_timestamp} - {end_timestamp} {subtitle_i}’. We trim subtitles if they cannot fit into the input size. Results and analysis can be seen in Section 3.3 and Table 3.

Appendix E Technical details on the temporal window study

In this section, we present additional details and results for the temporal window study (see Section 3.2 and Figure 7). The goal of the temporal window study is to evaluate the performance of various methods on SFD at different temporal levels—shot, scene, and movie—across all modalities. Along with the discussed FrozenBiLM and LLoVi, we examine Video-LLaVA and LangRepo.

At the shot-level, we truncate the input prompt to include only information from a specific shot. For FrozenBiLM and Video-LLaVA, it means that we only include subtitles and frames extracted from the shot. For LLoVi and LangRepo, we only include subtitles and captions extracted from the shot. Hence, we get logits for each answer option and shot, aggregated in a matrix of shape $n_{shots}\times n_{options}$: we obtain a final prediction by taking the maximum logit over both dimensions. Note that the shot-level inference takes too much time to compute for LangRepo ($>2,000$ A100 GPU-hours per experiment), as this method involves many steps of rephrasing and summarizing.

The scene-level inference is similar to the shot-level one, the main difference being that we constrain the input information (subtitles and frames/captions) to the scene-level, which arbitrarily corresponds to 10 shots in our case. We end up with a logit matrix of size $n_{scenes}\times n_{options}$.

The movie-level inference is straightforward: it is the same as in Section D.

Table 6: Temporal window study.

		Vision	Language	Vision-Language
FrozenBiLM	Shot-Level	22.1	30.9	30.8
	Scene-Level	22.7	34.8	34.8
	Movie-Level	23.4	38.2	38.6
Video-LLaVA	Shot-Level	16.0	24.4	24.0
	Scene-Level	23.7	20.1	20.1
	Movie-Level	34.2	21.3	24.7
LLoVi	Shot-Level	32.8	38.5	50.1
	Scene-Level	34.2	51.2	55.4
	Movie-Level	30.8	64.2	55.6
LangRepo	Shot-Level	-	-	-
	Scene-Level	26.9	27.9	28.1
	Movie-Level	29.0	32.1	31.0
Human	Movie-Level	59.0	70.9	89.8

The numerical results of the temporal window study are reported in Table 6.

Along the temporal direction (i.e. row-wise for each method), we observe that: (i) generally, all methods exhibit consistent gains with larger temporal windows confirming the importance of long-term understanding for our designed questions, and (ii) exceptions are noted, such as with Video-LLaVa in Language and Vision-Language settings, likely due to its low precision (near random chance at 20%) which diminishes the impact of the temporal window.

For different modalities (i.e. column-wise), we make the following observations: (i) Corroborating the argument in the main paper (Section 3.3), the Language-only models, in general, achieve better performances, e.g. LLoVi reaches the best performance at movie-level in the language-only setting (64.2%). (ii) The vision-only setting results in the weakest performance, revealing the largest gap compared to human performance (last row). This suggests significant potential for improvement in visual models. (iii) Current modality-fusion methods show limited effectiveness. VLM models like FrozenBiLM and Video-LLaVA only demonstrate marginal improvements when fusing vision and language, with performance increment from 38.2% to 38.6% for FrozenBiLM and from 21.3% to 24.7% for Video-LLaVa. However, all LLM-based methods experience a regression after modality fusion.

Overall, the human benchmark highlights that there remains a substantial gap between the best methods and human performance, particularly in vision-language integration over extended temporal windows.

Appendix F Human Study

In this section, we present additional details for the conducted human study (see Section 3.3, ‘User Study’). We conducted a comprehensive human study to evaluate the upper-limit performance of our benchmark. Participants were asked to watch a movie and subsequently answer multiple-choice questions based on the content. To analyze the impact of different modalities, we implemented two ablation conditions: a muted version, where participants had access to the vision-only information without audio or subtitles, and a blind version, where participants were provided with the audio-only information, in the form of textual subtitles, without any video.

To avoid any bias, we ensured that different sets of participants were assigned to the full, vision-only, and audio-only settings, preventing any overlap and potential influence from experiencing multiple conditions. Movies were randomly assigned to users to ensure a diverse range of data. The entire study was facilitated through an integrated interface, which restricted access to external information such as movie loglines and synopses, with only the movie title being displayed (see Figure 9).

Figure 9: Human study interface.

Appendix G Technical details on data leakage experiment

Table 7: Data leakage results.

Name	SFD	MovieQA	LVU
Open-source models
Gemma 2B	19.7	18.3	28.8
Gemma 7B	20.7	21.5	52.4
LLaMA 3 8B	22.1	34.5	69.9
LLaMA 3 70B	24.1	51.9	64.1
Mistral 7B	28.9	44.1	61.4
Mixtral 8x7B	33.5	55.4	70.2
Commercial models
Claude 3 Haiku	28.9	55.4	68.5
Claude 3 Sonnet	36.0	64.4	71.0
GPT-3.5	26.3	56.7	75.0
GPT-4	31.5	71.3	76.0

Figure 10: Impact of model knowledge on data leakage. Higher MMLU performance results in LLMs memorizing more information about commercial movies.

In this section, we include additional information on the data leakage study mentioned in the main paper Section 3.1. The objective of our data leakage study is to demonstrate that current large language models (LLMs) possess prior knowledge about commercial movies. This is because LLMs have been exposed to vast amounts of movie-related textual information during pretraining, including synopses, blog posts, news articles, and more. The experiment is set up as an open-ended question-answering task where the input consists of a movie title and a corresponding question, and the output is a free-form text generated by the LLM. We then evaluate the answer using an LLM to compute the similarity between the predicted and the correct answer, following [37].

We choose ten of the most recent LLMs, among which six are open-source models: Gemma 2B, Gemma 7B, LLaMA-3 8B, LLaMA-3 70B, Mistral 7B, Mixtral 8x7B; and four commercial models: Claude 3 Haiku, Claude 3 Sonnet, GPT-3.5, GPT-4. The open-source models are run using the Hugging Face transformers library . The commercial models are run using their official API: the OpenAI API  for GPT and the Anthropic API  for Claude 3.

We choose three movie datasets: MovieQA [56], LVU [64], and SFD (ours). For MovieQA and SFD, the question-answer pairs are directly available. In the case of LVU, we re-formulated the classification tasks into templated questions to fit the open-ended question-answering format. For example, the director classification task is converted into the question ‘Who directed the movie {movie_title}?’.

The detailed results of the data leakage experiment are presented in Table 7 and Figure 10. Their analysis shows that LLMs exhibit significantly higher data leakage on MovieQA and LVU datasets compared to SFD. Models like GPT-4 and Claude 3 Sonnet answer correctly to more than 70% of the questions of both datasets, without any context except the movie title. The scatter plot emphasizes a correlation between data leakage and model knowledge (i.e. MMLU performance), with higher MMLU scores correlating with increased leakage issues on compromised datasets. This trend is consistent across both open-source and commercial models, highlighting the vulnerability of larger and more advanced models to memorizing training data.

In sharp contrast, the relatively low and flat performance trend on SFD suggests it is a more robust benchmark for evaluating model capabilities. These findings emphasize the need for robust evaluation datasets to accurately assess model performance and mitigate the impact of data leakage.

Appendix H Samples from SFD

In the following pages, we show some samples from SFD. For each sample, we display the title, the logline, the synopsis, a few frames, some subtitles, and the designed multiple-choice questions. See Figures 11, 12,13, 14.

Figure 11: Sample From SFD: #1

Figure 12: Sample From SFD: #2

Figure 13: Sample From SFD: #3

Figure 14: Sample From SFD: #4

Appendix I Prompts

We provide the prompts for the data generation process: question generation prompt (see Figure 8), distractor generation prompt (see Figure 9), and question classification prompt (see Figure 10). As detailed in Section 2.2 , those prompts were used with GPT-4 (’gpt-4-turbo-2024-04-09’) through the OpenAI API.

Table 8: Question generation prompt.

Table 9: Distractor generation prompt.

Table 10: Question classification prompt.

Appendix J Datasheet for Datasets

Motivation

For what purpose was the dataset created? Was there a specific task in mind? Was there a specific gap that needed to be filled? Please provide a description.

This dataset and benchmark are created to assess long-form video understanding capabilities of modern multimodal systems. It is different from other video question-answering datasets as it features longer videos and associated story-related questions. Additionally, it stands out as a dataset that is available online and not exposed to current large language models (limited data leakage issues).

Who created this dataset (e.g., which team, research group) and on behalf of which entity (e.g., company, institution, organization)?

This dataset was created by Ridouane Ghermi, Xi Wang, Vicky Kalogeiton, and Ivan Laptev.

Who funded the creation of the dataset? If there is an associated grant, please provide the name of the grantor and the grant name and number.

This project is funded by V.Kalogeiton ANR-22-CE23-0007 and a Hi!Paris project.

Any other comments?

Composition

What do the instances that comprise the dataset represent (e.g., documents, photos, people, countries)? Are there multiple types of instances (e.g., movies, users, and ratings; people and interactions between them; nodes and edges)? Please provide a description.

Our dataset instance consists of a short movie (around 5-20 mins), some metadata (e.g. movie title, logline, synopsis, country of origin, language, release year), and a list of multiple-choice questions and answers.

How many instances are there in total (of each type, if appropriate)?

There are 1,078 movies and 4,885 questions. Each question comprises one correct answer and four wrong answers.

Does the dataset contain all possible instances or is it a sample (not necessarily random) of instances from a larger set? If the dataset is a sample, then what is the larger set? Is the sample representative of the larger set (e.g., geographic coverage)? If so, please describe how this representativeness was validated/verified. If it is not representative of the larger set, please describe why not (e.g., to cover a more diverse range of instances, because instances were withheld or unavailable).

The dataset is not a sample of instances from a larger set.

What data does each instance consist of? “Raw” data (e.g., unprocessed text or images) or features? In either case, please provide a description.

Each instance includes a movie as a video file. Metadata comes as .csv files, in the form of text. We also provide processed features.

Is there a label or target associated with each instance? If so, please provide a description.

Each movie comes along with several multiple-choice questions, each with five possible options. The correct option is designated by an index (from 0 to 4).

Is any information missing from individual instances? If so, please provide a description, explaining why this information is missing (e.g. because it was unavailable). This does not include intentionally removed information, but might include, e.g., redacted text.

No.

Are relationships between individual instances made explicit (e.g., users’ movie ratings, social network links)? If so, please describe how these relationships are made explicit.

We provide a dataframe that links question ids and video ids.

Are there recommended data splits (e.g., training, development/validation, testing)? If so, please provide a description of these splits, explaining the rationale behind them.

SFD is a test set only.

Are there any errors, sources of noise, or redundancies in the dataset? If so, please provide a description.

No, the dataset has been manually curated.

Is the dataset self-contained, or does it link to or otherwise rely on external resources (e.g., websites, tweets, other datasets)? If it links to or relies on external resources, a) are there guarantees that they will exist, and remain constant, over time; b) are there official archival versions of the complete dataset (i.e., including the external resources as they existed at the time the dataset was created); c) are there any restrictions (e.g., licenses, fees) associated with any of the external resources that might apply to a future user? Please provide descriptions of all external resources and any restrictions associated with them, as well as links or other access points, as appropriate.

All movies are available on YouTube; We only provide the corresponding URLs. It should remain constant over time as it comes from YouTube channels that curate a movie catalogue. The dataset is distributed under the CC BY-NC-SA 4.0 (Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International) license.

Does the dataset contain data that might be considered confidential (e.g., data that is protected by legal privilege or by doctor-patient confidentiality, data that includes the content of individuals non-public communications)? If so, please provide a description.

No.

Does the dataset contain data that, if viewed directly, might be offensive, insulting, threatening, or might otherwise cause anxiety? If so, please describe why.

No.

Does the dataset relate to people? If not, you may skip the remaining questions in this section.

Yes, the dataset contains movies with actors.

Does the dataset identify any subpopulations (e.g., by age, gender)? If so, please describe how these subpopulations are identified and provide a description of their respective distributions within the dataset.

No.

Is it possible to identify individuals (i.e., one or more natural persons), either directly or indirectly (i.e., in combination with other data) from the dataset? If so, please describe how.

Yes, it is possible to identify actors.

Does the dataset contain data that might be considered sensitive in any way (e.g., data that reveals racial or ethnic origins, sexual orientations, religious beliefs, political opinions or union memberships, or locations; financial or health data; biometric or genetic data; forms of government identification, such as social security numbers; criminal history)? If so, please provide a description.

No, everything is fictional.

Any other comments?

Collection Process

How was the data associated with each instance acquired? Was the data directly observable (e.g., raw text, movie ratings), reported by subjects (e.g., survey responses), or indirectly inferred/derived from other data (e.g., part-of-speech tags, model-based guesses for age or language)? If data was reported by subjects or indirectly inferred/derived from other data, was the data validated/verified? If so, please describe how.

Movies are available online and downloaded from YouTube using yt-dlp, along with metadata. Multiple-choice questions are generated by a commercial LLM, GPT-4, and further manually curated.

What mechanisms or procedures were used to collect the data (e.g., hardware apparatus or sensor, manual human curation, software program, software API)? How were these mechanisms or procedures validated?

Video URLs are scraped using the official YouTube API. Videos and metadata are downloaded using yt-dlp. Questions are generated using GPT-4 through the official OpenAI API and further refined by manual curation.

If the dataset is a sample from a larger set, what was the sampling strategy (e.g., deterministic, probabilistic with specific sampling probabilities)?

Not applicable.

Who was involved in the data collection process (e.g., students, crowdworkers, contractors) and how were they compensated (e.g., how much were crowdworkers paid)?

Students voluntarily helped to curate the dataset.

Over what timeframe was the data collected? Does this timeframe match the creation timeframe of the data associated with the instances (e.g., recent crawl of old news articles)? If not, please describe the timeframe in which the data associated with the instances was created.

All data are collected and curated during the first half of 2024. Movies were uploaded to the video-sharing platform between 2017 and 2024.

Were any ethical review processes conducted (e.g., by an institutional review board)? If so, please provide a description of these review processes, including the outcomes, as well as a link or other access point to any supporting documentation.

No.

Does the dataset relate to people? If not, you may skip the remaining questions in this section.

No.

Did you collect the data from the individuals in question directly, or obtain it via third parties or other sources (e.g., websites)?

Not applicable.

Were the individuals in question notified about the data collection? If so, please describe (or show with screenshots or other information) how notice was provided, and provide a link or other access point to, or otherwise reproduce, the exact language of the notification itself.

Not applicable.

Did the individuals in question consent to the collection and use of their data? If so, please describe (or show with screenshots or other information) how consent was requested and provided, and provide a link or other access point to, or otherwise reproduce, the exact language to which the individuals consented.

Not applicable.

If consent was obtained, were the consenting individuals provided with a mechanism to revoke their consent in the future or for certain uses? If so, please provide a description, as well as a link or other access point to the mechanism (if appropriate).

Not applicable.

Has an analysis of the potential impact of the dataset and its use on data subjects (e.g., a data protection impact analysis) been conducted? If so, please provide a description of this analysis, including the outcomes, as well as a link or other access point to any supporting documentation.

Not applicable.

Any other comments?

–

Preprocessing/cleaning/labeling

Was any preprocessing/cleaning/labeling of the data done (e.g., discretization or bucketing, tokenization, part-of-speech tagging, SIFT feature extraction, removal of instances, processing of missing values)? If so, please provide a description. If not, you may skip the remainder of the questions in this section.

We extracted several annotations from the dataset, see Sections C.

Was the “raw” data saved in addition to the preprocessed/cleaned/labeled data (e.g., to support unanticipated future uses)? If so, please provide a link or other access point to the “raw” data.

Yes.

Is the software used to preprocess/clean/label the instances available? If so, please provide a link or other access point.

Yes. For annotation extraction, we will provide the tools and scripts in the GitHub repository. For question generation, we will provide the prompts but the usage requires an access to the OpenAI API (which is not free).

Any other comments?

–

Uses

Has the dataset been used for any tasks already? If so, please provide a description.

The dataset is accompanied by two tasks: multiple-choice and open-ended question answering.

Is there a repository that links to any or all papers or systems that use the dataset? If so, please provide a link or other access point.

Yes, we will make this available on the dataset GitHub repository.

What (other) tasks could the dataset be used for?

We strongly believe that, with appropriate annotations, this dataset can be extended for many challenging use cases, e.g. spatio-temporal localization, video grounding, movie summarization, causal reasoning.

Is there anything about the composition of the dataset or the way it was collected and preprocessed/cleaned/labeled that might impact future uses? For example, is there anything that a future user might need to know to avoid uses that could result in unfair treatment of individuals or groups (e.g., stereotyping, quality of service issues) or other undesirable harms (e.g., financial harms, legal risks) If so, please provide a description. Is there anything a future user could do to mitigate these undesirable harms?

No.

Are there tasks for which the dataset should not be used? If so, please provide a description.

No.

Any other comments? –

Distribution

Will the dataset be distributed to third parties outside of the entity (e.g., company, institution, organization) on behalf of which the dataset was created? If so, please provide a description.

Yes, but we do not distribute raw video content; instead, we provide only URLs redirecting to YouTube, where the full copyright and licensing rights of creators are acknowledged and YouTube’s Terms of Service regarding exclusive rights are respected. To further protect the metadata and other information, we apply the CC BY-NC-SA 4.0 (Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International) license to our dataset.

How will the dataset be distributed (e.g., tarball on website, API, GitHub) Does the dataset have a digital object identifier (DOI)?

The dataset is distributed on the HuggingFace platform and as a GitHub repository. We provide video URLs and annotations. Additionally, we provide code to download videos, pre-process the data and run baselines.

When will the dataset be distributed?

The dataset is already available.

Will the dataset be distributed under a copyright or other intellectual property (IP) license, and/or under applicable terms of use (ToU)? If so, please describe this license and/or ToU, and provide a link or other access point to, or otherwise reproduce, any relevant licensing terms or ToU, as well as any fees associated with these restrictions.

The dataset is distributed under the CC BY-NC-SA 4.0 (Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International) license. The data was collected from publicly available sources. All rights and credits for the original movie content go to the respective owners. This dataset and any derivatives are intended for non-commercial use only and must be shared under the same license.

Have any third parties imposed IP-based or other restrictions on the data associated with the instances? If so, please describe these restrictions, and provide a link or other access point to, or otherwise reproduce, any relevant licensing terms, as well as any fees associated with these restrictions.

Not applicable.

Do any export controls or other regulatory restrictions apply to the dataset or to individual instances? If so, please describe these restrictions, and provide a link or other access point to, or otherwise reproduce, any supporting documentation.

Not applicable.

Any other comments?

–

Maintenance

Who will be supporting/hosting/maintaining the dataset?

The dataset is hosted on both GitHub and HuggingFace. It will be maintained by the authors.

How can the owner/curator/manager of the dataset be contacted (e.g., email address)?

We provide a contact information email, and will also answer any pull requests in the repository.

Is there an erratum? If so, please provide a link or other access point.

The erratum is available on the GitHub repository.

Will the dataset be updated (e.g., to correct labeling errors, add new instances, delete instances)? If so, please describe how often, by whom, and how updates will be communicated to users (e.g., mailing list, GitHub)?

The dataset is a test benchmark, it will remain stable to ensure fair evaluation of future methods.

If the dataset relates to people, are there applicable limits on the retention of the data associated with the instances (e.g., were individuals in question told that their data would be retained for a fixed period of time and then deleted)? If so, please describe these limits and explain how they will be enforced.

Not applicable.

Will older versions of the dataset continue to be supported/hosted/maintained? If so, please describe how. If not, please describe how its obsolescence will be communicated to users.

Not applicable.

If others want to extend/augment/build on/contribute to the dataset, is there a mechanism for them to do so? If so, please provide a description. Will these contributions be validated/verified? If so, please describe how. If not, why not? Is there a process for communicating/distributing these contributions to other users? If so, please provide a description.

If others want to contribute to the dataset, by adding more tasks for example, we would gladly host them in our repository. Also, we will provide as many details and tools as possible to reproduce our annotation process and create more tasks.

Any other comments?

N/A.