DiffIR2VR-Zero: Zero-Shot Video Restoration with Diffusion-based Image Restoration Models

Abstract

This paper introduces a method for zero-shot video restoration using pre-trained image restoration diffusion models. Traditional video restoration methods often need retraining for different settings and struggle with limited generalization across various degradation types and datasets. Our approach uses a hierarchical token merging strategy for keyframes and local frames, combined with a hybrid correspondence mechanism that blends optical flow and feature-based nearest neighbor matching (latent merging). We show that our method not only achieves top performance in zero-shot video restoration but also significantly surpasses trained models in generalization across diverse datasets and extreme degradations (8$\times$ super-resolution and high-standard deviation video denoising). We present evidence through quantitative metrics and visual comparisons on various challenging datasets. Additionally, our technique works with any 2D restoration diffusion model, offering a versatile and powerful tool for video enhancement tasks without extensive retraining. This research leads to more efficient and widely applicable video restoration technologies, supporting advancements in fields that require high-quality video output. See our project page for video results: jimmycv07.github.io/DiffIR2VR_web.

1 Introduction

Video restoration is a valuable topic that transforms low-quality video into high-quality video. It usually involves video denoising, super-resolution, and deblurring. The state-of-art methods that employ convolutional neural networks (CNNs) [2, 32, 57] or transformers [19, 49, 78] trained on large-scale data achieve incredible effectiveness. However, the regression-based methods often result in blurry outputs without realistic details (Fig. 2(a)). Furthermore, the degradations they address are typically well-defined (e.g., bicubic downsampling, given noise standard deviation), and models are often tailored to specific degradations only. This limitation restricts their generalization capabilities, as different settings often require additional paired data and retraining the model.

Diffusion models recently are adapted to image restorations [85, 45] Because of their powerful generative ability, they can hallucinate realistic details. But this ability inherently comes with high randomness. As a result, directly performing per-frame inference to process videos leads to severe flickering (Fig. 2(b)). This phenomenon is even more pronounced in Latent Diffusion Models (LDM) because the decoder will magnify the randomness.

A potential solution to reduce the temporal flickering is to fine-tune or train a single-image diffusion model by inserting 3D convolution and temporal attention layers into it. However, the modified model needs to be trained on videos to impose temporal consistency, which often requires unfordable computational resources (e.g., 32 A100-80G GPUs for video upscaling  [102]). Moreover, different tasks necessitate retraining a model.

Given limited computational resources, we present a novel zero-shot video restoration framework that transforms low-quality input videos into temporally consistent high-quality outputs. We design two training-free modules— hierarchical latent warping, hybrid flow-guided spatial-aware token merging—to enforce temporal consistency in both latent and token (feature from the attention layer) spaces. Our method can be applied to any pre-trained image diffusion model without additional training or fine-tuning. Extensive experiments demonstrate that our video restoration method outperforms state-of-the-art approaches in both video quality and temporal consistency, even under extreme degradation conditions. To summarize, our main contributions are as follows:

• We propose a novel, zero-shot video restoration method that achieves realistic results and maintains temporal consistency, compatible with any image-based diffusion models.

• Our training-free framework manipulates both latent and token spaces to enforce semantic consistency across frames, introducing hierarchical latent warping to maintain consistency within and between batches and improving token merging with flow correspondence and spatial information.

• Our method demonstrates state-of-the-art restoration results, especially in scenarios of extreme degradation and large motion, which handle various levels of degradation with a single model, offering greater generalizability and robustness compared to traditional regression-based methods.

Figure 1: Zero-shot temporal-consistent diffusion model for video restoration. Given a pre-trained diffusion model for single-image restoration, our method generates temporally consistent restored video with fine details without any further training.

Figure 2: 4$\times$ video super-resolution results. (a) Traditional regression-based methods such as FMA-Net [96] are limited to the training data domain and tend to produce blurry results when encountering out-of-domain inputs. (b) Although applying image-based diffusion models such as DiffBIR [45] to individual frames can generate realistic details, these details often lack consistency across frames. (c) Our method leverages an image diffusion model to restore videos, achieving both realistic and consistent results without any additional training.

2 Related Work

Video Restoration.

Video restoration aims to restore high-quality frames from degraded videos, addressing issues such as noise, blur, and low resolution [7, 9, 31, 39, 96, 101, 47, 48]. This task is more challenging than image restoration because it requires maintaining temporal consistency across frames. Learning-based approaches often employ architectures like optical flow warping [30, 58, 67, 68, 88], deformable convolutions [7, 8, 17, 77, 80, 81, 103], and attention mechanisms to handle temporal dependencies [5, 40, 43, 44, 98]. One major limitation is their dependency on paired high-quality (HQ) and low-quality (LQ) data for training [10, 86, 93], which is even more difficult to obtain for videos than for images. Moreover, most existing approaches [37, 36, 38, 40, 44] assume predefined degradation processes, reducing their effectiveness in real-world applications where degradations are unknown and diverse, thus leading to poor generalization performance. Additionally, these models often need retraining for different degradation levels or types [43, 46, 55, 95, 96], highlighting their limited generalization capabilities. Last but not least, these methods tend to lose significant detail, similar to image restoration [11, 42, 82, 99]

Diffusion Models for Image Restoration.

With significant advancements in diffusion models [13, 18, 25, 26, 63], many diffusion-based approaches have been proposed for image restoration [21, 26, 56, 70, 73, 79, 92]. One straightforward method involves training a diffusion model from scratch [63, 66, 85, 97], conditioned on low-quality images [66, 1]. However, this approach requires substantial computational resources. To reduce these costs, a common strategy is to introduce constraints into the reverse diffusion process of pre-trained models, as demonstrated by DDRM [34]. While efficient, these methods [13, 15, 21, 34, 72, 83, 89] depend on predefined image degradation processes or pretrained super-resolution (SR) models, which limits their generalizability. Recent works have enhanced performance by fine-tuning frozen pre-trained diffusion models with additional trainable layers [79, 92, 100], as seen in StableSR [79] and DiffBIR [45]. Despite their effectiveness, these methods encounter challenges in video restoration, where the inherent randomness of the diffusion process can cause temporal inconsistencies across frames. Our method allows these methods to work on video without any training.

Diffusion Models for Video Task.

Building on the success of text-to-image diffusion models [3, 13, 22, 24, 25, 53, 62, 65, 100], recent research explores using diffusion models for video tasks [20, 27, 28, 29, 50, 51, 52], extending pre-trained image diffusion models to video processing. Methods [12, 69], like Upscale-A-Video [102] and MGLD-VSR [94], achieve this by integrating and fine-tuning temporal layers. Upscale-A-Video adds temporal layers to UNet and VAE-Decoder, ensuring sequence consistency. Similarly, MGLD-VSR uses motion dynamics from low-resolution videos and calculated optical flows to align latent features. These methods require paired video data and substantial computational resources. Alternatively, zero-shot methods use existing image diffusion models to generate videos without training [6, 14, 16, 23, 61, 84, 91, 93, 100], employing techniques like token merging [4], noise shuffling, and latent warping. VidToMe [41] and TokenFlow [23] enhance temporal consistency by merging and aligning attention tokens across frames; Rerender-A-Video [90] employs latent warping [76, 87] and frame interpolation; RAVE [33] uses noise shuffling to maintain frame consistency in longer videos with reduced time complexity. These techniques are capable of generating impressive video sequences with minimal effort. However, they often produce blurry results and struggle with semantic consistency in demanding video restoration tasks. Inspired by these methods, our training-free framework manipulates latent and token spaces to ensure semantic consistency across frames, introducing hierarchical latent warping and improving token merging with flow correspondence and spatial information.

Figure 3: Pipeline of our proposed zero-shot video restoration method. We process low-quality (LQ) videos in batches using a diffusion model, with a keyframe randomly sampled within each batch. (a) At the beginning of the diffusion denoising process, hierarchical latent warping provides rough shape guidance both globally, through latent warping between keyframes, and locally, by propagating these latents within the batch. (b) Throughout most of the denoising process, tokens are merged before the self-attention layer. For the downsample blocks, optical flow is used to find the correspondence between tokens, and for the upsample blocks, cosine similarity is utilized. This hybrid flow-guided, spatial-aware token merging accurately identifies correspondences between tokens by leveraging both flow and spatial information, thereby enhancing overall consistency at the token level.

3 Method

Given a low-quality video with $n$ frames $\left\{lq_{1},lq_{2},\ldots,lq_{n}\right\}$, our goal is to restore it into a high-quality video $\left\{hq_{1},hq_{2},\ldots,hq_{n}\right\}$ using off-the-shelf image-based diffusion models. However, as illustrated in Fig. 2 and Fig. 7, directly applying these models to each frame individually results in temporal inconsistency due to the inherent stochasticity of the diffusion models, particularly in cases of extreme degradation. Our method, as depicted in the Fig. 3, addresses this by enforcing temporal stability in both latent and token space during restoration through two main components: Hierarchical Latent Warping (Sec. 3.2) and Hybrid Flow-guided Spatial-aware Token Merging (Sec. 3.3). In this section, we first briefly introduce the background of diffusion models and video token merging in Sec. 3.1. We then introduce the hierarchical latent warping strategy in Sec. 3.2, hybrid flow-guided spatial-aware token merging in Sec. 3.3, and scheduling of them in Sec. 3.4.

Figure 4: An illustration of our key modules. Without requiring any training, these modules can achieve coherence across frames by enforcing temporal stability in both latent and token space. Hierarchical latent warping provides global and local shape guidance; Hybrid spatial-aware token merging before the self-attention layer improves temporal consistency by matching similar tokens using optical flow in the down blocks and cosine similarity in the up blocks of the UNet.

Figure 5: Token correspondences. Correspondences found by cosine similarity and by optical flow. (Top) At the beginning of the denoising process, the latents in the UNet downblocks are too noisy for cosine similarity to be effective, while optical flow estimated from LQ frames remains reliable. (Bottom) Flow and cosine similarity often identify different correspondences, so a hybrid approach is more effective.

3.1 Preliminaries

Diffusion Models.

Diffusion models are a type of generative model that models a data distribution $p_{\text{data}}$ through gradual diffusing and denoising. The forward process diffuses a clean image $x_{0}$ by Gaussian noises in $T$ steps, given by

$${x_{t}=\sqrt{\alpha_{t}}x_{t-1}+\sqrt{1-\alpha_{t}}\epsilon_{t-1}}\Rightarrow x_{t}=\sqrt{\bar{\alpha}_{t}}x_{0}+\sqrt{1-\bar{\alpha}_{t}}\epsilon,$$ (1)

where $t\sim[1,T]$, $\epsilon_{t},\epsilon\sim\mathcal{N}(\mathbf{0},\mathbf{I})$, and $\bar{\alpha}_{t}=\prod_{s=1}^{t}\alpha_{s}$. The latent variable $x_{T}$ will be nearly a standard Gaussian distribution when $T$ is large enough. A denoiser $\epsilon_{\theta}$, usually implemented with UNet [64], is trained to estimate the noise $\epsilon_{t}$ by minimizing $\mathbb{E}_{t\sim\left[1,T\right],x_{0},\epsilon_{t}}\left[||\epsilon_{t}-\epsilon_{\theta}(x_{t},t)||^{2}\right]$. During inference, the inverse process starts from an i.i.d. noise $x_{t}$, and produce a clean image $x_{0}\sim p_{\text{data}}$ by gradual denoising with the well-trained denoiser over $\bar{T}$ steps [26, 71, 74], where $\bar{T}\in[1,T]$. These unconditional generative models can be further enhanced with additional guidance, such as text prompts and images for guided generations. One will inject the control signal $c$ into the denoiser $\epsilon_{\theta}(x_{t},t,c)$ through training [100] or optimization [34].

Video Token Merging.

Video Token Merging (VidToMe) [41] enhances the temporal consistency of generated videos by utilizing image diffusion models to merge similar tokens within frame chunks in the attention blocks. This token merging not only improves temporal coherence but also reduces the computational overhead of the attention mechanism by decreasing the size of token chunks.

Given a token chunk $\textbf{T}\in\mathbb{R}^{B\times A\times C}$, where $A=w*h$, the algorithm first separates the tokens into source tokens $\textbf{T}_{\text{src}}\in\mathbb{R}^{B\times A-1\times C}$ and a target token $\textbf{T}_{\text{tar}}\in\mathbb{R}^{B\times 1\times C}$. It then calculates the cosine between each source and target token, determining their corresponding similarity levels, denoted $score\in\mathbb{R}^{((B-1)*A)\times A}$. The algorithm then identifies the most similar target token for each source token by taking the maximum value in the last column.

$$s(\textbf{T}_{\text{src}},\textbf{T}_{\text{tar}})=\frac{\textbf{T}_{\text{src}}\cdot\textbf{T}_{\text{tar}}}{\left\|\textbf{T}_{\text{src}}\right\|\left\|\textbf{T}_{\text{tar}}\right\|}\,,\,\,\,c=\max_{\{\textbf{t}\in\textbf{T}_{\text{tar}}\}}(s(\textbf{T}_{\text{src}},\textbf{t})),$$ (2)

where $s(\cdot,\cdot)$ is the cosine similarity score and $c$ indicates the correspondences. Next, the $r$ most similar paired source-target tokens are merged, and the remaining tokens are concatenated as the output. Merged tokens are subsequently unmerged after self-attention to preserve the original shape by simply assigning the merged source-target tokens the exact same value. The token merging and unmerging are defined as follows:

$$\textbf{T}_{\text{merge}}=\mathcal{M}(\textbf{T}_{\text{src}},\textbf{T}_{\text{tar}},\;c,\;r)\,,\,\,\,\textbf{T}_{\text{unmerge}}=\mathcal{U}(\textbf{T}_{\text{merge}},\;c)\,,$$ (3)

where $\mathcal{M}$ and $\mathcal{U}$ denote the merging and unmerging operations, respectively.

3.2 Hierarchical Latent Warping

We introduce a hierarchical latent warping module that operates in the latent space. As illustrated in Fig. 5, this module provides rough shape guidance on both global and local scales by hierarchically propagating latents within keyframes and further from keyframes to their respective batches. Let $\hat{x}^{i}_{t\rightarrow 0}$ be the predicted $\hat{x}_{0}$ latent for the $i^{th}$ keyframe at denoising step $t$. We warp the latents between keyframes as follows:

$$\hat{x}^{i}_{t\rightarrow 0}\leftarrow M_{ji}\cdot\hat{x}^{i}_{t\rightarrow 0}+\left(1-M_{ji}\right)\cdot\mathcal{W}(\hat{x}^{j}_{t\rightarrow 0},f_{ji})$$ (4)

where $j=i-1$ and $f_{ji}$, $M_{ji}$ denotes the optical flow and the occlusion mask from $lq_{j}$ to $lq_{i}$ estimated by GMFlow [87]. After keyframe latent warping, these latents are further warped to the remaining frames following the same procedure. The flows and masks are downsampled to match the latent size, and these operations are omitted for simplicity. This module primarily functions in the early stages of the denoising process, ensuring that corresponding points between frames share similar latents both globally and locally from the beginning.

3.3 Hybrid Flow-guided Spatial-aware Token Merging

While latent manipulation can achieve a certain degree of consistency, manipulating latents during the later stages of the denoising process would result in blurry outcomes. Additionally, the token space is highly semantically related to the image. Therefore, we propose hybrid flow-guided spatial-aware token merging to achieve consistency in the token space.

Flow-guided.

Even with low-quality video, we can identify correspondences between frames based on color, indicating that flow calculated from $lq$ can still provide useful guidance. As shown in the top of Fig. 5, in the early stages of the denoising process, when the latents are still very noisy, cosine similarity struggles to find the correct correspondences, especially in the downsample blocks of the UNet. Therefore, we use flow for correspondences at the downsample blocks in the UNet and employ the confidence from forward-backward consistency check as a criterion to determine $r$ most similar paired source token $\textbf{T}_{\text{src}}$ and target token $\textbf{T}_{\text{tar}}$:

$$\sigma=\exp(-\left\|f_{\text{src}\rightarrow\text{tar}}(X(\textbf{T}_{\text{src}}))+f_{\text{tar}\rightarrow\text{src}}\left(X(\textbf{T}_{\text{src}})+f_{\text{src}\rightarrow\text{tar}}(X(\textbf{T}_{\text{src}}))\right)\right\|_{2}^{2}),$$ (5)

where $\sigma$ is the confidence, $X(\textbf{T}_{\text{src}})$ is the spatial location of $\textbf{T}_{\text{src}}$, and $f_{src\rightarrow tar}$, $f_{tar\rightarrow src}$ denotes the forward and backward flow between $\textbf{T}_{\text{src}}$ and $\textbf{T}_{\text{tar}}$. Thus, the proposed flow-guided token merging is termed:

$$\textbf{T}_{\text{merge}}=\mathcal{M}(\textbf{T}_{\text{src}},\;\textbf{T}_{\text{tar}},\;f_{\text{src}\rightarrow\text{tar}},\;\sigma,\;r).$$ (6)

Fig. 5 provides a clearer illustration of our proposed component. Additionally, as shown at the bottom of Fig. 5, flow and cosine similarity identify different correspondences, so a hybrid approach can provide comprehensive guidance, leading to improved temporal consistency and overall video quality.

Spatial-awareness and Padding Removal.

Directly finding correspondences relying on cosine similarity can easily lead to mismatches in places with uniform textures, especially video backgrounds (e.g., sky, sand, grass), bottom of  Fig. 5, resulting in blurrier outcomes. Given that corresponding points in adjacent frames are typically spatially close in videos, leveraging spatial information is crucial for accurate correspondence. We can effectively utilize this information by weighting the cosine similarity scores with the tokens’ spatial distances. The weighted scores are defined as:

$$s_{ij}^{\prime}=s_{ij}\cdot e^{-\tau}\,,\,\,\text{with}\,\,\tau=\left\lfloor\left[{\|X(i)-X(j)\|_{2}^{2}}\right]/{R}\right\rfloor\,,$$ (7)

where $X(i)$, $X(j)$ is the spatial location of the $i^{th}$ source token and the $j^{th}$ target token; $R$ is a hyperparameter defining the radius of the region with uniform weight.

Images are often padded to propagate through the UNet, and we find that this padding can significantly impact the correspondences found in the tokens. While it is acceptable for padding to correspond to padding in another token, when the token feature dimension is low, cosine similarity may mistakenly identify padding as corresponding to the actual image content. This issue persists until the later stages of the denoising process. Please refer to the appendix for the non-padding correspondence figure. To address this, we remove padding before merging and add it back when unmerging.

Merging Ratio Annealing.

For restoration tasks that demand fine details, maintaining a high merging ratio in the later stages of the denoising process can result in blurred and unrealistic outcomes. To address this, we employ ratio annealing to gradually reduce the merging ratio, preserving detail and realism in the restored video. The merging ratio of the $i^{th}$ denoising step is computed as:

$$r_{i}=r\cdot\cos\left(\frac{\pi}{2}\cdot\text{max}\left(\text{min}\left(\delta\cdot\frac{i-i_{\text{beg}}}{i_{\text{end}}-i_{\text{beg}}},1\right),0\right)\right),$$ (8)

where $i_{\text{beg}}$, $i_{\text{end}}$ are predefined steps indicating the beginning and end of the merging process, and $\delta$ represents a hyperparameter for controlling the annealing speed.

3.4 Scheduling

As depicted in Fig. 3, at the initial stage of the diffusion denoising process, hierarchical latent warping offers rough shape guidance on a global scale by warping latents between keyframes and on a local scale by propagating these latents within the batch. During the majority of the denoising process, tokens are processed with our hybrid spatial-aware token merging before entering the attention layer. This component further improves temporal consistency by matching similar tokens, utilizing both flow and spatial information.

Table 1: Quantitative comparisons. (Left) 4$\times$ and 8$\times$ video super-resolution on the SPMCS [75] and DAVIS [60] datasets. (Right) video denoising of various noise levels on the REDS30 [55] dataset. The best and second performances are marked in red and blue, respectively. $E_{\text{warp}}^{*}$ denotes $E_{\text{warp}}(\times 10^{-3})$ and $E_{\text{inter}}$, LPIPS${}_{\text{inter}}$ denotes interpolation error and LPIPS.

				SD $\times$4		DiffBIR
	Metrics	VidToMe	FMA-Net	Frame	Ours	Frame	Ours
SPMCS	PSNR $\uparrow$	20.516	21.910	20.573	20.636	21.534	21.843
	SSIM $\uparrow$	0.471	0.617	0.490	0.517	0.544	0.572
	LPIPS $\downarrow$	0.352	0.230	0.298	0.286	0.261	0.258
	$E_{\text{warp}}^{*}\downarrow$	0.531	0.157	1.058	8.290	0.571	0.571
	$E_{\text{inter}}\downarrow$	10.102	3.271	13.817	11.961	10.257	9.712
	LPIPS${}_{\text{inter}}\downarrow$	0.218	0.015	0.241	0.226	0.177	0.158
DAVIS $\times$4	PSNR $\uparrow$	23.948	25.215	23.504	23.843	23.780	24.182
	SSIM $\uparrow$	0.608	0.727	0.584	0.618	0.601	0.621
	LPIPS $\downarrow$	0.298	0.347	0.277	0.272	0.264	0.262
	$E_{\text{warp}}^{*}\downarrow$	0.512	0.186	0.912	0.745	0.654	0.474
	$E_{\text{inter}}\downarrow$	14.615	11.558	18.125	17.431	16.529	14.666
	LPIPS${}_{\text{inter}}\downarrow$	0.278	0.078	0.292	0.274	0.266	0.232
DAVIS $\times$8	PSNR $\uparrow$	22.570	22.690	20.268	20.519	21.964	22.331
	SSIM $\uparrow$	0.527	0.594	0.446	0.424	0.502	0.519
	LPIPS $\downarrow$	0.454	0.528	0.470	0.434	0.362	0.367
	$E_{\text{warp}}^{*}\downarrow$	0.523	0.351	2.199	1.759	0.964	0.699
	$E_{\text{inter}}\downarrow$	14.117	13.978	24.496	21.746	17.981	15.853
	LPIPS${}_{\text{inter}}\downarrow$	0.379	0.132	0.457	0.442	0.372	0.333

					DiffBIR
$\sigma$	Metrics	VRT	Shift-Net	VidToMe	Frame	Ours
75	PSNR $\uparrow$	25.050	21.033	23.791	24.585	24.520
	SSIM $\uparrow$	0.787	0.381	0.618	0.649	0.649
	LPIPS $\downarrow$	0.275	0.735	0.296	0.276	0.275
	$E_{\text{warp}}^{*}\downarrow$	0.314	1.757	0.765	0.751	0.706
	$E_{\text{inter}}\downarrow$	17.825	27.094	21.751	21.798	21.166
	LPIPS${}_{\text{inter}}\downarrow$	0.095	0.501	0.287	0.275	0.264
100	PSNR $\uparrow$	24.582	22.573	24.606	24.524	24.534
	SSIM $\uparrow$	0.744	0.484	0.676	0.648	0.652
	LPIPS $\downarrow$	0.346	0.518	0.318	0.275	0.271
	$E_{\text{warp}}^{*}\downarrow$	0.294	1.126	0.781	0.763	0.696
	$E_{\text{inter}}\downarrow$	17.079	23.424	21.460	21.835	20.639
	LPIPS${}_{\text{inter}}\downarrow$	0.095	0.375	0.278	0.281	0.267
random	PSNR $\uparrow$	24.989	21.113	23.692	24.579	24.508
	SSIM $\uparrow$	0.780	0.386	0.615	0.650	0.649
	LPIPS $\downarrow$	0.284	0.728	0.303	0.276	0.270
	$E_{\text{warp}}^{*}\downarrow$	0.363	1.896	0.772	0.755	0.713
	$E_{\text{inter}}\downarrow$	18.147	27.565	21.929	21.743	21.140
	LPIPS${}_{\text{inter}}\downarrow$	0.099	0.542	0.291	0.282	0.272

Table 2: Ablation studies for 8$\times$ VSR on DAVIS [59] test sets. (Left) different correspondence matching methods.  (Right) the proposed components applied at different stages of the denoising process. We apply our two proposed components, hierarchical latent warping (HLW) and hybrid spatial-aware token merging (HS-ToMe) at the early, mid, and late stages of the denoising process.

Down blocks	Up blocks	Spatial- aware	LPIPS $\downarrow$	$E_{\text{warp}}^{*}\downarrow$	LPIPS${}_{\text{inter}}\downarrow$
Flow	Flow	–	0.518	1.214	0.563
Cos	Cos	–	0.390	0.736	0.350
Cos	Flow	–	0.507	1.049	0.545
Flow	Cos	–	0.375	0.677	0.347
Flow	Cos	✓	0.367	0.699	0.333

HLW (Sec. 3.2)				HS-ToMe (Sec. 3.3)
Early	Mid	Late		Early	Mid	Late	LPIPS $\downarrow$	$E_{\text{warp}}^{*}\downarrow$	LPIPS${}_{\text{inter}}\downarrow$
–	–	–		–	–	–	0.362	0.964	0.372
✓	–	–		✓	–	–	0.368	0.887	0.369
✓	✓	–		✓	✓	✓	0.43	0.804	0.383
✓	✓	✓		✓	✓	✓	0.411	0.704	0.339
✓	–	–		✓	✓	✓	0.367	0.699	0.333

Figure 6: Comparison of temporal profile. We examine a row of pixels and track changes over time. The profiles from Flow + Flow and Cosine + Flow methods exhibit noise, indicating flickering artifacts. The Cosine + Cosine method shows smoother profiles but contains some discontinuities. Flow + Cosine demonstrates improved consistency but retains some distortions. Utilizing flow, cosine, and spatial-aware techniques, our method achieves the most seamless and consistent transitions, effectively minimizing artifacts.

Figure 7: Qualitative comparisons on 8$\times$ video super-resolution. As shown in the first row, the low-quality input lacks almost all details. In the zoomed-in patches, our method produces clearer and more consistent results.

Figure 8: Qualitative comparisons on video denoising in REDS30 [54] dataset for $\sigma=100$. Our method effectively denoises and generates detailed results while maintaining temporal coherence.

4 Experiments

Testing Dataset.

For video super-resolution, we evaluate on SPMCS [95] and DAVIS [59] testing sets, with two downsample scales ($\times$4, $\times$8), following the same degradation pipeline of RealBasicVSR [10] to generate LQ-HQ video pairs. For video denoising, we evaluate on REDS30 [54] with 3 different noise levels (std. $=$ 75, 100, and std. is uniformly sampled from the range [50, 100]).

Evaluation Metrics.

We evaluate the restoration performance based on two aspects: (1) image quality, using LPIPS, SSIM, and PSNR; (2) temporal consistency, using warping error $E_{\text{warp}}$, interpolation error, and interpolation LPIPS. Since LPIPS better reflects visual quality, we propose interpolation LPIPS, based on the interpolation error used in a previous study  [41], to more accurately measure video continuity from a visual perspective. This involves interpolating a target frame from its previous and next frames and computing the LPIPS between the estimated and target frames.

Implementation Details.

The experiment is conducted on an NVIDIA RTX 4090 GPU. We apply our method to DiffBIR [85] and SDx4 upscaler [1], both image-based diffusion models, to demonstrate the propsed method’s compatibility with different models. Noted that for models that are restricted to a super-resolution scale of 4$\times$, we will apply the process twice and then use bicubic downsampling to achieve 8$\times$ results.

4.1 Comparisons with State-of-the-Art Methods

To verify the effectiveness of our approach, we compare it with several state-of-the-art methods, including FMA-Net [96] for video super-resolution, and VRT [43] and Shift-Net [39] for video denoising. We also compare our method to per-frame restoration and the application of VidToMe [41], a zero-shot video editing method, onto the same model as ours.

Video Super-resolution.

As shown in Tab. 1, regression-based methods like FMA-Net [96] perform better on datasets like SPMCS that have minimal motion. However, their generalization ability diminishes significantly with increased motion or severe degradation. VidToMe [41] can generate highly consistent results, but they are often very blurry, leading to poor visual quality. In contrast, our method enhances temporal consistency while maintaining the generation quality of the original diffusion model, making it the most competitive approach. Fig. 7 provides visualizations of two challenging VSR cases. FMA-Net fails to produce sharp results due to domain gaps between training and testing. Diffusion-based image restoration method DiffBIR [45] and SD$\times$4 upscaler [1] can generate sharp results with details, while per-frame processing makes the result video temporal inconsistent and jitters across frames. On the contrary, our zero-shot video restoration framework restores a low-quality input video into a temporally consistent high-quality video.

Video Denoising.

Video denoising, compared to VSR, is a simpler task for regression models, as they can often find the correct pixel value given a sufficiently large batch size. However, our method consistently outperforms others in terms of visual quality (LPIPS) and remains highly robust even as degradation becomes severe. Fig. 8 visualizes the denoising results on the REDS30 dataset. Shift-Net [39] fails to remove all noise, likely due to the out-of-domain noise level; VRT [43] produces smooth results but lacks fine details. Although DiffBIR [45] generates highly detailed images, it suffers from poor temporal consistency, as evident in the changes to the pedestrian’s head and the statue’s face. In contrast, our method preserves both fine details and temporal consistency, effectively balancing these two aspects.

4.2 Ablation Study

Ways of Identifying Correspondence.

Tab. 2 presents an ablation study comparing different approaches (optical flow and cosine similarity) for finding correspondences and their order in the UNet. As detailed in Sec. 3.3, the hybrid approach of using optical flow at the downsample blocks and cosine similarity at the upsample blocks achieves the best performance. Additionally, our proposed spatial-aware token merging further enhances performance by utilizing spatial information to guide correspondences. The comparisons in Fig. 6 also indicate that our results are smoother, demonstrating better temporal stability.

Applied Stages in the Denoising Process.

Tab. 2 presents an ablation study evaluating the application of our two proposed components, hierarchical latent warping (HLW, Sec. 3.2) and hybrid spatial-aware token merging (HS-ToMe, Sec. 3.3), at the early, mid, and late stages of the denoising process. The results indicate that applying latent warping in the mid or late stages can significantly degrade the generated outcomes. Furthermore, ensuring consistency in the token space is crucial for achieving coherent and high-quality results.

5 Conclusion

We introduce a novel zero-shot video restoration framework utilizing pre-trained image-based diffusion models, eliminating the need for extensive retraining. Our approach integrates hierarchical latent warping and hybrid flow-guided, spatial-aware token merging, significantly enhancing temporal consistency and video quality under various degradation conditions. Experimental results demonstrate that our framework surpasses existing methods both in quality and consistency.

Limitations.

Our zero-shot video restoration framework has some limitations. Random keyframe sampling may not always select the most representative frames, potentially affecting restoration quality, especially if frames with severe degradation are chosen. Additionally, the sensitivity of the LDM decoder to minor variations in input latents can cause flickering, particularly in dynamic scenes. Future improvements will focus on refining keyframe selection and stabilizing the decoder output to enhance the practical application of diffusion-based video restoration methods.

Appendix A Appendix / supplemental material

In this supplementary material, we first provide additional details on the testing datasets and evaluation metrics. Subsequently, we present more visual comparisons of various methods.

A.1 Correspondences identified by cosine similarity without padding removal

Fig. 9 shows that padding value will affect the matching severely.

Figure 9: The padded regions affect the matching severely.

A.2 Additional Application: Consistent Video Depth

Our zero-shot framework is applicable to any pre-trained image-based diffusion models and could improve the predicted video consistency. Therefore, we integrate our proposed zero-shot framework into a state-of-the-art latent diffusion-based monocular depth estimator: Marigold [35]. Fig. 10 shows that integrating our proposed framework into Marigold helps improve the temporal consistency of video depth estimation.

Input
Marigold [35]
Ours
Input
Marigold [35]
Ours
Input
Marigold [35]
Ours

Figure 10: Integrating our proposed framework into Marigold [35] helps improve the temporal consistency of video depth estimation.