Abstract

We propose the first video diffusion framework for reference-based lineart video colorization. Unlike previous works that rely solely on image generative models to colorize lineart frame by frame, our approach leverages a large-scale pretrained video diffusion model to generate colorized animation videos. This approach leads to more temporally consistent results and is better equipped to handle large motions. Firstly, we introduce Sketch-guided ControlNet which provides additional control to finetune an image-to-video diffusion model for controllable video synthesis, enabling the generation of animation videos conditioned on lineart. We then propose Reference Attention to facilitate the transfer of colors from the reference frame to other frames containing fast and expansive motions. Finally, we present a novel scheme for sequential sampling, incorporating the Overlapped Blending Module and Prev-Reference Attention, to extend the video diffusion model beyond its original fixed-length limitation for long video colorization. Both qualitative and quantitative results demonstrate that our method significantly outperforms state-of-the-art techniques in terms of frame and video quality, as well as temporal consistency. Moreover, our method is capable of generating high-quality, long temporal-consistent animation videos with large motions, which is not achievable in previous works. Our code and model are available at this https URL.

Paper: https://arxiv.org/abs/2409.12960

Project Page: https://luckyhzt.github.io/lvcd

Code: (coming soon)

Supplementary Demo clips: https://luckyhzt.github.io/lvcd/supplementary/supplementary.html

(b74#0 - nya~) on Discrod (2024)

Image description: A young woman with vibrant yellow hair sitting on a set of stairs at night. She is dressed in a black hoodie. Her gaze is directed towards the camera, and she wears a warm smile on her face. The stairs are illuminated by the soft glow of streetlights, casting a gentle light on the scene. The background is dark, suggesting a quiet, urban setting. The overall mood of the image is serene and inviting, capturing a moment of tranquility in the midst of the city's hustle and bustle.

(turkey910) (2024)

Image description: A girl is seated on a bench against the backdrop of a cityscape at night. She is wearing a white hoodie adorned with a dark blue stripe and withe floral patterns and black leggings. Illuminated by the gentle glow of nearby lights, buildings stand tall but silent, their windows aglow with warmth.

(ToriBirdZ) (2023)

Image description: A woman wearing an eye mask looks down at the camera with mild surprise, She is wearing a red jacket with white fur lining, a white graphic top, a pendant, and black ripped jeans.

(Manityro) (2023)

Image Caption: An illustration of Izumi Noa from Patlabor sitting on a set of concrete steps. She is wearing a white hoodie with a small yellow logo on the arm and black shoes. The background shows a city skyline with tall buildings behind a green hill.

Demo at https://cotrans.touhou.ai/

(.v.i.l.) on Discord (2023)

Image Caption: The face of a vampire is illuminated by the moonlight. She has white hair and pointed ears. She wears a dark cloak and a white blouse with a black corset with red laces. She is surrounded by red flowers.

Abstract:

Significant advancements have been achieved in the realm of large-scale pre-trained text-to-video Diffusion Models (VDMs). However, previous methods either rely solely on pixel-based VDMs, which come with high computational costs, or on latent-based VDMs, which often struggle with precise text-video alignment. In this paper, we are the first to propose a hybrid model, dubbed as Show-1, which marries pixel-based and latent-based VDMs for text-to-video generation. Our model first uses pixel-based VDMs to produce a low-resolution video of strong text-video correlation. After that, we propose a novel expert translation method that employs the latent-based VDMs to further upsample the low-resolution video to high resolution. Compared to latent VDMs, Show-1 can produce high-quality videos of precise text-video alignment; Compared to pixel VDMs, Show-1 is much more efficient (GPU memory usage during inference is 15G vs 72G). We also validate our model on standard video generation benchmarks. Our code and model weights are publicly available at https://github.com/showlab/Show-1.

Abstract:

Significant advancements have been achieved in the realm of large-scale pre-trained text-to-video Diffusion Models (VDMs). However, previous methods either rely solely on pixel-based VDMs, which come with high computational costs, or on latent-based VDMs, which often struggle with precise text-video alignment. In this paper, we are the first to propose a hybrid model, dubbed as Show-1, which marries pixel-based and latent-based VDMs for text-to-video generation. Our model first uses pixel-based VDMs to produce a low-resolution video of strong text-video correlation. After that, we propose a novel expert translation method that employs the latent-based VDMs to further upsample the low-resolution video to high resolution. Compared to latent VDMs, Show-1 can produce high-quality videos of precise text-video alignment; Compared to pixel VDMs, Show-1 is much more efficient (GPU memory usage during inference is 15G vs 72G). We also validate our model on standard video generation benchmarks. Our code and model weights are publicly available at https://github.com/showlab/Show-1.

cross-posted from: https://lemmy.dbzer0.com/post/6153066