Tencent and Tsinghua University Collaborate on High-Resolution Video Expansion Technology: Follow-Your-Canvas
Beijing, China – Tencent’s HunYuan team, in collaboration with Tsinghua University and other institutions, has unveiled a groundbreaking high-resolution video expansion technology called Follow-Your-Canvas. This innovative technology empowers users to seamlessly extend video content toany desired resolution while maintaining the original video’s quality and style.
Follow-Your-Canvas tackles the challenge of scaling video content to higher resolutions, aprocess often hindered by GPU memory limitations. By employing distributed processing and layout alignment, the technology effectively overcomes these constraints, allowing for the handling of large-scale video extension tasks.
Key Features and Capabilities:
- High-ResolutionOutput: Follow-Your-Canvas enables the expansion of video content to any resolution, such as scaling from 4K to 8K or even higher.
- Unbound by Memory Constraints: The technology can handle large-scale videoextension tasks without being limited by GPU memory size.
- Spatiotemporal Consistency: During the expansion process, Follow-Your-Canvas maintains the spatial and temporal consistency of the video, ensuring that the final output retains the original video’s style and quality.
- Generation of Rich New Content: The technologygenerates new content that seamlessly blends with the original video’s style within the designated expansion area, enhancing the overall visual effect.
- Large-Scale Video Extension: Follow-Your-Canvas excels in handling large-scale video extension tasks, such as expanding a 512×512 resolution video to1152x2048 (approximately 9 times the original resolution).
Technical Principles:
- Spatial Window Segmentation: The video is divided into multiple spatial windows, each processed independently for content generation. These windows are then seamlessly merged, allowing for the handling of videos of any size and resolution withoutbeing limited by GPU memory.
- Layout Encoder: A layout encoder extracts global layout information from the source video and injects it into the generation process for each window, ensuring that newly generated content aligns with the original video’s layout.
- Relative Region Embedding (RRE): RRE provides the relativepositional relationship between the source video and the target window, further guiding the generation process for each window. This ensures that the generated expanded content harmonizes with the original video’s layout, improving spatial and temporal consistency.
- Distributed Generation: Content generation for each window is processed in parallel across multiple GPUs, and thegenerated windows are seamlessly merged into the final video.
Applications and Potential:
Follow-Your-Canvas has a wide range of potential applications across various industries:
- Video Size Adjustment: Transforming vertically-shot mobile videos into horizontal format for different playback needs, such as social media sharing or viewing onwidescreen TVs.
- Ultra-Wide Screen Video Generation: Expanding standard videos to ultra-wide screen format, offering a wider field of view ideal for cinema or ultra-wide screen displays.
- Panoramic Video Production: Transforming regular videos into panoramic videos using video expansion technology, enhancing the viewing experience andsuitable for virtual reality (VR) and 360-degree video production.
- High-Resolution Video Output: Handling video outputs up to 16K resolution, catering to applications demanding high definition, such as film production or high-quality commercials.
Availability and Resources:
Follow-Your-Canvasis open-source and available for developers to explore and utilize. The project’s official website, GitHub repository, and technical paper can be accessed through the following links:
- Project Website: follow-your-canvas.github.io
- GitHub Repository: https://github.com/mayuelala/FollowYourCanvas
- arXiv Technical Paper: https://arxiv.org/pdf/2409.01055
Conclusion:
Follow-Your-Canvas represents a significant advancement in video expansion technology, addressing key limitations and offering a powerful tool for enhancing video content. Its ability toscale video resolutions without compromising quality and its adaptability to various applications make it a valuable asset for professionals and enthusiasts alike. As the technology continues to evolve, we can expect even more innovative applications and advancements in the field of video manipulation.
Views: 0