Kwai Unveils Open-Source LivePortrait: A Framework for High-Fidelity Portrait Animation
Beijing, China – Kwai, the popular short-form video platform, has released LivePortrait, an open-source framework for generating high-quality animated portraits. This innovative technology enables users to seamlessly transfer expressionsand poses from a driving video onto static or dynamic images, creating expressive and engaging videos.
LivePortrait leverages a novel implicit keypoint framework, trainedon a massive dataset of high-quality images and videos. This approach significantly enhances the model’s generalization capabilities and allows for precise control over the generated animations. Notably, LivePortrait achieves impressive speeds, generating a single frame in just 12.8 milliseconds on an RTX 4090 GPU, with further optimization potential.
The open-source nature of LivePortrait has garnered significant interest within the AI community. The project’s GitHub repository provides comprehensive documentation, tutorials, and resources for developers to explore and integrate this powerful technology.
Key Features of LivePortrait:
- Expression and Pose Transfer: LivePortrait seamlessly transfers expressions and poses from a driving video to static or dynamic portraits, resulting in videos with rich and nuanced facial movements.
- High Efficiency: The frameworkboasts exceptional processing speeds, generating a single frame in a mere 12.8 milliseconds on an RTX 4090 GPU.
- Generalization Capabilities: Trained on a vast dataset of high-quality images and videos, LivePortrait exhibits excellent generalization capabilities, adapting to diverse portrait styles and identities.
- Controllability: The use of implicit keypoints and lightweight MLP networks allows for fine-grained control over the animation generation process.
- Multi-Style Support: LivePortrait handles various portrait styles, including real-life photographs and stylized images like anime.
- High-Resolution Animation: The frameworksupports the generation of high-resolution animations, delivering crisp and detailed visual output.
- Alignment and Redirection Modules: LivePortrait incorporates alignment and eye/mouth redirection modules to handle complex scenarios like cropping and group photos, preventing pixel misalignment.
Technical Underpinnings of LivePortrait:
LivePortrait’s foundation lies in a two-stage training process:
- Base Model Training: The first stage focuses on optimizing the appearance extractor, motion extractor, warping module, and decoder. These components are trained from scratch.
- Stitching and Redirection Module Training: After training the base model, the appearance extractor, motion extractor, warping module, and decoder are frozen. The second stage optimizes the stitching and redirection modules while keeping the base model unchanged.
Video-Image Mixed Training: LivePortrait treats each image as a single-frame video segment and trains the model simultaneously on both videos and images, enhancing its generalization capabilities.
Enhanced Network Architecture: LivePortrait unifies the canonical implicit keypoint estimation network, head pose estimation network, and expression deformation estimation network into a single model. It utilizes ConvNeXt-V2-Tiny as the architecture, directly estimating the canonical implicit keypoints, head pose, and expression deformation of theinput image.
Keypoint-Guided Implicit Keypoint Optimization: LivePortrait incorporates 2D keypoints to capture subtle expressions, using keypoint-guided loss as a guide for implicit keypoint optimization.
Cascaded Loss Function: The framework employs a cascaded loss function, including the face vid2vid implicit keypoint invariance loss, keypoint prior loss, head pose loss, and deformation prior loss, along with perceptual and GAN losses, to enhance texture quality.
Applications of LivePortrait:
- Social Media Content Creation: Users can transform their photos or videos into dynamic content, enhancing interactivity and engagementon social media platforms.
- Virtual Anchors and Live Streaming: LivePortrait enables the creation of virtual avatars for live streaming or video production, eliminating the need for real-life presenters. This technology is ideal for gaming streams, educational presentations, and more.
- Film and Animation Production: LivePortrait canbe utilized in post-production for character expression capture and animation generation, streamlining production workflows and reducing costs.
Availability and Resources:
- Project Website: https://liveportrait.github.io/
- GitHub Repository: https://github.com/KwaiVGI/LivePortrait
- Hugging Face Model Library: https://huggingface.co/spaces/KwaiVGI/LivePortrait
- arXiv Technical Paper: https://arxiv.org/pdf/2407.03168
LivePortrait represents a significant advancement in portrait animation technology, offering a powerful andversatile tool for creators across various fields. Its open-source nature fosters collaboration and innovation within the AI community, paving the way for exciting new possibilities in digital content creation.
【source】https://ai-bot.cn/liveportrait/
Views: 1