PortraitGen AI-Powered Video Portraits fromUSTC

Chinese University Unveils AI Portrait Video Editing Tool: PortraitGen

Hefei, China – The University of Science and Technology of China (USTC) has announced the release of PortraitGen, a groundbreaking AI-powered video editing tool designed to revolutionize the way we manipulate and enhance human portraits invideo content.

PortraitGen leverages advanced 3D Gaussian splatting technology and a neural Gaussian texture mechanism to transform 2D portrait videos into4D Gaussian fields, enabling high-quality 3D editing with temporal consistency. The tool supports multi-modal editing, including text-driven, image-driven editing, and relighting, allowing for quick and efficient manipulation of video subjects.Users can seamlessly stylize, change clothing, adjust lighting, and more, all while preserving natural facial features and expressions.

Key Features of PortraitGen:

Multi-Modal Portrait Editing: PortraitGen offers both text-drivenand image-driven editing modes. Users can input text descriptions to specify desired actions, expressions, and scene changes or utilize reference images for style transfer, virtual try-on, and other creative applications.
Relighting: Powered by IC-Light technology, PortraitGen enables dynamic adjustment of lighting effects within videos based on text descriptions. This ensures seamless integration of lighting with the scene for a realistic and natural look.
Face-Aware Editing: PortraitGen’s facial perception module meticulously preserves facial structures and individual characteristics during editing. This ensures that edited portraits maintain natural expressions and movements.
Style Transfer and Virtual Try-on:The tool supports style transfer and virtual try-on features. Users can apply global style transformations, such as converting portraits to an animated style, or add virtual clothing and accessories to video subjects using reference images.
Multi-Camera and Complex Scene Handling: PortraitGen can handle multi-camera videos, maintaining consistency in styleand character appearances. Its Gaussian texture technology enables complex style rendering for videos, including Lego-style and pixel art styles.
Fast Generation and High Frame Rate Output: PortraitGen delivers rapid editing capabilities, generating videos with a high frame rate of up to 100 frames per second (FPS). This makes itideal for efficient video production workflows.

Technical Principles Behind PortraitGen:

3D Gaussian Splatting (3DGS): This technique represents scenes using 3D Gaussians, defining the center point, direction, size, opacity, and color properties of each Gaussian through a 3Dcovariance matrix. This enables the construction of dynamic 3D fields.
Neural Gaussian Texture Mechanism: A 3D Gaussian field is maintained within the UV space of the SMPL-X model. The Gaussians deform based on the underlying mesh deformation tracked from the input video. UV mapping and a 2Dneural renderer convert feature maps into RGB signals.
Face-Aware Editing Module: This module performs two rounds of editing on the head region, enhancing facial structure perception and improving editing quality.
Expression Similarity Guidance: Rendered images and input source images are mapped to the EMOCA latent expression space. A loss function ensuresexpression similarity.
Multi-Modal Editing Technology: Knowledge from large-scale 2D generative models is integrated to enable text-driven editing, image-driven editing, and relighting.

Applications of PortraitGen:

Film and Television Production: PortraitGen can be used to create or modify characterappearances, implement special effects makeup, or achieve stylized scene transitions in films, TV series, and short films.
Artistic Creation: Artists and illustrators can use PortraitGen to create portrait art with specific styles, such as converting portraits to pixel art or oil painting styles.
Advertising and Marketing: Inthe advertising industry, PortraitGen can be used to customize portrait editing based on brand image or product characteristics, attracting target audiences.
Fashion Industry: Fashion designers and retailers can leverage virtual try-on capabilities to showcase clothing and accessories in virtual environments, providing customers with new shopping experiences.
Social Media andShort Videos: Content creators and influencers can use PortraitGen to edit their portrait videos, adding creative effects and enhancing content engagement and interactivity.
Game Development: PortraitGen can be used to quickly generate or edit character appearances in game design, enhancing game personalization and richness.

PortraitGen’s release signifies a significantadvancement in AI-powered video editing technology. Its ability to seamlessly manipulate human portraits in videos opens up a world of possibilities for filmmakers, artists, marketers, and content creators alike. As the technology continues to evolve, we can expect even more innovative applications and creative potential to emerge.

>>> Read more <<<