This AI deepfake is next level: Control expressions & motion

Deep Fake Technology: Live Portrait Tool

Overview of Live Portrait Tool

• The speaker introduces a powerful deep fake tool called "Live Portrait," which animates a still photo using the movements from any video of a person talking.

• This tool is noted for its ability to handle tricky expressions, making it one of the few capable of animating photos with complex facial movements.

Features and Accessibility

• Developed by the team at QuShow, known for their AI video generator Cing, Live Portrait is free and open-source, allowing users to download and run it locally.

• Users can input videos of someone moving their face alongside any photo, enabling the mapping of expressions onto that image without restrictions on famous individuals.

Animation Capabilities

• The tool can animate various types of images including black-and-white photos and oil paintings, demonstrating versatility in handling different artistic styles.

• It effectively animates multiple faces within a single image while maintaining consistency across all animated features without distortions or warping.

Advanced Examples

• Demonstrations show how well the tool manages exaggerated expressions and 3D character animations, proving its effectiveness beyond realistic photographs.

• The technology allows for creative applications even with stationary images; characters' faces move while their bodies remain still.

Video Integration and Customization

• Users can also apply this technology to videos. For instance, they can map expressions from one video onto another moving subject's face seamlessly.

• Additional settings allow users to control specific facial features like eye openness and mouth movement through adjustable dials for more personalized animations.

Unique Applications

Getting Started with Life Portrait AI Tool

Introduction to Installation Process

• The speaker introduces the architecture and methodology of the Life Portrait AI tool, mentioning a link for further information in the description.

• A Dell Precision 5690 is used for this demonstration, highlighting its capability to run AI tools locally with an RTX 5000 Ada GPU.

Setting Up the Environment

• The speaker creates a new folder named "live portrait" on their desktop for organization purposes before proceeding with installation.

• Instructions are provided to clone the GitHub repository using command prompt; users must have Git installed beforehand.

Installing Git

• The speaker guides through downloading and installing Git for Windows, emphasizing default settings during installation.

• After successful installation, they confirm that the live portrait repository has been cloned into the designated folder.

Navigating Directories

• Users are instructed to change directories into the capitalized "Live Portrait" folder using command line commands.

• The next step involves creating a Conda environment with Python version 3.9.1; instructions are given for those who need to install Conda.

Installing Anaconda/Miniconda

• The speaker recommends installing Miniconda as a lightweight alternative to full Anaconda, which includes unnecessary packages.

• They guide viewers through downloading and installing Miniconda while ensuring compatibility with Python 3.11 for better support of free AI tools.

Finalizing Installation Steps

• After completing Miniconda installation, users must add it to their system path by editing environment variables.

How to Set Up and Use Live Portrait Environment

Activating the Environment

• After creating the environment, it must be activated using cond activate. This command tells Anaconda to activate the "live portrait" environment.

• Upon successful activation, "live portrait" will appear in parentheses at the beginning of each command line, indicating that the environment is active.

Installing Required Packages

• The next step involves installing necessary dependencies such as Torch, NumPy, OpenCV Python, and ONNX Runtime GPU. This process may take time due to large package sizes (e.g., Torch is 2.7 GB).

• Once all packages are installed successfully, users need to download pre-trained weights from Google Drive or another source for model inference.

Downloading Pre-Trained Weights

• Users should download two zip files containing essential models and neural networks into their cloned repository folder.

• After downloading, unzip both files into a designated "pre-trained weights" folder within the repo. This includes folders named "Insight face" and "Live portrait."

Verifying Folder Structure

• Inside the "pre-trained weights" folder, there should be two subfolders: "Insight face," which contains a models directory with a Buffalo folder; and "Live portrait," which includes base models and retargeting models.

• Confirming this structure ensures that all necessary components for running inference are correctly set up.

Running Inference with Gradio Interface

• To run inference more intuitively, users can execute python app.py, which launches a Gradio interface for easier interaction.

• Clicking on the provided link opens up an interactive interface where users can upload videos and images for testing purposes.

Testing with Example Videos

• Users can drag example videos from their assets folder into the interface to test functionality. They can also upload photos onto which they want to clone expressions.

How to Use Live Portrait for Animation

Adjusting Facial Features

• The tool allows users to adjust the mouth's openness with a ratio slider, where dragging it to zero closes the mouth and dragging it to 0.8 opens it fully.

• Users can also manipulate eye closure; setting both features to maximum creates an exaggerated and potentially horrifying expression.

• After adjustments, clicking "retargeting" updates the facial expressions based on the new settings.

Setting Up Live Portrait

• To reopen Live Portrait after exiting, navigate to your cloned repository folder and open Command Prompt by typing CMD.

• Activate the Conda environment using cond activate live portrait, ensuring uppercase letters are used correctly.

• Run the application by typing Python app.py, which initializes a Gradio interface for testing images.

Testing Animation with Different Characters

• The presenter tests animation capabilities using a Disney Pixar character video, achieving impressive results in animating facial expressions.

• An attempt is made with an anime character; despite initial doubts about capturing tricky expressions, the outcome is surprisingly effective.

Implications of Animation Technology

• The ability to map human expressions onto animated characters opens up creative possibilities for content creation, allowing anyone to create their own anime scenes easily.

• This technology could enable users to control not just animations but also voice modulation through various tools available online.

Advanced Applications and Comparisons

• A demonstration using a video of Jeff Bezos showcases how well the tool tracks head movements and maps them onto still images without audio initially included.

• Post-processing involves adding original audio manually, highlighting that while this tool excels at visual fidelity, audio integration requires additional steps.

Limitations and Future Potential

• Compared to other face animator tools like Hello (which only sync audio), this method provides comprehensive control over both movement and expression in animations.

How to Create Quick Animations with AI Tools

Overview of Animation Generation

• The speaker discusses the efficiency of a new animation tool, noting that it generates animations in about 30 seconds, significantly faster than other open-source video generators.

Comparison of Generated Animations

• A comparison is made between a generated 2.5D portrait of a dog and the original human input, highlighting impressive eye-tracking capabilities but also some limitations in mouth movements during expressions.

Human-to-Human Animation

• The speaker tests the tool with a portrait of a woman, expecting better results due to both inputs being human; the output shows excellent synchronization of facial movements.

Tool Capabilities and User Control

• The generated video closely resembles the original woman, demonstrating no visible artifacts or flaws. This showcases the power and potential applications of this AI tool for personal filmmaking.

Accessibility and Community Engagement

• The speaker emphasizes that this powerful animation tool is free and open-source, allowing anyone to download and use it without restrictions. They encourage viewers to share their experiences with the tool.

Conclusion and Future Content

• Viewers are invited to comment on their experiences with the tool. The speaker mentions ongoing efforts to discover new AI tools and encourages engagement through likes, shares, and subscriptions.