THIS is the Biggest Thing Since CGI

What Are Gaussian Splats?

Introduction to Holograms and 4D Technology

• The speaker introduces holograms as a new medium, emphasizing their ability to be viewed from multiple angles and dimensions.

• The discussion highlights the potential of this technology, suggesting it could represent the future of visual media.

Evolution from Nerfs to Gaussian Splats

• The speaker reflects on previous excitement about neural radiance fields (Nerfs), which allowed for photorealistic 3D scene capture without extensive manual work.

• Despite initial promise, Nerfs were abandoned due to processing speed issues and usability challenges, paving the way for Gaussian splats.

Understanding Gaussian Splats

• Gaussian splats are introduced as an evolution in 3D capture technology that is gaining traction in the industry.

• The speaker explains that both splats and traditional photo scans involve capturing overlapping images but differ fundamentally in how they represent light and depth.

Technical Process of 3D Capture

• Capturing a 3D object involves taking numerous photos from various angles, followed by a process called structure from motion to determine camera positions.

• Traditional photo scanning creates a point cloud that can be turned into a textured 3D model; however, these models often lack realistic light behavior.

Radiance Fields vs. Traditional Methods

• Light interaction with objects is discussed; real-world lighting affects color perception based on viewing angle.

• Radiance fields capture how light travels from points in space rather than just shapes, addressing view dependency for more realistic rendering.

Differences Between Nerfs and Gaussian Splats

• While both Nerfs and Gaussian splats are forms of radiance fields, they operate differently: Nerfs use complex mathematical functions while splats exist in actual 3D space.

• The speaker emphasizes that Gaussian splats resemble point clouds but are enhanced with unique properties allowing for real-time rendering.

Characteristics of Gaussians

• A demonstration shows photorealistic scenes rendered in real time using fuzzy blobs known as Gaussians that overlap smoothly.

• These Gaussians can take various shapes (e.g., spherical or flat), making them versatile for representing different materials like glass.

Gaussian Splatting: A New Approach to 3D Representation

Understanding Gaussian Splatting

• Gaussian splats can vary in size, representing either detailed objects or large smooth surfaces. Each splat may contain millions of individual Gaussians that store information about position, rotation, shape, opacity, and view-dependent color.

• Storing every possible shade of a color for all viewing angles is impractical due to the infinite directions. Instead, Gaussian splats utilize spherical harmonics to efficiently represent colors by storing a single base color and adjusting it mathematically based on the viewing angle.

• This method allows for a vast range of colors to be represented with minimal data storage. The efficiency stems from mathematical minimalism—only essential values are stored while still achieving photorealistic results.

Practical Application of Gaussian Splatting

• To create effective Gaussian splats, capturing crisp images is crucial; any blur can compromise quality. Techniques include locking shutter speed (ideally at 1/1000), white balance, ISO settings, and focus.

• After gathering data through scanning, two additional stages are necessary before generating the final splat: tracking and processing through software like Posed Shot or Lick Field Studio. These tools automate tracking and training processes for ease of use.

Examples and Demonstrations

• The website Super Splat showcases various examples of Gaussian splats created by users. Notable instances demonstrate how well these techniques capture intricate details such as reflections and textures in everyday objects.

• One example features a chocolate item where spherical harmonics effectively change its appearance based on the viewer's angle—illustrating realistic rendering capabilities through individual Gaussians arranged like brushstrokes in art.

Advanced Capabilities of Gaussian Splatting

• Gaussian splats excel at depicting high-frequency details such as hair or leaves. For instance, scans reveal depth in animal fur that traditional photography cannot achieve.

• Transparency effects allow viewers to see structures within small creatures like bees without photographic limitations. This capability enhances understanding of complex biological forms.

Creating Splats from Renders

• Beyond real-life captures, Gaussian splatting can also be applied to rendered images by compiling multiple views into a dataset. This approach saves time compared to re-rendering scenes from different angles after initial creation.

• By using this technique with renders (e.g., chicken shawarma), creators can manipulate perspectives without extensive computational resources typically required for full re-renders.

Real-Time High Detail Rendering Techniques

Utilizing Photo Mode in Video Games

• The speaker discusses using photo mode in video games to create high-detail scans, emphasizing its effectiveness due to the removal of UI and ability to pause the game.

• A specific example is given where a character was isolated and placed into a tracked scene, showcasing innovative use of gaming technology for scanning.

Challenges in Scanning Environments

• It is noted that any missing detail from the data set will result in absent features in the final scan, highlighting the importance of comprehensive coverage during capture.

• The speaker mentions that capturing all perspectives is tedious with just a phone; thus, a 360 camera would be more efficient for thorough data collection.

Advancements with 360 Cameras

• Introduction of the Antigravity A1 drone as a solution for capturing 360-degree scans quickly and effectively, reducing time significantly compared to traditional methods.

• Features like SkyGenie and SkyPath are highlighted, allowing automated camera movements and obstacle avoidance while flying indoors.

Practical Application of Drone Technology

• The speaker describes how controlling the drone feels similar to video gaming, enhancing user experience while performing scans.

• Emphasizes efficiency by stating that what would take two full batteries on other drones can be accomplished with half a battery on this new model.

Steps for Creating Gaussian Splat Data Sets

• Outlines three essential steps: capture, tracking, and training. The focus here is on transferring footage onto a computer using Antigravity Studio for further processing.

• Discusses exporting footage as an equirectangular image for optimal results when creating data sets rather than focusing solely on aesthetic shots.

Tracking Challenges with 360 Footage

• Highlights difficulties in tracking 360 footage; only one program (Metashape) effectively handles spherical maps but requires additional scripting work.

• Mentions converting 360 footage back into regular formats necessary for tracking before proceeding to training stages.

Photogrammetry and Historical Preservation

Upgrading Old Photo Scans

• The speaker discusses the principles of photogrammetry, emphasizing that old photo scans can be upgraded to Gaussian splats if the original dataset is available.

• They mention reprocessing scans from Cerro Gordo, highlighting its potential for historical preservation in a more photorealistic manner.

Inspiration Point and Its Digital Legacy

• The speaker shares a personal story about Inspiration Point, a favorite location that burned down during wildfires in LA.

• A new scan was created after the fire, making it the only 3D representation of this landmark, ensuring its digital legacy endures even if physically destroyed.

Applications of Gaussian Splats in Visual Effects

• Gaussian splats are presented as effective tools for capturing reality and enhancing visual effects; they allow artists to manipulate scenes creatively.

• The speaker describes using Octane Render to relight scenes and add depth with particle effects, showcasing how reality can be bent visually.

Real-Time Rendering Capabilities

• Gaussian splats operate in real-time within Unreal Engine, enabling their use on LED walls for immersive experiences.

• The speaker demonstrates being at The Showroom in Los Angeles, where virtual cameras track movements seamlessly with real locations.

Exploring 4D Gaussian Splat Technology

• An interview with Jiaming, CEO of 4D Views, introduces the concept of 4D Gaussian splats as holograms resembling those seen in sci-fi films.

• They discuss unique camera angles and effects possible with this technology, such as bullet time and extreme aperture adjustments.

How Do Animated Gaussians Work in 4D?

Unique Approach to 4D Gaussian Splatting

• The speaker discusses the innovative method of animating Gaussians over time and space, contrasting it with traditional methods that create new sets of Gaussians for each frame, leading to artifacts and large file sizes.

• Each Gaussian point has a velocity and time representation, allowing them to flow continuously rather than jumping between frames. This creates a seamless animation where points can appear or disappear dynamically.

• By interpolating these animated Gaussians, any frame rate can be achieved (e.g., 1,000 FPS or even 10,000 FPS), enabling high-quality visual experiences without sacrificing performance.

Compression Techniques for Efficient Data Handling

• The continuous representation allows one moving Gaussian point to represent hundreds of frames of information. This compression reduces the data size significantly to about 30-60 megabits per second—100 times smaller than original 2D video files.

• The small file size enables playback on devices like smartphones and VR headsets (e.g., Meta Quest 3), making advanced technology accessible for everyday use.

Advantages of Volumetric Capture

• Using a volumetric capture stage instead of green screens provides perfect alpha mats and flexibility in camera angles. This allows subjects to be placed in any desired 3D environment seamlessly.

• The ability to isolate elements from backgrounds enhances the versatility of scenes created with this technology, including complex ray tracing effects.

Future Applications and Developments

• The speaker draws parallels between current technology and futuristic concepts seen in films like Star Trek, emphasizing how volumetric capture is becoming a reality today.

• Anticipated applications include Google Maps using Gaussian splats for realistic views instead of traditional Street View. Real estate platforms like Zillow may also adopt this technology for immersive property listings.

Challenges and Prospects Ahead

• While splats are easy to use now, they require many cameras and significant computing power for model training. Efforts are underway to reduce these requirements significantly within the next couple of years.

• As holograms become more prevalent alongside images and videos, the future looks promising for integrating this technology into various fields beyond entertainment.