I try the tech that WILL replace CG one day
Introduction
In this section, the speaker introduces the topic of neural radiance fields and how they can change filmmaking forever.
Neural Radiance Fields
- The speaker introduces a new technology called neural radiance fields (Nerf).
- Nerfs have the potential to change filmmaking forever.
- The speaker has made a career out of experimenting with new technologies to create short films.
- Photo scanning has always been just a shortcut on the road to rendering 3D scenes.
Limitations of Photo Scanning
In this section, the speaker discusses the limitations of photo scanning and why it is not always possible to scan certain objects.
Challenges with Photo Scanning
- Some things are simply impossible to scan due to reflections or other factors.
- Photo scanning relies on pixels being consistent, which can be challenging when dealing with reflective surfaces like chrome balls.
- When the math behind photo scanning isn't perfect, unusable garbage is produced.
Introducing Neural Radiance Fields
In this section, the speaker explains how neural radiance fields work and demonstrates their potential for creating photorealistic images.
Neural Radiance Fields in Action
- The speaker takes photos and processes them into a neural radiance field instead of using photogrammetry.
- Despite some janky results, the ball has accurate reflections and transparency in the bottle.
- Paul Franklin responds positively to these results, seeing a VFX revolution on the horizon.
- Reflections are key to achieving photorealism in CGI.
Achieving Photorealism with Reflections
In this section, the speaker explains how reflections are key to achieving photorealism in CGI.
The Importance of Reflections
- Photorealism is achieved by making an image so utterly realistic that it convinces your brain that it is real.
- Reflections are the secret to photorealism, as every object reflects light from every other object.
- Everything we see is just light that has reached our eyeballs, and unless you're looking directly at a source of light, it first has to reflect off of something along the way.
Neural Rendering and Scanning Technology
In this section, the speaker discusses how neural rendering technology is used to create 3D scans that look like real-life videos. They also introduce a new scanning technology that allows for easy background replacement and portal creation.
Neural Rendering and Deep Fakes
- Nerfs rely on neural rendering, which can learn the color of every point in a 3D space depending on where it's viewed from.
- The ability to replicate reality quickly, easily, and cheaply is the entire point of Nerf technology.
- The speaker briefly mentions a sponsored movie called "Bones and All" about cannibalism.
New Scanning Technology
- The speaker introduces a new scanning technology that allows for easy background replacement and portal creation.
- Users take multiple photos with their phone app, upload them to the server, and receive a Nerf scan within minutes.
- The website version of the app allows users to import their own camera animation separately.
Background Replacement and Portal Creation
- One use for this scanning technology is simple background replacement using green screens.
- Another use is creating portals by 3D tracking shots and applying them to any Radiance field desired.
- The speaker shares some rendered shots they created using this technology.
The Power of Nerf
In this section, the speaker discusses how Nerf can be used to create photorealistic scenes and how it compares to other methods.
Using Nerf for Photorealism
- Reflections in a scene can be subtle but are important for creating photorealistic scenes.
- Nerf can create photorealistic scenes that look like footage.
- To achieve the best results with Nerf, capture both the plate and the Nerf outdoors at golden hour with the same camera and lighting conditions.
Comparing Red Camera vs. Webcam Scans
- A scan from a red camera will have a cinematic look while a scan from a webcam will have a cheap webcam look.
- Compression, dynamic range, and lens quality all impact finished scans.
Using Nerf for Complex Camera Moves
- The magic of Nerfs is that it can do complex camera moves even if they are not smooth.
- Radiance field scans allow for picking and choosing camera moves after the fact.
The Value of Neural Render
- While geometry can be extracted from a Nerf scan, reflectiveness is lost in the process.
- The most valuable part of a Nerf is its neural render which allows for treating it like video footage rather than just extracting textures and models.
Introduction
In this section, the speaker talks about the potential of new technology and how it can be used to create impressive video effects.
The Potential of New Technology
- The speaker believes that it's just a matter of time before no one will be able to tell the difference between a Nerf and actual video.
- The speaker wants to demonstrate how crazy useful this new tech can be by doing an Inception Shot.
Recording a Quick Shot
In this section, the speaker records a quick shot to get a rebel scan.
Getting a Rebel Scan
- The rebel scans have gotten away from us this time because of all the chaotic stuff going on here.
- If I wanted to actually shoot a video it would have been too hard but first time I was able to get a scan so in theory I can go back and actually do all the production filmmaking after the fact.
Conclusion
In this section, the speaker concludes by encouraging artists to push this Tech as far as it will go.
Pushing This Tech Further
- It's our job as artists to push this Tech as far as it will go. None of these shots in this video were like production ready but I hope that maybe they inspire some sort of nugget of an idea in you so that you can get into this stuff as well and push the bounds too.
- There are tons of options available for using new technology, including Luma, Nvidia's instant nerfing plugin for Blender, and Polycam's partnership with Nerf Studio.
- Anyone who is not yet convinced of the potential of this new technology is encouraged to just look two papers down.