You Can Actually Chat With Images Now! (MiniGPT-4)

AI Advancements in Mini GPT4

In this video, the speaker discusses six cool advancements in AI, with a focus on mini GPT4. Mini GPT4 brings multi-modality to our chats and allows us to upload pictures and ask questions about them.

Multi-Modality in Chats

• Mini GPT4 brings multi-modality to our chats.

• Users can upload pictures and ask questions about them.

• The AI diagnoses the issue with an uploaded image of a plant and explains what to do next.

• The AI writes an advertisement for new mugs shown in an image.

Generating Website Code from Handwritten Text

• The AI generates website code from handwritten text.

• There is a demo available on mini gpt-4.github.io where users can play around with the technology.

Image Description

• The AI describes an astronaut standing in front of a planet with bright light in the background.

• It is not possible to determine the sex of the astronaut from the image provided.

• The AI provides a detailed description of the astronaut's spacesuit.

Overall, mini GPT4 is making significant strides towards more advanced chat capabilities by bringing multi-modality into conversations. Additionally, it has proven useful for generating website code from handwritten text and providing detailed descriptions of images.

Image Recognition AI

In this section, the speaker discusses their experience with an image recognition AI and its ability to answer questions about uploaded images.

Slow Response Time

• The AI takes a long time to answer even simple questions.

• It took at least 387 seconds to get an answer for one question.

• It usually takes around 10 minutes to answer each question.

House on a Cliff

• The second image uploaded was of a modern house on a cliff overlooking Los Angeles at sunset.

• The AI did a good job describing the house and gave keywords such as modern, hilltop, cityscape, and sunset.

• The AI was unable to determine the location or era of the house in the image.

Green Alien Image

• The third image uploaded was of a green alien standing on a rocky surface with a planetary background.

• The AI correctly identified that it was a cartoon character with exaggerated features such as large head and eyes and wearing a spacesuit.

• It correctly identified that the alien had four fingers on each hand.

Describing Images

In this section, the speaker discusses their attempt to get the AI to describe specific elements in an image.

Elements in Green Alien Image

• The speaker attempted to get the AI to identify specific colors behind the alien but it only described various shades of brown and gray for rocks and blue sky with white clouds in the background.

• The AI correctly identified that the alien appeared excited or enthusiastic while giving thumbs up gesture with one hand while holding onto rocky surface with other hand.

Texture of Alien Skin

• The AI described the skin of the alien as smooth and slightly shiny with a slight texture that suggests it may be made of a rubbery or plastic material.

• It identified the color of the skin as light green with a slightly darker shade.

Mini GPT4 and Dyno V2: Enhancing Vision Language Understanding with Advanced Large Language Models and State of the Art Computer Vision Models with Self-Supervised Learning

In this section, we learn about two computer vision tools - Mini GPT4 and Dyno V2. Mini GPT4 is a language model that can add images to its understanding in the future, while Dyno V2 is a self-supervised learning computer vision tool that can map the depth of videos.

Mini GPT4

• Mini GPT4 is a language model that enhances vision language understanding with advanced large language models.

• It has a GitHub page where it can be installed, but it requires a beefy graphics card.

• It gives us an idea of what GPT4 might be like in the future when it starts adding multimodal capabilities such as images.

Dyno V2

• Dyno V2 is a state-of-the-art computer vision model with self-supervised learning.

• It maps the depth of videos and delivers strong performance without requiring fine-tuning.

• It learns from any collection of images and can learn features such as depth estimation that current standard approaches cannot.

• Meta AI has open-sourced it, allowing people to use it, build off of it, and iterate off of it.

Meta's Animated Drawings Research

In this section, we learn about Meta's animated drawings research which allows users to animate children's drawings.

• The animated drawings research allows users to take children's drawings and animate them.

• The code for this research has been open-sourced on GitHub so users can download it and run it on their computers.

• Users can upload photos to sketch.metademolab.com and animate them.

• The website provides different animations, but it does not seem to allow users to upload their own videos yet.

Animated Drawings and Apple's Facelift Neural 3D Relightable Faces

In this section, the speaker talks about two new technologies. The first is called Animated Drawings, which allows users to animate their children's drawings. The second technology is Apple's Facelift Neural 3D Relightable Faces, which can map depth onto a single image and make it look 3D.

Animated Drawings

• Animated Drawings is a fun way to animate your kids' drawings.

• It took some effort to animate an image made in Stable Diffusion that was supposed to look like the speaker as Buzz Lightyear.

• Over time, this technology will be used for more realistic imagery.

Apple's Facelift Neural 3D Relightable Faces

• Apple announced Facelift Neural 3D Relightable Faces, which maps depth onto a single image and makes it look 3D.

• From an AI nerd's point of view, it doesn't introduce anything super exciting but it leaves room for improvement.

• The speaker believes that Apple will move into the game in a much bigger way in the coming weeks and months.

Adobe Firefly with Video

In this section, the speaker talks about Adobe Firefly with Video. This tool generates music and sound effects for videos using AI. It also analyzes words spoken in videos and finds b-roll footage to go along with them.

Adobe Firefly with Video

• Adobe Firefly with Video generates music and sound effects for videos using AI.

• It analyzes words spoken in videos and finds b-roll footage to go along with them.

• It can take a script and create a storyboard from it, as well as generate AI images for that storyboard.

DaVinci Resolve 18.5

In this section, the speaker talks about Blackmagic Design's DaVinci Resolve 18.5. The new features inside of 18.5 are AI features.

DaVinci Resolve 18.5

• DaVinci Resolve 18.5 has new AI features.

• There is no further information provided in the transcript about these new features.

AI News and Tools Overview

In this video, the speaker provides an overview of recent developments in the field of AI. He also shares some useful tools and resources for those interested in learning more about AI.

Recent Developments in AI

• The speaker discusses recent developments in the field of AI, including advancements in natural language processing and computer vision.

• He encourages viewers to follow him on Twitter (@MrEFlow) for real-time updates on new developments.

• The speaker invites viewers to sign up for his free newsletter at futuretools.io, where he curates a list of cool AI tools and sends out a weekly email with news updates.

Useful Tools and Resources

• Futuretools.io is a website where the speaker curates a list of useful AI tools that can be used for personal or business purposes.

• Viewers can sign up for the free newsletter at futuretools.io to receive weekly updates on new tools and news related to AI.

Conclusion

• The speaker encourages viewers to subscribe to his channel if they are interested in staying up-to-date with the latest news and research related to AI.