Using Ollama To Build a FULLY LOCAL "ChatGPT Clone"

Building Chat GPT with Open-Source Models using Olama

In this section, the speaker introduces Olama, a tool for running large language models on your computer and building applications on top of them. The speaker explains how to download and set up Olama, as well as showcases its capabilities.

Getting Started with Olama

• Download Olama from the official website.

• Currently available for Mac OS and Linux, with a Windows version coming soon.

• Windows users can potentially use WSL (Windows Subsystem for Linux) to run Olama.

• Open the downloaded file to install Olama.

• Once installed, an icon will appear in the taskbar.

Available Models in Olama

• Visit the "Models" section on the Olama website to see the available open-source models.

• Popular models include Code Llama, Llama 2, Mistol, Zephyr, Falcon, and Dolphin 2.2.

Running Models with Olama

• To run a model through the command line:

• Type olama run <model_name> in the terminal (e.g., olama run nral).

• If the model is not already downloaded, it will be automatically downloaded.

• Multiple models can be run simultaneously:

• Open multiple windows of Olama and run different models in each window.

• Models can queue up and run sequentially.

Use Cases for Multiple Models

• Having multiple models allows for dispatching different tasks to appropriate models.

• Useful for Autogen-like scenarios where different models handle specific tasks sequentially.

Customizing Model Behavior

• Create a model file using Visual Studio Code or any text editor:

• Start with from <model_name> (e.g., from llama2) to specify which model to use.

• Set parameters like temperature and system prompt in the model file.

• Use the olama create <model_name> command to create a model profile using the model file.

• Run the customized model with olama run <model_name>.

Olama Integrations

• Olama offers various integrations, including web and desktop UIs, terminal integrations, libraries like Lang Chain and Llama Index, as well as extensions and plugins.

• These integrations make it easy to build applications on top of Olama.

Conclusion

Olama is a powerful tool for running open-source language models on your computer. It allows for running multiple models simultaneously, customizing their behavior, and integrating them into different applications.

Generating a Completion

In this section, the speaker explains how to generate a completion using Python. The necessary libraries are imported, and the URL for the API is set.

• Import the requests and json libraries.

• Set the URL to "Local Host" with port 11434.

• Define headers and data for the API request.

• Use the prompt "Why is the sky blue?" for testing.

• Make a POST request to the URL with headers and data.

• Print the response if it's successful (status code 200), otherwise print an error message.

Streaming JSON Objects

The speaker explores how to handle streaming JSON objects in the response.

• The initial attempt resulted in streamed responses with multiple pieces of information.

• According to documentation, a stream of JSON objects is returned by default.

• To disable streaming, add stream=false to the request parameters.

Extracting Response from JSON

The speaker modifies code to extract only the desired response from JSON.

• Changes made include retrieving response text, loading it as JSON, parsing it, and extracting the actual response from model-generated JSON.

• The modified code successfully prints only the answer without additional information.

Adding Gradio Front End

The speaker adds a Gradio front end for user interaction in a browser.

• Code is pasted into Gradio framework to create an interface for user input and model-generated responses.

• A function called generate_response is created to handle user prompts and return model responses.

• Gradio interface is launched with input prompt and output response fields.

Testing the Gradio Interface

The speaker tests the Gradio interface by interacting with the model.

• The Gradio interface is opened in a browser.

• A joke prompt is entered, and the model responds with a joke.

• The interaction demonstrates successful communication between user and model through the Gradio interface.

Adding Conversation History

The speaker enhances the model by adding conversation history to maintain context.

• To enable conversation history, an array called conversation_history is created.

• When generating a response, the conversation history is appended with each prompt.

• The full conversation history is joined with newline characters and passed as input to the model.

• After receiving a response, it is added to the conversation history before returning it.

Testing Conversation History

The speaker tests if conversation history allows for continuity in conversations.

• A series of prompts are given to test if the model retains previous messages in its responses.

• The model successfully incorporates previous messages into its generated responses.