GenAI ChatGPT Sourav Session

Introduction to GPT-2 and Its Architecture

Overview of GPT-2

• The speaker introduces the use of GPT-2 for demonstrating concepts like temperature, top P, and top K sampling.

• Different versions of GPT-2 are mentioned: large, medium, excel, with the smallest version having 124 million parameters.

• GPT-2 is described as a decoder-only transformer architecture pre-trained on a large corpus of English data in a self-supervised manner.

Training Objective

• The model was trained to predict the next word in a sentence using an automatic process that generates inputs and labels from raw text.

Course Structure and Future Sessions

Upcoming Classes

• The speaker outlines the schedule for upcoming classes: one on ritual augmented generation (RAG) and another on AI agents and APIs.

• A reminder is given to download all shared materials from Google Drive for future reference due to potential space issues.

Course Content Reflection

• The speaker emphasizes that this course only scratches the surface of AI/ML topics; self-learning is encouraged for deeper understanding.

Understanding Tokens in GPT-2

Tokenization Details

• The vocabulary size of GPT-2 is stated as 50,257 unique tokens mapped to a dimension of 1280.

Model Architecture Breakdown

• Each block in the architecture includes layer normalization followed by attention layers and MLP layers.

Decoding Strategies: Greedy Search vs. Beam Search

Greedy Search Methodology

• In greedy search, the token with the highest probability is selected as the next word based on previous tokens.

Limitations of Greedy Search

• It often leads to repetitive outputs since it does not explore diverse combinations effectively; high-probability words can mask better options.

Introduction to Beam Search

• Beam search allows exploration of multiple paths by keeping track of several hypotheses at each step, enhancing output diversity.

Implementation Details

• Parameters such as no-repeat N-Gram size can be set to avoid repetition in generated sequences while maintaining creativity.

Output Generation Examples

Generated Outputs Comparison

• Three different outputs are generated using beam search; minor differences are noted among them compared to greedy search results.

Observations on Output Quality

• While beam search improves upon greedy methods, outputs may still show similarities due to inherent limitations in exploring diverse language structures.