De 5 Tera a 25 Giga | Compressão de Dados

How Does Data Compression Work?

Introduction to Data Compression

• Fabio Akita introduces the concept of data compression, emphasizing that even non-programmers are familiar with "zipping" files.

• He encourages viewers to think critically about how data compression works, particularly in video formats for high-definition viewing.

Understanding Video Resolution

• Akita explains 4K resolution as four times the pixel density of Full HD (1080p), detailing the pixel dimensions: 3840 pixels vertically and 2160 pixels horizontally.

• He highlights that doubling both height and width results in quadrupling the total number of pixels, illustrating basic geometry principles.

Color Representation in Digital Media

• The discussion shifts to color representation, where he notes that standard colors use 24 bits (3 bytes), allowing for 256 shades per RGB component.

• A single 4K image requires approximately 24 megabytes due to its pixel count and color depth, leading into a discussion on video frame rates.

Bandwidth Requirements for High Definition Video

• Akita calculates that streaming a video at 30 frames per second would require over 711 megabytes per second, which is impractical for storage without compression.

• He poses a challenge regarding how programmers can manage such large amounts of data effectively through compression techniques.

Simplifying Video Compression Concepts

• The speaker acknowledges the complexity of video compression but aims to simplify it for better understanding among viewers.

• He emphasizes that all digital media ultimately consists of binary data (bits), reinforcing the idea that programming involves manipulating these bits efficiently.

Exploring Color Representations Beyond RGB

• Akita critiques static thinking in color representation, suggesting there are more efficient methods than traditional RGB encoding.

Understanding Color Science and YUV Format

Introduction to Color Representation

• The discussion begins with the concept of color science, emphasizing the representation of colors as distances in a three-dimensional space rather than mere mixtures of paints.

• Key color spaces such as sRGB, Adobe RGB, and DCI-P3 are introduced, highlighting their roles as subsets of theoretical color possibilities known as color gamut.

Transition from RGB to YUV

• The historical context of YUV is explained, originating from black-and-white television broadcasting where luma (Y) and chroma (U and V) were first conceptualized.

• A visual explanation is provided on how RGB components can be separated into individual channels for red, green, and blue before converting to YUV format.

Practical Applications and Benefits

• The practical implications of using YUV over RGB are discussed, particularly in terms of video connections like RCA cables versus HDMI.

• It is noted that while both formats use 24-bits per pixel, the conversion process allows for more efficient data handling due to human sensitivity to luma changes compared to chroma.

Understanding Chroma Subsampling

• The importance of separating luma from chroma in YUV is highlighted; this separation enhances clarity in brightness while reducing detail in color information.

• An example illustrates how connecting a computer to an older TV can result in blurred text due to the limitations of the TV's support for different formats.

Compression Techniques in Video Formats

• The concept of chroma subsampling is introduced with examples like 4:4:4 and 4:2:0 formats that optimize data storage without significant perceptual loss.

• Details about how reducing chroma samples leads to substantial data savings while maintaining image quality are discussed.

Conclusion on Standards and Usage

• Clarification on standards such as YCbCr versus older formats like YPbPr emphasizes the evolution within video technology.

Understanding Video Compression Techniques

Chroma Subsampling and Its Importance

• Chroma subsampling reduces the amount of color data needed for video representation, making it ideal for broadcasting or streaming while maintaining quality.

• The process involves decreasing chroma samples (color information) while preserving brightness samples, allowing for a visually similar image with less data.

• By reducing the size of video files (e.g., from 5TB to 3TB), we can manage storage better, although even 3TB is still substantial for long videos.

Understanding Downsampling and Lossy Compression

• Downsampling refers to reducing chroma sample numbers to save bits by discarding information that is less perceptible to human vision.

• This technique results in lossy compression, meaning original details cannot be recovered once simplified; it's crucial to understand this limitation.

Digital Representation of Images and Sound

• Both images and sounds are represented digitally through discrete samples; more pixels lead to higher resolution in visuals, while sound waves are similarly discretized.

• The Fourier Transform is essential in audio processing, converting continuous sound waves into discrete representations suitable for digital manipulation.

Frame Rate and Audio Sampling Rates

• Video smoothness relies on frame rate; typically, 30 frames per second (fps) suffices for films, whereas gaming requires at least 60 fps for optimal experience.

• Audio quality standards include CD-quality music at 44.1 kHz sampling rate with 16-bit depth; high-fidelity music demands even higher rates like 192 kHz with 24-bit depth.

Historical Context of Audio Formats

• A standard CD holds about 650 MB of data; an album's size can quickly fill this capacity due to high-quality audio formats.

• Early gaming consoles struggled with compressed audio formats like MP3 due to hardware limitations, leading them to use uncompressed audio tracks instead.

Transitioning from Analog to Digital

Understanding the Fast Fourier Transform

The Importance of Fast Fourier Transform (FFT)

• The discrete Fourier transform (DFT) is computationally intensive, particularly on older hardware from the 1980s.

• In the 1960s, a significant breakthrough led to the development of the Fast Fourier Transform (FFT), reducing complexity from quadratic to logarithmic (N.log N).

• FFT is regarded as one of the most important numerical algorithms ever created, emphasizing its relevance in digital signal processing.

Visualizing Fourier Transforms

• For a better visual understanding of Fourier transforms, it is recommended to watch videos by Grant from 3Blue1Brown, who provides animated explanations.

• Groups of pixels in images can be treated similarly to frequency series, allowing for analysis through transformations.

Discrete Cosine Transform (DCT)

• DCT serves as an alternative to DFT and FFT when working with images since it only requires real numbers instead of complex numbers.

• In image processing, DCT decomposes luma and chroma components into sums of cosine functions at various frequencies.

Image Processing Techniques

• Images are divided into blocks of 8x8 pixels for processing; this "divide and conquer" strategy simplifies handling large data sets.

• Each pixel's intensity is adjusted before applying DCT, transforming values from a range of 0–255 to -128–127 for effective computation.

Coefficients and Frequency Components

• A new matrix representing weights for each frequency function is generated using DCT; these coefficients help reconstruct the original image.

• Coefficients corresponding to lower frequencies contribute more significantly to reconstruction than those associated with higher frequencies.

Understanding Frequency Ranges

• High-frequency components are less distinguishable visually but play a role in overall image quality; they correspond to details that may not be perceived by human eyes.

Understanding Image Compression Techniques

The Process of Color Downsampling and Transformation

• After years of research, the process begins with color downsampling, where chroma is reduced across three channels for an 8x8 pixel block. This step is repeated for all blocks in the image.

• The next phase involves saving images in formats like JPEG, where users select quality levels (e.g., 80%) that impact detail retention during compression.

Quantization Phase Explained

• This phase is known as quantization, which involves dividing coefficients from a matrix using a specific table. Numbers closer to the top left are smaller, while those at the bottom right are larger.

• Each quality level corresponds to a different division table; lower quality results in larger numbers used for division, leading to smaller or zeroed coefficients after rounding.

Rearranging Pixels and Preparing for Compression

• The quantization process leads to many coefficients becoming zero, which contributes little to image reconstruction. High-frequency details are less distinguishable by human eyes.

• Pixels are rearranged in a zigzag pattern instead of linearly. This method clusters zeros together and prepares data for effective compression.

Introduction to Compression Algorithms

• Up until this point, no actual compression has occurred; only preparations have been made. The zigzag arrangement reveals sequences of zeros that can be compressed.

• A simple example illustrates Run Length Encoding (RLE), where repeated characters (e.g., "AABCDDDDEFFFFFGGGH") can be represented more compactly (e.g., "A2B1D4E1F5G3H"), achieving significant size reduction.

Advanced Compression Techniques: Huffman Coding

• RLE provides initial gains but may falter with high entropy strings lacking repetition. More sophisticated methods like Huffman coding utilize frequency tables.

• By counting character frequencies and constructing a binary tree based on these frequencies, we create an efficient encoding scheme that reduces overall data size without loss of information.

Understanding Character Representation in Computing

Understanding Binary Encoding and Compression Techniques

The Basics of Character Encoding

• The uppercase letter 'A' in English corresponds to the hexadecimal code 0041, while 'B' is 0042. This pattern continues for other letters.

• In binary, these characters are represented as sequences of zeros and ones; for example, 'A' is 0100 0001 in binary.

• Each character occupies a minimum of 8 bits when using UTF-8 encoding, meaning a string like "AABC" would take up 18 bytes.

Alternative Encoding with Frequency Trees

• To optimize space, a frequency tree can be created to encode characters differently than UTF-8.

• Starting from the root node of the tree, left branches represent zero and right branches represent one. For instance, 'A' could be encoded as 0000.

Efficiency Gains Through Custom Encoding

• Using this method reduces the size significantly; four occurrences of 'D', which would normally take up 4 bytes in UTF-8, can now be represented with just 2 bits each.

• Letters that appear more frequently (like 'D' and 'F') require fewer bits compared to less common letters (like 'A' or 'C'), demonstrating efficient use of space.

Compression Results

• The original string took up 18 bytes but was compressed down to only 60 bits (12 bytes), achieving a reduction of two-thirds in size.

Practical Applications of Huffman Coding

• In larger texts such as source code or JSON files, compression techniques like Run Length Encoding combined with Huffman coding can lead to significant space savings.

Understanding Decompression Process

• Decompression involves traversing back through the frequency tree based on the binary codes to reconstruct the original text accurately.

Image Compression Techniques Explained

• Similar principles apply in image processing where chroma subsampling and discrete cosine transform (DCT) are used before applying Run Length and Huffman coding for effective compression.

Real-world Impact on File Sizes

• A raw uncompressed image file could reach around 20 megabytes; however, after applying various compression techniques including Huffman coding, it can shrink down to approximately 900 kilobytes—over a twenty-fold reduction.

Conclusion: JPEG Compression Insights

Understanding JPEG Limitations

The Lossy Nature of JPEG Compression

• JPEG format is inherently lossy, meaning it discards information during compression, making it impossible to recover original details.

• Repeatedly editing and saving a JPEG results in further loss of definition, particularly affecting images with sharp edges like drawings or comics.

• Low-quality JPEGs saved at low quantization levels exhibit visible artifacts due to the grid-like processing of image blocks.

Comparison with Other Formats

• Professional photographers prefer RAW formats over JPEG for better color information retention, allowing extensive post-editing capabilities.

• BMP and TIFF formats are similar to RAW as they retain all data without significant compression; however, TIFF applies minimal lossless compression.

The Evolution of GIF Format

Origins and Characteristics

• GIF was developed by CompuServe in the late 1980s to facilitate image sharing on early internet platforms, replacing larger bitmap formats.

• GIF limits color representation to 256 colors (8 bits), significantly reducing the color palette from over 16 million available in other formats.

Functionality and Compression Techniques

• GIF allows for transparency using a single color but lacks true alpha channel support; this limitation affects visual quality.

• The format supports multiple images within one file for animation but requires careful management of the limited color palette across frames.

Understanding LZW Compression Algorithm

Mechanism of LZW Compression

• GIF employs LZW (Lempel-Ziv-Welch), an algorithm that compresses data by replacing repeated strings with pointers indicating their previous occurrences.

• An example illustrates how repeating phrases can be compressed by referencing earlier instances instead of rewriting them entirely.

Efficiency Insights

Understanding Compression Algorithms and Their Evolution

The Challenge of Memory in Early Compression

• The need for pointers to reference text in compression algorithms requires significant memory, which was expensive in the late 1970s. Early microcomputers had limited RAM (around 32 kilobytes).

• A sliding window technique is introduced, allowing only a portion of the text to be held in memory while decompressing. This limits pointer references to within this smaller window.

• Increasing the size of the sliding window improves compression but also increases memory usage, creating a trade-off between efficiency and resource consumption.

Historical Context: ZIP and Its Impact

• Phil Katz's PKZIP emerged as a major success in the 1980s, utilizing the LZ77 algorithm. It became widely adopted after Microsoft’s endorsement.

• Various compression formats like ZIP, RAR, and 7zip evolved from Lempel-Ziv algorithms, each with unique strategies for data handling and compression efficiency.

Advancements in Compression Techniques

• The development of formats such as RAR by Eugene Roschal and 7zip by Igor Pavlov showcased significant advancements in file compression techniques during the early '90s.

• Many modern formats combine Huffman coding with variations of Lempel-Ziv algorithms to enhance data compression effectiveness.

Gzip: A Standard for Web Data

• Gzip became prevalent on web servers since its introduction in the early '90s. It uses the deflate algorithm—a combination of LZ77 and Huffman coding.

• Libraries supporting these algorithms are available across programming languages, making them foundational for lossless data compression.

The GIF Controversy and PNG Emergence

• GIF format was created using LZW algorithm but faced patent issues when Unisys claimed licensing fees from users in 1994.

The Evolution of Image Formats and Video Compression

The Adoption of PNG and GIF Usage

• In 1996, the W3C officially adopted the PNG format, which is superior to GIF but lacks animation support.

• Despite its advantages, GIF remains popular for animated memes due to the late adoption of APNG (Animated PNG), created by Mozilla in 2008 and supported by Chromium only from 2017.

• PNG supports a wide color range (up to 32 bits) and offers better transparency than GIF, making it ideal for web development.

Compression Techniques and Format Comparisons

• JPEG compresses images at a ratio of about 20:1, while PNG achieves around 4:1 compression; WebP offers marginally better compression than PNG.

• For video files, uncompressed formats can be excessively large; converting RGB to YUV reduces size but still results in massive files (e.g., down to 3 terabytes).

Strategies for Video File Size Reduction

• Compressing each frame as JPEG can reduce file sizes significantly—from 5 terabytes down to approximately 256 gigabytes.

• However, even with this reduction, downloading such a file would take an impractical amount of time on standard internet connections.

Streaming Requirements and Video Codecs

• To stream effectively without buffering issues, the maximum file size must be less than approximately 42 gigabytes for a two-hour movie at optimal speeds.

• Motion JPEG (M-JPEG), used in older video formats like MPEG1/2/4, compresses frames individually using techniques similar to JPEG.

Understanding Compression Algorithms

• The terminology surrounding image and video codecs can be confusing due to multiple names for similar technologies across different companies.

Understanding Video Compression: H.264 and Beyond

The Basics of Video Compression

• A codec like H.264 begins by recording a keyframe (interframe), which is a complete image, followed by intraframes that capture only the changes (deltas).

• The concept is similar to Git commits, where only modified sections are saved rather than the entire file. Keyframes are essential for distinguishing between full images and deltas.

• Typically, a 2-hour movie encoded in H.264 can reduce from 5 terabytes to about 20-25 gigabytes while maintaining good quality, suitable for streaming.

Streaming Capabilities

• With a connection speed of 6 Mbps, it would take approximately 1 hour and 18 minutes to download such a movie, allowing for streaming as it downloads.

• The final size of videos varies based on content type; animated films compress better than action films with shaky cameras due to delta intraframe challenges.

Advanced Compression Techniques

• Modern codecs utilize optimizations like motion compensation to enhance compression efficiency based on scene dynamics.

• H.264 was officially released around 2003 after years of development; its successor, H.265 (HEVC), launched around 2013 and offers nearly 50% smaller files with comparable quality but requires more processing power.

Editing Considerations

• While H.264 and H.265 are great for consumption due to their efficient compression methods, they are not ideal for editing because they use chroma downsampling.

• Professional video editing should utilize lossless formats like Apple ProRes or DNxHR instead of lossy codecs like H.264 or H.265 to retain color information during edits.

Practical Implications for Video Production

• For instance, recording in DNxHR can yield large file sizes (e.g., ~120 GB for two hours), but provides higher color fidelity compared to compressed formats.

• Understanding the differences between formats like ProRes 4444 versus ProRes 422 helps clarify how chroma information is preserved or lost during compression.

Non-linear Editing Challenges

• In non-linear editing software (NLE), each frame request may require reconstructing from intraframes back to keyframes, complicating the editing process with lossy codecs.

• Professional codecs store only keyframes similar to Motion JPEG (M-JPEG), making them more efficient for editing despite larger file sizes since no reconstruction is needed.

Understanding Data Compression Techniques

The Importance of File Formats in Video Editing

• H.264 format complicates video editing due to higher processing and memory requirements, making it less efficient for users.

• Emphasizes the advantage of using external SSDs via Thunderbolt for storage, as disk space is relatively inexpensive.

• Mentions previous discussions on NAS (Network Attached Storage) and its relevance to data management.

Audio Compression Formats

• Discusses lossless audio formats like FLAC, WAV, and AIFF, paralleling them with RAW image formats used in photography.

• Highlights lossy audio compression methods such as MP3 and AAC, commonly paired with video formats like H.265 or Blu-Ray.

Sampling and Data Simplification

• Explains the concept of sampling in both video (frames per second) and audio (samples per second), crucial for understanding data compression.

• Describes how resolution adjustments and frequency transformations are employed to achieve high compression while minimizing information loss.

File System Compression Considerations

• Introduces NTFS file system's ability to compress files automatically but warns about potential performance overhead during read/write operations.

• Suggests that investing in an affordable external USB hard drive is more practical than compromising system speed for minor disk space savings.

Exploring Data Compression Algorithms

• The video's goal is to provide a foundational understanding of data compression concepts applicable across various programming languages.