24. Object Storage - Everything You Need to Know: Part - 2

Understanding Multipart Uploads for Large Files

Challenges with Single PUT Requests

• A single PUT request has a hard limit of 5GB, which is enforced by the API and not by backend specifications.

• Large files like videos or CAD files require different architectural approaches due to their size limitations.

• If an upload fails (e.g., due to Wi-Fi issues), it cannot be resumed; users must restart from zero, leading to inefficiencies.

• Throughput limitations arise from TCP connections, which cannot fully utilize network bandwidth, especially during packet loss.

• Download managers improve speeds by opening multiple connections for parallel downloads, mitigating throughput issues.

User Experience Concerns

• Single PUT requests provide rough progress tracking and lack features like pause or resume during uploads.

Introduction to Multipart Upload

Step 1: Create Multipart Upload

• Initiate a multipart upload by providing the object details; this returns an upload ID that acts as a staging area for parts.

Step 2: Upload Parts

• Each part can be uploaded independently via HTTP requests in any order, allowing retries without affecting other parts.

Step 3: Complete Multipart Upload

• Finalize the upload by sending the upload ID and part information; this operation is fast as it only records metadata rather than combining data physically.

Limitations of Multipart Upload

Hard Limits on Parts

• There’s a cap of 10,000 parts per multipart upload; choosing incorrect part sizes can lead to failed uploads at high costs.

Dynamic Part Sizing Strategy

• Calculate part sizes dynamically based on file size while ensuring they remain manageable for retries.

ETag Considerations in Multipart Uploads

Integrity Checks with ETags

• The ETag returned after multipart uploads differs from standard MD5 hashes; it reflects individual part hashes instead of overall content integrity.

Handling Incomplete Multipart Uploads

Lifecycle Management

• Unfinished uploads consume storage and incur costs indefinitely; implementing lifecycle rules can mitigate these expenses effectively.

Architecture for Efficient File Uploading

Workflow Overview

• The browser initiates an API call to start the multipart upload process, including file details and authentication checks.

Backend Processing Steps

• The backend computes ideal part sizes before creating a multipart upload entry in the database with a pending status.

Parallel Requests from Browser

• The browser sends multiple concurrent requests (4–6 recommended), uploading each part directly to the bucket without server relay.

Optimizing File Downloads

Direct Access vs. Proxying Downloads

• Avoid routing downloads through your application server as it consumes resources unnecessarily and slows down transfer rates.

Recommended Download Patterns

Public Content Delivery

• For public assets like images or CSS files, make buckets public or use CDNs for efficient access without compromising security.

Pre-Signed URLs for Secure Access

• Use pre-signed GET requests for secure access control but be aware of caching implications due to unique URL generation per request.

Caching Strategies with Pre-Signed URLs

• Adjust expiry times on pre-signed URLs to enhance cache hit rates while balancing security needs.

Understanding CDN and Signed URLs

The Role of CDNs in Authorization

• CDNs can handle authorization through signed URLs or cookies, allowing the process to occur at the edge rather than relying on backend servers.

• This method is particularly beneficial for video streaming, as it avoids the need for backend servers to sign numerous URLs during a session.

Range Requests Explained

• Range requests allow clients to request specific byte ranges from a file instead of downloading the entire object, improving efficiency.

• When a client sends an invalid range request, the server responds with status 416 (Range Not Satisfiable), indicating an error in the requested range.

Combining HTTP and Object Storage

Benefits of HTTP Support in Object Storage

• The combination of HTTP's capability for partial content retrieval and object storage's support for ranged objects unlocks new functionalities.

• For instance, when seeking a specific point in a video, browsers can skip ahead without downloading unnecessary data by calculating byte offsets.

Resumable Downloads and Parallel Connections

Mechanisms Behind Resumable Downloads

• Download managers utilize range headers to resume downloads from where they left off after interruptions like network failures.

• Parallel downloads enhance speed by opening multiple connections for different ranges of the same object.

Demonstrating Range Requests

Practical Application of Range Requests

• A terminal command example shows how requesting specific bytes results in status 206 (Partial Content), confirming successful range requests.

• By examining only part of a file (e.g., first eight bytes), one can identify file types without needing to download entire files.

Adaptive Bitrate Streaming Techniques

Video Streaming Architecture Insights

• Streaming platforms use adaptive bitrate techniques to serve different quality versions based on user connection speeds.

• This involves transcoding videos into various qualities and segmenting them into smaller parts managed via manifest files.

Cost Considerations in Object Storage

Pricing Models Impacting Architecture Choices

• Costs associated with object storage include storage fees per gigabyte, operation costs per request, and egress charges for data leaving cloud providers.

• Egress costs are significant; some providers like Cloudflare R2 offer free egress which can be advantageous for high-volume applications.

Recap on Object Storage Principles

Key Takeaways About Object Storage

• Object storage scales effectively by sacrificing mutability features found in traditional file systems while providing durability through erasure coding.

• Objects consist of keys, bytes, and metadata within a flat namespace; effective management requires careful key generation and metadata handling.