23. Object Storage - Everything You Need to Know: Part -1

Implementing File Uploads in Backend Systems

Introduction to File Upload Requirements

• The discussion begins with a review of the backend system's capabilities, including authentication and database management.

• A new requirement arises: users need to upload files, specifically for updating their profile pictures.

Initial Approach to File Handling

• The intuitive method involves receiving files via form data from the frontend and saving them in the local file system.

• After storing the file, it is essential to record its path in the database for future retrieval.

Problems with Naive Implementation

• This naive approach may seem straightforward but has significant flaws that will be explored later.

• Users can successfully retrieve their uploaded files if everything works as intended.

Challenges with File Size

• Profile picture uploads typically range from 300 KB to 10 MB; however, larger files pose challenges.

• Introducing video uploads complicates matters due to their potential size, ranging from a few megabytes to several gigabytes.

Memory Management Issues

• A scenario is presented where a user attempts to upload a 4GB video, leading to memory exhaustion on a server instance with only 4GB of RAM.

• When the server runs out of memory while processing this request, it results in container failure and loss of all stored files.

Ephemeral Storage Concerns

• Containers are ephemeral; upon restart, any locally stored files are lost permanently.

• With multiple instances running behind load balancers, accessing uploaded files becomes problematic as they may not exist on every instance.

Understanding Object Storage Solutions

Misconceptions About File Ownership

• The assumption that an application owns its uploaded files leads to critical failures when scaling up.

Transitioning Towards Object Storage

• Files differ significantly from typical JSON data handled by servers; they require distinct storage solutions like object storage.

Exploring Object Storage Benefits

Overview of Object Storage

• Object storage addresses issues related to file handling by utilizing pre-signed URLs for secure uploads and downloads.

Handling Large Files Efficiently

• Techniques such as multipart uploads and resumable uploads are discussed for managing large file transfers effectively.

Identifying Key Problems Addressed by Object Storage

Problematic Nature of Ephemeral Disk Space

• The ephemeral nature of service disks means data is lost upon container restarts unless persistent volumes are used.

Horizontal Scaling Limitations

When using multiple server instances behind load balancers, routing requests can lead to missing files if not managed correctly.

Fixed Disk Size Challenges

• Disk capacity increases require downtime since they cannot be expanded horizontally like server instances.

Durability Concerns

• Single disk failures can result in total data loss without proper replication strategies in place.

Availability vs Durability Distinction

• Availability refers to immediate access while durability ensures long-term existence through replication practices.

Download Performance Issues

• Serving large files ties up server resources unnecessarily; content delivery networks (CDNs), designed for this purpose, should be utilized instead.

Transactional Integrity Between Files and Database Rows

• Lack of transactional control between file systems and databases can lead to orphaned or inaccessible data under certain failure conditions.

Understanding File System Operations

Atomic Operations in File Systems

• Rename operations must be atomic to prevent system failures; they should either succeed completely or fail without leaving a corrupted state.

• Partial writes need consistent visibility; for example, editors like VS Code create temporary files during edits to ensure users only see complete versions of files.

Coordination Between Processes

• When multiple applications write to the same file, locking mechanisms are necessary to avoid conflicts, especially in log files.

• Network file systems (e.g., NFS) exist to manage these coordination issues but come with their own trade-offs.

The Concept of Object Storage

Minimal Interface for Scalability

• Object storage focuses on providing a minimal interface with just four operations: put, get, delete, and list objects by name.

• Objects in storage cannot be modified in place; any change requires downloading the entire object and re-uploading it.

Characteristics of Object Storage

• There are no real directories or folders; what appears as folders is merely an illusion created by naming conventions using prefixes.

• Renaming an object involves copying it to a new name and deleting the old one, incurring costs equivalent to the object's size.

Trade-offs of Using Object Storage

Performance Considerations

• Latency in object storage is higher than traditional file systems due to HTTP requests crossing network boundaries.

Advantages Despite Limitations

• The lack of in-place modification eliminates many distributed systems problems related to coordination and consistency.

• No hierarchical structure means no trees need maintenance or locking, simplifying scalability across global servers.

Addressability and Compatibility

Universal Access via HTTP

• Every object can be accessed directly via HTTP, allowing various devices and applications seamless interaction with object storage.

Trade-offs Beyond Storage

Broader Implications in Distributed Systems

• The trade-offs seen in object storage also apply broadly within distributed systems concerning caching and stateless servers.

Structure of an Object

Components of an Object

• An object consists of a key (identity), value (data), system metadata (size, type), and user metadata (custom attributes).

Buckets as Containers

• Objects reside within buckets—named containers that must have globally unique names across cloud platforms like AWS.

Illusion of Folders

Misconceptions About Hierarchies

• What appears as folder structures is simply a naming convention where slashes are treated as characters rather than directory separators.

Key Design Considerations

Best Practices for Keys

• Avoid using user-provided filenames as keys due to potential collisions that could overwrite existing objects silently.

Security Concerns

• Path traversal risks arise from special characters in filenames that may escape intended directory structures.

• User-uploaded filenames can contain problematic characters leading to backend bugs.

Recommended Strategies

• Generate unique keys internally instead of relying on user input while storing original filenames separately for reference when needed.

Behind-the-scenes Operations

Request Handling Process

• Upon receiving a request, it first lands on stateless front-end nodes which authenticate before processing data through separate metadata and data planes.

Metadata vs Data Plane Functions

• The data plane handles large volumes with simple operations while the metadata plane manages numerous operations requiring strong consistency guarantees.

Object Storage and Metadata Management

Replication Strategies

• Setting up replication to a second bucket in a different AWS account can prevent access by attackers if credentials are compromised.

• This strategy enhances security by isolating data across accounts.

Understanding the Metadata Plane

• The metadata plane is conceptualized as a distributed B-tree, which serves as a sorted key-value index for object storage.

• List operations on an object storage bucket can be slow due to range scans over this distributed index, leading to performance issues.

Best Practices for Object Storage Operations

• Avoid using list operations in request paths; instead, maintain an index of files within your database.

• Relying on the list objects API can lead to latency issues and scalability problems.

Consistency Models in Object Storage

Atomic Rename Challenges

• Renaming objects is complex because it involves changing keys that determine partition locations without altering the underlying data.

Evolution of S3 Consistency

• Historically, S3 was eventually consistent, causing immediate read requests after uploads to return 404 errors due to outdated index replicas.

• As of December 2020, S3 supports strong read-after-write consistency, ensuring successful uploads return a 200 status even with immediate subsequent requests.

Conditional Writes in Object Storage

Introduction of Conditional Writes

• AWS S3 introduced conditional writes allowing uploads only if certain conditions are met (e.g., "if none match" header).

• This feature prevents silent overwrites and enables better error handling through specific status codes like 412 for precondition failures.

Use Case Scenarios

• In scenarios where two clients attempt simultaneous updates, conditional writes ensure that one client's changes do not overwrite another's without detection.

Implementing Pre-signed URLs

Direct Upload Architecture

• A common architecture involves clients uploading files directly from their browsers to S3 using pre-signed URLs rather than routing through backend servers.

Security Considerations

• Pre-signed URLs allow temporary access without exposing AWS credentials while enforcing strict permissions on upload actions.

Enforcing Size Limits with Policies

Policy-Based Upload Control

• Using pre-signed post URLs allows setting conditions such as content length ranges and content types directly enforced by the object storage service.

Handling Abandoned Upload Workflows

• Implementing lifecycle rules helps manage abandoned uploads by deleting pending entries after a specified duration.

This structured summary captures critical insights from the transcript while adhering strictly to timestamp requirements for easy reference.