21.2. Backend Scaling and Performance Engineering: Part-2

Understanding Horizontal Scaling and Statelessness

Introduction to Horizontal Scaling

• The video continues from a previous discussion on horizontal scaling, emphasizing the importance of understanding this concept in context.

• The speaker mentions a division into two parts due to frame corruption issues, clarifying that this was not intentional.

Key Concept: Statelessness

• Statelessness is identified as a crucial property enabling horizontal scaling; it allows multiple instances of backend applications to function without exclusive data tied to any single instance.

• Horizontal scaling involves adding more machines with the same capacity rather than increasing the power of one machine, which requires careful planning across the entire stack.

Implications of Stateful vs. Stateless Design

• If an instance holds exclusive data (stateful), it can lead to errors when requests are distributed among instances, disrupting service continuity.

• The need for statelessness means that all servers must have access to shared data; if one server fails or is removed, others should still operate seamlessly.

Examples Illustrating Statelessness

• In authentication flows, session information must be stored in a centralized location (like Redis), rather than in-memory on individual instances to avoid user experience issues.

• For file storage, uploaded files should be saved in accessible object storage (e.g., S3), ensuring all instances can retrieve necessary files regardless of where they were uploaded.

Load Balancers: A Critical Component

Role of Load Balancers

• Load balancers are essential for managing traffic between users and multiple server instances; they determine how requests are routed based on specific algorithms.

• All incoming requests first go through the load balancer before being forwarded to appropriate server instances based on its logic.

Load Balancing Algorithms

• The round-robin algorithm distributes requests evenly across servers but may not account for varying request complexities or server capacities.

• Weighted round-robin enhances this by sending more requests to higher-capacity servers based on predefined weights assigned according to their resources.

Advanced Load Balancing Techniques

• Least connections algorithm directs traffic towards servers with fewer active connections, optimizing resource use during heavy loads.

• Variations like least response time and resource-based algorithms further refine request distribution by considering current server performance metrics.

Handling Server Failures

Health Checks Mechanism

• To manage potential server failures effectively, load balancers implement health checks that continuously monitor each server's status by sending test requests at regular intervals.

• If a server fails these checks (not responding with success codes), it is temporarily blacklisted from receiving user requests until it passes subsequent tests again.

This structured approach ensures robust handling of horizontal scaling challenges while maintaining seamless user experiences across distributed systems.

Scaling Backends and Database Architecture

Introduction to Load Balancers

• A significant aspect of scaling backends involves understanding load balancers, which distribute incoming traffic across multiple servers to enhance performance and reliability.

Externalizing State for Horizontal Scaling

• Horizontal scaling is favored over vertical scaling; externalizing state is crucial. This means not tying any state or session data to a specific server instance.

• All caches, files, and session data should be externalized using services like Redis for caching and S3 for file storage, allowing seamless addition of more servers behind load balancers.

Challenges in Database Scaling

• Databases are stateful components that cannot be easily duplicated like server instances. Each database holds unique data that must remain consistent across instances.

• The complexity arises from ensuring that multiple database instances return the same data when queried, making database scaling particularly challenging.

Read Replicas as a Solution

• Read replicas provide an intuitive method for scaling databases by creating secondary instances that handle read requests while the primary instance manages write operations.

• These replicas can be geographically distributed to reduce latency for users in different regions, improving response times significantly.

Benefits and Trade-offs of Read Replicas

• Utilizing read replicas reduces the load on the primary database instance since they handle most read queries (up to 70%).

• However, consistency issues arise when updates occur; if a user updates their profile but immediately fetches data from a read replica, they may receive outdated information due to replication lag.

Addressing Consistency Issues

• Solutions include routing subsequent read requests after writes directly to the primary instance or tracking replication lag before serving reads.

• Another approach is implementing planned latency in front-end applications so that updated data is fetched only after ensuring it has been replicated.

Sharding as a Database Scaling Technique

• Sharding involves partitioning large tables into smaller segments across multiple database instances based on defined criteria (e.g., date ranges).

• This technique addresses both query latency and resource allocation by distributing requests among several shards instead of relying on one massive table.

Current Trends in Database Management

• Distributed databases like PlanetScale and CockroachDB are emerging solutions that automatically manage sharding and replication complexities.

Conclusion: Understanding Modern Database Practices

• While backend engineers may not implement these systems directly, understanding concepts such as replication, sharding, and distributed databases is essential for effective configuration and management.

Understanding Latency and CDNs

The Role of Latency in Web Performance

• Latency is not solely caused by the initial request; it also involves the service layer, which includes business logic and database interactions.

• In optimal conditions (same region data center), mid-level to complex queries can still incur 50-100 milliseconds of latency.

• External API calls can add an additional 200 milliseconds, leading to a total latency of approximately 500-800 milliseconds for users far from the server.

How CDNs Mitigate Latency

• Content Delivery Networks (CDNs) address latency by placing edge locations closer to users, reducing travel distance from thousands of kilometers to just hundreds.

• This geographical proximity can decrease response times from 100 milliseconds down to as low as 2-3 milliseconds, significantly enhancing performance.

Benefits of Using CDNs

• One major advantage is reduced load on primary servers since CDN nodes handle user requests directly, distributing traffic effectively.

• By caching content at various CDN nodes (e.g., Tokyo, Mumbai), overall server resource usage decreases by about 50%, alleviating scaling concerns.

Types of Content Suitable for CDNs

• Static content such as JavaScript bundles, CSS files, HTML documents, images, and videos are ideal candidates for CDN caching due to their infrequent changes.

• API responses that do not change often (like product catalogs in e-commerce sites) can also be cached effectively using CDNs.

Cache Management Strategies

• Techniques like cache invalidation or purging allow for dynamic updates when content changes occur.

• For instance, tagging cached content enables targeted purging based on specific conditions like user uploads or data changes.

Security Advantages of CDNs

Protection Against DDoS Attacks

• Modern DDoS attacks involve numerous bots overwhelming a server's resources. A CDN acts as a buffer against this influx of malicious traffic.

Cloudflare's Role in Security

• By routing all traffic through its network before reaching the primary server, Cloudflare helps mitigate potential damage from DDoS attacks while maintaining service availability.

Edge Computing vs. Traditional Servers

Definition and Functionality

• Edge computing refers to processing that occurs at CDN nodes rather than centralized data centers. This reduces latency further by handling requests closer to users.

Use Cases for Edge Computing

• Authentication processes benefit greatly from edge computing; validating session information at the edge node minimizes round-trip time compared to traditional methods.

Constraints and Limitations of Edge Computing

Resource Limitations

• Edge nodes typically have limited resources compared to robust primary servers due to their infrastructure being primarily managed by ISPs.

Processing Constraints

• Technologies like Cloudflare Workers utilize constrained environments that limit certain functionalities (e.g., file system access), impacting what tasks can be offloaded from main servers.

Asynchronous Processing for Improved User Experience

Concept Overview

• Asynchronous processing allows operations that do not require immediate feedback—such as sending emails—to be handled without blocking user interactions with the application.

Implementation Example

• Instead of waiting for an email confirmation after inviting a user via an API call—which could take up to 400 milliseconds—responses can be sent immediately after database updates while processing continues in the background.

Understanding Asynchronous Processing in Backend Systems

Overview of Queue-Based Processing

• The server acts as a producer, pushing tasks into a queue (e.g., Redis Q or Rabbit MQ), which is essential for managing jobs efficiently.

• Consumers, also known as workers, can either run within the same codebase or be separate, allowing for horizontal scaling based on traffic demands.

Asynchronous Operations Explained

• Workers retrieve tasks from the queue and execute them without requiring immediate user feedback, exemplified by email notifications that users do not expect to receive instantly.

• Sending emails and notifications are classic examples of asynchronous operations where users do not anticipate immediate results after performing an action.

Real-world Examples of Asynchronous Processing

• YouTube's video upload process illustrates asynchronous behavior; after uploading, various tasks like thumbnail generation and encoding occur in the background without user intervention.

• Deleting user accounts in SaaS applications often involves multiple database deletions. Instead of processing synchronously (which could take several seconds), a task is queued to handle deletions asynchronously.

Improving User Experience with Asynchronous Tasks

• A naive synchronous approach to account deletion could lead to poor user experience due to long wait times; instead, basic checks are performed first before queuing the actual deletion process.

• Implementing a robust queue system (like Bull MQ with Redis) allows for efficient handling of background jobs while improving perceived latency for users.

Exploring Microservices Architecture

Introduction to Microservices

• Microservices architecture is increasingly popular among companies looking to scale their backend applications by dividing them into independent services.

• Monolith applications consist of all functionalities bundled together as a single deployable unit, making them easier to manage but potentially limiting scalability.

Advantages of Microservices

• Microservices facilitate team scaling rather than just application performance scaling; they allow large teams (typically over 100 developers) to work independently on different modules.

• Deployment dependencies in monolith architectures can hinder rapid development across teams working on different modules; microservices mitigate this issue by enabling independent deployments.

Scaling and Technology Flexibility

• Different resource requirements among services necessitate independent scaling capabilities that microservices provide—allowing specific services like payment processing to scale without affecting others.

• Using varied technology stacks becomes feasible with microservices; for instance, one service may require Node.js while another benefits from Rust’s performance capabilities.

Challenges and Considerations in Adopting Microservices

Trade-offs Involved with Microservices

• Transitioning from monolith to microservice architecture introduces network complexities since inter-service communication relies on network calls rather than direct function calls.

• Debugging requests across multiple services becomes more complex due to distributed systems architecture requiring logs from various sources for troubleshooting.

Data Consistency Issues

• Each microservice typically manages its own database instance, raising concerns about data consistency and replication lag when changes occur across different databases.

The Role of Serverless Computing in Modern Applications

Introduction to Serverless Architecture

• Serverless computing has gained traction as an effective method for scaling backend applications through platforms like Vercel and Netlify that abstract server management away from developers.

Traditional vs. Serverless Models

• Traditionally, deploying applications involved provisioning virtual machines (VM), configuring servers with necessary software, and ongoing management responsibilities throughout their lifecycle.

Understanding Capacity Planning and Autoscaling

The Challenge of Capacity Planning

• Capacity planning involves predicting user traffic and determining the necessary server resources before deployment, including RAM, CPU cores, and hard drive capacity.

• Underprovisioning can lead to server crashes or unresponsiveness during traffic spikes, resulting in potential loss of users and revenue.

• Overprovisioning incurs unnecessary costs; for example, paying for 32GB of RAM when only 8GB is needed can lead to significant financial waste.

The Solution: Autoscaling

• Autoscaling automatically adjusts the number of active servers based on current resource usage, helping to manage underprovisioning and overprovisioning issues.

• However, autoscaling has drawbacks such as time delays in spinning up new instances which may not meet sudden demand spikes quickly.

Limitations of Autoscaling

• Setting maximum instance limits too low can revert back to underprovisioning issues while setting them too high can lead to exorbitant costs during unexpected traffic surges.

• Autoscaling is reactive rather than proactive; it responds only after resource thresholds are crossed, potentially leading to service interruptions.

Transitioning from Traditional Servers to Serverless Architecture

Introduction to Serverless Computing

• Serverless architecture eliminates the need for developers to manage server infrastructure directly; they focus solely on writing code (functions).

• An API gateway acts as an intermediary that routes requests to appropriate functions without requiring knowledge about underlying server configurations.

Benefits of Serverless Architecture

• In a serverless model, costs are incurred only during actual function execution rather than maintaining always-on servers, significantly reducing expenses.

Challenges with Serverless Computing

• Cold start times present a major challenge since new instances must be provisioned upon receiving requests, leading to latency issues.

• Other limitations include execution time caps (e.g., AWS Lambda functions have a maximum runtime), statelessness which complicates traditional stateful interactions like database connections.

When Should You Use or Avoid Serverless Architecture?

Ideal Use Cases for Serverless

• Event-driven tasks such as video processing or image resizing are well-suited for serverless due to their sporadic nature and ability to scale efficiently on demand.

Situations Where Serverless May Not Be Suitable

• Applications requiring low latency or long-running processes may struggle with cold starts and execution limits inherent in serverless architectures.

Key Takeaways on Scaling Solutions

Measuring Performance Before Optimization

• Always begin by identifying specific performance bottlenecks through observability tools before implementing solutions like caching or indexing.

Preference for Simplicity

• Opt for simpler solutions where possible; complex architectures like microservices may introduce more challenges than benefits unless adequately managed.

The Importance of Simplicity in System Design

Horizontal Scaling and Complexity

• When discussing horizontal scaling with tools like Kubernetes, it's crucial to recognize that these solutions can be complex. Only opt for them when absolutely necessary.

• Simple solutions are preferred as they are easier to understand, debug, and operate. Starting with a large vertical scaling solution is often simpler than adopting a horizontal scaling architecture from the outset.

Database Indexing vs. Caching

• Implementing proper indices for database queries is a more straightforward approach compared to using caching mechanisms like Redis in front of all database queries.

• A monolithic architecture is generally simpler than microservices due to the added complexity and potential failure points introduced by additional components.

Accepting Complexity Judiciously

• Complexity incurs costs; every new component requires monitoring and understanding. Only accept complexity when simplicity fails to meet your needs.

• Scale your systems based on current user needs rather than over-engineering for hypothetical future growth.

Building Systems for Realistic Growth

Observability and Learning from Metrics

• It's essential to implement observability (logs, metrics, traces) early on. This helps identify bottlenecks as your system grows.

• Relying on generic advice from large companies may not apply directly to your specific situation; measuring your own system's performance provides more valuable insights.

The Necessity of Early Observability

Importance of Production Grade Observability

• Implementing production-grade observability from day one prevents random errors and crashes by providing clear visibility into system performance.

• As systems grow, having robust observability allows proactive adjustments before issues arise.

Mindset Towards Performance Optimization

Continuous Learning in System Design

• Performance optimization is an ongoing learning process; experience helps you understand how to scale effectively based on actual user traffic rather than theoretical models.

• Your role as a backend engineer involves building resilient systems that gracefully handle problems when they occur, emphasizing the importance of measurement and diagnosis skills.