13. Caching, the secret behind it all

What is Caching?

Definition and Importance of Caching

• Caching is a mechanism that reduces the time and effort required to perform tasks by storing a subset of data for quicker access.

• A more technical definition describes caching as keeping a subset of primary data based on usage frequency, access speed, and other parameters.

• This technique significantly enhances performance in high-demand applications where latency is critical, often measured in microseconds or milliseconds.

Examples of Caching in Action

Google Search Engine

• When users search queries like "what is the weather today," Google's complex algorithms would be computationally expensive without caching mechanisms.

• Caching allows Google to store results from previous queries, reducing server load and response times for frequently searched information.

• If a query result is cached (a cache hit), it can be retrieved instantly; if not (a cache miss), the system computes the result and caches it for future requests.

Netflix Content Delivery

• Netflix uses encoding to prepare multiple resolutions of content for different devices and network speeds, optimizing bandwidth use while minimizing server load.

• The platform employs Content Delivery Networks (CDNs) with strategically placed edge servers worldwide to deliver content efficiently with minimal buffering.

• By analyzing regional viewing trends, Netflix decides which content subsets to cache at edge locations, balancing cost and resource efficiency against user demand.

Caching Strategies in Social Media Platforms

Twitter's Trending Topics

• Twitter analyzes millions of tweets in real-time to identify trending topics but avoids recalculating this data for every user request through caching strategies.

• Cached trending topics remain stable over hours or days, allowing quick retrieval without heavy computation each time a user accesses them.

• The platform utilizes in-memory databases like Redis to store these computed trends efficiently, ensuring fast access for users when they check trending sections.

Levels of Caching Relevant to Backend Engineering

Types of Caching

• Three primary levels of caching are commonly encountered: Network level (e.g., CDNs), Hardware level, and Software-based caching systems that interact with hardware components.

Network Level Cache

1. Content Delivery Networks (CDN): These networks cache content closer to end-users via geographically distributed servers known as edge nodes or points of presence (POPs).

2. DNS Queries: DNS also employs caching techniques to minimize latency by resolving domain names into IP addresses quickly using local caches within recursive resolvers provided by ISPs or third-party services like Google DNS or Cloudflare.

Understanding DNS Resolution Process

Local Cache Check

• The process begins with checking if the IP address for example.com is in the local cache. If found, it returns the corresponding IP to the user’s device.

• If there is a cache miss, the query is sent to other resolvers to find the IP address. This initiates further steps in DNS resolution.

Querying Root Servers

• When a resolver does not have the result, it queries root servers, which are numerous and distributed globally but do not store specific domain IP addresses. Instead, they provide referrals for top-level domains (TLDs).

• For example.com, the root server directs to the TLD server for .com. This step narrows down where to look next in finding an authoritative name server.

Authoritative Name Server

• The TLD server also lacks direct information about example.com but provides access to its authoritative name server, which holds definitive data about that domain's IP address.

• The recursive resolver continues querying until it reaches this authoritative name server and retrieves the required IP address. This highlights its recursive nature as it delves deeper into various servers until successful retrieval occurs.

Importance of Caching in DNS

• Caching plays a crucial role in DNS efficiency by reducing repetitive queries across multiple servers for every request made by users; local caches help speed up this process significantly.

• Operating systems like Windows or Mac maintain local caches that check first before querying external resolvers, thus optimizing performance and reducing load on network resources.

Hierarchical Caching Mechanisms

Browser-Level Caching

• Modern web browsers such as Chrome and Firefox implement their own DNS caching mechanisms after operating system checks; this further reduces latency when accessing frequently visited sites.

Recursive Resolver Caches

• Recursive resolvers provided by ISPs or services like Google Public DNS also maintain their own caches to expedite responses without needing to reach out to all servers repeatedly for each request made by users.

Authoritative Name Server Caches

• Some authoritative name servers may also implement caching strategies, allowing them to respond faster without needing additional requests from other servers when serving common queries or data points.

Hardware-Level Caching Concepts

CPU Cache Levels

• Hardware-level caching includes L1, L2, and L3 caches within CPUs designed for rapid access during computations; these layers optimize processing speeds significantly compared to main memory or hard disk storage methods due to their proximity and design efficiency.

Random Access Memory (RAM) Characteristics

• RAM allows quick data access through electrical signals rather than mechanical means used by hard disks; however, its volatile nature means data is lost when power is turned off—highlighting trade-offs between speed and persistence versus capacity limitations inherent in primary storage solutions like RAM versus secondary storage options like hard drives or SSDs.

In-Memory Databases: Redis & Memcached

Overview of In-Memory Databases

• Technologies such as Redis and Memcached utilize RAM for storing key-value pairs instead of traditional databases stored on disks; this results in much faster read/write operations due to reduced latency associated with accessing primary memory directly rather than going through slower disk-based systems.

Key Features:

• In-memory Storage: Data resides primarily in RAM leading to quicker access times compared with traditional databases.

• Key-value Structure: Unlike relational databases requiring strict schemas, these allow flexible structures where any type of value can be associated with a unique key.

Caching Strategies

Lazy vs Write-through Caching

Lazy Caching:

• Resources are only cached upon being requested; if absent from cache during a request cycle, they are fetched from primary storage then stored for future use—this method conserves resources but may introduce delays initially while fetching new entries into cache.

Write-through Caching:

• Updates occur simultaneously both in database and cache whenever changes happen (e.g., POST/PUT requests); while ensuring freshness of cached data increases overhead since both locations must be updated concurrently during write operations.

Eviction Policies

Managing Cache Size

No Eviction Policy:

• Without configured eviction policies, attempts at inserting new data into full caches will fail—leading potentially inefficient resource management.

Least Recently Used (LRU):

• Tracks usage frequency determining which items were accessed least recently; older entries get evicted first when space needs freeing up.

Least Frequently Used (LFU):

• Monitors how often keys are accessed over time; less frequently accessed keys are removed first when new entries need accommodation.

Time-to-Live (TTL):

• Keys can be assigned expiration times after which they automatically become invalidated—allowing dynamic management of what remains accessible within limited cache space.

Use Cases for In-Memory Databases

Database Query Optimization

• Complex SQL queries involving multiple joins can benefit greatly from caching results temporarily so repeated calls do not burden underlying databases unnecessarily—improving response times significantly especially under high traffic conditions.

Session Management

• Storing session tokens post-authentication within Redis enhances performance since retrieving session details becomes much faster than querying traditional databases repeatedly during user interactions.

API Call Efficiency

• External API calls can lead quickly toward rate limits if unoptimized; using caching mechanisms allows frequent requests against static datasets without overwhelming third-party services while maintaining application responsiveness overall.

Caching and Rate Limiting in API Development

Understanding Caching for Weather Data

• Caching is essential when dealing with APIs to avoid hitting rate limits or incurring high billing costs. Weather data, which does not change frequently, is a prime candidate for caching.

• By fetching weather data from an API and caching it, subsequent requests can utilize this cached data instead of making repeated calls to the API. A Time-To-Live (TTL) of one hour can be set for the cache.

• After the TTL expires, the cache invalidates automatically, prompting a fresh fetch from the API for new requests.

Implementing Rate Limiting

• Rate limiting is often implemented using technologies like Redis or other in-memory caches to manage request frequency effectively.

• Middleware plays a crucial role in rate limiting by intercepting requests before they reach the server's routes. It extracts headers such as "X-Forwarded-For" to identify user IP addresses.

Mechanism of Rate Limiting

• The middleware checks incoming requests against predefined limits (e.g., 50 requests per second). This helps prevent abuse from bots and conserves computational resources.

• If a client exceeds their allowed request count within a specified duration (e.g., one minute), the middleware blocks further requests and returns an error response (HTTP status code 429).

Advantages of In-Memory Databases

• Storing rate limit counters in in-memory databases like Redis enhances performance by reducing latency compared to relational databases like PostgreSQL or MySQL.

• Using relational databases for rate limiting could lead to increased latency due to slower query times and higher load on database resources.

Conclusion on Caching and Rate Limiting Practices

• Effective caching strategies minimize API latency while reducing database load. In-memory databases are preferred for their speed advantages over traditional relational databases.

• Understanding these concepts equips developers with practical knowledge about implementing efficient caching mechanisms and rate limiting strategies using tools like Redis.