Tornando sua App Web Mais Rápida! | 4 Técnicas de Otimização

Name: Tornando sua App Web Mais Rápida! | 4 Técnicas de Otimização
Uploaded: 2022-12-12T12:42:16.000Z
Duration: 1 h 38 min 2 s

Understanding the Gap Between Beginner Web Apps and Professional Applications

Introduction to the Topic

Fabio Akita introduces a complex topic aimed at beginner programmers who are learning to create their first web applications.

He poses a critical question about why simple e-commerce applications created through tutorials differ significantly from real-world applications like those of Amazon or Mercado Livre.

Key Concepts vs. Framework Specifics

Akita emphasizes that today's focus will be on understanding core concepts rather than specific programming languages or frameworks, which can be found in various tutorials and documentation.

He explains that all frameworks can achieve similar outcomes; the key is understanding how things work beyond just following step-by-step instructions.

The HTTP Request/Response Cycle

When a user accesses a website (e.g., amazon.com.br), the browser sends an HTTP request, which travels over the internet to reach Amazon's servers.

The server processes this request, generates an HTML page, and sends back an HTTP response with a status code (200 for success).

Building Basic Web Applications

Most beginners can create basic web pages using popular frameworks like Rails, Django, Express, etc., by connecting to databases and rendering HTML templates.

Each framework has its own command for starting a local server (e.g., rails server, python manage.py runserver).

Common Misconceptions About Deployment

Akita highlights misconceptions about deploying web applications directly on remote servers as if they were running locally; this approach is flawed.

He discusses why local development servers typically run on non-standard ports (like 3000 or 8080), while browsers expect requests on standard ports (80 or 443).

Understanding Server Architecture

Many scripting languages operate primarily on single threads; thus, deploying multiple instances of an application across different CPU cores is essential for efficiency.

To handle incoming traffic effectively without exposing individual application processes directly to users, load balancers like NGINX are used. They distribute requests among multiple backend processes.

Conclusion: Importance of Load Balancing

Understanding Load Balancing and Scalability in Web Applications

The Role of Load Balancers

A load balancer operates transparently, allowing backend processes to handle requests as if they were directly connected to the user, similar to local development environments.

Basics of Scalability

If a process takes 100 milliseconds to respond, it can handle up to 10 requests per second. Exceeding this limit results in queued requests.

By deploying multiple processes on the same server (e.g., four), the server can support up to 40 requests per second, illustrating the concept of throughput.

Enhancing Application Performance

As traffic increases, developers must either scale vertically by upgrading existing machines or horizontally by adding more servers.

Using a reverse proxy like NGINX enhances security and optimizes performance while managing SSL certificates for secure connections.

Scaling Strategies

In Kubernetes environments, an Ingress controller serves a similar purpose as NGINX for load balancing across multiple application processes.

Large services like Google or Amazon utilize DNS resolution techniques that return multiple IP addresses for load balancing across geographically distributed servers.

Practical Considerations for Developers

While a single NGINX server is often sufficient for many applications, understanding advanced scaling techniques is beneficial.

When faced with increased traffic (e.g., 80 requests per second), developers can either upgrade their machine or add new servers under the load balancer.

Vertical vs. Horizontal Scaling

Upgrading existing machines is termed "vertical scalability," while adding new machines is referred to as "horizontal scalability."

Challenges in Scaling Applications

Although scaling seems straightforward, complexities arise from multi-threaded behaviors and resource management within operating systems.

Recommendations for New Projects

Understanding Connection Pooling in Database Management

The Need for Multiple Processes and Dynos

To handle 400 requests per second, the application requires at least 13 dynos, with each process managing 10 requests.

Unlike Java or Elixir, scripting languages manage threads differently; they can have multiple threads but may not run them all in parallel consistently.

A connection pool is essential as database connections are resource-intensive; opening too many connections can exhaust system resources.

Managing Resource Limits

Each session of a database connection consumes memory and CPU; without limits, either memory or processing power will be exhausted, leading to application failure.

Historically, applications would crash when exceeding limits; modern practices involve setting timeouts to prevent indefinite loading times.

Implementing Connection Pools

Connection pooling allows applications to reuse existing connections rather than opening new ones continuously, which helps manage resource consumption effectively.

For example, if an application has three processes with five connections each on a dyno, it could consume up to 15 connections from the database pool.

Understanding Database Plans and Connections

When selecting a database plan (e.g., Heroku Postgres), it's crucial to consider the maximum number of allowed connections rather than just storage capacity.

If an application needs more connections than provided by its plan (e.g., needing 195 while only having access to 120), it may require upgrading or implementing a connection manager like pgbouncer.

The Role of Connection Managers

Pgbouncer acts as an intermediary that manages connection requests efficiently by queuing excess requests instead of directly connecting to the database.

Optimization Techniques for Web Applications

The Importance of Caching in Web Applications

A web application can waste resources by making multiple database queries for the same homepage request, leading to inefficiencies when many users access the same content.

Implementing caching involves storing the last response to avoid reprocessing. A simple method is to create a local file named "cache-homepage" that stores the HTML generated from database queries.

On subsequent requests, if the cache file exists and is not older than a set time (e.g., 5 minutes), it can be served directly without querying the database again.

Impact of Caching on Database Connections

With caching implemented, instead of opening numerous connections to the database for each request, only one connection is needed initially. This drastically reduces resource consumption.

By limiting database connections through caching, an application can handle more traffic efficiently—reducing required connections from 195 per second to nearly zero during cache validity.

Evaluating Optimization Strategies

Optimizations like caching can significantly reduce response times and server load without changing programming languages or frameworks. This allows for better scalability with existing resources.

Switching programming languages or frameworks may yield minor performance improvements (5%-10%), but effective caching strategies can lead to substantial gains (5x or 10x).

Advanced Caching Solutions

While files are used for basic explanations, real-world applications typically utilize faster databases like Memcache or Redis for caching due to their speed and ability to handle more simultaneous connections.

These NoSQL solutions lack strict ACID guarantees but are suitable for cache purposes where occasional inconsistencies are acceptable.

Understanding Database Performance Profiles

Traditional relational databases (MySQL, Postgres) excel at read operations but have slower write speeds due to their ACID compliance. This ensures data integrity but limits throughput under heavy write loads.

In contrast, NoSQL databases like Redis offer different performance characteristics that prioritize speed over strict consistency, making them ideal for high-volume applications needing rapid data retrieval.

Understanding NoSQL Databases and Their Use Cases

The Trade-offs of Speed vs. Guarantees

Cassandra offers less guarantee than traditional databases, allowing for faster write operations.

Applications like Google Analytics can tolerate data loss; missing a single click does not significantly impact overall analytics.

In scenarios where ACID guarantees are not critical, using faster databases like MongoDB or Cassandra is beneficial.

Characteristics of Different Databases

There is no one-size-fits-all solution for database selection; understanding the specific needs and profiles is crucial.

For most non-startup businesses, a relational database (e.g., Postgres) combined with caching servers (e.g., Memcache or Redis) suffices.

Complex queries involving multiple joins can slow down systems significantly, especially during high-demand periods.

Managing Database Load and Performance

High traffic situations can lead to performance bottlenecks when generating large reports, causing delays in response times.

Implementing multiple servers helps manage load: a primary server for writes and replicas for read operations ensures efficiency.

Scaling Solutions for E-commerce Systems

During peak times (e.g., Black Friday), more users may attempt to place orders than the primary server can handle.

A common e-commerce checkout process involves waiting on external payment systems, which can introduce delays during high traffic.

Handling External Dependencies and Traffic Spikes

Increased user demand during events like Black Friday may overwhelm the database connections leading to timeouts.

The transaction process must be efficient; if external factors slow down payment processing, it affects user experience negatively.

Managing User Queues and Payment Processing

Efficient Queue Management

The concept of giving users a "ticket" to return later helps manage long queues without keeping them actively connected, conserving server resources.

When users are ready to pay, the solution involves creating additional queues. Associating order numbers with database IDs can lead to performance issues if the database is overloaded.

A unique number generation system should be implemented, decoupling order numbers from database IDs to enhance efficiency.

Utilizing Message Queues

Instead of directly writing orders into the database, data is sent to a queue service upon checkout completion, allowing immediate user feedback while processing occurs in the background.

Various queue services like Redis, RabbitMQ, Apache Kafka, and AWS SQS can be utilized for managing message queues effectively.

Advantages of Dedicated Queue Services

Using dedicated queue services allows for faster message handling compared to direct database writes when under heavy load.

True queuing protocols (e.g., RabbitMQ) provide delivery guarantees similar to relational databases' ACID properties.

Worker Processes and Asynchronous Jobs

Workers are lightweight applications that listen for messages on specific channels (e.g., payment processing), enabling separation of tasks and reducing user wait times.

Each channel has its own worker responsible for executing specific actions based on queued messages.

Balancing Resources and Performance

Modern web frameworks offer various methods for handling asynchronous jobs; examples include ActiveJob in Ruby on Rails and Bull in Node.js.

While increasing workers may seem beneficial, they still require connections to the main database. Proper resource management is crucial to avoid overwhelming the system.

Conclusion: Resource Management Strategy

Web Application Architecture and Caching Strategies

Introduction to Web Application Architecture

The speaker compares configuring web architecture to playing an RPG, emphasizing the importance of resource management.

Initial setup involves a single process running locally, connecting to a database like Postgres via localhost:3000.

In production environments (AWS, Azure, Google Cloud), a load balancer (e.g., NGINX) directs traffic to multiple application instances.

Load Balancing and Connection Management

Each application instance connects to the database with a limited number of reusable connections through connection pooling.

Caching is crucial for optimizing performance by storing frequently accessed data that doesn't change often.

Implementing Caching Solutions

Memcache is introduced as a caching service that intelligently manages memory by removing older cache entries when necessary.

User-specific caching can enhance performance; for example, user profile data can be cached to reduce database queries.

Scaling Database Operations

As applications grow, integrating third-party services increases response times; thus, managing resources becomes essential.

Read operations can be distributed across replica databases while write operations are handled separately to optimize performance.

Handling HTTP Requests Efficiently

Immediate responses are prioritized in HTTP requests; external service calls (like payments) should be processed asynchronously.

A task queue system (e.g., SQS or Kafka) is recommended for handling background tasks without blocking user requests.

Performance Monitoring and Optimization

It's important not just to add components randomly but also to analyze request times using monitoring tools like New Relic's RPM.

Different types of pages have varying response times; content-heavy pages may respond quickly while checkout processes take longer.

Performance Optimization Techniques

Importance of Detailed Performance Analysis

Alternatives like Scout exist, but New Relic provides a more detailed analysis, identifying key performance offenders in applications.

It can pinpoint specific controllers, models, and queries that are causing slowdowns, eliminating guesswork in optimization efforts.

Emphasizes that 80% of performance issues typically stem from 20% of the application; thus, not everything needs to be rewritten.

Data-Driven Optimization Strategies

Focus on optimizing a few critical functionalities (2 out of 10) can lead to significant resource savings across the infrastructure.

Optimizations should rely on real user data from production environments rather than assumptions or local testing.

Once the main offenders are identified and corrected, measuring their impact post-deployment is crucial for assessing effectiveness.

Front-End Optimization with CDN

Front-end developers should learn to utilize Content Delivery Networks (CDNs) to enhance performance by reducing server requests for assets.

When HTML is served with multiple asset tags (images, CSS), it leads to numerous requests back to the server which can slow down response times.

Implementing CDN Effectively

Instead of hardcoding image URLs directly into HTML, use absolute URLs pointing to CDNs for better efficiency and flexibility during development and deployment.

Utilize helper functions within frameworks (e.g., Node Express with EJS templates) to dynamically generate URLs based on environment settings.

Benefits of Using CDNs

Properly implemented CDNs cache images after the first request, significantly reducing load on the original application server and improving response times for subsequent users.

CDNs often have servers distributed globally; this means users accessing your application from different regions will experience faster load times due to localized content delivery.

Additional Optimization Techniques

Explore caching strategies using tools like Memcache or Redis and consider asynchronous job processing with services such as AWS SQS or Kafka for improved performance.

Monitoring real-time metrics in production is essential for understanding application behavior under actual usage conditions.