Entendendo "Devops" para Iniciantes em Programação (Parte 2) | Série "Começando aos 40"

New Section

In this section, Fabio Akita introduces the topic of DevOps and reflects on the evolution of hosting technologies from shared hosting to virtual private servers (VPS) and beyond.

Evolution of Hosting Technologies

• Fabio discusses the unpreparedness of infrastructure for the rapid growth in demand from unicorns around 2005, leading to the emergence of VPS services like Linode.

• Virtual Private Servers (VPS) are explained as a service where users can select CPU, memory, disk space, and operating system configurations for a monthly fee, offering a virtualized alternative to physical servers.

• Amazon's response to scalability challenges is highlighted through the launch of EC2 for virtual machines, S3 for blob storage, and SQS for queues in 2006.

• The introduction of Elastic Block Storage (EBS) by AWS is discussed as mountable volumes that can be attached to EC2 instances for additional storage capacity.

• AWS's innovative billing model based on pay-per-use hourly charges instead of fixed monthly fees revolutionizes cost management in hosting services.

Infrastructure Transformation with AWS

This segment delves into how Amazon Web Services (AWS) transformed web infrastructure by introducing granular billing and control mechanisms.

AWS Billing Model Revolution

• The granularity of AWS billing allows users to pay only for resources used per hour or megabyte, enabling flexibility in resource allocation based on real-time needs.

• The control provided by APIs empowers users to manage resources without manual intervention, facilitating rapid scaling across different geographic regions with redundancy.

• A paradigm shift is noted in handling machines akin to software components due to AWS's approach towards commercializing infrastructure management.

Challenges in Infrastructure Management

This part explores the complexities and risks associated with traditional infrastructure management practices.

Risks in Traditional Infrastructure

• The analogy of infrastructure as a "House of Cards" underscores its fragility and susceptibility to errors due to intricate configurations involving hardware components.

• An anecdote illustrates the catastrophic consequences of a single script error propagated across multiple servers via SSH, emphasizing the critical role of meticulous sysadmin practices.

Provisioning and Evolution of Infrastructure Technologies

The discussion delves into the evolution of infrastructure technologies, emphasizing the shift towards a more controlled and efficient approach in managing databases, server scripts, and software deployment.

Prohibition on Unrestricted Access (440s - 476s)

• Emphasizes treating database operations with caution akin to handling nuclear material.

• Highlights the conflict between traditional infrastructure culture and the need for faster scalability in enterprises.

Transition to Improved Scalability (510s - 544s)

• Discusses how Virtual Private Servers (VPS) aided developers by providing virtual machines but lacked full scalability features.

• Introduces a paradigm shift with AWS offering virtual machines as a service, enabling rapid machine recreation and enhanced security measures.

Emergence of Infrastructure as Software (578s - 615s)

• Details the advantages of AWS over traditional VPS, leading to quicker image creation and standardized testing environments.

• Describes the transformation in sysadmins' roles from physical hardware management to software-based infrastructure maintenance.

Evolution from IaaS to OpenStack

The conversation transitions into discussing the emergence of Infrastructure as a Service (IaaS), focusing on open-source efforts like OpenStack as alternatives to proprietary solutions like AWS.

Rise of OpenStack and Competitors (687s - 760s)

• Introduces Infraestrutura como Serviço or IaaS concept, highlighting efforts like Eucalyptus and OpenStack mirroring AWS functionalities.

• Chronicles the development timeline of OpenStack post-AWS era, leading to commercial options such as Microsoft Azure and Google Cloud challenging AWS dominance.

Role in Secure Environments (760s - 798s)

• Explores paravirtualization concepts, hardware-accelerated virtualization instructions, configuration tools like Chef, and orchestration platforms such as OpenStack for secure environments.

Understanding Containers and Virtualization

In this section, the speaker discusses the concept of isolation in processes and resources within a machine, highlighting the evolution from projects like Parallels and Virtualbox to Control groups (cgroups) for process isolation.

Evolution from Parallels to cgroups

• The focus shifts from tools like Parallels or Virtualbox to isolating processes and resources within a machine.

• Control groups (cgroups) aim to make the kernel deceive processes, enabling two processes to run on the same machine without awareness of each other's existence.

• Differences between distributions like RedHat, Ubuntu, or Arch lie in how binaries are packaged and configured within the distribution, starting with the kernel and drivers.

Linux Containers: LXC and Namespace

This part delves into Linux Containers (LXC), encompassing cgroups in the Linux kernel along with namespaces for process labeling.

Linux Containers Components

• Introduction of LXC or Linux Containers in 2008 by Google, incorporating cgroups and namespaces for process labeling.

• Namespaces allow tagging processes while cgroups set limits for process groups.

• Different namespaces manage various aspects such as networking (virtual ports), mounts (volume management), akin to how chroot masks filesystem access.

Containers vs. Hypervisors

Contrasting containers with hypervisors regarding shared kernels versus multiple virtualized kernels.

Container Distinction

• Cgroups and namespaces serve as equivalents for masking various machine resources beyond just CPU and memory, extending to network and I/O resources.

• A container restricts processes while sharing a common kernel among multiple containers—a departure from hypervisors managing separate kernels per virtual machine.

Evolution of Git Tools & Infrastructure Management

Reflecting on the transition from traditional version control systems to Git tools like GitHub or GitLab for code management.

Transition to Git Tools

• Shift towards Git post-2006 marked a significant change in code versioning practices within developer communities.

• Emergence of platforms like GitHub facilitated not only code versioning but also automation capabilities within development workflows.

Heroku: Simplifying Infrastructure Deployment

Introducing Heroku as a platform integrating VPS capabilities with Linux containers via cgroups for streamlined deployment using Git repositories.

Heroku Advantages

• Heroku revolutionizes infrastructure setup by combining VPS features with container technology through cgroups integration.

Heroku's Impact on Tech Startups

The discussion highlights how Heroku revolutionized the tech startup landscape by simplifying infrastructure for developers, leading to the emergence of new categories like PaaS and SaaS.

Heroku's Role in Simplifying Infrastructure

• Heroku made infrastructure trivial for developers, bridging the gap between VPS/AWS and Kubernetes by focusing on usability and a smart business model.

Evolution of Tech Startups Post-2006

• The tech startup scene boomed post-2006, with a lack of infrastructure knowledge hindering deployment on AWS. This gap created opportunities for startups offering specialized services like Sendgrid, RedisLabs, and Cloudinary.

Categories Introduced by Heroku

• Heroku pioneered the PaaS category, while companies like New Relic and Sendgrid contributed to the rise of SaaS. Integration of various SaaS into Heroku's platform simplified service addition and billing for projects.

Educational Impact of Heroku on Developers

The transcript delves into how Heroku educated developers about application scalability through initiatives like "the twelve factors" guide.

Developer Education by Heroku

• Choosing Heroku over self-infrastructure management is advised for software developers lacking sysadmin experience. Emphasizes considering long-term maintenance costs over immediate hourly expenses.

Influence on Development Practices

• Heroku influenced developers to prioritize quality coding practices over quantity, shifting from code-heavy approaches to scalable application development based on factors like RAM allocation.

Adoption of Best Practices

Understanding Docker and Containerization

In this section, the speaker delves into the evolution of Docker, emphasizing its role in managing large applications through microservices and containerization.

Evolution of Docker

• Docker is essential when a large application cannot fit into a single container, prompting the need to trim unnecessary components or partition it into multiple containers.

• Docker utilizes the Union File System to create incremental images, akin to Git commits, enabling easy rollback to previous versions and efficient packaging for distribution.

• The introduction of Dockerfiles standardized image descriptions, resembling configuration recipes seen in CF Engine or Chef cookbooks. These files specify machine configurations, package installations, and initialization commands.

• Dockerfiles streamline image creation and command-line usage, setting a precedent that even Kubernetes follows. It distinguishes Docker from virtualization technologies like Virtualbox by providing a simplified approach to application deployment.

• Contrary to full virtualization or paravirtualization, Docker organizes cgroups, namespaces, and union filesystem tools efficiently on top of existing resources without running another operating system layer.

Container Magic: Running Programs Efficiently

This segment elucidates how containers operate by sharing kernels while isolating userland programs within distinct environments for optimal resource utilization.

Operating Principles

• Containers enable diverse programs to run concurrently on a single operating system without additional OS layers. Each program operates alongside others with kernel-level isolation for enhanced efficiency.

• Despite running on RedHat or Ubuntu hosts, containers can execute different Linux distributions like Arch or Alpine internally. This setup mimics hypervisor-like behavior without actual virtualization due to shared kernels.

• Containers leverage shared kernels but isolate userland programs effectively. By removing redundant kernel instances within containers hosting similar userlands like Ubuntu's core components only once across multiple instances.

• Optimal container performance necessitates lightweight userlands such as CoreOS or Alpine instead of full Ubuntu setups. This strategy minimizes resource consumption while ensuring essential binaries support various applications efficiently.

Optimizing Container Performance

The discussion shifts towards optimizing container performance by emphasizing minimal dependencies per container for streamlined execution.

Performance Optimization

• Containers should encapsulate individual processes rather than multiple applications. Specialized containers dedicated solely to specific tasks like databases (e.g., Postgres), caching (e.g., Redis), or web serving (e.g., NGINX).

• Running Docker on Windows or Mac involves initiating a hypervisor first before deploying Linux-based containers within virtual machines. Communication between client-side Docker and VM-hosted Docker occurs via internal network protocols for seamless operation.

• For optimal performance parity with Linux hosts, Windows/Mac users must run Linux-based hypervisors beneath their containers due to inherent kernel differences affecting direct execution efficiency. Single-process focus ensures each container operates independently for streamlined functionality.

New Section

In this section, the speaker discusses the use of Dockerfiles to specify volumes and ports for sharing files and exposing containers. Additionally, they introduce Docker Machine for deploying containers on cloud servers like AWS EC2 or DigitalOcean.

Understanding Dockerfiles and Deployment

• Each container has its own Dockerfile specifying volumes and ports for file sharing.

• Docker Machine simplifies deploying containers on cloud servers like AWS EC2 by providing support for Docker on virtual machines.

• Orchestration tools like Docker Swarm, Mesos, Marathon, and Kubernetes help manage complex infrastructures with multiple virtual machines and containers.

Deployment with Kubernetes

This part delves into Kubernetes as a leading orchestrator in 2019, available across various infrastructure service providers such as Google Cloud, Microsoft Azure, and AWS.

Kubernetes Functionality

• Kubernetes views virtual machines as Nodes where multiple Docker containers can run within Pods.

• It offers centralized infrastructure management to handle container or node failures efficiently.

• Supports scenarios with multiple zones for optimal performance catering to diverse audiences globally.

Evolution of Container Ecosystem

The discussion shifts towards the evolution of container technologies beyond Docker, emphasizing initiatives like Open Container Initiative (OCI) standardizing container runtime and image formats.

Container Ecosystem Evolution

• Components like cgroups and namespace in Linux kernel are crucial for container functionality.

• Emergence of projects like gvisor, clearcontainers, katacontainers due to OCI standards impacting Kubernetes' dependency on Docker runtime.

Future Trends: Kubernetes Dominance

The speaker highlights the rising prominence of Kubernetes over Docker in the orchestration market and emphasizes the importance of mastering both technologies for modern sysadmins.

Future Technological Landscape

• With Kubernetes overshadowing Docker's market share in orchestration, understanding both technologies is essential for sysadmins today.