Shell Scripting for DevOps|Shell Scripting Zero 2 Hero|Shell Scripting Interview Questions| #devops

Introduction to Shell Scripting

Overview of the Tutorial

• Abhishek introduces the tutorial on shell scripting, aiming to take viewers from beginner to advanced levels.

• The focus will be on understanding shell scripting from a DevOps engineer's perspective, including its role in daily activities.

• The session will cover both basic and complex components of shell scripting, with examples provided for clarity.

Importance of Automation

• Automation is defined as reducing manual tasks; applicable across various fields beyond just DevOps.

• Examples are given where repetitive or tedious tasks can benefit from automation, enhancing efficiency in daily activities.

Understanding Shell Scripting

Practical Applications

• A simple example illustrates how to automate printing numbers from 1 to 10 using the echo command.

• As task complexity increases (e.g., printing numbers up to 1000), automation becomes essential for feasibility.

Creating Files with Shell Scripts

• Another example discusses creating multiple files using the touch command, emphasizing the need for automation when scaling tasks (e.g., creating 100 or 2000 files).

Getting Started with Writing Shell Scripts

Basic Requirements

• To write a shell script, one must create a file with a .sh extension; this is similar to other programming languages like Python (.py).

Setting Up Your Environment

• Viewers are encouraged to set up their environment by either using AWS or Oracle VirtualBox for practicing shell scripting.

Creating Your First Shell Script

Steps Involved

• Abhishek demonstrates how to create a new file on a Linux machine using the touch command followed by naming conventions for scripts.

Understanding Basic Linux Commands

Navigating Files in Linux

• The process of listing files on a Windows machine involves using the mouse to navigate through drives (C, D, etc.), while in Linux, this is done via command line.

• The command ls is used to list files in a directory. For example, typing ls will show all files including newly created ones like "first shell script.sh".

• To view files with timestamps and additional details such as permissions and ownership, the command ls -ltr can be utilized.

Utilizing Manual Pages for Command Reference

• If users forget the syntax or purpose of commands like ls, they can use the manual command by typing man <command> (e.g., man ls) to access detailed descriptions.

• The touch command is explained as a utility that changes file access and modification times; if a file does not exist, it creates one with default permissions.

Learning Key Commands

• Users are encouraged to explore options available for commands by appending them with the man command (e.g., checking options for ls).

• Three essential commands learned so far include:

• LS: Lists files/folders.

• Touch: Creates new files or updates timestamps.

• Man: Provides manual/reference information for any command.

Creating and Editing Files

• The speaker demonstrates creating a file using the touch command and verifying its existence with ls before proceeding to write a shell script within that file.

• To edit files in Linux, users can utilize text editors like VI (available by default), while Vim may require installation.

Terminal Usage and Tools

• Unlike Windows where right-clicking allows easy file creation, Linux requires different methods such as double-clicking content to copy filenames directly into terminal commands.

How to Use VI and Vim for Shell Scripting

Opening Files with VI and Vim

• The command to open a file in VI is straightforward, while Vim offers a more user-friendly interface. Downloading Vim is easy, but if it's unavailable, you can still use VI.

• Using the vim command directly (e.g., vim second_shell_script.sh) creates a new file automatically if it doesn't exist, allowing immediate writing.

Understanding the Touch Command

• The touch command serves a different purpose than Vim; it’s essential for automation tasks where multiple files need to be created without opening them.

• In scenarios requiring the creation of numerous files (e.g., 1000 files), using Vim would not be practical as it opens each file, which could lead to system overload.

Importance of Shebang in Shell Scripts

• To start writing a shell script, the first line should include a shebang (#!) followed by the path to the shell interpreter (e.g., /bin/bash). This indicates which interpreter should execute the script.

• The shebang line is crucial as it defines how the script will run on different systems. For example, common executables include Bash (/bin/bash), Dash (/bin/dash), and Ksh (/bin/ksh).

Executable Shell Options

• Different shells have unique syntax variations. While they share similarities in commands, slight differences exist that can affect script execution.

• Specifying an executable in your shell script informs Linux which interpreter to use when executing your commands. This is vital for ensuring compatibility across various environments.

Conclusion on Shell Script Execution

Understanding Shell Scripting: Bash vs. Sh

Choosing the Right Shell for Scripting

• The most widely used shell for scripting is Bash, which simplifies syntax compared to other shells like ksh.

• There are significant differences in how loops are written in Bash versus ksh, with Bash offering a more user-friendly approach.

• In scripts, you may encounter two common shebang lines: #!/bin/sh and #!/bin/bash, indicating different shell interpreters.

Understanding Shebang Lines

• The shebang line #!/bin/sh historically redirected to /bin/bash due to Linux's linking concept, but this is changing.

• Linking allows scripts using /bin/sh to execute as if they were running under Bash, although this may not always be the case now.

Recent Changes in Default Shell Behavior

• Some operating systems like Ubuntu have shifted their default from Bash to Dash, affecting script compatibility.

• It’s crucial for scripters to use #!/bin/bash explicitly to avoid issues when sharing scripts across different environments.

Key Differences Between Shell Types

• The difference between /bin/sh, /bin/bash, and /bin/dash lies in their linking behavior; previously both sh and bash pointed to bash, but now some systems default to dash.

Writing a Simple Shell Script

• A basic shell script can be created simply to print "My name is Abhishek" using the echo command in Bash.

• Unlike Windows where text can be printed directly, Linux requires using commands like echo within scripts for output.

Saving and Editing Scripts

How to Open, Write, and Save Files in Linux

Understanding File Operations in Linux

• In a Linux environment, it is essential to specify the purpose of opening a file—whether for reading, writing, or copying content.

• To enter insert mode for writing in a file using Vim, press Esc followed by i. This indicates that you are ready to input text.

• The status of being in insert mode is confirmed by an "INSERT" indicator at the bottom left of the screen. You can then start typing your content.

• To save changes made to a file after editing, press Esc, type :wq!, and hit Enter. This command saves the file and exits Vim.

• If you exit without saving (using :q!), any changes will be lost upon reopening the file.

Viewing File Contents Without Opening

• Instead of repeatedly opening and closing files in Vim, use the cat command to print the contents directly in the terminal.

• The cat command allows users to view file contents quickly without needing to edit or close them.

Executing Shell Scripts

• To execute a shell script, you can either use sh <filename> or prefix with ./<filename>. Both methods allow execution from the terminal.

• Using dot-slash (./) is common for executing executable files like shell scripts. It signifies that you're running a script located in your current directory.

Understanding Permissions in Linux

• When attempting to execute a newly created script, you may encounter "permission denied." This highlights Linux's security model regarding file execution rights.

• Even as the creator of a file, permissions must be explicitly granted before execution. This ensures only authorized users can run scripts.

Granting Permissions with chmod

• The command used for changing permissions on files is chmod. Understanding this command is crucial for managing access rights effectively.

Understanding the chmod Command in Linux

Overview of chmod Command

• The chmod command is used to change file permissions in Linux. It requires specifying the desired permissions along with the filename.

• Permissions are categorized into three main types: user (owner), group, and others. Each category has specific access rights.

Permission Categories

• The first category pertains to the root user (administrator), who has elevated privileges.

• The second category refers to group permissions, which allow a set of users access based on their group affiliation.

• The third category defines individual user permissions, specifically for the creator or current user of the file.

Granting Permissions Using Numbers

• Permissions can be granted using numeric values; for example, chmod 777 gives full access (read, write, execute) to all users.

• To learn more about chmod, one can use man chmod, which provides detailed documentation on its usage.

Understanding Numeric Values for Permissions

• The number '7' is significant as it represents full permissions: read (4), write (2), and execute (1).

• For instance, using chmod 444 grants read-only permission to everyone—user, group, and others—denying write and execute capabilities.

Practical Application of chmod

• When executing chmod 444 XYZ, only read access is granted. This means no modifications can be made by any user.

• If a file's permission is set to 444 and an attempt is made to edit it, a "permission denied" error will occur due to lack of write access.

Summary of Commands Learned

Shell Scripting Basics

Summary of Key Learnings

• The session emphasizes the importance of practicing shell scripting daily to master it effectively.

• The initial focus is on basic commands, starting with file operations and their creation methods.

File Creation Commands

• Two primary commands for creating files are discussed: touch (creates a file without opening it) and vim (creates and opens a file).

• The cat command is introduced for reading the contents of a file without opening it in an editor.

Listing Files and Permissions

• To list files in a directory, the ls command is used, which stands for "list."

• The chmod command allows users to change permissions on files, enhancing security and access control.

Manual Pages and Writing in Scripts

• The man command provides manual pages for other commands, offering detailed documentation (e.g., man ls).

• Instructions on writing within a shell script using Vim are provided; entering insert mode requires pressing 'i' after hitting 'Esc'.

Navigating Directories

• Users can check their current working directory using the pwd command, which displays the path they are currently in.

• To navigate into directories, users can utilize the cd (change directory) command.

Creating Directories

• Directory creation in Linux is done using the mkdir command (make directory), allowing users to create new folders easily.

• After creating a folder with mkdir, users can verify its existence by listing files with options like ls -ltr.

Changing Directories and Verifying Location

• To enter a newly created folder, use the cd <folder_name> command.

Understanding Basic Linux Commands and Shell Scripting

Overview of Key Linux Commands

• The PWD command is used to identify the present working directory in a Linux system, helping users understand their current location among numerous folders.

• The MKDIR command allows users to create new directories. This is essential for organizing files within the file system.

• The CD command stands for "change directory," enabling users to navigate between different directories easily.

Learning Shell Scripting Basics

• After learning about various commands, the next step involves writing a simple shell script that automates tasks like creating files and directories.

• A shell script will be created using the VIM editor, which combines file creation and opening into one step, streamlining the process.

Writing a Shell Script

• It’s important to start shell scripts with a shebang (#!/bin/bash) to specify the interpreter that should execute the script.

• Comments are crucial in scripting; they help explain code functionality. In shell scripts, comments are denoted by #, allowing developers to annotate their code without affecting execution.

Creating Files and Directories in Scripts

• To create files within a script, use the touch command. For example, two files can be created as part of an automated task.

• The script will also include commands to create a folder (named "Abhishek") and navigate into it before creating files inside.

Setting Permissions for Executing Scripts

• After saving the script, permissions must be set using chmod 777, granting full access rights (read/write/execute) to all users on the machine.

• Alternatively, more restrictive permissions can be applied (e.g., chmod 770), limiting access based on user groups while still allowing execution rights.

Understanding Shell Scripting Basics

Introduction to Shell Script Permissions

• The speaker explains the permission model in shell scripting, where users can read (4), write (2), and execute (1) scripts. Granting all permissions totals 7.

Executing a Sample Shell Script

• A sample shell script named sample_shell_script.sh is executed, which does not print output but creates a folder named "Abhishek."

Exploring Created Files

• Inside the "Abhishek" folder, two files are created as part of the script's task, demonstrating successful execution.

Simplicity of Writing Shell Scripts

• The speaker emphasizes that writing shell scripts is straightforward and summarizes key components involved in creating them.

Re-explaining Due to Technical Issues

• After a power cut interrupted screen sharing, the speaker restarts the explanation of the shell script and its functionalities.

Shell Scripting in DevOps

Role of Shell Scripting in Organizations

• The speaker discusses how shell scripting plays a crucial role in DevOps for infrastructure maintenance and code management using Git repositories.

Preference for Linux Over Windows

• Organizations prefer Linux due to its security and lightweight nature, making it suitable for managing infrastructure and code with tools like Git.

Automation Tasks with Shell Scripts

• Regular automation tasks such as cron jobs or executing Ansible scripts require writing shell scripts for efficiency within DevOps practices.

Example Scenario: A DevOps Engineer's Responsibilities

Understanding Shell Scripting in DevOps

Managing Virtual Machines with Shell Scripting

• A scenario is presented where a DevOps engineer manages 10,000 Linux-based virtual machines (VMs) and needs to monitor their health regularly.

• Developers report issues such as high CPU usage, low memory, or slow performance on specific VMs, prompting the need for efficient monitoring solutions.

• Instead of manually checking each VM using Linux commands, the engineer opts to automate this process through shell scripting.

• The shell script logs into the problematic VM automatically and evaluates parameters like CPU usage and memory status to identify issues.

• The script can be scheduled to run periodically without manual intervention, providing regular updates on the health of all 10,000 VMs.

Automation Benefits and Use Cases

• The automated script sends email notifications summarizing the health status of the VMs, highlighting any that are suspicious or have resource issues.

• Shell scripting is emphasized as an essential skill for DevOps engineers due to its ability to streamline various tasks beyond just monitoring node health.

• Learning Python is also recommended for more complex automation tasks that may not be feasible with shell scripts alone; however, mastering shell scripting first is crucial.

• Engineers can justify their use of custom scripts over existing tools by explaining that their scripts provide more comprehensive data than those tools allow.

• Various use cases for shell scripting include infrastructure automation and configuration management alongside daily operational tasks.

Key Learnings from Shell Scripting

• The session covers fundamental concepts behind shell scripting and its importance in automating routine checks on system resources like CPU and RAM.

• An interview question regarding differences between bin/sh and bin/bash was discussed as part of understanding different shells used in scripting environments.

• Essential shell commands were reviewed including file manipulation commands (ls, mkdir, rm) and permission settings (chmod).

How to Monitor CPU and RAM on Your Laptop

Understanding CPU and RAM Monitoring Commands

• The speaker discusses common errors related to CPU and RAM when overusing a laptop, emphasizing the importance of monitoring these components.

• To check CPU information, the command nproc is recommended, which lists the CPUs available on the machine.

• For memory status, the free command can be used to identify total, used, and free memory on the system.

Using Advanced Monitoring Tools

• The top command is introduced as a powerful tool for monitoring running processes on a machine. It provides insights into which processes consume more or less memory.

• The output from top includes details such as total processors running, active processes, their states (running/sleeping), process IDs (PIDs), CPU usage, and memory usage.

Node Health Evaluation Parameters

• Basic parameters for evaluating node health include CPU and RAM metrics along with commands discussed earlier.

• Interview questions may focus on how to monitor node health using tools like top or custom shell scripts.

Advanced Shell Scripting Concepts

• The speaker touches upon advanced topics in shell scripting such as trapping signals. For instance, pressing Ctrl+C sends a signal that can stop an executing script.

• Trapping signals allows scripts to continue running even if interrupt commands are issued by users.

Feedback and Future Content Suggestions

• Viewers are encouraged to provide feedback for future content focusing on advanced shell scripting concepts through comments.