Shell Scripting for DevOps | Zero 2 Hero Part-2 | Shell Scripting Interview Q&A | #devops

Name: Shell Scripting for DevOps | Zero 2 Hero Part-2 | Shell Scripting Interview Q&A | #devops
Uploaded: 2022-12-01T14:00:52.000Z
Duration: 2 h 20 min 30 s

Introduction to Advanced Shell Scripting

Overview and Context

Abhishek welcomes viewers back to his channel, expressing gratitude for the support on the previous video about shell scripting.

He encourages viewers to watch the prior episode before proceeding with advanced concepts in shell scripting.

Recap of Previous Video

The focus remains on Bash scripting due to its widespread availability across operating systems.

A brief recap is provided, highlighting that the last session ended with writing a simple shell script that creates files and folders.

Key Commands for Node Health Analysis

Important Commands Discussed

Review of commands learned previously:

DF Command: Displays information about available disk space.

Free Command: Shows memory usage statistics.

Nproc Command: Provides details about CPU count on the virtual machine.

Emphasizes the importance of these parameters for DevOps engineers when assessing hardware specifications.

Process Monitoring

Introduction of the Top Command, which lists currently running processes and their resource consumption (CPU and memory).

These four commands (DF, Free, Nproc, Top) are essential tools for analyzing node status effectively.

Creating a Custom Node Health Script

Purpose of the New Script

The goal is to write a custom shell script named node_health.sh that detects node health issues in a virtual machine.

Practical Application

The script will be stored in a GitHub repository for easy access; users can download it when they encounter issues with their virtual machines.

Best Practices in Shell Scripting

Writing Structured Scripts

Abhishek emphasizes starting scripts with a shebang followed by specifying Bash as the executable. This prevents errors if another default shell is used instead.

Documentation within Scripts

Importance of including comments at the beginning of scripts:

Author's name (e.g., "Abhishek")

Date written (e.g., "1st December")

Purpose and prerequisites for using the script

Understanding Shell Script Best Practices

Importance of Metadata in Scripts

Always include metadata information in your script files to clarify their purpose. For example, a file named addition.sh should specify whether it performs addition of two, three, or more numbers.

Essential Commands for System Information

Utilize commands like df -h for disk space, free -g for memory usage, and nproc to check CPU resources when scripting.

Granting Permissions for Scripts

Use the command chmod 777 to grant all permissions temporarily while testing scripts. However, ensure appropriate permissions are set in a production environment.

Enhancing Output Clarity with Echo Statements

To improve user understanding of script outputs, use echo statements before each command. This helps identify which output corresponds to which command.

Debugging Shell Scripts Effectively

Instead of using multiple echo statements, enable debug mode by adding set -x at the beginning of your script. This will display each executed command along with its output.

Balancing Debugging and Security

If certain commands should not be visible to users (for security reasons), you can comment out the set -x line to prevent displaying those commands during execution.

Understanding Processes on Your Machine

Understanding Processes in Virtual Machines

Overview of Processes

The speaker describes a scenario where multiple processes are running simultaneously, such as watching a YouTube video while browsing Facebook. This illustrates the concept of concurrent processes on a system.

In virtual machines, especially within organizational contexts like Amazon, numerous processes can be active due to various applications and microservices being deployed.

Each application (e.g., YouTube, Chrome) operates under its own process. In large systems like Amazon, hundreds of microservices may each initiate distinct processes.

Identifying Running Processes

On Windows, users can easily view running processes through graphical interfaces or command prompts. However, Linux requires command-line tools for this purpose.

The command ps -ef is introduced as a method to list all running processes in Linux. The ps stands for "process status," and the -ef option provides detailed information about each process.

Detailed Process Information

Using ps -ef, users can see not only active but also background and stopped processes (e.g., daemon and zombie processes).

An example is given where Python runs by default on every virtual machine alongside specific Amazon-related processes.

Filtering Specific Processes

To find specific process IDs (PIDs), one must first identify relevant Amazon processes using the ps -ef command.

After identifying these processes, the speaker emphasizes the importance of filtering results to obtain only necessary information using scripting techniques.

Utilizing Grep for Process Identification

The grep command is introduced as a powerful tool to filter output from commands like ps -ef. By appending | grep Amazon, users can isolate only those Amazon-related processes from the full list.

This approach allows users to efficiently retrieve relevant data without sifting through extensive lists of unrelated information.

Extracting Process IDs

Once filtered results show relevant Amazon processes, further refinement may be needed if only PIDs are required.

The significance of PIDs is highlighted; they are essential for managing tasks such as terminating or analyzing specific running applications.

Command Breakdown

A breakdown of the combined command (ps -ef | grep Amazon) clarifies that it prints all process details while filtering out unnecessary lines based on user-defined criteria with grep.

Understanding the Pipe Command in Shell Scripting

Introduction to the Pipe Command

The grip command is used to filter output, such as names containing "PSI" (e.g., Sai Krishna, Sai Hari).

The pipe parameter (|) is crucial for integrating commands by sending the output of one command to another.

For example, if a script returns numbers and you want only specific ones (like "5"), you can use grip 5 after piping.

Practical Example with Shell Script

A simple shell script named test.sh can be created that echoes several numbers.

By executing ./test.sh | grip 1, it filters and prints only those numbers containing "1", demonstrating the power of the pipe command.

Detailed Explanation of Commands

The command structure involves using ps -ef to list running processes and then piping this output into a grip command to filter for specific processes (e.g., Amazon).

An alternative method is using grip 1 < ./test.sh, which achieves similar filtering from the script's output.

Summary of Key Learnings

Key concepts learned include:

Writing file information is essential for understanding scripts.

Using set -x helps print commands in debug mode.

Finding running processes with ps -ef.

Filtering outputs with the grip command.

Utilizing pipes to send outputs between commands effectively.

Common Interview Question: Pipe Behavior

An interview question often posed involves predicting the output of a date command piped into an echo statement.

The expectation is that it should print formatted date information; however, it does not work as intended when executed as date | echo "today is...".

Understanding Output Channels

The reason behind this behavior lies in how commands handle their outputs. The date command sends its output to standard input (STDIN), while pipes only capture standard output (STDOUT).

Understanding Command Line Utilities and Scripting Best Practices

The Basics of Command Output

When asked about the output of a command, it's important to note that the date command outputs directly to standard output (STDOUT), which cannot be piped into another command if it doesn't send information to STDIN.

Filtering Processes with PS and AWK

To filter processes related to Amazon, use ps -ef | grep Amazon. However, only process IDs are needed, which can be extracted using the awk command.

The awk command is essential for processing text data. It allows users to extract specific fields from output, making it a powerful tool in scripting and data manipulation.

Understanding AWK Functionality

AWK is described as a pattern scanning and processing language. It's particularly useful for retrieving specific columns from structured text data.

For example, when using ps -ef, you can retrieve user or process information by specifying column numbers with AWK.

Practical Application of AWK

By piping commands together (e.g., ps -ef | grep Amazon | awk 'print $2'), you can isolate specific columns such as process IDs effectively.

This method demonstrates how to filter out unnecessary information while focusing on relevant data points like user names or process IDs.

Combining GREP and AWK

In scenarios where you need to extract names from a file, combining GREP with AWK enhances efficiency. For instance, using grep name filename followed by an appropriate AWK print statement retrieves desired information succinctly.

Best Practices in Scripting

When utilizing pipes in scripts, it's crucial to implement error handling practices such as using set -e and set -o pipefail. These commands ensure that scripts exit upon encountering errors during execution.

Understanding Shell Script Error Handling

Importance of set -e in Shell Scripts

The speaker emphasizes the necessity of using set -e in shell scripts to ensure that if a command fails, the script stops executing further commands. This prevents subsequent steps from running when they depend on the success of previous ones.

An example is provided where creating a user must precede creating a file and adding the username to it. If the user creation fails and set -e is not used, the script continues, leading to an empty file without any username.

The absence of error handling can result in misleading outputs for end users who may believe everything executed correctly, despite critical failures occurring earlier in the script.

It is highlighted that every time a script is written, it should exit upon encountering an error to maintain integrity and expected functionality.

Limitations of set -e

While set -e is crucial for stopping execution on errors, it does not handle errors effectively when pipes are involved. This limitation necessitates additional commands for comprehensive error management.

When using pipes, if one command fails but another succeeds, set -e will only check the last command's status. This can lead to unnoticed failures within earlier commands in a pipeline.

Enhancing Error Handling with Pipe Failures

To address issues with piped commands, it's recommended to use set -o pipefail. This ensures that if any command in a pipeline fails, the entire pipeline returns an error status.

By combining both set -e and set -o pipefail, scripts become more robust against errors across all commands and pipelines.

Best Practices for Using Set Commands

Users can combine multiple set options (like set -ex) into one line; however, separating them into different lines is advised for clarity and ease of modification later on.

Keeping set options distinct allows easier adjustments based on organizational requirements or debugging needs without losing track of what each option does.

Conclusion: Clarity Over Conciseness

Understanding Log File Management in DevOps

Engaging with the Audience

The speaker encourages viewers to leave comments and questions, assuring them that all inquiries will be addressed in the comment section or future videos.

Transitioning to Key Topics

The discussion shifts towards best practices for DevOps engineers, emphasizing a structured approach to avoid confusion.

Common Practices in Error Handling

When applications fail, the first step is typically to check log files for errors. This is a fundamental practice among DevOps engineers.

Log files can be extensive; thus, finding specific errors requires efficient searching techniques.

Analyzing Log Files

A sample log file is referenced to illustrate how application logs are formatted and the volume of information they contain.

Engineers often encounter various log levels (Trace, Info, Error), focusing primarily on error-level logs for troubleshooting.

Utilizing Command Line Tools

To extract error messages from log files on Linux systems, commands like cat and grep are commonly used.

As logs accumulate over time, they may be stored externally (e.g., Google Cloud Storage or AWS S3), necessitating different retrieval methods.

Retrieving Logs with Curl Command

The curl command is introduced as a tool for retrieving log files from external storage locations via their URLs.

Similarities between curl and API request tools like Postman are highlighted; both serve to facilitate data retrieval but in different contexts.

Practical Application of Curl Command

By using curl, engineers can access logs stored online and filter them for errors using additional commands like grep.

The speaker explains how this process mirrors making GET requests in programming languages such as Python using libraries like Requests.

Conclusion on Automation and Efficiency

Emphasizing simplicity, the speaker notes that utilizing commands like curl streamlines the process of accessing necessary information from external sources.

Understanding Curl and Wget Commands in DevOps

Overview of Curl Command

The curl command can be used as an alternative to Postman or the Python request module for retrieving information, such as log files or downloading packages.

Curl is versatile; for instance, it can download Python from the internet using a simple command.

Comparison Between Curl and Wget

Wget is another command that serves a similar purpose but functions differently. While wget downloads files directly to your system, curl outputs data to the terminal.

Using wget requires two steps: first downloading the file and then using commands like cat and grep to extract specific information from it.

In contrast, curl allows you to retrieve information in one step without saving it locally unless specified.

Practical Applications of Each Command

Depending on your needs—whether you want to save output or just view it—you should choose between curl (for direct output) and wget (for saving files).

Understanding these differences is crucial for interviews, particularly regarding how each command operates within DevOps contexts.

Exploring the Find Command in Linux

Introduction to Find Command

The find command is essential for locating files within a Linux system, especially when dealing with numerous files across various directories.

Usage Scenario

For example, if you're unsure where a specific file (like pm.d) exists within your system's directory structure, you can use find with root access.

Executing Find Command

To execute the find command effectively, use find / -name "pm.d" which searches through all directories starting from root (/).

Navigating User Permissions in Linux

Importance of Root User Access

It's important to understand user permissions; while root access allows full control over the system, it's risky due to potential irreversible changes.

Switching Users Safely

Understanding User Privileges and Shell Scripting in Linux

User Privileges and Commands

The command sudo allows users to execute commands with root privileges, enabling them to perform tasks that require higher access levels.

Users can remain on their standard user account (e.g., Ubuntu) but use sudo when they need to run a command as the root user occasionally.

To switch to the root user while executing a command, one can use sudo su -, which grants temporary access to root privileges.

When using the find command, appending sudo allows users to search through directories that would otherwise be restricted due to permission errors.

The find command is powerful for locating files across the entire file system based on specified criteria.

Writing Conditional Statements in Shell Scripting

DevOps engineers should learn how to write conditional statements such as if loops, if else loops, and for loops in shell scripting.

The syntax of if conditions is consistent across programming languages; however, specific implementations may vary slightly.

An if statement begins with if, followed by a condition. If true, it executes specified actions; otherwise, it moves to an else clause.

In shell scripting, ending an if loop is done by writing "fi," which is "if" spelled backward—this indicates the conclusion of the conditional block.

A practical example involves comparing two variables (a and b), where you check which variable holds a greater value and print accordingly.

Implementing Loops in Shell Scripting

For loops are used for executing repetitive tasks or iterating over a set of values within scripts.

They are essential for performing actions multiple times without rewriting code—common across various programming languages but implemented differently depending on context.

Understanding For Loops in Shell Scripting

Introduction to Loops

The task of printing all days in a week or numbers from 1 to 100, for example, can be automated using loops instead of manual entry.

Various types of loops exist in shell scripting: while loop, for loop, until loop, and select loop; however, the focus will be on the for loop due to its common usage.

Syntax and Structure of For Loop

The basic syntax for a bash for loop is for I in 1..100; do echo $I; done, where you specify a range (e.g., 1 to 100).

The first statement defines the condition (e.g., if I is between 1 and 100), followed by an action that should occur if the condition is met.

Execution Flow of For Loop

Upon meeting the condition (I = 1), it executes the specified action (printing I), then increments I until it reaches the upper limit (100).

The completion of the for loop is indicated by the done statement.

Understanding Conditional Statements

Familiarity with if-else conditions alongside loops is essential; resources are available online to troubleshoot syntax issues.

Key Concepts in Shell Scripting

Automation Practices

Discussed various scripts used by DevOps engineers including node automation and best practices when writing scripts.

Filtering Information from Log Files

Techniques were introduced on how to filter information from large log files using commands like curl and wget.

The Trap Command Explained

Purpose of Trap Command

The trap command is used to handle signals within scripts. It’s considered tricky but important for managing script execution effectively.

Understanding Signals in Linux

Signals are actions performed via keyboard or commands that instruct Linux processes. An example includes using kill -9 <process_name> to terminate a process.

Handling Process Termination

When executing commands like kill or pressing Ctrl+C during script execution, signals are sent indicating that a process should stop running.

Overview of Linux Signals

Types of Signals

Understanding the Trap Command in Linux

Overview of Signals in Linux

The discussion begins with an overview of various signals in Linux, focusing on the "trap" command. An example is given using Ctrl+C as a signal that interrupts script execution.

Functionality of the Trap Command

The trap command allows users to manage how their scripts respond to specific signals, such as Ctrl+C, preventing unintended interruptions during execution.

Setting Up Signal Trapping

Users can configure their scripts to either ignore certain signals or execute alternative actions (like sending notifications) when those signals are received.

Syntax and Usage

The basic syntax for using the trap command involves specifying the action to take followed by the signal you want to trap. For instance, trap 'echo "Don't use Ctrl+C"' SIGINT will display a message if Ctrl+C is pressed.

Practical Applications of Trap Command

In real-world scenarios, trapping signals can prevent incomplete data entries into databases. If a user interrupts a script while it’s populating data, it may lead to inconsistencies.

A practical example includes setting up a trap so that if Ctrl+C is pressed during database operations, all partially entered data can be deleted instead of leaving half-completed records.

Conclusion and Further Learning Opportunities

The speaker encourages viewers to engage by asking questions or requesting further explanations about commands like curl or SMTP in future videos.