15 Minutes- Spark Clusters in Databricks Explained -Tips & Tricks | Azure Databricks Tutorials

Overview of Cluster Types in ASO Data Braks

Introduction to Clusters

• The video provides a comprehensive overview of different types of clusters available in ASO Data Braks, emphasizing the importance of understanding these concepts before creating a cluster.

• Viewers are encouraged to watch previous videos on ASO Data Braks and instance creation for foundational knowledge.

All-Purpose Compute Cluster

• An all-purpose compute cluster can be utilized for various tasks, including executing notebooks and running jobs within data pipelines.

• This type of cluster is versatile and suitable for development work, making it ideal for general use cases.

Job Compute Cluster

• A job compute cluster is specifically designed for executing notebooks as jobs, not suited for development work like the all-purpose compute cluster.

• Cost-wise, job clusters are cheaper than all-purpose compute clusters due to their focused functionality.

Instance Pools

• Instance pools consist of idle resources that can be allocated or deallocated based on workload requirements, similar to a swimming pool filled with VMs.

• Users can assign more instances from the pool when processing power is needed and release them back when workloads decrease.

Creating an All-Purpose Compute Cluster

• To create a new cluster, users click on the "create cluster" option and fill out various settings; understanding these options is crucial.

• The policy dropdown offers several types (e.g., unrestricted), affecting available configurations; unrestricted allows access to all settings necessary for creating a robust cluster.

Node Configuration Options

• Users must choose between multi-node or single-node options based on performance needs; multi-node supports high-performance operations but incurs higher costs.

• Selecting multi-node enables configuration of additional settings such as worker types and driver types, which are not available in single-node setups.

Access Modes Explained

• Three access modes exist: single user (exclusive access), shared (accessible by multiple users), and no isolation shared.

Accessing ASU Data Lake and Credential Pass-Through

Understanding Access to ASU Data Lake

• Users must have access to the ASU Data Lake for authentication when using Azure Databricks, ensuring that only authorized users can access data.

• The concept of credential pass-through needs to be enabled when creating a shared cluster; failure to do so results in an error message.

• Enabling credential pass-through is done through a checkbox in the advanced options section during cluster setup.

Cluster Isolation and Performance Implications

• The "no isolation" shared option allows multiple users to operate in the same environment, which may lead to performance issues if one user runs heavy workloads.

• In real-time projects, companies typically prefer single-user or shared clusters for better performance management.

Cluster Configuration and Performance Optimization

Databricks Runtime Version

• Users can select from various Spark versions categorized as standard or machine learning (ML), with the latest version being Spark 3.4.0 as of now.

• Different versions are updated frequently by Databricks, allowing users to choose based on their library requirements.

Photon Acceleration and Worker Types

• Photon acceleration improves performance for SQL-based operations while minimizing costs; however, it is not mandatory at this stage.

• Worker types determine CPU and memory allocation for executing Spark jobs; different categories are available based on workload needs.

Configuring Workers and Autoscaling

• Users can configure minimum and maximum workers based on workload demands, enhancing performance during large operations.

• Enabling autoscaling allows dynamic adjustment of worker nodes according to workload fluctuations, optimizing resource usage.

Driver Type Configuration

Role of Driver Type in Spark Applications

How to Create a Cluster in Azure Databricks

Understanding Driver and Worker Nodes

• The driver node identifies the different types of tasks needed for an operation and instructs the worker nodes to perform these tasks.

• The driver node can be likened to a team lead, while the worker nodes execute the tasks as directed by the driver node.

Configuring Cluster Settings

• The "Terminate Clusters on Inactivity" option allows clusters to automatically shut down after 120 minutes of inactivity, which is crucial for cost-saving.

• Adjusting this setting from 120 minutes to 15 minutes ensures that even if forgotten, the cluster will turn off automatically, further reducing costs.

Creating a Cluster

• For demonstration purposes, a single-node cluster is chosen instead of a multi-node option due to current workload requirements.

• Before creating the cluster, it's essential to change its name (to "Dev cluster") and confirm all settings including inactivity time adjustments.

Conclusion and Future Steps