MapReduce Programming | Hadoop Tutorial for Beginners | Hadoop [Part 12]

MapReduce and Data Processing in Banking

Overview of a Bank Use Case

• The speaker discusses a practical example involving a bank's use of MapReduce for processing transaction data from multiple branches.

• Different branches have varying transaction volumes, with Bangalore having 1 million transactions, Hubli with 10,000, Chennai with 2 million, and Goa with 100,000.

Understanding Data Skewness

• The transaction data includes customer numbers and amounts transacted; the goal is to calculate totals by branch.

• Each branch becomes a key in the MapReduce framework while each transaction amount serves as the value.

• Data skewness occurs when some branches have significantly more transactions than others, leading to uneven workload distribution among reducers.

Challenges with Reducer Distribution

• If only two reducers are defined for four branches, one reducer may take much longer due to handling more data (e.g., Bangalore's 3 million values).

• A custom partitioner can be implemented to control which keys go to which reducers, optimizing performance and ensuring balanced workloads.

Comparison of MapReduce and Spark

• In production environments, hash partitioning is often avoided due to its unpredictability; custom solutions are preferred for efficiency.

• Spark differs from MapReduce by keeping data in memory rather than writing it back to disk after mapping. This results in faster processing speeds—up to 100 times faster than traditional MapReduce methods.

Programming Language Considerations

• Writing MapReduce programs in Java can be complex and has a steep learning curve compared to Spark’s support for Scala or Python.

• While Python support exists for MapReduce, programming in Spark is generally quicker and more concise.

Key Takeaways on Data Representation

• Transaction data typically requires only essential columns (e.g., branch code and amount), reducing overall data size significantly (from terabytes down to manageable sizes).

• Effective processing relies on representing all necessary information as key-value pairs; this format allows various operations across different types of datasets.

Understanding MapReduce and Data Processing

Key Concepts in MapReduce

• The importance of processing logic in the mapper phase is emphasized, suggesting that all necessary data cleaning should occur here to avoid inefficiencies later in the process.

• Custom partitioning is discussed as a coding aspect within MapReduce programs. Java developers must compile their code into a class file and package it as a JAR for submission, while Python developers can use Hadoop streaming without compilation.

• The handling of intermediate results is crucial; if a node crashes, the unpersisted data will be lost. This highlights the need for proper data persistence strategies during processing.

Data Persistence and Recovery

• Intermediate results are stored on the local file system rather than being replicated like in traditional systems. If a machine fails, recovery requires re-running processes from scratch.

• During installation, configuring a MapReduce scratch directory is essential for storing output across machines. This setup ensures that data persists through various phases of processing.

• When shuffling starts after mapping, data is loaded into RAM instead of reading from hard disk storage to enhance performance and reduce latency.

Memory Management in Processing

• RAM serves as non-persistent storage; any restart will clear its contents. In contrast, hard disks provide persistent storage where data remains intact even after reboots.

• Temporary outputs during processing stages must be managed carefully to prevent loss due to crashes or failures. Proper persistence formats are necessary for identifying and retrieving this data later.

Framework Operations

• The shuffle and sort phase follows mapping, where temporary persisted data is loaded into RAM based on key-value pairs for further processing by reducers.

• A distributed framework manages these operations seamlessly; it tracks which mappers have run and requests necessary data accordingly without user intervention.

• A Java object within the framework maintains an associative array that indexes each key-value pair's location, facilitating efficient access during subsequent phases of processing.

Challenges in Fine-Tuning MapReduce