VIDEO
HIGHLIGHT
Log InSign up
Oozie Hadoop Tutorial | HDFS Processing | Hadoop Tutorial for Beginners | Hadoop [Part 7]
SummaryTranscriptChat

Oozie Hadoop Tutorial | HDFS Processing | Hadoop Tutorial for Beginners | Hadoop [Part 7]

Understanding Hadoop and Resource Management

Overview of Apache OZ Tool

  • The tool discussed is Apache OZ, which is used alongside Hadoop for scheduling programs.
  • Users can schedule jobs to analyze large datasets (e.g., 100 TB), but immediate submission may fail due to resource contention.

Resource Allocation in Hadoop

  • Different roles (developer, tester, researcher) require varying access levels to the Hadoop cluster; resource allocation is managed through queues (Q).
  • Parallelism is emphasized; multiple teams can submit jobs to their respective queues rather than directly to the cluster.

Queue Management and Scheduling

  • Example given: a Hadoop cluster with 10 data nodes, each having specific RAM and processor cores. Proper management prevents monopolization of resources by any single job.
  • Multiple queues can be created with defined resource allocations (e.g., developer Q gets 60% of resources).

Scheduler Types and Policies

  • Various schedulers exist (capacity scheduler, fair scheduler); users must specify a queue when submitting jobs.
  • Default scheduling policy is FIFO (First In First Out), but custom policies can also be implemented.

Data Storage and Processing in Hadoop

  • HDFS allows storage of various file types without strict format requirements; processing logic must be written separately.
  • Hive serves as a data warehouse on Hadoop that accepts SQL queries but operates differently from traditional RDBMS systems.

Comparison Between Hadoop and RDBMS

  • Performance improvements are possible with Hive indexing, but it should not be compared directly with RDBMS due to fundamental differences in operation.
Playlists: Hadoop Tutorial for Beginners | Big Data Hadoop Tutorial for Beginners | Hadoop Tutorial
Video description

#BigData | What is Big Data Hadoop? How does it helps in processing and analyzing Big Data? In this course, you will learn the basic concepts in Big Data Analytics, what are the skills required for it, how Hadoop helps in solving the problems associated with the traditional system and more. About the Speaker: Raghu Raman A V Raghu is a Big Data and AWS expert with over a decade of training and consulting experience in AWS, Apache Hadoop Ecosystem including Apache Spark. He has worked with global customers like IBM, Capgemini, HCL, Wipro to name a few as well as Bay Area startups in the US. #OozieHadoopTutorial #BigDataHadoop #Hadoop #GreatLakes #GreatLearning About Great Learning: - Great Learning is an online and hybrid learning company that offers high-quality, impactful, and industry-relevant programs to working professionals like you. These programs help you master data-driven decision-making regardless of the sector or function you work in and accelerate your career in high growth areas like Data Science, Big Data Analytics, Machine Learning, Artificial Intelligence & more. - Watch the video to know ''Why is there so much hype around 'Artificial Intelligence'?'' https://www.youtube.com/watch?v=VcxpBYAAnGM - What is Machine Learning & its Applications? https://www.youtube.com/watch?v=NsoHx0AJs-U - Do you know what the three pillars of Data Science? Here explaining all about the pillars of Data Science: https://www.youtube.com/watch?v=xtI2Qa4v670 - Want to know more about the careers in Data Science & Engineering? Watch this video: https://www.youtube.com/watch?v=0Ue_plL55jU - For more interesting tutorials, don't forget to Subscribe our channel: https://www.youtube.com/user/beaconelearning?sub_confirmation=1 - Learn More at: https://www.greatlearning.in/ For more updates on courses and tips follow us on: - Google Plus: https://plus.google.com/u/0/108438615307549697541 - Facebook: https://www.facebook.com/GreatLearningOfficial/ - LinkedIn: https://www.linkedin.com/company/great-learning/ Great Learning has collaborated with the University of Texas at Austin for the PG Program in Artificial Intelligence and Machine Learning and with UT Austin McCombs School of Business for the PG Program in Analytics and Business Intelligence.