Sesión 2 Análisis de Datos

Introduction to Data Analysis Methodologies

Welcome and Course Overview

• Humberto Marín Vega introduces the second session of the data analysis preparatory module, emphasizing attendance rules and course structure.

• Attendance is mandatory during synchronous classes, with students required to log their names via a provided link or QR code from 9 AM to 9 PM Mexico City time.

• If unable to attend live sessions, students must comment on the recording for attendance verification.

Course Structure and Evaluation

• The preparatory course has no creation requirements; students have two attempts for the final evaluation available on January 17th.

• This session focuses on Module One: Introduction to Data Analysis, laying foundational knowledge necessary for working with data.

Understanding Data Analysis Methodologies

Importance of Methodologies

• Today's focus is on Section 1.2: methodologies for data analysis, aiming for students to understand various methodologies' purposes and applications by class end.

• A methodology in data analysis consists of steps, techniques, and processes that transform data into useful information for decision-making.

Traditional Approach to Data Analysis

• The traditional approach follows a clear sequence: defining problems/hypotheses, collecting data, cleaning/preparing it, conducting exploratory analysis, applying statistical techniques/models, and interpreting results.

• Key phases include problem definition (what needs investigation), data collection (creating a dataset), cleaning/preparation (removing noise), application of models/statistics (predictive techniques), followed by interpretation/communication of results.

Advantages and Disadvantages of Traditional Methods

Evaluating Traditional Approaches

• Advantages include ease of understanding and widespread use in scientific research; however, disadvantages involve inflexibility in adapting to changing problems or large datasets.

• While still useful in many contexts, traditional methods may not be optimal for modern projects involving extensive information processing.

Introduction to CRISP-DM Methodology

Overview of CRISP-DM

• The discussion transitions into CRISP-DM (Cross Industry Standard Process for Data Mining), highlighting its significance as a standard framework specifically designed for advanced analytics projects.

• CRISP-DM is versatile across various sectors such as education, health care, business, marketing—applicable wherever data-driven insights are needed.

Phases of CRISP-DM

• The first phase involves business understanding—defining the problem at hand without delving into data yet. Understanding objectives and potential impacts is crucial before proceeding further.

Understanding Business Solutions and Data Processing

Identifying Business Problems

• The first step in understanding business solutions is to identify the specific problem that needs resolution, along with the data that will be manipulated.

• It is crucial to become experts in the business context to gain a detailed comprehension of the information being handled and the decisions to be made.

Understanding Data

• The second phase involves understanding what data exists, its sources, and its reliability.

• It's important to check for anomalies such as null values or outliers that could affect model performance.

Data Preparation

• The third phase focuses on data preparation, which includes cleaning (removing nulls and duplicates), variable selection, and transformation into a usable format.

• This stage often consumes significant time due to the need for thorough data handling before modeling can occur.

Modeling Techniques

• In the fourth phase, statistical techniques or algorithms are applied for decision-making purposes.

• Evaluation follows modeling to verify if objectives set at the beginning are met before deploying results into real-world applications.

Advantages of CRISP-DM Model

• The CRISP-DM model is iterative and flexible; phases can be revisited if issues arise during any stage of development.

• For example, one might return from modeling back to data preparation if inconsistencies are found in the dataset.

Applicability Across Sectors

• CRISP-DM's flexibility allows it to be applicable across various sectors including healthcare, market analysis, finance, and electoral studies.

• Its comprehensive nature makes it one of the most realistic methodologies for real-world projects due to its adaptability.

KDD: Knowledge Discovery in Databases

Overview of KDD Process

• KDD focuses on discovering useful knowledge within large volumes of data through several stages: selection of relevant data, preprocessing errors, transformation for analysis, mining patterns, interpretation, and evaluation.

Stages of KDD

• The process begins with selecting relevant datasets followed by preprocessing where errors like null values are corrected.

• Transformation prepares cleaned data for analysis while mining seeks patterns that lead to meaningful interpretations.

Advantages of KDD Methodology

• KDD excels when working with large datasets; it's effective at generating knowledge rather than just predictions or models.

Limitations Compared to CRISP-DM

• A notable limitation is KDD's lack of emphasis on business context compared to CRISP-DM’s focus on becoming business experts.

SENMA Methodology Overview

Introduction to SENMA

• The SENMA model, which stands for Sample Explore Modify Model Access, is developed by SAS Perl and focuses on predictive modeling.

• It is particularly designed for statistical and predictive modeling, starting with the sampling stage to extract a representative data sample.

Stages of SENMA

• Sample: This initial stage involves taking a sample of information from the dataset for analysis.

• Explore: In this phase, statistical analysis is conducted to understand the behavior of the data within the sample. This helps in identifying patterns useful for modeling later on.

• Modify: This stage focuses on cleaning and transforming data to enhance accuracy before applying models.

• Model: Here, predictive models are applied based on the prepared data.

• Access: The final stage evaluates the quality of results obtained from the models applied. Overall, SENMA consists of five stages: Sample, Explore, Modify, Model, and Access.

Comparison with Traditional Data Analysis

Traditional Data Analysis Phases

• Traditional data analysis includes six phases: problem definition or hypothesis formulation, data collection, cleaning/preparation, exploratory data analysis (EDA), application of statistical techniques/models, and interpretation/communication of results.

Differences in Approaches

• While traditional methods emphasize business understanding and expertise in domain knowledge (as seen in Cristian's approach), they differ from more analytical-focused methodologies like SENMA that prioritize technical aspects over business comprehension.

Agile Methodologies in Data Analysis

Importance of Agile Methodologies

• Agile methodologies can be adapted for current projects involving data analysis due to their flexibility and ability to quickly adapt to changes during project execution. This adaptability makes them suitable for dynamic environments where requirements may shift frequently.

Examples of Agile Methodologies

• Notable agile methodologies include Scrum (which works with short cycles called sprints), Kanban (focused on visual workflow management), and Link Analytics (emphasizing key metrics definition and rapid hypothesis validation). These approaches are beneficial when projects require constant adaptation due to changing conditions or requirements.

Machine Learning Lifecycle - MLOps

Introduction to MLOps

• Transitioning into modern practices beyond mere data analysis leads us into Machine Learning Operations (MLOps), which encompasses all necessary practices ensuring that models function effectively once deployed into production settings rather than remaining theoretical constructs confined to laboratories.

Stages in MLOps Lifecycle

1. Define Problem: Clearly articulate what needs predicting; identify model utility and target user base who will utilize outcomes generated by the model.

2. Data Collection & Preparation: Gather relevant datasets; clean and transform them adequately so they are ready for training purposes.

3. Further stages include understanding the model itself, validating it post-training, deploying it into operational environments, followed by ongoing monitoring and maintenance efforts post-deployment.

• Each step is crucial as it ensures that machine learning applications remain effective throughout their lifecycle while adapting as needed based on real-world performance feedback.

• Understanding these stages allows practitioners to build robust systems capable of delivering reliable predictions consistently over time.

These notes encapsulate key insights from each section while providing timestamps for easy reference back to specific parts of the transcript or video content discussed above.

Machine Learning Lifecycle and Methodologies

Stages of Machine Learning Model Development

• The training phase involves using machine learning algorithms to learn patterns from identified data. This is crucial for model development.

• Validation follows training, ensuring the model functions correctly and generalizes well to new data. This step is essential for assessing model performance.

• Deployment entails implementing the model in a real-world system, such as an application or institutional platform, allowing it to operate in real-time.

• Continuous monitoring and maintenance (MLOps) are vital since data evolves over time; without supervision, models can lose accuracy. Regular updates are necessary to adapt to changing needs.

• The lifecycle's importance lies in its adaptability to modern AI environments, enabling continuous operation and automation of processes like model retraining with new data inputs.

Comparative Analysis of Data Analysis Methodologies

• A comparative table illustrates various methodologies for data analysis, helping summarize key differences among them effectively. Understanding these distinctions is critical for selecting the appropriate methodology for specific problems.

Traditional Approach

• The traditional approach is linear, progressing step-by-step without easily reverting to previous stages; this suits academic research where problems are well-defined from the start but lacks flexibility for evolving projects.

Iterative Business-Oriented Approach

• An iterative approach focuses on solving real organizational problems rather than just analyzing data; it's applicable across sectors and covers the entire process from problem understanding to solution deployment, making it suitable for complex projects like data mining.

Knowledge Discovery in Databases (KDD)

• KDD emphasizes discovering knowledge by identifying patterns within large datasets; however, it does not prioritize implementation or deployment of models, making it less practical when immediate application is required.

Statistical Modeling (SENMA)

• SENMA focuses on rapid construction of predictive statistical models but may lack depth in understanding business problems; it's often associated with SAS projects aimed at efficiency rather than comprehensive analysis.

Agile Methodologies

• Agile methodologies highlight flexibility and adaptability to change, making them ideal for projects with constantly shifting requirements that demand ongoing results; they are prevalent in modern data science initiatives due to their collaborative nature.

Machine Learning Lifecycle Summary

• The machine learning lifecycle emphasizes production-oriented approaches that ensure models remain functional over time through continuous monitoring and maintenance; this automation aspect is crucial as it allows sustained operational effectiveness beyond initial deployment phases. Understanding each methodology's unique purpose helps identify the best fit for specific analytical challenges faced by organizations today.

Methodology Selection for Data Analysis

Key Considerations Before Choosing a Methodology

• The choice of methodology depends on the specific problem to be solved, as different methodologies excel in various areas such as research, knowledge discovery, rapid modeling, or production.

• Three critical questions should guide the selection process:

1. What problem do I want to solve?

2. What type of data do I have?

3. How important is it to implement the solution in production?

Practical Case Study: University Student Retention

• A university aims to identify factors influencing student dropout rates before the second year, focusing on data like age, grades, attendance, socioeconomic status, and tutoring participation.

• The analysis goes beyond mere data examination; it seeks to understand complex phenomena involving both academic and social aspects.

Objectives of the University’s Analysis

• The university has several clear objectives:

• To deeply understand what influences student dropout rates.

• To analyze available data for patterns and trends related to this issue.

• They aim to build a predictive model that identifies at-risk students for timely intervention.

Importance of Implementation

• Implementing the model in production is crucial for real-time usage and continuous adaptation based on changing data over time.

Recommended Methodology: CRISP-DM

• Traditional approaches may fall short as they often lack deployment considerations; CRISP-DM is recommended because it encompasses understanding institutional perspectives and allows for comprehensive analysis.

• This methodology supports analyzing social and academic factors while facilitating model construction and evaluation with an emphasis on real-time deployment.

Current Technologies for Data Analysis

Overview of Data Analysis Technologies

• The session will cover current technologies used in data analysis, emphasizing their purposes and appropriate contexts for use.

Definition of Data Analysis Technologies

• Technologies include tools, languages, platforms, and services that enable collecting, storing, processing, analyzing, visualizing, and modeling data or information.

Integration of Tools in Projects

• No single tool operates independently; projects typically involve combining various technologies organized by categories.

Programming Languages Focus: Python

• Python is highlighted as a leading programming language in data science due to its user-friendliness and extensive library support. Further detailed discussions about Python are scheduled for an upcoming session.

Implementation of Key Libraries in Data Analysis

Overview of Key Libraries

• The discussion begins with the introduction of essential libraries for data analysis, including Pandas for data manipulation and analysis.

• NumPy is highlighted as a tool for numerical calculations, while Matplotlib and Seaborn are mentioned for data visualization.

• A reference to Scikit-learn, which provides preloaded machine learning models that can be applied directly by users.

Programming Languages in Data Analysis

• Python is emphasized as a versatile language used for both exploratory analysis and advanced modeling in production systems.

• The second programming language discussed is R, known for its statistical capabilities and strong presence in academic research, featuring powerful libraries like Tidyverse.

SQL: A Query Language

• SQL (Structured Query Language), although not a programming language, is categorized here due to its practical applications in data manipulation.

• SQL is primarily used to query, extract, and manipulate data stored in relational databases; it’s essential knowledge for most data analysis projects.

Relational Databases

• Common relational databases include MySQL, PostgreSQL, SQL Server, and Oracle. These databases store structured information in tables ideal for consistency and control.

• Relational databases allow relationships between different tables through keys (e.g., student ID), enabling complex queries across datasets.

Non-relational Databases

• Non-SQL databases emerged to address new needs; they do not rely on traditional table structures or rigid schemas.

• Examples include document-based databases like MongoDB that store information in JSON format, suitable for unstructured or semi-structured data.

Types of NoSQL Databases

• Cassandra serves as a columnar database designed to handle large volumes of distributed information effectively.

• Redis represents key-value stores where data is stored as pairs; these are fast and often used for caching temporary data.

Big Data and Business Intelligence Tools

Introduction to Big Data Platforms

• Big Data platforms are essential for processing large datasets that cannot be handled by a single computer, enabling distributed information processing.

• Technologies like Hadoop HDFS allow data storage, while Apache Spark facilitates faster in-memory data analysis.

• Hive provides SQL-like querying capabilities for large volumes of data, and Kafka supports real-time data usage.

• These technologies are primarily utilized in large-scale or enterprise environments.

Business Intelligence (BI) Tools Overview

• The discussion transitions to Business Intelligence tools, which are designed for data visualization, reporting, and decision support.

• BI tools focus on clear communication of information rather than complex modeling; they aim to present data visually for better decision-making.

Power BI: A Leading BI Tool

• Power BI is developed by Microsoft and widely used in business settings; it connects to various data sources including Excel and cloud services.

• It allows users to create interactive dashboards without programming knowledge, facilitating the exploration of metrics such as student performance in universities.

Tableau: Advanced Visualizations

• Tableau is known for its advanced visual capabilities, allowing users to create sophisticated visualizations through a drag-and-drop interface.

• It is particularly useful for discovering patterns or relationships within large datasets, making it popular among analysts and data scientists.

Calipens: Associative Model Exploration

• Calipens features an associative model that highlights related data when a specific item is selected, aiding exploratory analysis without predefined paths.

• This tool helps uncover hidden relationships between datasets dynamically during analysis.

Google Data Studio (Looker Studio)

• Google Data Studio (now Looker Studio), a free cloud-based BI tool, integrates seamlessly with Google products like Sheets and Analytics.

• It enables quick reporting and online dashboards that can be shared in real-time without installation requirements.

Cloud Platforms for Data Analysis

• Cloud platforms facilitate data analysis without needing personal servers or high-performance computers; processes run on remote servers accessed via the internet.

• They offer scalability based on current needs—using fewer resources when handling smaller datasets and scaling up as needed.

Overview of Data Analysis Tools and Techniques

Google Club Tools for Data Management

• The Google Club features tools like Bit Query for analysis, Locker for visualization, and Verte for machine learning models. These platforms integrate storage, analysis, visualization, and modeling into a single environment.

Advanced Machine Learning Tools

• The discussion transitions to advanced machine learning tools used for AI that enable systems to learn from data and make predictions or classifications automatically.

• Various tools cater to different user expertise levels: some are designed for programmers while others target technical users without expert knowledge. Examples include Scikit-learn for classic models and TensorFlow or Keras for deep learning.

ETL vs ELT Processes

• ETL (Extract, Transform, Load) processes are crucial yet often overlooked. They involve extracting data from various sources, transforming it by cleaning and correcting errors, then loading it into a final repository.

• ELT reverses the order: data is first loaded before being transformed. Tools like Talend and Apache NiFi facilitate these processes while ensuring data readiness for analysis.

Techniques in Data Analysis

• Data analysis relies on a combination of technologies rather than a single tool; each technology serves a specific function within the overall process.

• Understanding the purpose of each tool is essential; it's not about mastering all but knowing when to use them effectively. Technologies complement methodologies rather than replace them.

Conclusion of Topic 1.3

• Successful real-world data analysis projects combine methodology, data, and technology to support decision-making processes.

• The session concludes with an invitation to continue practical applications in upcoming sessions starting Monday.